Real-time data quality monitoring methods for marine hydrological and meteorological automatic observation systems

Abstract

This invention discloses a real-time data quality monitoring method for automatic marine hydrological and meteorological observation systems, belonging to the field of data quality monitoring technology. The invention first preprocesses real-time marine observation data; then, based on the distance relationships between similar observation elements, it detects anomalies in the distance correlation of the observation data; based on the spatial and temporal pools of an HTM network, it simulates the spatial and temporal relationships of the observation data, and achieves data quality monitoring through pattern matching and prediction. This invention utilizes an HTM network to transform the data quality monitoring problem in a non-Markovian environment into a pattern matching and prediction problem. Through reasonable training, it simulates the spatial and temporal relationships of the observation time series, achieving real-time data quality monitoring; it adds anomaly detection based on the distance relationships between similar observation elements, improving the accuracy of the quality monitoring method; considering missing data and network packet loss, it preprocesses the observation data to ensure the robustness and stability of the quality monitoring process.

Description

Technical Field

[0001] This invention belongs to the field of data quality monitoring technology, and in particular relates to a real-time data quality monitoring method for marine hydrological and meteorological automatic observation systems. Background Technology

[0002] The marine hydrological and meteorological automatic observation system connects intelligent sensors distributed in different geographical locations through communication technology to realize the automatic collection, transmission, sharing and storage of hydrological and meteorological data.

[0003] In marine hydrological and meteorological automatic observation systems, the quality of observation data is often not guaranteed due to problems such as unstable network communication environment, malfunction of intelligent sensors, and abnormal operation of transmission software.

[0004] The main problems are as follows:

[0005] 1. Currently, the data transmission frequency in marine hydrological and meteorological automatic observation systems is generally on the order of minutes. It is impossible to detect anomalies in massive amounts of real-time observation data in a timely manner by relying solely on on-duty personnel for inspection.

[0006] 2. Traditional methods for monitoring the quality of marine observation data, such as the 3σ maximum permissible error test, Dixon algorithm, and Grubbs algorithm, are mainly for delayed data such as monthly and annual reports.

[0007] HTM network, as a time series quality monitoring algorithm based on unsupervised learning, has the advantages of not requiring parameter tuning and being robust; however, when anomalies occur frequently or even become the norm, the algorithm is prone to adapting to abnormal patterns and misjudging abnormal points as normal.

[0008] Meanwhile, in the complex marine observation environment, there are problems such as network packet loss and missing data, which can affect the effectiveness of data quality monitoring. Summary of the Invention

[0009] The purpose of this invention is to provide a real-time data quality monitoring method for automatic marine hydrological and meteorological observation systems. This method involves preprocessing real-time marine observation data; further, detecting anomalies in the distance correlation of observation data based on the distance relationships of similar observation elements; and simulating the spatial and temporal relationships of observation data using HTM network spatial and temporal pools. Data quality monitoring is achieved through pattern matching and prediction. This solves the problem that existing methods, in complex marine observation environments, suffer from adverse factors such as network packet loss and missing data, which affect the effectiveness of data quality monitoring.

[0010] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0011] This invention relates to a real-time data quality monitoring method for automatic marine hydrological and meteorological observation systems, comprising the following steps:

[0012] Step S001: Preprocess the observation data;

[0013] Step S002: Detect anomalies in the correlation between data from multiple measuring points by utilizing the distance relationship between similar observation elements;

[0014] Step S003: Based on the HTM network, perform anomaly detection on the spatial and temporal correlation of observation data at a single site.

[0015] Furthermore, the method for preprocessing the observation data in step S001 is as follows:

[0016] Step S011: Obtain anomaly labels from observed data

[0017]

[0018] in, This represents the observation data of sensor type m at measurement point τ at the current time k; This indicates an abnormal label for the observation data of sensor type m at measurement point τ at the current time k, where 0 indicates normal data and 1 indicates abnormal data.

[0019] Step S012: Based on the anomaly labels of the observed data Abnormal observation data is removed, and observation data marked as 1 is not subject to subsequent algorithm detection.

[0020] Furthermore, in step S002, when performing anomaly detection on the correlation between data from multiple measurement points, the following algorithm is executed:

[0021] Step S21: Calculate the distance between the observation data of measuring points τ and n from the m-type sensors at the current time k.

[0022] Obtain the distance matrix of sensor observation data of type m from measuring point τ and other measuring points n in the marine hydrological and meteorological automatic observation system at the current time k.

[0023] in, This represents the observation data of type m sensors at measurement point n at the current time k; Γ represents all measurement points in the marine hydrological and meteorological automatic observation system that are not marked as anomalies.

[0024] Step S22: Determine whether the distance between the current measuring point and the same type of observation data from other measuring points in the marine hydrological and meteorological automatic observation system is greater than the distance threshold. If it is greater than the distance threshold, mark the current measuring point as having an abnormal distance from other measuring points.

[0025] Step S23: Calculate the number of abnormal pairs in the observation data. If the number of abnormal pairs in the observation data is greater than half the number of all measurement points not marked as abnormal, the current observation data is judged to be abnormal, and the abnormal data is removed without proceeding to the subsequent algorithm.

[0026] Furthermore, in step S22, the method for determining anomalies in the distance between the current measuring point and other measuring points is as follows:

[0027] Obtain distance anomaly labels from sensor observation data of type m at current time k, including measurement point τ and other measurement points n. 0 indicates normal data distance, 1 indicates abnormal data distance;

[0028]

[0029] Obtain the distance anomaly label matrix from the observation data of measuring point τ and all unmarked measuring points n in the marine hydrological and meteorological automatic observation system at the current time k.

[0030]

[0031] Where, Δ τ,n,m Let t be the distance threshold for the observation data of the m-type sensors at measuring points τ and n.

[0032] Furthermore, the algorithm for determining the anomaly of the current observation data based on the number of anomaly pairs in the observation data distance in step S23 is as follows:

[0033] Obtain the distance anomaly log label of the m-type sensor observation data at the current time k at the measurement point τ.

[0034]

[0035] This indicates that the data is normal. This indicates data anomalies; data marked as 1 will not be further processed by the algorithm.

[0036] in, denoted as the sum of the distance anomaly pairs of the observation data of the current time k at the measuring point τ and all the measuring points n not marked as anomalies in the marine hydrological and meteorological automatic observation system; num(Γ) is the number of all the measuring points not marked as anomalies in the marine hydrological and meteorological automatic observation system.

[0037] Furthermore, the algorithm for anomaly detection of spatial and temporal correlation in the observation data of a single station in step S003 is as follows:

[0038] Step S31: Initialize the HTM network;

[0039] Step S32: Calculate the active state of neural columns, the active state of neurons, and the predicted state of the HTM network;

[0040] Step S33: Update the proximal synaptic connectivity gain, proximal synaptic connectivity, and basal synaptic connectivity of the active state, and simulate the spatial and temporal relationship of the observed time series.

[0041] Step S34: Calculate the difference between the neuron's predicted state and active state at the previous moment. If the difference exceeds the abnormal threshold, determine that the spatial and temporal correlation of the sensor observation data is abnormal.

[0042] Furthermore, the algorithm for calculating the active state of the HTM network neural columns, the active state of neurons, and the predicted state in step S32 is as follows:

[0043] Calculate the active state of neural column i at time k in the HTM network. This indicates that the nerve column is inactive. Indicates the active state of the nerve column;

[0044]

[0045] Among them, b i This represents the proximal synaptic connectivity gain of nerve column i; Z represents the proximal synaptic connectivity state of nerve column i, with 0 indicating a proximal synaptic disconnection state and 1 indicating a proximal synaptic connectivity state; i This indicates the sensor observation data corresponding to nerve column i. sparse discrete representation; This represents the i-th order of elements in matrix {·} after sorting them from largest to smallest. Each element value; o represents a neural column within the inhibition region radius Υ; b o This represents the proximal synaptic connectivity gain of nerve column O; Indicates the proximal synaptic connectivity state of nerve column O; Z o This indicates the observation data corresponding to nerve column o. Sparse discrete representation; θ a Indicates the threshold for the active state of a nerve column;

[0046] Calculate the active state of neuron j in neural column i at time k in the HTM network.

[0047] in, This indicates the active state of nerve column i at time k. This represents the predicted state of neuron j in nerve column i at time k-1 in the previous time step, with 0 indicating that it is not in the predicted state and 1 indicating that it is in the predicted state.

[0048] Calculate the predicted state of neuron j in neural column i at time k in the HTM network.

[0049]

[0050] in, represents the connectivity state of the basal synapse u of nerve column i, with 0 representing the non-connected state of the basal synapse and 1 representing the connected state of the basal synapse; ⊙ represents matrix dot product; This represents the neuron activity state matrix at the current time k. θ represents the active state of neuron j in nerve column i at the current time k; p This represents the threshold for predicting the state of a neural column.

[0051] Furthermore, the algorithm for updating the proximal synaptic connectivity gain, proximal synaptic connectivity, and basal synaptic connectivity of the active state in step S33, and simulating the spatial and temporal relationship of the observed time series, is as follows:

[0052] according to Update the proximal synaptic connectivity gain b of the active neural column i. i ;

[0053] Where exp(·) represents the exponential function; T represents the smoothing period; This represents the average active state of nerve column i from time k to time k-T+1; β represents the mean active state of nerve column i from time k-1 to time kT; β represents the modulation gain.

[0054] according to Update the proximal synaptic connectivity V of active neural column i. i ;

[0055] Where, q + q represents the proximal synaptic connectivity gain; - Indicates proximal synaptic connectivity inhibition; I represents the identity matrix; This represents the matrix of active states of the neural columns at the previous time step k-1. This indicates the active state of nerve column i below k-1 at the previous time.

[0056] according to Update the basal synaptic connectivity of neuron j in active nerve column i.

[0057] Where, p + p represents the basement synaptic connectivity gain; - This indicates inhibition of basal synaptic connectivity; This represents the neuron activity state matrix at the current time k. This indicates the active state of neuron j in nerve column i at the current time k.

[0058] Furthermore, in step S34, the difference between the predicted state and the active state of the neuron at the previous time step is:

[0059]

[0060] Among them, A k This represents the neuron activity state matrix at the current time k; This represents the neuron's predicted state matrix at the current time k. θ represents the predicted state of neuron j in nerve column i at time k; f This represents the threshold for abnormal spatiotemporal relationships.

[0061] The present invention has the following beneficial effects:

[0062] This invention applies Hidden Trace Meter (HTM) networks to an automatic marine hydrological and meteorological observation system, transforming the data quality monitoring problem in a non-Markovian environment into a pattern matching and prediction problem. Through appropriate training, it simulates the spatial and temporal relationships of observed time series, thereby achieving real-time data quality monitoring. However, HTM networks are unsupervised learning algorithms, and due to the lack of data labels, they can easily misclassify long-term, regular anomalies as normal. This method adds anomaly detection based on the distance relationship between similar observation elements before HTM network processing, increasing the effectiveness of the quality monitoring method. Considering the data gaps and network packet loss during data acquisition and transmission in the automatic marine hydrological and meteorological observation system, the observation data is preprocessed to increase the robustness and stability of the quality monitoring method.

[0063] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0064] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart of the real-time data quality monitoring method for an automatic marine hydrological and meteorological observation system according to the present invention.

[0066] Figure 2 This is a diagram showing the effect of the method of the present invention in detecting anomalies in temperature data. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] This invention relates to a real-time data quality monitoring method for automatic marine hydrological and meteorological observation systems. Please refer to [link / reference]. Figure 1 As shown, the real-time marine observation data is first preprocessed; then, based on the distance relationship of similar observation elements, anomalies in the distance correlation of multi-source observation data are detected; based on the spatial and temporal pools of the HTM network, the spatial and temporal relationships of the observation data are simulated, and data quality monitoring is achieved through pattern matching and prediction. This method can effectively detect anomalies in the spatial and temporal correlation of real-time data in the marine hydrological and meteorological automatic observation system under complex environments such as network packet loss and missing data, while the identification speed can meet the real-time requirements of the quality monitoring process of the marine hydrological and meteorological automatic observation system. A real-time data quality monitoring method for marine hydrological and meteorological automatic observation systems includes the following steps: Step S001: Preprocessing the observation data according to the characteristics of the marine hydrological and meteorological automatic observation system; Step S002: Detecting anomalies in the correlation of data between multiple measuring points using the distance relationship of similar observation elements; Step S003: Detecting anomalies in the spatial and temporal correlation of observation data at a single station based on the HTM network.

[0069] Example 1:

[0070] As an embodiment of the present invention, preferably, according to the "Marine Observation Specification" (GB / T 149.2—2019), published by the State Administration for Market Regulation and the Standardization Administration of China, published on March 25, 2019, and implemented on October 1, 2019, the method for preprocessing observation data to detect and remove missing data and network packet loss anomalies is as follows:

[0071] Step S011: Obtain anomaly labels from observed data

[0072]

[0073] in, This represents the observation data of sensor type m at measurement point τ at the current time k; This indicates an abnormal label for the observation data of sensor type m at measurement point τ at the current time k, where 0 indicates normal data and 1 indicates abnormal data.

[0074] Step S012: Based on the anomaly labels of the observed data Abnormal observation data are removed, and observation data marked as 1 are not further processed by the algorithm. Specifically, in this embodiment, the marine observation data is collected by the XZY3 automatic observation equipment produced by the National Marine Technology Center. The marine observation data quality monitoring system receives real-time observation data from each measuring point via the network, detects and removes data with missing measurements and network packet loss anomalies. According to the "Marine Observation Specifications," 99.998 missing values ​​for salinity data and 999.8 missing values ​​for air temperature data are removed. Therefore, the anomaly markings after observation data preprocessing are shown in Table 1.

[0075] As an embodiment of the present invention, preferably, when performing anomaly detection on the correlation between data from multiple measurement points in step S002, the following algorithm is executed:

[0076] Step S21: Calculate the distance between the observation data of measuring points τ and n from the m-type sensors at the current time k.

[0077] Obtain the distance matrix of sensor observation data of type m from measuring point τ and other measuring points n in the marine hydrological and meteorological automatic observation system at the current time k.

[0078] in, This represents the observation data of type m sensors at measurement point n at the current time k; Γ represents all measurement points in the marine hydrological and meteorological automatic observation system that are not marked as anomalies. Specifically, in this embodiment:

[0079] Salinity data distance matrix:

[0080] Temperature data distance matrix:

[0081] Step S22: Determine whether the distance between the current measuring point and similar observation data from other measuring points in the marine hydrological and meteorological automatic observation system is greater than a distance threshold. If it is greater than the distance threshold, mark the current measuring point as having an abnormal distance from other measuring points. The method for determining the abnormal distance between the current measuring point and other measuring points is as follows:

[0082] Obtain distance anomaly labels from sensor observation data of type m at current time k, including measurement point τ and other measurement points n. 0 indicates normal data distance, 1 indicates abnormal data distance;

[0083]

[0084] Obtain the distance anomaly label matrix from the observation data of measuring point τ and all unmarked measuring points n in the marine hydrological and meteorological automatic observation system at the current time k.

[0085]

[0086] Where, Δ τ,n,m Let Γ represent the distance threshold for sensor observation data of measuring points τ and n of type m, and let Γ represent all measuring points not marked as anomalies in the automatic marine hydrological and meteorological observation system. Specifically, in this embodiment:

[0087] Salinity data distance threshold matrix

[0088] Temperature data distance threshold matrix

[0089] The salinity data distance anomaly label matrix can be obtained. Temperature data distance from anomaly label matrix

[0090] Step S23: Calculate the number of abnormal pairs in the observation data. If the number of abnormal pairs in the observation data is greater than half of the number of all measurement points not marked as abnormal, the current observation data is judged to be abnormal, and the abnormal data is removed without proceeding to the subsequent algorithm.

[0091] The algorithm for determining anomalies in current observation data based on the number of anomaly pairs in the observation data distance is as follows:

[0092] Obtain the distance anomaly log label of the m-type sensor observation data at the current time k at the measurement point τ.

[0093]

[0094] This indicates that the data is normal. This indicates data anomalies; data marked as 1 will not be further processed by the algorithm.

[0095] in, This represents the sum of the number of distance anomaly pairs in the observation data of sensor type m at the current time k, including the observation data of measuring point τ and all unmarked measuring points n in the automatic marine hydrological and meteorological observation system; num(Γ) is the total number of unmarked measuring points in the automatic marine hydrological and meteorological observation system. Specifically, in this embodiment, the observation data obtained after distance correlation detection is shown in Table 2.

[0096] As an embodiment of the present invention, preferably, the algorithm for anomaly detection of spatial and temporal correlation of observation data at a single station in step S003 is as follows:

[0097] Step S31: Initialize the HTM network. Specifically, in this embodiment, the number of nerve columns is set to 2048, the number of neurons contained in each nerve column is 32, and the synaptic connectivity values ​​are randomly initialized;

[0098] Step S32: Calculate the active state of neural columns, the active state of neurons, and the predicted state of the HTM network;

[0099] Step S33: Update the proximal synaptic connectivity gain, proximal synaptic connectivity, and basal synaptic connectivity of the active state, and simulate the spatial and temporal relationship of the observed time series.

[0100] Step S34: Calculate the difference between the neuron's predicted state and its active state at the previous time step. If the difference exceeds the abnormal threshold, determine that the spatial and temporal correlation of the sensor observation data is abnormal.

[0101] Table 1

[0102]

[0103] Table 2

[0104]

[0105]

[0106] As an embodiment of the present invention, preferably, the algorithm for calculating the active state of HTM network neural columns, the active state of neurons, and the predicted state in step S32 is as follows:

[0107] Calculate the active state of neural column i at time k in the HTM network. This indicates that the nerve column is inactive. Indicates the active state of the nerve column;

[0108]

[0109] Among them, b i This represents the proximal synaptic connectivity gain of nerve column i; Z represents the proximal synaptic connectivity state of nerve column i, with 0 indicating a proximal synaptic disconnection state and 1 indicating a proximal synaptic connectivity state; i This indicates the sensor observation data corresponding to nerve column i. sparse discrete representation; This represents the i-th order of elements in matrix {·} after sorting them from largest to smallest. Each element value; o represents a neural column within the inhibition region radius Υ; b o This represents the proximal synaptic connectivity gain of nerve column O; Indicates the proximal synaptic connectivity state of nerve column O; Z o This indicates the observation data corresponding to nerve column o. Sparse discrete representation; θ a Indicates the threshold for the active state of a nerve column; θ a Υ and Υ are preset values. Specifically, in this embodiment, the sorting constraint... Set the threshold θ to 40 for the active state of the nerve column. a The value is set to 13, and the inhibition domain radius Y is set to 64. The neural column needs to enter the active state to satisfy the condition that the number of proximal synapses in the connected state multiplied by the connectivity gain is greater than the threshold of the active state of the neural column, and the value of this product must be ranked first.

[0110] Calculate the active state of neuron j in neural column i at time k in the HTM network.

[0111] in, This indicates the active state of nerve column i at time k. This represents the predicted state of neuron j in nerve column i at time k-1 in the previous time step, with 0 indicating that it is not in the predicted state and 1 indicating that it is in the predicted state.

[0112] Calculate the predicted state of neuron j in neural column i at time k in the HTM network:

[0113]

[0114] in, represents the connectivity state of the basal synapse u of nerve column i, with 0 representing the non-connected state of the basal synapse and 1 representing the connected state of the basal synapse; ⊙ represents matrix dot product; This represents the neuron activity state matrix at the current time k. θ represents the active state of neuron j in nerve column i at the current time k; p θ represents the threshold for predicting the state of a neural column. p This is a preset value. Specifically, in this embodiment, the predicted state threshold θ p Take 10; among the basal dendrites that the neuron needs to connect to to enter the predicted state, there exists one that satisfies the condition that the number of basal synapses is greater than the predicted state threshold.

[0115] Furthermore, the algorithm for updating the proximal synaptic connectivity gain, proximal synaptic connectivity, and basal synaptic connectivity of the active state in step S33, and simulating the spatial and temporal relationship of the observed time series, is as follows:

[0116] according to Update the proximal synaptic connectivity gain b of the active neural column i. i ;

[0117] Update the proximal synaptic connectivity gain b of the active neural column i.i ;

[0118] Where exp(·) represents the exponential function; T represents the smoothing period; This represents the average active state of nerve column i from time k to time k-T+1; β represents the average active state of nerve column i from time k-1 to time kT; β represents the adjustment gain; T and β are preset values. Specifically, in this embodiment, the smoothing period T is a preset value, T is 1000, and the adjustment gain β is 1; the smoothing period is mainly used to calculate the average active state of the nerve column, so that the proximal synaptic connectivity gain changes smoothly, reflecting the overall trend of the active state of the nerve column;

[0119] according to Update the proximal synaptic connectivity V of active neural column i. i ;

[0120] Where, q + q represents the proximal synaptic connectivity gain; - Indicates proximal synaptic connectivity inhibition; I represents the identity matrix; This represents the matrix of active states of the neural columns at the previous time step k-1. This represents the active state of nerve column i at time k-1 in the previous step; proximal synaptic connectivity gain q + Proximal synaptic connectivity inhibition q - This is a preset value. Specifically, in this example, the proximal synaptic connectivity gain q... + Set to 0.1, q represents proximal synaptic connectivity inhibition. - The value is set to 0.1; the proximal synaptic connectivity gain and suppression values ​​range from [0,1] and can be adjusted according to the effect.

[0121] according to Update the basal synaptic connectivity of neuron j in active nerve column i.

[0122] Where, p + p represents the basement synaptic connectivity gain; - This indicates inhibition of basal synaptic connectivity; This represents the neuron activity state matrix at the current time k. This represents the active state of neuron j in nerve column i at the current time k. Specifically, in this example, the basal synaptic connectivity gain p + Take 0.003; basal synaptic connectivity inhibition p - The value is set to 0.0005. The basement synaptic connectivity gain and suppression values ​​range from [0,1] and can be adjusted according to the effect.

[0123] Furthermore, in step S34, the difference between the predicted state and the active state of the neuron at the previous time step is:

[0124]

[0125] Among them, A k This represents the neuron activity state matrix at the current time k; This represents the neuron's predicted state matrix at the current time k. θ represents the predicted state of neuron j in nerve column i at time k; f The spatiotemporal relationship anomaly threshold, θ. f This is a preset value. Specifically, in this embodiment, the spatiotemporal relationship anomaly threshold θ f Taking 0.9, the anomaly markers of the quality monitoring algorithm are shown in Table 3:

[0126] Table 3

[0127]

[0128] A real-time data quality monitoring method for an automatic marine hydrological and meteorological observation system, as shown in the following figure, demonstrates the effectiveness of anomaly detection in temperature data. Figure 2 As shown, by using distance correlation and HTM network, the quality monitoring of real-time ocean observation data is achieved. Compared with the prior art, the beneficial effects of this invention are as follows: the observation data is preprocessed to mark packet loss and time delay anomalies, and missing data is filtered and removed to increase the robustness of the quality monitoring algorithm. Then, based on the distance correlation of similar observation elements between different measuring points, anomalies in multi-source observation data are detected. Furthermore, an HTM network is constructed to judge real-time observation data anomalies by learning the spatial and temporal characteristics of the observation time series.

[0129] The quality monitoring method comprehensively considers the correlation of multi-source data and the spatiotemporal relationship of single-source data, and has a low false alarm rate and low false alarm rate. It can meet the accuracy requirements of practical applications. Furthermore, the time for monitoring one data point is 4ms, while the data acquisition interval for observation data is generally 1 minute, which can meet the timeliness requirements of real-time data quality monitoring.

[0130] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0131] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for real-time data quality monitoring of an automatic marine hydrological and meteorological observation system, characterized in that, Includes the following steps: Step S001: Preprocess the observation data; Step S002: Utilize the distance relationship between similar observation elements to perform anomaly detection on the correlation between data from multiple measurement points. The anomaly detection algorithm is as follows: Step S21: Calculate the distance between the observation data of measuring points τ and n from the m-type sensors at the current time k. Obtain the distance matrix of sensor observation data of type m from measuring point τ and other measuring points n in the marine hydrological and meteorological automatic observation system at the current time k. in, This represents the observation data of type m sensors at measurement point n at the current time k; Γ represents all measurement points in the marine hydrological and meteorological automatic observation system. Step S22: Determine whether the distance between the current measuring point and the observation data of the same type of sensor in other measuring points in the marine hydrological and meteorological automatic observation system is greater than the distance threshold. If it is greater than the distance threshold, mark the current measuring point as having an abnormal distance from other measuring points. Step S23: Calculate the number of abnormal pairs in the observation data. If the number of abnormal pairs in the observation data is greater than half of the total number of other measuring points, the current observation data is judged to be abnormal, and the abnormal data is removed without proceeding to the next algorithm. Step S003: Based on the HTM network, perform anomaly detection on the spatial and temporal correlation of observation data at a single station. The algorithm for anomaly detection on the spatial and temporal correlation of observation data at a single station is as follows: Step S31: Initialize the HTM network; Step S32: Calculate the active state of the neural columns, the active state of the neurons, and the predicted state of the HTM network; Step S33: Update the proximal synaptic connectivity gain, proximal synaptic connectivity, and basal synaptic connectivity of the active state, and simulate the spatial and temporal relationship of the observed time series; Step S34: Calculate the difference between the predicted state and the active state of the neuron at the previous moment. If the difference exceeds the abnormal threshold, determine that the spatial and temporal correlation of the sensor observation data is abnormal. In step S22, the method for determining anomalies in the distance between the current measuring point and other measuring points is as follows: Obtain distance anomaly labels from sensor observation data of type m at current time k, including measurement point τ and other measurement points n. 0 indicates normal data distance, 1 indicates abnormal data distance; Obtain the distance anomaly label matrix of the sensor observation data of measuring point τ and all other measuring points n in the marine hydrological and meteorological automatic observation system at the current time k. Where, Δ τ,n,m The distance threshold for sensor observation data of measuring points τ and n is given by type m. The algorithm for determining the anomaly of the current observation data based on the number of anomaly pairs in the observation data distance in step S23 is as follows: Obtain the distance anomaly log label of the m-type sensor observation data at the current time k at the measurement point τ. This indicates that the data is normal. This indicates data anomalies; data marked as 1 will not be further processed by the algorithm. in, denoted as the sum of the number of distance anomaly pairs of sensor observation data of measuring point τ and all other measuring points n in the marine hydrological and meteorological automatic observation system at the current time k; num(Γ) is the total number of measuring points in the marine hydrological and meteorological automatic observation system.

2. The real-time data quality monitoring method for an automatic marine hydrological and meteorological observation system according to claim 1, characterized in that, The method for preprocessing the observation data in step S001 is as follows: Step S011: Obtain anomaly labels from observed data in, This represents the observation data of sensor type m at measurement point τ at the current time k; This indicates an abnormal label for the observation data of sensor type m at measurement point τ at the current time k, where 0 indicates normal data and 1 indicates abnormal data. Step S012: Based on the anomaly labels of the observed data Abnormal observation data is removed, and observation data marked as 1 is not subject to subsequent algorithm detection.

3. The real-time data quality monitoring method for an automatic marine hydrological and meteorological observation system according to claim 1, characterized in that: The algorithm for calculating the active state of HTM network neural columns, the active state of neurons, and the predicted state in step S32 is as follows: Calculate the active state of neural column i at time k in the HTM network. This indicates that the nerve column is inactive. Indicates the active state of the nerve column; Among them, b i V represents the proximal synaptic connectivity gain of nerve column i; i % represents the proximal synaptic connectivity state of nerve column i, with 0 representing the proximal synaptic disconnection state and 1 representing the proximal synaptic connectivity state; Z i This indicates the sensor observation data corresponding to nerve column i. sparse discrete representation; This represents the i-th order of elements in matrix {·} after sorting them from largest to smallest. Each element value; o represents a neural column within the inhibition region radius r; b o V represents the proximal synaptic connectivity gain of nerve column o; o % indicates the proximal synaptic connectivity state of nerve column o; Z o This indicates the observation data corresponding to nerve column o. Sparse discrete representation; θ a Indicates the threshold for the active state of a nerve column; Calculate the active state of neuron j in nerve column i at time k in the HTM network. in, This indicates the active state of nerve column i at time k. This indicates the active state of neuron j in nerve column i below k-1 at the previous time, with 0 indicating inactive and 1 indicating active. Calculate the predicted state of neuron j in neural column i at time k in the HTM network: in, denoted by , where 0 represents the basal synapse u of nerve column i is connected, and 1 represents the basal synapse disconnected state; e represents matrix dot product. This represents the active state matrix of neuron j at time k. θ represents the active state of neuron j in nerve column i at the current time k; p This represents the threshold for predicting the state of a neural column.

4. The real-time data quality monitoring method for an automatic marine hydrological and meteorological observation system according to claim 1, characterized in that, The algorithm for updating the proximal synaptic connectivity gain, proximal synaptic connectivity, and basal synaptic connectivity of the active state in step S33, and simulating the spatial and temporal relationship of the observed time series, is as follows: according to Update the proximal synaptic connectivity gain b of the active neural column i. i ; Where exp(·) represents the exponential function; T represents the smoothing period; This represents the average active state of nerve column i from time k to time k-T+1; β represents the mean active state of nerve column i from time k-1 to time kT; β represents the modulation gain. According to V i ←q + (V i e A k-1 )-q - (V i e(IA k-1 Update the proximal synaptic connectivity V of the active neural column i. i ; Where, q + q represents the proximal synaptic connectivity gain; - Indicates proximal synaptic connectivity inhibition; I represents the identity matrix; This represents the active state matrix of nerve column i at time k-1 in the previous time step. This indicates the active state of nerve column i below k-1 at the previous time. according to Update the basal synaptic connectivity of neuron j in active nerve column i. Where, p + p represents the basement synaptic connectivity gain; - This indicates inhibition of basal synaptic connectivity; This represents the active state matrix of neuron j at time k. This indicates the active state of neuron j in nerve column i at the current time k.

5. The real-time data quality monitoring method for an automatic marine hydrological and meteorological observation system according to claim 1, characterized in that, In step S34, the difference between the predicted state and the active state of neuron j at the previous time step is: Among them, A k This represents the active state matrix of neuron j at time k. This represents the prediction matrix of neuron j at time k. θ represents the predicted state of neuron j in nerve column i at time k; f This represents the threshold for abnormal spatiotemporal relationships.

Images