Data quality control method based on spatial-temporal feature fusion, medium and device

Abstract

The invention relates to a data quality control method, medium and device based on spatio-temporal feature fusion, and belongs to the technical field of data processing, the method comprises the following steps: obtaining data with spatio-temporal features after quality control over the years, and inputting the data into a spatio-temporal database; selecting data in the same geographic area as the data to be subjected to quality control by utilizing the space-time database to make a data set; performing data preprocessing on the manufactured data set, constructing a training data quality control network, and inputting the preprocessed data set into the quality control network for training until a model of the quality control network converges; and carrying out data preprocessing on data to be subjected to quality control according to the same method, and finally inputting the data to the converged model to obtain a quality control mark of each data point. The invention also provides a medium and a device for operating the method. The method can realize the quality control of the spatio-temporal data, and has the characteristics of high detection accuracy, high detection speed, good robustness and the like.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and specifically relates to a data quality control method, medium and device based on spatiotemporal feature fusion. Background Art

[0002] Spatiotemporal data refers to data that describes information from two dimensions: time and space. It contains not only the numerical values ​​of the data, but also the specific time and spatial location where these values ​​occur. For example, meteorological data (wind speed, wind direction, air pressure and temperature, relative humidity) and hydrological data (sea surface temperature, salinity, wave direction, wave height) recorded at a specific place and time, the changes in traffic flow on a certain road at different times, or the location and time of earthquake events on the earth's surface are all spatiotemporal data.

[0003] Objective and accurate observation results are the basis for subsequent scientific research. Spatiotemporal data is highly complex, which is reflected in the multidimensionality, dynamic variability, spatial heterogeneity and uncertainty of the data. The interaction of these factors can cause abnormal phenomena in the observation equipment during data collection. For example, drastic changes in tides, currents, temperature and salinity may affect the accuracy of sensors, resulting in bias or noise in the acquired data. Therefore, before the data is actually used, it is necessary to perform quality control on the data, mark or delete outliers in the data, and ensure the objectivity of the data. Common spatiotemporal data quality control methods include format check, range check, spike check, gradient check, and Rheinda check and box plot check based on statistical methods. Traditional quality control methods are usually based on a set of static rules or patterns to identify data anomalies. However, these methods are difficult to learn potential, unmined data anomalies from massive data, and cannot adaptively respond to new data anomalies, so more flexible and intelligent data quality control methods are needed. At present, outlier detection models based on classification networks are gradually being used. However, due to the scarcity of abnormal data, the data labels after quality control are seriously unbalanced, the low discrimination of abnormal data and the insufficient complexity of classification models, it is difficult for the model to fit the model parameters under limited abnormal information, which ultimately leads to poor quality control results.

[0004] In summary, traditional quality control methods have the following limitations. 1) When performing quality control on time series data, static rules or patterns are used, which are difficult to adapt to the challenges caused by random changes in equipment position; 2) The correlation of multiple variables is not considered, mined, or used to perform quality control on data in more dimensions, or it is difficult to use a formula to quantify the correlation of different variables; 3) Time and space data cannot be integrated simultaneously for quality control; 4) Quality control rules rely on expert experience, require more human intervention, and even require more visual inspections to improve the accuracy of quality control results. Summary of the invention

[0005] In order to solve the above problems, the purpose of this method is to provide a data quality control method based on spatiotemporal feature fusion to achieve quality control of spatiotemporal data, which has the characteristics of high detection accuracy, fast detection speed and good robustness.

[0006] The present invention is achieved through the following technical solutions: A data quality control method based on spatiotemporal feature fusion, the specific steps of the method are as follows: Step 1: Obtain the data with spatiotemporal characteristics after quality control over the years and enter them into the spatiotemporal database; Step 2: Use the spatiotemporal database to select data in the same geographical area as the data to be quality controlled to make a data set; take the acquisition device as the unit, sort in ascending order by time, generate sliding window data for all the data on each device according to the sliding window size (win_size) and the sliding window step (win_step), and then store each sliding window data as a window of the device; take the sliding window as the unit, perform spatiotemporal similarity retrieval and time statistical information retrieval, and append the retrieved spatiotemporal and temporally adjacent data to the end of the window; Step 3: Perform data preprocessing on the data set produced in step 2, wherein the preprocessing includes data set division, training set oversampling processing and data format processing; Step 4: construct a training data quality control network, and input the data set preprocessed in step 3 into the quality control network for training until the model of the quality control network converges; Step 5: Use the spatiotemporal database to divide the spatiotemporal data to be detected into sliding windows according to the same parameters (sliding window size, sliding window step, k neighbor data), expand the spatiotemporal similarity retrieval k neighbor data to the time window, and retrieve historical data based on time statistics; then preprocess the data according to step 3, and finally input it into the converged model in step 4 to obtain the quality control mark of each data point; for overlapping sliding windows, if there is a window that is judged to be abnormal, then the point is an abnormal point, and if all windows where the point exists are judged to be normal, it means that it is normal data in all relevant contexts.

[0007] As a preferred implementation, the spatiotemporal similarity retrieval in step 2; taking the sliding window as a unit, reading and traversing all the data rows in each window, inputting each row record into the spatiotemporal database for spatiotemporal similarity retrieval, that is, retrieving k data that are close neighbors to the record in time and space (the data needs to have a certain similarity in numerical distribution, size, and direction), and replacing them with default records if the data is less than k; then, inserting the retrieved records into the back of the record in the window from near to far, and finally expanding the number of data rows in each window from the sliding window size to the sliding window size × (1+k), that is, k data of the neighbors + the original data; As a preferred implementation, the time statistical information retrieval in step 2 is performed; taking the sliding window as a unit, and calculating the maximum and minimum values ​​of the longitude and latitude coordinates in each window to obtain the rectangular coordinate area of ​​the window, and then calculating the time range of each window, and inputting the rectangular coordinate area range and the time range into the spatiotemporal database to perform statistical information retrieval for the area, that is, retrieving data within the same time range and within the rectangular coordinate area in all years of the area; truncating according to the maximum historical data length, filling with the default value if it is insufficient, and then appending all retrieved data to the end of the window.

[0008] As a preferred implementation, the step 3 is specifically as follows: Step 3.1: Divide the data into training set and test set based on the acquisition device ratio; As a more preferred implementation, the data is divided into a training set and a test set in a ratio of 7:3 or 8:2.

[0009] Step 3.2: Oversampling of training set data; Since there are abnormal and normal sample imbalances in the data, oversampling technology is used to compensate for this problem; As a more preferred implementation, the oversampling method is to use a traditional anomaly detection algorithm to reversely generate anomalies of a set ratio, with the initial ratio set to 10%, and increase the ratio if the model does not converge; then replace the generated anomalies and anomaly labels with the record rows in the original window of the training set; Step 3.3: Data format processing; taking each window in the data set as a unit, select the multivariate labels that need to participate in quality control and the labels of the quality-controlled variables; use corresponding feature engineering processing for different variables, and then standardize and normalize the data to eliminate the dimensional influence between indicators, so as to facilitate the subsequent learning, training and reasoning of the prediction model; and perform hot-unique encoding on the labels.

[0010] As a preferred implementation, the step 4 is specifically as follows: Step 4.1: The constructed data quality control network includes the following layers: Self-Attention Networks, GAT (Graph Attention Networks) based on attention mechanism, BiLSTM (Bidirectional Long Short-Term Memory Networks), TF-Decoder (Transformer Decoder Networks) based on attention mechanism, TCN (Temporal Convolutional Networks), and FC (Fully Connected Neural Networks).

[0011] Step 4.2: Input the prepared training set into the control network described in step 4.1. Taking window data as the unit, first send the sliding window size × (1+k) data in the window data to the multivariate self-attention network Self in sequence at the time interval of 1+k. Attention, GAT, a dynamic graph network based on the attention mechanism, and BiLSTM, a bidirectional long short-term memory network. At the same time, the time statistical information retrieval data in the window data is sent to the TCN network in increasing time to obtain the abstract hidden state representation of the statistical data in time. Then, the output of TCN is used as the K value and V value of the attention mechanism, and the output of BiLSTM is used as the Q value to be sent to the TF-Decoder network for cross-attention connection to merge the two time series. Finally, the output of TF-Decoder is sent to the FC network to obtain the outlier classification probability of each time step, and the loss function value is calculated by comparing it with the label value of the time step. Then, the output error is back-propagated through the stochastic gradient descent method and the error back-propagation algorithm, and the error is apportioned to the computing units of each layer, so as to obtain the error signal of each layer unit, and then adjust the weight value. When the loss function value converges, the model converges. If the model does not converge, increase the proportion of generated outliers, that is, increase the learnable abnormal information.

[0012] The present invention also provides a computer-readable storage medium for data quality control based on spatiotemporal feature fusion, wherein the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the data quality control method based on spatiotemporal feature fusion.

[0013] The present invention also provides a data quality control device based on spatiotemporal feature fusion, the device comprising a spatiotemporal database module, a data set making module, a data preprocessing module, a data quality control model generating module and a data quality control module; The spatiotemporal database module executes the step 1, i.e., obtaining data with spatiotemporal characteristics after quality control over the years and entering them into the spatiotemporal database; The data set making module executes the step 2, i.e., making a data set using the spatiotemporal database of the spatiotemporal database module; The data preprocessing module executes the step 3, i.e., performs data preprocessing on the prepared data set, wherein the preprocessing includes data set division, training set oversampling processing and data format processing; The data quality control model generation module executes the step 4, i.e., constructs a training data quality control network, inputs the data set preprocessed by the data preprocessing module into the control network for training, until the model converges, and generates a data quality control model; The data quality control module runs step 5, that is, the data quality control module has a data input end, inputs the spatiotemporal data to be detected into the data quality control model, and obtains the quality control mark of each data point; for overlapping sliding windows, if there is a window that is judged to be abnormal, then the point is an abnormal point, and all windows where the point exists are judged to be normal, indicating that it is normal data in all relevant contexts.

[0014] Compared with the prior art, this method has the following beneficial effects: 1. This method learns from more details. The detection method takes advantage of deep learning in feature extraction, uses the self-attention mechanism to automatically capture the correlation of multiple variables; uses graph neural networks to learn relevant information from neighboring data; uses bidirectional long-short term recurrent networks to capture contextual information in the window; uses spatiotemporal convolutional networks to learn from temporal statistics in the region; contains more detailed information and can automatically capture more complex high-dimensional abstract concepts.

[0015] 2. This method is highly comprehensive and can be used to train a classification network to check for outliers in data, or to train a prediction network to interpolate outlier data. Both models can reuse the underlying network modules.

[0016] 3. This method has high precision and good stability. The method has achieved high scores in various indicators such as precision, accuracy, recall rate and F1 index, showing excellent performance. In addition, the method shows strong adaptability in quality control and can effectively cope with quality control that adapts to various variables, ensuring continuous stable performance in different environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is the architecture diagram of the data quality control method of this method; Figure 2 It is a schematic diagram of the sliding window partitioning algorithm of this method; Figure 3 Schematic diagram of the self-attention network principle of this method; Figure 4 This is a schematic diagram of the GAT network working structure of this method; Figure 5 This is a schematic diagram of the TF-Decoder network of this method; Figure 6 This is a diagram showing the abnormal classification effect of an embodiment of the method. DETAILED DESCRIPTION

[0018] In order to better understand the technical solution of the present method, the present method will be further described below in conjunction with the specific specification drawings and a specific marine buoy embodiment. It should be noted that the described embodiments are only part of the embodiments of the present method, not all of them. Based on the embodiments given by the present method, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present method.

[0019] like Figure 1 As shown in the figure, the architecture diagram of the data quality control method of this method, the specific steps are as follows: Step 1: Obtain the ocean drift data after quality control over the years and enter it into the spatiotemporal database: The ocean drift data after quality control over the years can be obtained from the float open data platform, such as the National Oceanic and Atmospheric Administration (NOAA). The spatiotemporal database technology selection can be built using PostgreSQL, Timescaler, and PostGIS, using Timescaler time partitioning technology to store in blocks and establish a BTree index for the time field; and using PostGIS to establish an RTree index for the location coordinate field, and reasonably divide the data into tables. Ultimately, an ocean drift database with a storage capacity of 100 million, a time span of more than 22 years, and millisecond-level spatiotemporal retrieval capabilities can be realized.

[0020] Step 2: Use the spatiotemporal database to select data in the same geographical area as the data to be quality controlled to create a data set. Specifically: Step 2.1: Sliding window division. Select the same geographical location as the quality control data, and retrieve data from the spatiotemporal database in batches using floating devices as units. Sort the retrieved data in ascending order of time, and generate sliding window data based on the sliding window size win_size and the sliding window step win_step for the data on each device, such as Figure 2 As shown in the figure. Currently, the widely used data sampling interval is one hour, and 24 sample point data are collected in one day. Therefore, win_size can be set to 24, that is, the data quality is controlled in units of days.

[0021] Step 2.2: Spatiotemporal similarity retrieval. It is known that the spatiotemporal neighbor data of ocean drifts imply similarity information between data, that is, the distribution interval, numerical size, and numerical direction of neighboring spatiotemporal data are roughly similar. Specifically, the sliding window can be used as a unit to read and traverse all the data in each window. Each record is input into the spatiotemporal database for spatiotemporal similarity retrieval, that is, to retrieve k data that are close to the record in time and space. If the data is less than k, the default record is used to replace it. Then, the retrieved records are inserted into the back of the record in the window from near to far. Finally, the number of data rows in each window is expanded from win_size to win_size×(1+k). From the analysis of the existing drift data, a rule is drawn: the maximum difference in the daily drift position is widely below 30 kilometers, and the absolute difference in sea surface temperature is widely below 0.5 degrees Celsius. Therefore, when performing spatiotemporal similarity retrieval, the distance can be limited to a circular space of 30 kilometers; the time can be limited to 24 hours, that is, 12 hours before and 12 hours after the current sample point.

[0022] Step 2.3: Time statistical information retrieval. It is known that the historical statistical data of ocean floats contains the statistical information of the data. For example, the volatility of the data can be understood through the standard deviation and variance, and the trend of data changes can be understood through the average value. Therefore, the sliding window can be used as a unit, and the maximum and minimum values ​​of the longitude and latitude coordinates in each window are calculated to obtain the rectangular coordinate area of ​​the window; and the time range of each window is calculated. The rectangular coordinate area range and the time range are input into the spatiotemporal database for information retrieval of the area, that is, the floating data of the same time range and rectangular coordinate area in all years of the area are retrieved; according to the maximum historical data length, the floating data are truncated, and the default value is filled if it is insufficient. Then, all the retrieved data are appended to the end of the window. In the actual retrieval process, the spatial range can be limited to the minimum range that can meet the maximum historical data length; and the time range can be limited to a certain month from the perspective of extensive data statistics, because the data has an annual variation law, the average temperature difference in the same month in different years will not be too large, so the time range can be limited to a certain month. For example, in the data set of this embodiment, the spatial range is limited to 20 kilometers, and the time range is limited to the same month, which can widely meet the requirement of the maximum historical length of 128.

[0023] Step 3: Preprocess the prepared data set, including data set division, training set oversampling, and data format processing. Specifically: Step 3.1: Divide the training set and test set into 8:2 ratios based on the float equipment, and randomly shuffle the order of the windows in the training set. Each window data is used as a training sample. The purpose of not dividing the data set by window is to facilitate the continuous display of quality control effects. Step 3.2: Oversampling of training set data. Since there is an imbalance between abnormal and normal samples in floating data, oversampling technology is used to compensate for this problem. The oversampling method used in this method is to reversely generate about 15% of abnormal values ​​using the traditional anomaly detection algorithm, and replace the generated abnormal values ​​and abnormal labels with the record rows in the original window of the training set. For example, the formula for the traditional peak detection algorithm is as follows: ; Here is the detection value, and is the value of the two observations before and after (unit: Celsius). If the result of the above formula exceeds the specified threshold, it is marked as a spike. For example, for temperature, when the pressure is less than If the result exceeds 6.0, then Therefore, we can use the “reverse direction” to infer The abnormal range is then selected from this range as the generated abnormal value. For example, the specific abnormal interval ranges of the traditional peak are: ; ; Step 3.3: Data format processing. Taking each window in the data set as a unit, select the labels of the multivariate and quality-controlled variables that need to participate in quality control; use the corresponding feature engineering processing for different variables, and then standardize and normalize the data to eliminate the dimensional influence between indicators, so as to facilitate the learning, training and reasoning of subsequent prediction models; and perform hot encoding on the labels. In feature engineering, the time variable is split into four features: year, month, day, and season, and the location information is characterized as a grid number. Preprocessing includes standardizing the time series data, that is, transforming the data into a distribution with a mean of 0 and a standard deviation of 1, which can be expressed as the following formula: ; in, is the mean of the time series data, is the standard deviation of the time series data, is the input value, is the output value.

[0024] Normalize the time series data, that is, map the sample data to the interval [-1,1]. It can be expressed as follows: ; in, is the minimum value of the time series data, is the maximum value of the time series data, Max is 1, is -1, is the input value, is the output value.

[0025] Step 4: Train the ocean float quality control network. Input the training data set prepared in step 3 into the network for training until the model converges. Specifically: Step 4.1: First, build an ocean float quality control network. The ocean float quality control network constructed by this method contains the following layers: Self Attention network, GAT network, BiLSTM network, TF-Decoder network, TCN network, and FC network. The functions of each network are as follows: Multivariate relationship learning: The Self Attention network uses the network's self-learning ability to automatically learn the correlation coefficient weights between multiple variables, without the need to manually select a set of feature variables with the highest correlation, and include as many variables as possible in the learning scope; that is, by introducing a self-attention mechanism into the input part of each time step, it automatically mines the relational weights between data features, and then weighted sums each variable to obtain a new weighted value. Studies have shown that there is a certain correlation between temperature and salinity. Generally, areas with higher salinity also have higher temperatures. Figure 3 In the graph, the thickness of the line represents the weight of the relationship between different feature variables. The thicker the line, the closer the relationship. For example, when calculating the value of a new feature variable of temperature, the self-attention mechanism will give priority to those feature variables with close relationships, such as temperature itself, salinity, etc., and weaken the mutual influence of feature variables with weaker relationships. The calculation formula of its attention mechanism is as follows: ; in, , and are input matrices, representing the query matrix, key matrix, and value matrix, respectively. is the vector dimension. For self-attention, if the input vector is ,but , and The matrices need to be generated through linear transformations. , , .

[0026] Capturing the relationship between neighboring data sample points: The GAT network is a graph attention network. This network introduces the attention mechanism into the feature aggregation process of the graph. It self-learns and adjusts weights through back-propagation. It uses the GlobalGraph Attention calculation method, which is completely independent of the structure of the graph. This design enables the model to adapt and optimize the aggregation of neighborhood information according to the loss function, so that it can handle the importance and correlation between nodes more finely. This mechanism not only improves the flexibility of the model, but also allows the model to more accurately capture and utilize complex relationships in graph-structured data. Specifically, the GAT network is used to capture the complex relationship representation between the observed sample point and the k sample points around it retrieved through spatiotemporal similarity, and then this relationship is applied to the data. For example Figure 4 As shown in the figure, in this hypothetical spatiotemporal database, it can be ensured that more than 80% of the sample points to be tested can find at least 3 neighboring points through the spatiotemporal similarity retrieval standard in step 2.2, that is, k is 3. For a certain quality control point i=1, let its 3 neighboring sample points be j={2,3,4}. If there are less than 3 sample points, the default neighbor records can be used instead. First, calculate the attention coefficient of node i and neighboring node j , the formula is: ; Then the result is normalized to get , the formula is: ; According to the calculated attention coefficient , aggregate the weighted sum of features, the formula is: ; : Node The characteristic matrix of : learnable weight matrix, That is, GAT outputs for each vertex New features (integrated with domain information), is the activation function. Finally, the node i Output to BiLSTM, the remaining nodes neglect.

[0027] Sequence context capture: The BiLSTM network can simultaneously take into account the contextual information of the time series and improve the prediction accuracy of the model. BiLSTM runs two independent LSTMs at each time step, one from the beginning to the end of the sequence, and the other from the end to the beginning of the sequence, and then concatenates the outputs in both directions at each time step. Finally, a fully connected linear transformation is performed to obtain the predicted output at that point.

[0028] Historical statistical information capture: The TCN spatiotemporal convolutional network is used to perform convolution operations on one-dimensional historical time series to obtain an abstract representation of historical statistical data. TCN uses causal convolution to ensure that the output depends only on the current and previous time steps, maintaining the temporal causality of the time series. The use of the dilated convolution mechanism enables the network to obtain a larger receptive field with fewer parameters, thereby effectively capturing long-term dependencies. By adding residual connections between convolutional layers, TCN improves the stability and training efficiency of the network and avoids the problem of gradient vanishing. In general, TCN first performs a convolution operation on the input one-dimensional time series, and continuously extracts feature information in the sequence through layers of convolution. In this way, TCN is able to generate high-dimensional abstract representations and reveal patterns and trends hidden in historical data.

[0029] Cross-connection of different time series: Construct an attention-based decoding block structure to achieve the fusion of two time series, such as Figure 5 As shown. In the attention-based decoder, the output of the BiLSTM network is first processed with a multi-head self-attention mechanism, that is, the information valid for the current time step is captured from the output context information, and multiple heads are used to process the data from different angles. It should be pointed out that, unlike the original attention-based decoder that requires setting the Mask matrix, this is processed for each time step, and there is no problem of future label leakage. The result is then residually connected and regularized and sent to the multi-head cross connection part as the input Q value of the cross attention. This part also uses the high-dimensional abstract statistical information captured by the TCN network as the K value and V value of the cross attention input. All time steps in the same window share the same TCN processing result, that is, the quality control of each time step can use the historical statistical information under the current window time and space, so that the quality control effect is more accurate.

[0030] Step 4.2: Input the prepared training set into the network, with window data as the unit. First, the win_size×(1+k) data in the window data are sent to the multivariate self-attention network SelfAttention, the dynamic graph network GAT based on the attention mechanism, the bidirectional long short-term memory network BiLSTM, the attention-based decoder block TF-Decoder, and the linear classification network FC in sequence at the time interval of 1+k; at the same time, the time history statistical information retrieval data in the window data is sent to the TCN network in increasing time to obtain a high-dimensional abstract representation of the time information of the statistical data. Then, the output of TCN is used as the K value and V value, and the output of BiLSTM is used as the Q value to the TF-Decoder network for cross-attention connection and merge the two time series. Finally, the output of TF-Decoder is sent to the FC network to obtain the probability of outlier classification at each time step. The loss function value is calculated by comparing with the label value of the time step. Then, the output error is back-propagated through the stochastic gradient descent method and the error back-propagation algorithm, and the error is distributed to the computing units of each layer, so as to obtain the error signal of each layer unit and adjust the weight value. When the loss function value converges, the model converges.

[0031] Step 5: Input the data collected by drifting float equipment at the same spatial position without quality control into the converged model in the same data format and processing method to obtain the quality control mark of each data point. For overlapping sliding windows, if there is a window that determines that the point is abnormal, then the point is determined to be an abnormal point, otherwise it is regarded as a normal data point.

[0032] like Figure 6 The effect is shown in the figure. y_pred is the predicted label value; y_true is the actual label value. For the points with wrong prediction (i.e. y_pred != y_true), use the "X" symbol to mark them; for the points accurately predicted as normal, no mark is used; and for the points accurately predicted as abnormal (i.e. y_pred == y_true == 1), use the "O" character to mark them.

[0033] The experimental results show that the effect of the model in this embodiment is very effective for detecting abnormal data. The test results show that whether the network is used for interpolation of ocean float time series prediction or trained as an ocean float outlier detection network, it has a smaller error and higher accuracy, and has a faster detection speed. Compared with the existing quality control algorithm, this method has better robustness and stronger generalization ability, and can easily achieve fine-tuning of quality control from one variable to another; while taking into account the advantages of detection speed and accuracy.

[0034] The present invention is not only applicable to the quality control of hydrological data collected by ocean floats, but also can be used for the quality management of GPS trajectory data, traffic flow data and meteorological data. This method is particularly effective when the data comes from multiple devices, is collected continuously in chronological order, and contains a variety of different collected variable information. The reason is that this method can learn the effective representation of data from the time context, historical statistical information, multivariate relationships and similar relationships of neighboring data, so as to identify outliers in the data; and there is no requirement for the specific meaning and data type of the data representation. Through this method, potential abnormal data points can be automatically identified, and the accuracy and reliability of data analysis can be improved. This makes it not only suitable for scientific research, but also widely used in multiple fields such as industrial monitoring, intelligent traffic management and meteorological detection. With the continuous increase in the amount of data, the number of collection devices and the number of collected variables, the data quality control effect using this method will become more and more significant.

Claims

1. A data quality control method based on spatiotemporal feature fusion, characterized in that: The specific steps of the method are as follows: Step 1: Obtain the data with spatiotemporal characteristics after quality control over the years and enter them into the spatiotemporal database; Step 2: Use the spatiotemporal database to select data in the same geographical area as the data to be quality controlled to make a data set; take the acquisition device as the unit, sort in ascending order by time, generate sliding window data for all the data on each device according to the sliding window size and sliding window step, and then store each sliding window data as a window of the device; take the sliding window as the unit, perform spatiotemporal similarity retrieval and time statistical information retrieval, and append the retrieved spatiotemporal and temporally adjacent data to the end of the window; Step 3: Perform data preprocessing on the data set produced in step 2, wherein the preprocessing includes data set division, training set oversampling processing and data format processing; Step 4: construct a training data quality control network, and input the data set preprocessed in step 3 into the quality control network for training until the model of the quality control network converges; Step 5: Use the spatiotemporal database to divide the spatiotemporal data to be detected into sliding windows according to the same parameters including sliding window size, sliding window step size, and k neighbor data, and use spatiotemporal similarity to retrieve the k neighbor data to expand them into the time window, and use time statistical information to retrieve historical data; then preprocess the data according to step 3, and finally input it into the model after convergence in step 4 to obtain the quality control mark of each data point; for overlapping sliding windows, if there is a window that is judged to be abnormal, then the point is an abnormal point, and if all windows where the point exists are judged to be normal, it means that it is normal data in all relevant contexts.

2. According to claim 1, a data quality control method based on spatiotemporal feature fusion is characterized in that: The spatiotemporal similarity retrieval in step 2 is as follows: taking the sliding window as a unit, reading and traversing all the data rows in each window, inputting each row record into the spatiotemporal database for spatiotemporal similarity retrieval, that is, retrieving k data that are close neighbors in time and space to the record, and replacing them with default records if there are less than k data; then, inserting the retrieved records into the back of the record in the window from near to far, and finally expanding the number of data rows in each window from the sliding window size to the sliding window size × (1 + k), that is, k neighboring data + the original one data.

3. The data quality control method based on spatiotemporal feature fusion according to claim 1 is characterized in that: The time statistical information retrieval in the step 2 is as follows: taking the sliding window as a unit, and calculating the maximum and minimum values ​​of the longitude and latitude coordinates in each window to obtain the rectangular coordinate area of ​​the window, and then calculating the time range of each window, and inputting the rectangular coordinate area range and the time range into the spatiotemporal database to perform statistical information retrieval for the area, that is, retrieving the data within the same time range and the rectangular coordinate area in all years of the area; truncating according to the maximum length of historical data, filling in the default value if it is insufficient, and then appending all retrieved data to the end of the window.

4. The data quality control method based on spatiotemporal feature fusion according to claim 1 is characterized in that: The step 3 is specifically as follows: Step 3.1: Divide the data into training set and test set based on the acquisition device ratio; Step 3.2: Oversampling of training set data; Step 3.3: Data format processing; Taking each window in the data set as a unit, select the multivariate that needs to participate in quality control and the label of the quality-controlled variable; Corresponding feature engineering processing is used for different variables, and then the data is standardized and normalized, and the labels are hot-encoded.

5. The data quality control method based on spatiotemporal feature fusion according to claim 4 is characterized in that: In the step 3.1, the data is divided into a training set and a test set in a ratio of 7:3 or 8:

2.

6. The data quality control method based on spatiotemporal feature fusion according to claim 4 is characterized in that: In the step 3.2, the oversampling process is performed: a set ratio of outliers is generated inversely using a traditional anomaly detection algorithm, with the initial set ratio being set to 10%. If the model does not converge, the ratio is increased; and then the generated outliers and outlier labels are replaced into the record rows in the original window of the training set.

7. The data quality control method based on spatiotemporal feature fusion according to claim 1 is characterized in that: The step 4 is specifically as follows: Step 4.1: The constructed data quality control network includes the following layers: self-attention network, dynamic graph network GAT based on attention mechanism, bidirectional long short-term memory network BiLSTM, decoder network TF-Decoder based on attention mechanism, time domain convolution network TCN, fully connected neural network FC; Step 4.2: Input the prepared training set into the control network described in step 4.

1. Taking the window data as the unit, first send the sliding window size × (1+k) pieces of data in the window data to the multivariate self-attention network Self Attention, the dynamic graph network GAT based on the attention mechanism, and the bidirectional long short-term memory network BiLSTM in sequence according to the time interval of 1+k; At the same time, the time statistical information retrieval data in the window data is sent to the TCN network in increasing time, and the abstract hidden state representation of the statistical data in time is obtained; Then, the output of TCN is used as the K value and V value of the attention mechanism, and the output of BiLSTM is used as the Q value to be sent to the TF-Decoder network for cross-attention connection to merge the two time series; finally, the output of TF-Decoder is sent to the FC network to obtain the outlier classification probability of each time step, and the loss function value is calculated by comparing it with the label value of the time step. Then, the output error is back-propagated through the stochastic gradient descent method and the error back-propagation algorithm, and the error is apportioned to the computing units of each layer, so as to obtain the error signal of each layer unit, and then adjust the weight value; when the loss function value converges, the model converges; if the model does not converge, increase the proportion of generated outliers, that is, increase the learnable abnormal information.

8. A computer-readable storage medium for data quality control based on spatiotemporal feature fusion, wherein the computer-readable storage medium stores a computer program, characterized in that: The computer program is suitable for being loaded by a processor and executing the data quality control method based on spatiotemporal feature fusion.

9. A data quality control device based on spatiotemporal feature fusion, characterized in that: The device comprises a spatiotemporal database module, a data set making module, a data preprocessing module, a data quality control model generating module and a data quality control module; The spatiotemporal database module runs step 1, i.e., obtaining data with spatiotemporal characteristics after quality control over the years and entering them into the spatiotemporal database; The data set making module runs step 2, i.e., using the spatiotemporal database to select data in the same geographical area as the data to be quality controlled to make a data set; The data preprocessing module runs step 3, i.e., performs data preprocessing on the prepared data set, wherein the preprocessing includes data set division, training set oversampling processing and data format processing; The data quality control model generation module runs step 4, i.e., constructs a training data quality control network, inputs the data set preprocessed by the data preprocessing module into the control network for training, until the model converges, and generates a data quality control model; The data quality control module runs step 5, that is, the data quality control module has a data input end, inputs the spatiotemporal data to be detected into the data quality control model, and obtains the quality control mark of each data point; for overlapping sliding windows, if there is a window that is judged to be abnormal, then the point is an abnormal point, and if all windows where the point exists are judged to be normal, it indicates that it is normal data in all relevant contexts.

Images