Time sequence data missing value filling method and electronic equipment

Abstract

The invention discloses a time sequence data missing value filling method and electronic equipment, and the method comprises the steps: obtaining to-be-filled time sequence data (having at least one time dimension missing value) for a business scene and multiple groups of reference time sequence data of the same type, and carrying out the similarity calculation of the to-be-filled time sequence data and the multiple groups of reference time sequence data; obtaining a first similar attention matrix between the multiple groups of reference time sequence data and the to-be-filled time sequence data; according to the similarity between each group of reference time sequence data and the to-be-filled time sequence data, performing correlation calculation on each reference value and each missing value in the corresponding reference time sequence data to obtain a first time sequence attention matrix between the multiple groups of reference time sequence data and the to-be-filled time sequence data, and according to the two attention matrixes and to-be-filled time series data, obtaining to-be-predicted time series data, inputting the to-be-predicted time series data into the trained nonlinear prediction model to fill each missing value, and obtaining filling time series data for the to-be-filled time series data.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular, to a method for filling missing values in time series data and an electronic device. Background Art

[0002] Currently, in modeling and prediction applications in many scenarios such as medical treatment, sales, finance, transportation, and fault detection, time series data is usually input into a model for processing and analysis to capture time dependence and trend changes and improve prediction accuracy.

[0003] However, during the acquisition of time series data, the obtained time series data is usually missing due to problems such as the failure of the data acquisition device itself, data collection errors or omissions, thereby reducing the reliability of time series data analysis and the accuracy of modeling, and affecting subsequent decisions.

[0004] To solve the above problems, currently, filling methods such as mean / median / mode are usually used to predict and fill the data of each dimension with missing values. The operation is simple, but it cannot capture the correlation information of time series data, resulting in a large amount of information loss and affecting the accuracy of the model prediction result. Summary of the Invention

[0005] In view of the above problems, the present application provides the following solutions:

[0006] The first aspect of the present application provides a method for filling missing values in time series data, and the method includes:

[0007] Obtain the time series data to be filled for the business scenario and multiple groups of reference time series data of the same type as the time series data to be filled; there are at least one missing value in the time dimension in the time series data to be filled;

[0008] Calculate the similarity between the time series data to be filled and each group of the reference time series data to obtain a first similarity attention matrix between the multiple groups of reference time series data and the time series data to be filled;

[0009] Calculate the correlation between each reference value and each missing value in the corresponding reference time series data according to the similarity between each group of the reference time series data and the time series data to be filled to obtain a first time series attention matrix between the multiple groups of reference time series data and the time series data to be filled;

[0010] Obtain the time series data to be predicted based on the first similarity attention matrix, the first time series attention matrix, and the time series data to be filled;

[0011] Input the to-be-predicted time series data into the trained non-linear prediction model to fill each of the missing values, and obtain the filled time series data for the to-be-filled time series data.

[0012] The second aspect of the present application provides an electronic device, which includes: at least one memory and at least one processor, where:

[0013] The memory is used to store multiple computer program instructions;

[0014] The processor is used to load and execute the computer program instructions to implement the following steps:

[0015] Obtain the to-be-filled time series data for the business scenario, and multiple groups of reference time series data of the same type as the to-be-filled time series data; there are at least one missing value in at least one time dimension of the to-be-filled time series data;

[0016] Calculate the similarity between the to-be-filled time series data and each group of the reference time series data respectively to obtain the first similarity attention matrix between the multiple groups of reference time series data and the to-be-filled time series data;

[0017] Calculate the correlation between each reference value and each of the missing values in the corresponding reference time series data according to the similarity between each group of the reference time series data and the to-be-filled time series data, and obtain the first time series attention matrix between the multiple groups of reference time series data and the to-be-filled time series data;

[0018] Obtain the to-be-predicted time series data based on the first similarity attention matrix, the first time series attention matrix, and the to-be-filled time series data;

[0019] Input the to-be-predicted time series data into the trained non-linear prediction model to fill each of the missing values, and obtain the filled time series data for the to-be-filled time series data. Description of the Drawings

[0020] In combination with the drawings and with reference to the following specific embodiments, the above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic, and the original elements and elements are not necessarily drawn to scale.

[0021] Figure 1 It is a schematic flowchart of a method for filling missing values in time series data proposed in the first embodiment of the present application;

[0022] Figure 2 It is a schematic flowchart of a method for filling missing values in time series data proposed in the second embodiment of the present application;

[0023] Figure 3 It is a schematic flowchart of a method for filling missing values in time series data proposed in Embodiment 3 of this application;

[0024] Figure 4 It is a schematic flowchart of a method for filling missing values in time series data proposed in Embodiment 4 of this application;

[0025] Figure 5 It is a schematic flowchart of a method for filling missing values in time series data proposed in Embodiment 5 of this application;

[0026] Figure 6 It is a schematic diagram of the network structure of a non - linear prediction model in a method for filling missing values in time series data proposed in the embodiments of this application;

[0027] Figure 7 It is a schematic diagram of the network structure of a PTL module in a method for filling missing values in time series data proposed in the embodiments of this application;

[0028] Figure 8 It is a schematic diagram of the network structure of an SPP module in a method for filling missing values in time series data proposed in the embodiments of this application;

[0029] Figure 9 It is a schematic flowchart of a method for filling missing values in time series data proposed in Embodiment 6 of this application;

[0030] Figure 10 It is a comparison chart of the effects of filling missing values in time series data;

[0031] Figure 11 It is a schematic flowchart of a method for filling missing values in time series data proposed in Embodiment 7 of this application;

[0032] Figure 12 It is a schematic diagram of the structure of a device for filling missing values in time series data provided in the embodiments of this application;

[0033] Figure 13 It is a schematic diagram of the hardware structure of an electronic device proposed in the embodiments of this application. Detailed implementation manners

[0034] The embodiments of this application will be described below with reference to the accompanying drawings in the embodiments of this application. The terms used in the implementation part of this application are only used to explain the specific embodiments of this application, rather than aiming to limit this application. The embodiments of this application will be described below with reference to the accompanying drawings. Those of ordinary skill in the art know that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0035] Throughout the present application, the terms "first", "second", etc. are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms may be interchanged under appropriate circumstances, which is merely a way of distinguishing objects with the same attributes when describing embodiments of the present application. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, such that a process, method, system, product, or device comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or are inherent to such process, method, product, or device.

[0036] Regarding the traditional data filling method described in the background art, since it uses single-dimensional feature information to predict missing values in time series data, although the process is simple, it will cause huge information loss, cannot well capture patterns such as seasonality or periodicity in time series data, and cannot obtain associated information in neighboring dates, etc., resulting in inaccurate filling of missing values. To improve this problem, a prediction model for predicting unknown missing values is trained based on known time series data features as training data, such as a prediction model based on the random forest algorithm or the Bayesian algorithm (or other probability algorithms), etc., to improve the accuracy of missing value prediction. However, this model filling method contains a prior assumption of sample independence, severs the correlation between similar samples, and cannot accurately fuse high-value information in similar samples, and the missing values that can be predicted and their accuracy are limited.

[0037] With the rapid development and application of artificial intelligence (AI) technology, the method based on machine learning / deep learning is further improved to fully fuse the effective information in time series data, improve the inference accuracy, and thus significantly and reliably improve the accuracy of missing value prediction. For example, the missing value filling of time series data is implemented based on the Recurrent Neural Network (RNN) / Convolutional Neural Network (CNN), etc. Among them, CNN expands the receptive field through the stacking of convolutional layers to obtain the dependence information between time series data. However, this method is limited by the number of network layers. If the network is too shallow, it cannot obtain long-distance dependence relationships, and if the network is too deep, it will significantly increase the training and inference costs.

[0038] Improved algorithms based on RNN, such as LSTM (Long Short-Term Memory), BiLSTM (Bi-directional Long Short-Term Memory), GPT (Generative Pre-trained Transformer), Transformer (a deep learning model based on self-attention mechanism), and GRU (Gated Recurrent Unit), can solve the problems of gradient vanishing and gradient explosion in traditional RNNs, have powerful information fusion and generation capabilities, can capture long-term dependencies in time series data, and are better applicable to complex time series data.

[0039] However, the above-listed machine learning / deep learning models have a large number of parameters, high training costs, and slow inference speeds, which affect the efficiency of filling missing values in time series data and require a large amount of training data. If the amount of training data is insufficient, it is easy to have a serious overfitting problem, reducing the accuracy and reliability of filling missing values.

[0040] To further solve the above technical problems, the embodiments of the present application provide a method for filling missing values in time series data, which proposes to use two sets of self-attention mechanisms to accurately capture the temporal correlation and sample correlation of data, effectively reduce the information loss of the model, and improve the efficiency and accuracy of filling missing values. The method for filling missing values in time series data of the embodiments of the present application will be introduced in detail below with reference to the accompanying drawings.

[0041] Refer to Figure 1 , which is a schematic flowchart of a method for filling missing values in time series data proposed in Embodiment 1 of the present application. This method for filling missing values in time series data can be applied to an electronic device, which can be a terminal device or a server with data processing capabilities, such as an application program running on the terminal device or a cloud service provided by a cloud server. As Figure 1 shown, the method for filling missing values in time series data provided in this embodiment may include but is not limited to the following steps:

[0042] Step S11: Obtain the time series data to be filled for the business scenario and multiple groups of reference time series data of the same type as the time series data to be filled; there are at least one missing value in the time dimension of the time series data to be filled.

[0043] In practical applications, in business scenarios such as meteorological / financial / medical and health / transportation data analysis, energy production / Internet of Things devices / environmental monitoring data processing, recommendation systems, big data analysis, or scientific research, the corresponding business data to be processed / analyzed is usually time series data containing one or more time dimensions. And due to certain factors, the actually collected / received time series data is incomplete, affecting the accuracy and integrity of subsequent data analysis. Usually, it is necessary to first fill in the missing values in the time series data, and then perform subsequent data analysis on the filled time series data.

[0044] Exemplarily, in the meteorological field, the meteorological data collected by relevant devices usually contains a large amount of time series data, such as meteorological data such as temperature, humidity, or wind speed observed at different latitudes / longitudes / altitudes / cities, etc. over a year or a quarter. However, during the collection of these data, there may be missing values due to a device failure or a transmission problem, such as missing meteorological data for one or more months, resulting in incomplete time series meteorological data. Facing this situation, the method provided in this application can be used to first fill in the missing values to obtain continuous meteorological data, so as to provide more accurate basic data for meteorological models or weather forecasts.

[0045] Similarly, in the recommendation field, a large amount of user behavior data (such as click, browse, and product sales records, etc.) stored in the database can be used as the time series data for training and optimizing the recommendation system. However, due to factors such as user inactivity or system failure, there may be missing values in the user behavior data, such as missing user behavior data such as browsing records in the last week or two. In the medical and health field, for time series data such as electronic health records, there may be missing heart rate and blood pressure record data for a certain period of time. Thus, it can be seen that in different business scenarios listed above, the actually collected or received time series data content is different, and the types and quantities of the time dimensions (which can be determined according to time units such as year, month, day, or hour, etc.) of the data content can be the same or different, depending on the situation.

[0046] In addition, it should be understood that in the actual business scenario, not all of the collected or received original time series data necessarily has missing values. For the original time series data with complete data, this application does not perform any processing and can be directly used as the basic data for subsequent data analysis. For the time series data detected with missing values from each group of original time series data, which is denoted as the time series data to be filled, the method for filling missing values in time series data proposed in this application needs to be executed on it. After obtaining the complete time series data, subsequent data analysis / processing can be continued. The number of missing values and their time dimensions, etc. among each group of time series data to be filled may be the same or different, depending on the situation.

[0047] Among them, for the method of obtaining the time series data to be filled, it is possible to determine whether there are missing values in the original time series data quickly through detection tools such as Pandas, or through the method of checking the continuity of timestamps, or through outlier detection / auto-correlation analysis methods based on data characteristics. It is also possible to detect missing values based on a model, such as training a time series prediction model to determine whether there are abnormal data or missing values in the time series data, or training an anomaly detection model based on a clustering algorithm to determine whether there are missing values or outliers in the original time series data. In addition, it is also possible to draw a time series graph or a heat map through a visualization tool to visually display the original time series data, so as to accurately and completely determine the time series data to be filled in the original time series dataset. This application does not limit the implementation method of how to detect the time series data to be filled with missing values, and it can be determined according to the situation.

[0048] For the detection methods including but not limited to those listed above, if they are implemented by the electronic device itself, the time series data to be filled in each group can be directly obtained through this detection method; if they are implemented by the data processing capabilities of the corresponding acquisition device itself, in the case where the electronic device and the acquisition device are different independent devices, the acquisition device can transmit the detected time series data to be filled in each group to the electronic device through wireless / wired communication, and the electronic device executes the method proposed in this application. This application does not limit the implementation method of obtaining the time series data to be filled for the business scenario in step S11.

[0049] In order to accurately and quickly predict each missing value in the time series data to be filled, this application can select multiple groups of reference time series data for the time series data to be filled from the complete original time series data of the same data type as the time series data to be filled. In one possible implementation, the original time series data can be clustered according to the data characteristics that are more concerned in the business scenario, the clustering type of each group of time series data to be filled is determined, and part or all of the complete original time series data belonging to this clustering type is selected as the reference time series data for this time series data to be filled. It can be seen that for different groups of time series data to be filled in the same business scenario, the reference time series data selected according to this method may be different, or may be different parts of the reference time series data of the same data type.

[0050] It should be noted that the method for obtaining the reference time series data corresponding to each group of time series data to be filled includes, but is not limited to, the clustering implementation methods listed above. It can also be implemented by using a pre-trained classifier or other data classification methods, or by screening based on the data characteristics concerned in the business scenario. Of course, if a large amount of original time series data is stored by classification, multiple groups of reference time series data for the time series data to be filled can also be directly obtained in the subset where each time series data to be filled is located (that is, composed of original time series data of the same type), so as to reduce the interference of noise data irrelevant to the missing values in the time series data to be filled, reduce the complexity of model prediction, and improve the prediction efficiency and accuracy of missing values. The present application does not limit the implementation method of step S11.

[0051] Step S12: Calculate the similarity between the time series data to be filled and multiple groups of reference time series data respectively to obtain a first similarity attention matrix between the multiple groups of reference time series data and the time series data to be filled;

[0052] Combined with the above analysis of the technical solution of the present application, in order to accurately fuse the high-value information in similar samples (that is, the effective information for missing values in time series data) and improve the prediction accuracy of missing values, the present application will consider the relevance between similar samples (that is, time series data) and can be implemented based on the corresponding attention mechanism. Therefore, after the electronic device obtains multiple groups of reference time series data corresponding to each group of time series data to be filled, it can further accurately calculate the similarity between the time series data to be filled and each group of reference time series data to construct the corresponding attention matrix, denoted as the first similarity attention matrix. The present application does not limit the similarity calculation method between different time series data, which can be one or more of, for example, Euclidean distance, DTW (Dynamic Time Warping, time similarity metric), correlation coefficient, or Fourier transform, and can be flexibly selected or adjusted according to the actual situation.

[0053] Among them, the size (that is, the number of rows and columns) of the first similarity attention matrix can be determined according to the number of groups of reference time series data. Therefore, the first similarity attention matrix can be a vector with 1 row and N columns, where N represents the number of groups of reference time series data of the time series data to be filled, and each element it contains is the similarity between the time series data to be filled and the reference time series data at the corresponding column position, or a value representing the similarity, such as obtained based on the normalization result of the similarity. The present application does not limit the content of the elements included in the first similarity attention matrix corresponding to each group of time series data to be filled.

[0054] Preferably, in the process of obtaining the first similarity attention matrix, the present application may also consider the distance between the time dimension of each data included in the reference time series data and the time dimension where the missing value is located in the time series data to be filled, so as to make the subsequent missing value calculation pay more attention to the data of the neighboring dates by assigning different weights or coefficients, etc., reduce the adverse impact of the marginal data, and improve the prediction accuracy of the missing values.

[0055] Step S13: Calculate the correlation between each reference value and each missing value in the corresponding reference time series data according to the similarity between each group of reference time series data and the time series data to be filled, so as to obtain the first time series attention matrix between each group of reference time series data and the time series data to be filled.

[0056] In addition to considering the correlation between the time series data to be filled and its corresponding groups of reference time series data, the present application also captures the important features and time dependence relationships in the time series data, determines the time series correlation between the missing values and the reference values in different time dimensions of the reference time series data, so as to effectively reduce the information loss of the model and improve the prediction accuracy of the missing values. Therefore, for the time series correlation between each group of time series data to be filled and its corresponding multiple groups of reference time series data, the correlation between each missing value and each reference value in the corresponding reference time series data can be calculated according to the similarity between each group of reference time series data and the time series data to be filled, and thus the first time series attention matrix representing the time series correlation for each missing value is obtained.

[0057] In practical applications, for the above correlation calculation method, a similarity measurement method based on distance, or a similarity measurement based on correlation (such as Pearson Correlation Coefficient or Mutual Information, etc.), or a similarity measurement method based on a model (such as single-head attention mechanism, multi-head attention mechanism or graph attention network, etc.) can be adopted alone or in combination to obtain the first time series attention matrix for each missing value in the time series data to be filled. According to the specific business scenario and the data characteristics of the time series data, a suitable time series correlation calculation method can be flexibly selected to implement step S13, and the implementation process is not described in detail in the present application.

[0058] Step S14: Obtain the time series data to be predicted based on the first similarity attention matrix, the first time series attention matrix, and the time series data to be filled.

[0059] Combined with the above descriptions of the similarity attention matrix and the temporal attention matrix, the present application can use the first similarity attention matrix and the first temporal attention matrix to process the to-be-filled temporal data. For example, through the first similarity attention matrix and the first temporal attention matrix, weighted summation is performed on the to-be-filled temporal data to combine the accurately captured sample correlation and the temporal correlation for the missing data, so as to generate an estimated value for the missing value in the to-be-filled temporal data. The corresponding missing value in the to-be-filled temporal data is replaced with the estimated value to initially obtain the complete temporal data. However, the complete temporal data obtained at this time often contains some redundancy or noise and can be used as the to-be-predicted temporal data to further perform accurate prediction of the missing value according to the following steps.

[0060] In a possible implementation manner, especially when the original temporal data set is a large-scale temporal data set, such that each to-be-filled temporal data corresponds to a large number of groups of reference temporal data, the computational amount of the attention matrix is large. As a result, directly using the first similarity attention matrix and the first temporal attention matrix to process the to-be-filled temporal data to obtain the processed temporal data (such as the complete temporal data after replacing the corresponding missing value with the estimated value as described above) has a high prediction cost. Therefore, the present application can remove redundant and noisy data through a pruning method to improve the prediction efficiency and interpretability of the model. For example, through the attention pruning method, unimportant attention weights (i.e., elements in the attention matrix) are removed to delete the data in the processed temporal data that is not helpful for predicting the missing value or even interferes with the prediction accuracy (which can be called invalid data), and the valid data that is more helpful for predicting the missing value is retained. At this time, the temporal data after attention pruning can be used as the to-be-predicted temporal data.

[0061] Optionally, the present application can achieve the above pruning process through one or a combination of two of setting a pruning threshold (such as setting the element value in the attention matrix less than the pruning threshold to zero, etc. The pruning threshold can be determined based on experience, experiments, historical data, etc., in combination with the tasks of the actual business scenario and the content of the temporal data, to avoid excessive pruning due to too high a pruning threshold, which affects the missing value filling effect. The present application does not limit its numerical size) and re-normalization processing (such as re-normalizing the attention weights in the same row of the attention matrix to ensure that the sum of the attention weights in the same row is 1, etc.), so as to achieve the effect of reducing the computational amount and enhancing the interpretability of the model.

[0062] Step S15: Input the to-be-predicted temporal data into the trained non-linear prediction model to fill each missing value, and obtain the filled temporal data for the to-be-filled temporal data.

[0063] In the embodiments of the present application, through analysis, it is known that the first similarity attention matrix and the first temporal attention matrix constructed using similarity and relevance reflect the linear relationship between vectors and are difficult to capture the non-linear information in the data. Especially for the prediction of missing values in the temporal data to be filled with non-linear characteristics, the prediction quality is affected.

[0064] In response to this, the present application pre-trains a non-linear prediction model that can reliably identify the non-linear information of similar temporal data (i.e., reference temporal data with a high similarity to the temporal data to be filled) and the adjacent time, and accordingly quickly and accurately fill each missing value in the temporal data to be filled, obtaining the filled temporal data after filling the missing values, which is used as the basic data for subsequent data analysis to ensure the accuracy and reliability of the task (such as weather prediction task, product recommendation task, or health anomaly alarm task, etc.) based on the data analysis results.

[0065] In some embodiments, the above non-linear prediction model can be a multi-scale convolutional neural network, a support vector machine, or a radial basis function network (RBFN), etc. Compared with the traditional prediction model, the non-linear prediction model trained in the present application can better capture the temporal information in the temporal data to be predicted and improve the prediction quality of missing values.

[0066] Preferably, in order to reduce the number of model parameters, reduce the training and inference costs, and improve the filling efficiency of missing values in temporal data, when constructing the initial non-linear prediction model, the present application can select network modules with a relatively simple network structure that can meet the prediction task requirements of the non-linear prediction model in the present application. The present application does not limit the network structure of the non-linear prediction model and can be determined according to the situation. It can be seen that for different missing value prediction requirements in different business scenarios, a non-linear prediction model with a matching network structure can be selected to implement missing value filling, or the parameters of the trained non-linear prediction model can be optimized or the structure can be adjusted to use the newly obtained non-linear prediction model to implement missing value filling in the corresponding business scenario, improving the diversity or flexibility of the non-linear prediction model, enabling the method proposed in the present application to effectively solve the problem of missing temporal data in multiple business scenarios, enhancing the effect of data analysis and modeling, and thus enhancing the conversion rate of enterprise solutions.

[0067] Based on the above analysis, after obtaining the time-series data to be filled for any business scenario and multiple groups of reference time-series data of the same type, the present application not only calculates the similarity between the time-series data to be filled and each group of reference time-series data to obtain the corresponding first similarity attention matrix to represent the associated information of similar time-series data; then, according to this similarity, it calculates the similarity between the reference values and each missing value in each group of reference time-series data to obtain the corresponding first time-series attention matrix to represent the time-series associated information of the data values. After that, based on these two attention matrices and the time-series data to be filled, the time-series data to be predicted is obtained, and then it is directly input into the trained non-linear prediction model. By optimizing the attention mechanism, the ability to extract and fuse non-linear information of the model is increased, and the filling of each missing value is quickly and accurately realized, improving the filling effect of the obtained filled time-series data.

[0068] Referring to Figure 2 , which is a schematic flowchart of a method for filling missing values in time-series data proposed in the second embodiment of the present application. This embodiment can describe an optional implementation method for obtaining the time-series data to be predicted in the method for filling missing values in time-series data proposed above, so as to reduce the model training and inference costs and improve the efficiency of filling missing values. For other implementation steps of this method, reference can be made to the corresponding parts of the context embodiments. This embodiment describes the implementation process of the time-series data to be predicted. As Figure 2 shown, an optional implementation method for obtaining the time-series data to be predicted based on the first similarity attention matrix, the first time-series attention matrix, and the time-series data to be filled proposed by the present application may include:

[0069] Step S21: Prune the first similarity attention matrix according to the information entropy of the first similarity attention matrix to obtain a second similarity attention matrix;

[0070] Step S22: Prune the first time-series attention matrix according to the information entropy of the first time-series attention matrix to obtain a second time-series attention matrix;

[0071] Step S23: Use the second similarity attention matrix and the second time-series attention matrix to process the time-series data to be filled to obtain the time-series data to be predicted.

[0072] Combined with the relevant descriptions of the above embodiments, the present application can adopt a pruning method to reduce the model calculation amount, and proposes to implement it by using a model pruning strategy based on information entropy. While effectively reducing the model training and inference costs, the robustness of the model is greatly improved. In this implementation method, the information entropy of the attention matrix is used to measure the importance of each attention weight in the attention matrix, and pruning is performed accordingly, that is, deleting the attention weights that contribute little to predicting the missing values.

[0073] It can be seen that the information entropy can measure the dispersion degree of the elements in the corresponding attention matrix. If the information entropy of the attention matrix is low, it indicates that the elements in the attention matrix are in a small number of positions, and the effective information volume for predicting missing values is higher; conversely, if the information entropy of the attention matrix is high, it indicates that the elements in the attention matrix are more evenly distributed, that is, the similar attention or temporal attention is more evenly distributed, and the corresponding attention matrix may contain redundant information. By removing the redundant information, the interference can be reduced and the computational amount of subsequent processing can be decreased, and the inference efficiency and accuracy of the non-linear prediction model can be improved, that is, the filling effect of missing values in time series data can be improved.

[0074] Based on the above analysis, the present application can utilize the operation principle of information entropy to respectively obtain the information entropy of the first similar attention matrix and the information entropy of the first temporal attention matrix, so as to realize pruning of the corresponding attention matrix, and obtain the corresponding pruned attention matrix, which is correspondingly denoted as the second similar attention matrix and the second temporal attention matrix.

[0075] Exemplarily, assuming that the element value (i.e., attention weight) at the i-th row and j-th column in the attention matrix (such as the first similar attention matrix or the first temporal attention matrix) is denoted as V(i,j), the information entropy of the j-th column of the attention matrix, denoted as H j , and the information entropy of the i-th row of the attention matrix, denoted as H i :

[0076]

[0077]

[0078] In the above formula, n represents the number of groups of reference time series data corresponding to the time series data to be filled. For different time series data to be filled, the value of n can be different or may be the same, and the value of n can be determined according to the situation. m represents the number of time dimensions included in the time series data. For example, if the time series data is monthly data in the corresponding business scenario, m = 12 (12 months), and if the time series data is annual data in the corresponding business scenario, m = 365 (365 days in a year), etc. It can be determined according to the smallest unit of the time dimension. The present application does not limit the values of m and n. Log() represents the logarithmic function. It can be seen that formula (1) represents the information entropy between the time series data to be filled and multiple groups of reference time series data on the same time dimension (i.e., the j-th time dimension); formula (2) represents the information entropy between the time series data to be filled and each group of reference time series data on all time dimensions.

[0079] Afterwards, in combination with the above description of information entropy, elements in the rows / columns of the attention matrix with information entropy higher than the pruning threshold can be removed or set to zero / null to optimize the attention mechanism of the model and improve the missing value filling effect. Exemplarily, in the case where the information entropy is greater than the pruning threshold (which can be represented by "θ"), the element values in the corresponding rows / columns of the corresponding first similarity / temporal attention matrix are set to null (which can be represented by ), the pruning method of the attention matrix can be expressed by the following formula:

[0080]

[0081] It can be seen that in this embodiment, for the first similarity / temporal attention matrix, if the information entropy of the element values in the i-th row is less than or equal to the pruning threshold θ, it indicates that the element values in this row need to be retained as valid information to predict the missing values of the time series data; if the information entropy of the i-th element value is greater than the pruning threshold θ, it indicates that the element values in this row belong to invalid information (i.e., redundant information) and need to be removed, and can be directly set to null so that it no longer participates in the subsequent missing value prediction calculation, greatly reducing the computational amount and prediction interference, which is beneficial to improving the efficiency and accuracy of missing value prediction.

[0082] In a possible implementation method, the present application can also construct a corresponding Mask mask array, and determine the Mask value at the corresponding position in the Mask mask array according to the comparison result between the information entropy and the pruning threshold, such as 0 or 1, etc. 0 indicates pruning of the element value at the corresponding position in the attention matrix; 1 indicates retaining the element value at the corresponding position in the attention matrix. However, it is not limited to the attention pruning implementation method described in this embodiment.

[0083] In an alternative implementation process of constructing the Mask mask array, the Mask value at the corresponding position for the same comparison result can be determined in combination with the activation function type in the non-linear prediction model trained by the present application. If the activation function in the non-linear prediction model is the first activation function (such as Softmax, a function that maps the input to a probability distribution), the element values with information entropy greater than the pruning threshold in the first similarity attention matrix and the first temporal attention matrix can be updated to negative infinity; if the activation function in the non-linear prediction model is the second activation function (such as ReLU, Rectified LinearUnit, a simple non-linear function that sets all negative numbers to 0), the element values with information entropy greater than the pruning threshold in the first similarity attention matrix and the first temporal attention matrix can be updated to zero. This pruning method can also be expressed in the form of a piecewise function such as the above formulas (3) and (4), which will not be elaborated in the present application.

[0084] It can be seen that in the embodiments of the present application, by analyzing the similarity and temporal correlation between the to-be-filled time series data and multiple groups of reference time series data, after obtaining the first similarity attention matrix and the first temporal attention matrix, an attention pruning method based on information entropy will be further adopted to prune these two attention matrices respectively, delete the redundant information that is not conducive to the missing value filling effect, and retain the effective information that helps to accurately implement the missing value filling. In this regard, this embodiment can use the level of information entropy to characterize whether the element values in the corresponding rows / columns of the corresponding attention matrix belong to redundant information or effective information, thereby determining whether to prune the element values in this row / column to obtain the corresponding pruned attention matrix for subsequent missing value filling, greatly reducing the computational amount and improving the efficiency and accuracy of time series data filling.

[0085] Referring to Figure 3 , which is a schematic flowchart of a method for filling missing values in time series data proposed in Embodiment 3 of the present application. This embodiment can refine the process of obtaining the first similarity attention matrix in the method for filling missing values in time series data proposed above. For other processing steps of this method for filling missing values in time series data, reference can be made to the corresponding parts of the context embodiments, and details are not described in this embodiment. Based on this, after obtaining the to-be-filled time series data for the business scenario and multiple groups of reference time series data of the same type as the to-be-filled time series data, as Figure 3 shown, the method for obtaining the first similarity attention matrix proposed in this embodiment may include but is not limited to the following steps:

[0086] Step S31: Determine the distance weights of each time dimension between the corresponding reference time series data and the to-be-filled time series data according to the dimension interval between the reference values in each group of reference time series data in each time dimension and the time dimension where the missing value is located, and the weight decay coefficient.

[0087] In the embodiments of the present application, based on the above analysis, it is known that using the above two attention mechanisms enables the missing value filling process of the to-be-filled time series data to focus on the reference time series data with higher similarity and the information of adjacent dates, reducing the adverse effects of marginal information. Exemplarily, the to-be-filled time series data 1 is most similar to the reference time series data 2 and the reference time series data 3 of the same type. When the data in June in the to-be-filled time series data 1 is missing, the data in May and July in the reference time series data 2 and the reference time series data 3 play the greatest role in filling the missing value in June in the to-be-filled time series data 1.

[0088] Therefore, in a possible implementation, the present application can use the weighted standard Euclidean distance to construct the first similarity attention matrix. During this construction process, considering the interval differences between different time dimensions and the time dimension where all missing values in the time series data to be filled are located, as well as their impacts on the missing value filling effect, corresponding distance weights can be constructed accordingly and added to the operation formula of the traditional Euclidean distance to improve the accuracy of the similarity between each group of reference time series data and the time series data to be filled obtained thereby.

[0089] According to the above analysis, after determining the time dimension where each missing value in the time series data to be filled is located, the difference calculation can be performed with the time dimension of each reference value in each group of reference time series data to obtain the dimension interval between the time dimension of each reference value in the corresponding group of reference time series data and each missing value in the time series data to be filled. It can be seen that the dimension intervals between the reference values under different time dimensions in each group of reference time series data and the time dimension where the missing values are located are different.

[0090] In addition, to prevent overfitting and control the model complexity, in the application of the similarity attention mechanism in this embodiment, a weight decay coefficient (or regularization coefficient) can be configured. Optionally, the present application can be set by manual adjustment, and can be determined or adjusted based on experience or the actual tasks and the original time series data set in the business scenario. It can also be adjusted through cross-validation or a validation time series data set to obtain an appropriate size of the weight decay coefficient. It can also use the hyperparameter adjustment method of Bayesian optimization to search for the optimal weight decay coefficient from the hyperparameter space, etc. The present application does not limit the size of the weight decay coefficient and its acquisition method.

[0091] After that, the distance weights corresponding to each time dimension between each group of reference time series data and the time series data to be filled can be calculated using, but not limited to, the following formula:

[0092]

[0093] In formula (5), λ represents the weight decay coefficient; d j can represent the sum of the intervals between the j-th time dimension in the reference time series data and the time dimension where all missing values in the time series data to be filled are located, that is, the above-mentioned dimension interval; Max() represents taking the maximum value; Min() represents taking the minimum value; W j represents the distance weight between the j-th time dimension in a group of reference time series data and the time dimension of all missing values in the time series data to be filled, j = 1, 2,..., m, where m is the number of time dimensions included in the time series data. According to the calculation method represented by formula (5), m distance weights can be determined for each group of reference time series data and the time series data to be filled.

[0094] Step S32: Using the distance weights corresponding to each time dimension of each group of reference timing data, calculate the mean square error between the corresponding reference timing data and the timing data to be filled, to obtain the distance between the corresponding reference timing data and the timing data to be filled.

[0095] Continuing the above analysis, in the case where the present application can use the Euclidean distance to calculate the mean square error to measure the difference / similarity between the reference timing data and the timing data to be filled, it can be implemented according to the following formula:

[0096]

[0097] Among them, D(i) represents the distance between the i-th group of reference timing data (which can be denoted as P i ) and the timing data to be filled (which can be denoted as Q), thereby representing the similarity between the reference timing data and the timing data to be filled. The smaller this distance, the higher this similarity; V j represents the variance of each timing data in the j-th time dimension. Using this variance for standardization, that is, to eliminate the dimensional difference between the characteristics of the timing data, and reduce the adverse effects of noise data and outliers on the missing value filling result. Q j can represent the data in the j-th time dimension of the timing data to be filled, and P ij can represent the data in the j-th time dimension of the i-th group of reference timing data. can represent the number of time dimensions where there are no missing values in both Q and P i . When there is a missing value in a certain time dimension, it does not participate in the calculation.

[0098] It should be understood that for the method of obtaining the distance between the reference timing data and the timing data to be filled, in addition to the Euclidean distance calculation method, other distance-based similarity algorithms such as the DTW (Dynamic Time Warping) algorithm, DDTW (Derivative Dynamic Time Warping) algorithm, WDTW (Weighted Dynamic Time Warping) algorithm, etc. can also be used, but distance weights W j and other hyperparameters need to be added to the traditional distance-based similarity algorithms, so that the attention calculation pays more attention to the information of the neighboring dates and reduces the influence of the edge information. The calculation process is not described in detail in the present application.

[0099] Step S33: Sum the distances between each group of reference timing data and the timing data to be filled, to obtain the total distance.

[0100] Step S34: Based on the distances between each group of reference timing data and the timing data to be filled, as well as the total distance, obtain the similarity between the corresponding reference timing data and the timing data to be filled, so as to form a first similarity attention matrix.

[0101] In the embodiment of the present application, after obtaining the distances between each group of reference timing data and the timing data to be filled, such as D(1), D(2), …, D(i), …, D(n), through normalization (or standardization) processing, the corresponding similarities between 0 and 1 can be obtained, and the sum of all similarities is 1. Optionally, the present application can adopt the method represented by the following formula to implement the normalization processing of each distance and obtain the corresponding similarity:

[0102]

[0103] Among them, S(i) can represent the similarity between the i-th group of reference timing data and the timing data Q to be filled; can represent the total distance of step S33. It should be noted that the method for normalizing each D(i) includes, but is not limited to, this proportional calculation method described in this embodiment.

[0104] According to the above method, the similarities between each group of reference timing data and the timing data to be filled can be expressed as: S(1), S(2), …, S(i), …, S(n). The first similarity attention matrix (i.e., a one-dimensional vector) is formed by these n similarities, so that the element values in the first similarity attention matrix are the similarities.

[0105] It can be seen that in the process of constructing the first similarity attention matrix between each group of reference timing data and the timing data to be filled in the embodiment of the present application, the differences in the dimension intervals between the parameter values of different time dimensions in the reference timing data and the time dimension where all missing values are located are fully considered, as well as the different influences of the similarities between the reference timing data and the timing data to be filled, so as to form a distance weight representing this influence. Then, through the mean square error or Euclidean distance algorithm, the distance representing the similarity between the corresponding reference timing data and the timing data to be filled is obtained. After normalization / standardization processing, the first similarity attention matrix is formed by the obtained similarities. Compared with the traditional attention mechanism, the present application can more effectively capture the correlation of similar parameter timing data, effectively reduce information loss, and lay a reliable foundation for accurately filling in missing values subsequently.

[0106] It should be noted that for the similarity calculation method between each group of reference time series data and the time series data to be filled, in addition to the distance-based similarity algorithm described above, a correlation-based similarity calculation method can also be adopted, such as algorithms like Pearson Correlation Coefficient or Cosine Similarity; or a similarity calculation method based on information theory, such as Mutual Information, etc. The appropriate similarity calculation method can be flexibly selected according to the actual business scenario and its task requirements, and the embodiments of this application will not list and elaborate on them one by one here.

[0107] Referring to Figure 4 , which is a schematic flowchart of a method for filling missing values in time series data proposed in the fourth embodiment of this application. In this embodiment, the process of obtaining the first time series attention matrix in the method for filling missing values in time series data proposed above can be described in detail. For other processing steps of this method for filling missing values in time series data, reference can be made to the corresponding parts of the embodiments in the context, and this embodiment will not be elaborated in detail. Based on this, after obtaining the time series data to be filled for the business scenario and multiple groups of reference time series data of the same type as the time series data to be filled, as Figure 4 shown, the method for obtaining the first time series attention matrix proposed in this embodiment may include but is not limited to the following steps:

[0108] Step S41, arrange the multiple groups of reference time series data and the time series data to be filled according to the similarity between each group of reference time series data and the time series data to be filled, to obtain a time series data matrix; among them, the smaller the arrangement distance between the reference time series data with a greater similarity and the time series data to be filled.

[0109] In the embodiments of this application, after obtaining the similarity between each group of reference time series data and the time series data to be filled according to the method described above, the multiple groups of reference time series data can be arranged according to the similarity, so that the smaller the arrangement distance between the reference time series data with a greater similarity and the time series data to be filled, that is, the adjacent arrangement position of the time series data to be filled is the reference time series data with the greatest similarity to it, to accurately obtain the time series correlation and further improve the influence of the reference time series data with the greatest similarity on filling the missing values. This application can form a time series data matrix with the arrangement results of the obtained n groups of reference time series data and a group of time series data to be filled, such as an (n + 1)-row and m-column time series data matrix.

[0110] Step S42, use the arrangement distance between each group of reference time series data and the time series data to be filled in the time series data matrix, and the weight decay coefficient, to calculate the correlation degree between each reference value and each missing value in the corresponding reference time series data, to obtain the correlation coefficient between each reference value and each missing value in the corresponding reference time series data.

[0111] In the embodiments of the present application, the Pearson correlation coefficient calculation method can be used to implement step S42, and the calculation formula is as follows:

[0112]

[0113] In formula (8), X j represents the reference value of the j-th time dimension in each group of reference time series data; X k represents the k-th missing value in the time series data to be filled, represents the mean of X j ; represents the mean of X k . W_Corr(X j , X k ) represents the correlation coefficient between each reference value of the j-th time dimension in the reference time series data and the k-th missing value in the time series data to be filled; λ represents the weight decay coefficient; d i represents the permutation distance between the i-th reference time series data and the time series data Q to be filled, that is, the weight decay of the calculation of time series correlation using similarity. The numerator part of formula (8) represents the covariance between X j and X k , and the denominator part represents the product of the standard deviations of X j and X k ; among them, in this calculation process, compared with the standard Pearson correlation coefficient calculation method, the present application constructs a corresponding weight coefficient using λ and d i to improve the accuracy and reliability of the obtained correlation coefficient for predicting missing values in time series data.

[0114] Step S43, using the correlation coefficients corresponding to each time dimension of each group of reference time series data, and the number of missing values existing in the time series data to be filled, obtain the attention values of the corresponding reference time series data and the time series data to be filled in each time dimension, so as to form a first time series attention matrix.

[0115] For the obtained correlation coefficients above, they can be used to measure the linear relationship between two time series data, such as the relationship between meteorological data, analyzing the correlation between stock prices, etc., so as to determine the attention values (which can also be called attention weights) for each missing value by monitoring the change of the correlation coefficient. Optionally, the present application can be calculated using the following formula:

[0116]

[0117] Among them, T(j) can represent the attention value of the reference time series data and the time series data to be filled in the j-th time dimension; It can represent the total number of missing values in the time series data to be filled. Therefore, in the embodiments of the present application, the correlation coefficients between each missing value and the parameter values of the reference time series data can be summed, and the total correlation coefficient obtained is divided by the total number of missing values to obtain the average correlation coefficient, which is directly used as the attention value between the reference value and the missing value in the corresponding time dimension, indicating the time series correlation degree. Then, the attention values between the reference time series data and the missing values in each time dimension are used to form the first time series attention matrix according to the above similarity arrangement result.

[0118] Referring to Figure 5 , which is a schematic flowchart of a method for filling missing values in time series data proposed in Embodiment 5 of the present application. In this embodiment, an optional refined implementation method for filling missing values through a trained non-linear prediction model in the method for filling missing values in time series data proposed above can be described. Based on this, after obtaining the time series data to be predicted according to the method described in the context embodiments, as Figure 5 shown, this embodiment can perform but is not limited to the following steps:

[0119] Step S51: Input the time series data to be predicted into the trained non-linear prediction model, and perform feature extraction at different levels through the single-shot multi-box detector and the skip connection layer of the non-linear prediction model to obtain multi-scale feature maps;

[0120] In the embodiments of the present application, according to the method described in the above embodiments, after processing the time series data to be filled by using the calculated first similarity attention matrix and the first time series attention matrix to obtain the time series data to be predicted, the present application proposes to use a trained non-linear prediction model to perform non-linear fine-tuning on the attention matrix to quickly and accurately capture non-linear information and fill each missing value to obtain the complete time series data, that is, the filled time series data.

[0121] In a possible implementation, in order to reduce information loss and enhance the applicability and practicality of the model, compared with the traditional prediction model, the non-linear prediction model of the present application can use the SSD network (Single Shot MultiBoxDetector) and the PTL (Pass Through Layer) module. Referring to Figure 6 the network structure shown, the SSD network is an efficient one-stage object detection algorithm, which can quickly and accurately detect the feature maps at different levels in the time series data to be predicted through the multi-scale feature map and the prior box mechanism.

[0122] As Figure 6 shown, for the time series data to be predicted (such as Figure 6For the input time series data to be filled in the shape of 1×n×n, for example, the time series data 1 to be filled, the time dimension is 12 (months), and the missing month is 6; there are corresponding similar reference time series data 2 and reference time series data 3. After the above processing, the time series data to be predicted input to the model is the time series data 1 to be filled, and the matrix 3×3 after arranging the reference time series data 2 and reference time series data 3 at the neighboring dates (such as May and July) can be expressed as:

[0124] [Data from February to May, data from February to June, data from February to July],

[0125] [Data from January to May, missing value, data from January to July],

[0126] [Data from March to May, data from March to June, data from March to July]

[0127] .

[0128] For the input time series data to be predicted, through multiple CBL modules and PTL modules of the SSD network, feature maps of different scales are extracted, such as Figure 6 shown, feature maps of various sizes such as c×n / 2×n / 2; 2c×n / 4×n / 4; 4c×n / 4×n / 4; 4c×n / 8×n / 8; 8c×n / 8×n / 8; 16c×n / 8×n / 8. Among them, CBL is a convolutional network building block, usually including Convolution (convolution layer), Batch Normalization (batch normalization layer) and LeakyReLU (activation function), to extract features through the convolution layer, stabilize and accelerate training through batch normalization, LeakyReLU introduces non-linearity, and the batch normalization layer and LeakyReLU can reduce the problem of gradient disappearance to improve the model stability; and the batch normalization layer can also reduce the sensitivity of the model to small changes in the input time series data to be predicted and prevent overfitting.

[0129] The PTL module is a network module for enhancing feature transfer and fusion. As shown in the schematic diagram of the operation principle of PTL in Figure 7 , the same type of data in the input feature map (which can be represented by the same color square in Figure 7 ) can be extracted into a feature map, so that the data types in the feature map are consistent. Based on this, this embodiment can fuse the high-level semantic information and low-level semantic information of the time series data to be predicted, effectively reduce the information loss in the semantic extraction process, and improve the integrity and reliability of the feature map.

[0130] Step S52, perform semantic information fusion on the feature maps of multiple scales, and input the fused feature map into the spatial pyramid pooling layer of the non-linear prediction model for processing to obtain a target feature map with a fixed scale; ​

[0131] As Figure 7 shown, after the processing of the SSD network and the PTL module, the feature maps of different sizes obtained are fused, for example, through a convolutional layer, to obtain a target feature map of a fixed scale such as 4c×n / 8×n / 8. It should be noted that the sizes of the feature maps in the non-linear prediction model include but are not limited to Figure 7 the sizes shown, and can be flexibly adjusted according to actual needs.

[0132] In the embodiment of the present application, in order to improve the applicability and practicality of the model, a spatial pyramid pooling layer (Spatial Pyramid Pooling, SPP) is deployed at the tail of the non-linear prediction model to process feature maps of any size input (such as Figure 7 the feature maps of sizes such as 4c×n / 8×n / 8; 8c×n / 8×n / 8; 16c×n / 8×n / 8 shown), so as to obtain a target feature map of a fixed scale. Optionally, as Figure 8 shown in the SPP network structure, it includes a Maxpool (maximum pooling) and a Flatten layer, so as to perform downsampling through the maximum pooling layer to reduce the spatial dimension of the feature map and retain important features. The multi-dimensional input data is flattened into one dimension through the Flatten layer, which is convenient for subsequent processing.

[0133] It should be noted that the network structures of the SSD network, the PTL module, and the SPP module include but are not limited to the network layers described above, and can be flexibly adjusted according to actual needs, such as the number of CBL modules and the number of PTLs in the SSD network, etc., to obtain more or fewer feature maps of different sizes, etc. The implementation process is not described in detail in this application.

[0134] Step S53, input the target feature map into the fully connected layer and the activation function layer of the non-linear prediction model for processing, and obtain the filled time series data for the time series data to be filled.

[0135] Following the above analysis, as Figure 6 shown, the one-dimensional data output by the SPP module is input into the fully connected layer for processing, and the processed data is input into the activation function layer for processing to complete the filling of missing values and obtain the completed filled time series data.

[0136] Combined with the above description of the network structure of the non-linear prediction model, compared with the traditional prediction model, it can better capture the time series information in the input data, and compared with time series models such as LSTM and Transformer, the non-linear prediction model has a small number of parameters, which improves the inference speed and thus improves the efficiency of filling missing values.

[0137] In addition, the non-linear prediction model proposed in this application has transferability. For the task of filling missing values in time series data under different business scenarios, by adaptively modifying the structure of the network tail of the non-linear prediction model or fine-tuning some parameters, etc., a missing value filling task more suitable for the actual business scenario can be obtained, and high-quality filled time series data can be obtained. It can be seen that the parameters of the non-linear prediction model or the structure of the spatial pyramid pooling layer corresponding to different business scenarios can be different.

[0138] In a possible implementation, combining the above analysis, after the business scenario changes, or when verifying the initially obtained filled time series data, if the verification fails, such as there are abnormal data in the filled time series data or the amount of abnormal data is greater than the threshold, etc., this application can optimize the existing non-linear prediction model in the above manner, that is, in response to the optimization request of the non-linear prediction model, adjust the parameters of the non-linear prediction model or the structure of the spatial pyramid pooling layer to obtain an optimized non-linear prediction model, so as to input the time series data to be predicted into the optimized non-linear prediction model and update the filled time series data for the time series data to be filled.

[0139] Combined with the refined description of other steps in the method for filling missing values in time series data proposed in this application above, this application can implement this method according to the following formula:

[0140]

[0141] Among them, Non_linear Model represents the non-linear prediction model, and Mask mask represents attention pruning processing; represents the first similar attention matrix; represents the first time series attention matrix, D M represents the time series data to be filled. It can be seen that in this application, after the time series data to be filled interacts with two attention matrices, pruning processing is performed through the Mask matrix, and the obtained time series data to be predicted is input into the non-linear prediction model for processing to obtain the final missing value filling result, that is, the filled time series data.

[0142] Refer to Figure 9 , which is a schematic flowchart of a method for filling missing values in time series data proposed in Embodiment 6 of this application. This embodiment can describe the training process of the above non-linear prediction model. Combining the above description of the network structure of the non-linear prediction model (such as Figure 6 shown, but not limited to this), as Figure 9 shown, the training process of the non-linear prediction model can include:

[0143] Step S91, obtain multiple training samples and multiple training reference samples corresponding to each training sample from multiple historical time series data collected for the business scenario;

[0144] Step S92: Extract missing values from each training sample to obtain corresponding training missing samples and training labels;

[0145] In the actual application of this application, multiple historical time-series data collected for business scenarios can come from open-source data sets, internal databases, etc. This application only takes the GSOD (Global Surface Summary of the Day) data set as an example for illustration. It contains meteorological daily observation data of more than 9,000 stations around the world. The time range of this meteorological daily observation data is from 1929 to the present. Among them, each piece of data contains more than 20 meteorological indicators such as average temperature (0.1 degrees Fahrenheit), average wind speed (0.1 knots), average sea-level pressure (0.1 mb), precipitation (0.1 inches), and snow depth (0.1 inches). In the application of some periods, due to data limitations or communication problems, it may be impossible to obtain meteorological data (time-series data) in a certain area, resulting in missing values in one or more meteorological data. This application can use the GSOD data set to construct a data set for training a non-linear prediction model.

[0146] In a possible implementation, four groups of time-series data, namely TEMP (temperature), DEWP (dew point temperature), WDSP (wind speed), and VISIB (visibility), of all observation points can be extracted from the large amount of meteorological data contained in the GSOD data set. For each group of time-series data, a certain number of time-series data can be randomly selected as training samples. For example, 2,000 training samples are selected, and then 5 (the number of missing values is not limited, this application only takes 5 as an example for illustration) time points (i.e., time dimension) are randomly selected from each training sample as missing values, and the value at the central position is used as the training label, that is, the true missing value.

[0147] It should be noted that the construction process of the training sample set for other business scenarios is similar. For the acquisition method of training reference samples, the acquisition method of reference time-series data described above can be referred to, and this embodiment will not be elaborated in detail. In the implementation process of obtaining training missing samples and their training labels, it includes but is not limited to the random selection method described above.

[0148] Step S93: Obtain the similar training attention matrix and the time-series training attention matrix between each training missing sample and multiple training reference samples corresponding to the training missing sample;

[0149] Step S94: Obtain corresponding missing samples to be predicted based on each training missing sample and the similar training attention matrix and the time-series training attention matrix corresponding to the training missing sample;

[0150] Regarding the method for obtaining the similarity training attention matrix and the temporal training attention matrix, reference can be made to the relevant descriptions of the obtaining processes of the first similarity attention matrix and the first temporal attention matrix described in the corresponding embodiments above, and details are not described herein in this embodiment. Subsequently, pruning can be performed on each attention matrix, and the pruned attention matrix is multiplied by the training missing samples to obtain the model input, i.e., the missing samples to be predicted. The implementation process is not described in detail in this application.

[0151] Step S95: Input the missing samples to be predicted into the initial non-linear prediction model to fill each missing value, and obtain the predicted filling samples for the corresponding training missing samples;

[0152] In this application, the network structure of the initial non-linear prediction model can be as Figure 6 shown. The implementation process of step S95 can refer to Figure 5 the relevant descriptions of steps S51 - S53 in the corresponding embodiments.

[0153] Step S96: Obtain the mean absolute percentage error between each predicted filling value in the predicted filling samples and the training labels of the corresponding training missing samples;

[0154] Step S97: Train the initial non-linear prediction model by minimizing the mean absolute percentage error to obtain a non-linear prediction model for the business scenario.

[0155] During each training process, the predicted loss can be calculated for the predicted filling values of the missing values in the predicted filling samples and the corresponding training labels. In this application, the mean absolute percentage error (MAPE) method can be used for calculation, as shown in the following formula:

[0156]

[0157] In formula (11), y i represents the training label of the i-th missing value, represents the predicted filling value of the i-th missing value, where i = 1, 2,..., t, and t is the number of missing values selected during the construction of the training sample set, such as 5, etc.

[0158] In the embodiments of the present application, the mean absolute percentage error calculated this time is used as the prediction loss, and accordingly, the parameters or network structure of the initial non-linear prediction model are adjusted, etc., so as to use the adjusted non-linear prediction model to perform the next training to reduce the mean absolute percentage error until the number of training times reaches the preset number of times, or the mean absolute percentage error is less than the error threshold (a relatively small value, the present application does not limit its size) or tends to be stable / convergent (varies within a relatively small range), it can be considered that the training termination condition is satisfied, and the finally trained non-linear prediction model is recorded for implementing the time series data missing value filling method proposed by the present application.

[0159] It should be noted that for the time series data missing value filling method proposed by the present application, each time series data can be continuous data or discrete data, and the implementation processes are similar. In some other embodiments, the non-linear prediction model can also be a generative model built based on the attention mechanism to achieve missing value filling through a generation task, and the implementation process is not elaborated in the present application.

[0160] In some embodiments, for the trained non-linear prediction model, the present application still conducts comparative experiments on four groups of time series data, namely TEMP, DEWP, WDSP, and VISIB, and selects a certain number of test missing samples for missing value filling tests to determine whether the missing value filling effect of the non-linear prediction model meets the requirements. If not, optimization training can still be continued. As Figure 10 shown, by comparing the filling effects of the time series data missing value filling method proposed by the present application with several traditional missing value filling methods (which can be represented by MAPE as an evaluation index, or the loss value calculated by other loss functions can also be used as an evaluation index, Figure 10 only taking MAPE as an example for illustration), through comparison, it can be known that

[0161] In the comparative experiments on the two groups of time series data, DEWP and VISIB, the MAPEs of the method proposed by the present application are 0.0711 and 0.098 respectively, which are the best results in all comparative experiments. In the TEMP group of time series data, the MAPE of missing value filling is 0.0831, which is only slightly inferior to the MAPE of 0.081 of the missing value filling based on random forest. Measured by the mean value, the MAPE of the method proposed by the present application is 0.1383, achieving the best missing value filling effect. Even in large-scale time series data tasks, relatively excellent missing value filling effects have been achieved, and the present application also fully considers the practicability, applicability, and inference (computing) cost of the method, and can quickly and accurately implement the missing value filling of time series data in various business scenarios. It can also flexibly adjust the model structure or parameters according to the requirements of the actual business scenario to ensure the missing value filling effect of time series data.

[0162] Refer toFigure 11 , which is a schematic flowchart of a method for filling missing values in time series data proposed in the seventh embodiment of this application. This embodiment can refine the description of how to obtain the time series data to be filled for the business scenario and the implementation method of multiple groups of reference time series data of the same type as the time series data to be filled. For example Figure 11 As shown, the method may include:

[0163] Step S111: Obtain each group of original time series data for the business scenario;

[0164] Step S112: Determine the time series data to be filled in each group of original time series data and the clustering characteristics of the missing values existing in the time series data to be filled;

[0165] For the acquisition process of the original time series data and the time series data to be filled, reference can be made to the description of the relevant part of Embodiment 1 above, and this application will not elaborate here.

[0166] In actual production, it may face a large amount of sample (user) information, which makes the model have huge computational costs and serious overfitting at the same time. Therefore, this application uses clustering to filter data (such as a large amount of original time series data or data sets), reduces the interference of irrelevant noise on the attention mechanism, and at the same time reduces the model complexity and increases the inference speed.

[0167] Among them, the clustering characteristics can be selected from the characteristics of the time series data to be filled itself, or other auxiliary information can be used, such as customer registration information in the retail business, credit in the financial scenario, risk response ability, such as the longitude, latitude, altitude, humidity and other characteristics included in the meteorological data, and the longitude, latitude and altitude of these three clustering characteristics. This application does not limit the determination method and content of the clustering characteristics, and can be determined according to the situation.

[0168] Step S113: Cluster each group of original time series data according to the clustering characteristics, and determine the clustering cluster where the time series data to be filled is located;

[0169] Step S114: Obtain multiple groups of reference time series data for the time series data to be filled from the original time series data included in the clustering cluster.

[0170] The present application can use a clustering algorithm to cluster all the original time-series data according to the determined clustering features, determine multiple clustering centers, i.e., data types, and the original time-series data around each clustering center forms a clustering cluster. Then, determine the clustering cluster where the time-series data to be filled is located. After that, screen multiple groups of reference time-series data from this clustering cluster, that is, preliminarily determine the original time-series data similar to the time-series data to be filled. It is also possible to determine all the original time-series data in this clustering cluster except the time-series data to be filled as the reference time-series data. To reduce the computational amount, part of the original time-series data can be selected from all the original time-series data in this clustering cluster except the time-series data to be filled as the reference time-series data. The present application places no restrictions on the quantity of the reference time-series data and its acquisition method.

[0171] Optionally, in addition to the implementation method of determining the original time-series data similar to the time-series data to be filled based on the clustering algorithm described above, other similarity algorithms listed above can also be used to determine, and then all or part of the original time-series data is selected from the original time-series data similar to the time-series data to be filled as the reference time-series data. Priority is given to reducing the magnitude of the reference time-series data on the premise of ensuring that the filling effect of the missing values is not greatly affected.

[0172] It should be understood that during the training process of the non-linear prediction model, the training reference samples for each training missing sample can also be determined according to the method described above. The content and quantity of the clustering features of the missing values for different training missing samples can be the same or different, and the implementation process is not elaborated in the present application.

[0173] The above introduces a method for filling missing values in time-series data provided by an embodiment of the present application. The following will introduce the device for executing the above method for filling missing values in time-series data.

[0174] Refer to Figure 12 , which is a schematic structural diagram of a device for filling missing values in time-series data provided by an embodiment of the present application. As Figure 12 shown, the device for filling missing values in time-series data may include:

[0175] A time-series data acquisition module 121, configured to acquire the time-series data to be filled for a service scenario and multiple groups of reference time-series data of the same type as the time-series data to be filled; there are at least one missing value in the time dimension of the time-series data to be filled;

[0176] A similarity calculation module 122, configured to calculate the similarity between the time-series data to be filled and the multiple groups of reference time-series data respectively, and obtain a first similarity attention matrix between the multiple groups of reference time-series data and the time-series data to be filled;

[0177] The relevance calculation module 123 is configured to calculate the relevance between each reference value and each missing value in the corresponding reference time series data according to the similarity between each group of the reference time series data and the time series data to be filled, so as to obtain a first time series attention matrix between the multiple groups of reference time series data and the time series data to be filled;

[0178] The to-be-predicted time series data obtaining module 124 is configured to obtain to-be-predicted time series data according to the first similarity attention matrix, the first time series attention matrix, and the time series data to be filled;

[0179] The filling module 125 is configured to input the to-be-predicted time series data into a trained non-linear prediction model to fill each missing value, so as to obtain filled time series data for the time series data to be filled.

[0180] Those skilled in the art can understand that the functions and technical effects of the modules in the above device embodiments are equivalent to the corresponding steps described in the foregoing method embodiments. For specific implementation details, reference can be made to the description in the method part, and details are not described herein again.

[0181] An embodiment of the present application further provides a computer program product, including computer-readable instructions, which, when running on an electronic device, enable the electronic device to implement any time series data missing value filling method provided by the embodiments of the present application.

[0182] An embodiment of the present application further provides a computer-readable storage medium, which carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can be enabled to implement any time series data missing value filling method provided by the embodiments of the present application.

[0183] Refer to Figure 13 , which is a schematic hardware structure diagram of an electronic device proposed by an embodiment of the present application. As Figure 13 shown, the electronic device may include: at least one memory 131 and at least one processor 132, where:

[0184] Communication may be possible between at least one memory 131 and at least one processor 132 through a bus. The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 13 only two lines are shown in

[0185] The memory 131 can be used to store multiple computer program instructions for implementing the method for filling missing values in time series data proposed in the embodiments of the present application; the processor 132 can load and execute the computer program instructions stored in the memory 131 to implement each step of the method for filling missing values in time series data proposed in the embodiments of the present application. The implementation process can refer to the description of the corresponding part in the method embodiment above.

[0186] In the embodiments of the present application, the memory 131 can include storage media such as floppy disks, USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs. The processor 132 can include any one or more of processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0187] According to actual needs, the electronic device can further include at least one communication component, which is used to receive the original time series data sent by the acquisition device or send it to other devices (such as servers, etc.) for processing, and can also be used to implement data or instruction transmission between the internal components of the electronic device.

[0188] In the embodiments of the present application, the communication component can include communication components corresponding to wireless communication methods such as wifi, Bluetooth, 5G / 6G, etc., so that the electronic device can implement data transmission with other devices through the communication component. It can also include one or more interfaces supporting wired communication methods, such as general-purpose input / output (GPIO) interfaces, USB interfaces, universal asynchronous receiver / transmitter (UART) interfaces, etc., to implement data transmission between the internal components of the electronic device. The present application does not limit the composition structure of the communication component for implementing this function and its corresponding communication transmission mechanism, and it can be determined according to the situation.

[0189] It should be understood that Figure 13The structure of the electronic device shown does not limit the electronic device in the embodiments of the present application. In practical applications, the electronic device may include more or fewer components than those Figure 13 shown, or combine certain components. When the electronic device is a terminal device such as a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, an intelligent transportation / medical device, a smart home device, or a robot, etc., it may also include components such as a microphone, a speaker, a display screen, various sensors, an antenna, a power module, a radio frequency component, an external port, and other input / output components, which can be determined according to the processing function requirements, and the present application will not give detailed examples one by one.

[0190] In addition, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the device embodiments provided in the present application, the connection relationship between the modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines.

[0191] Through the description of the above embodiments, in the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, training device, or data center to another website, computer, training device, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store, or a data storage device such as a training device or data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0192] In addition, the various embodiments in this specification are described in a progressive or parallel manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For the devices, electronic devices, products, and media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method part.

Claims

1. A method for filling missing values ​​in time series data, the method comprising: Acquire time series data to be filled for a business scenario, and multiple groups of reference time series data of the same type as the time series data to be filled; There is a missing value in at least one time dimension in the time series data to be filled; Calculate similarity between the time series data to be filled and the multiple groups of reference time series data respectively to obtain a first similarity attention matrix between the multiple groups of reference time series data and the time series data to be filled; According to the similarity between each group of the reference time series data and the time series data to be filled, the correlation between each reference value and each missing value in the corresponding reference time series data is calculated to obtain a first time series attention matrix between the multiple groups of reference time series data and the time series data to be filled; Obtaining the time series data to be predicted according to the first similarity attention matrix, the first time series attention matrix and the time series data to be filled; The time series data to be predicted is input into a trained nonlinear prediction model to fill in each of the missing values, so as to obtain filled time series data for the time series data to be filled.

2. According to the method of claim 1, obtaining the time series data to be predicted based on the first similarity attention matrix, the first time series attention matrix and the time series data to be filled includes: Pruning the first similarity attention matrix according to the information entropy of the first similarity attention matrix to obtain a second similarity attention matrix; Pruning the first temporal attention matrix according to the information entropy of the first temporal attention matrix to obtain a second temporal attention matrix; The second similarity attention matrix and the second time series attention matrix are used to process the time series data to be filled to obtain the time series data to be predicted.

3. The method according to claim 2, wherein: If the activation function in the nonlinear prediction model is the first activation function, updating the element values ​​of the first similarity attention matrix and the first temporal attention matrix whose corresponding information entropy is greater than the pruning threshold to negative infinite values; If the activation function in the nonlinear prediction model is the second activation function, the corresponding element values ​​in the first similarity attention matrix and the first temporal attention matrix whose information entropy is greater than the pruning threshold are updated to zero.

4. According to the method according to any one of claims 1 to 3, the similarity calculation of the time series data to be filled and the multiple groups of reference time series data is performed respectively to obtain a first similarity attention matrix between the multiple groups of reference time series data and the time series data to be filled, comprising: Determine the distance weight of each time dimension between the corresponding reference time series data and the time series data to be filled according to the dimensional interval between the reference value in each time dimension in each group of the reference time series data and the time dimension where the missing value is located, and the weight decay coefficient; Using the distance weights corresponding to the time dimensions of each group of the reference time series data, a mean square error calculation is performed on the corresponding reference time series data and the time series data to be filled, so as to obtain the distance between the corresponding reference time series data and the time series data to be filled; Sum the distances between each group of the reference time series data and the time series data to be filled to obtain a total distance; Using the distance between each group of the reference time series data and the time series data to be filled and the total distance, obtain the similarity between the corresponding reference time series data and the time series data to be filled; Among them, the similarity between each group of the reference time series data and the time series data to be filled constitutes a first similarity attention matrix.

5. According to the method according to any one of claims 1 to 3, the correlation between each reference value and each missing value in the corresponding reference time series data is calculated according to the similarity between each group of the reference time series data and the time series data to be filled, so as to obtain a first time series attention matrix between the multiple groups of reference time series data and the time series data to be filled, comprising: Arrange the multiple groups of reference time series data and the time series data to be filled according to the similarity between each group of the reference time series data and the time series data to be filled, to obtain a time series data matrix; wherein the greater the similarity, the smaller the arrangement distance between the reference time series data and the time series data to be filled; Utilizing the arrangement distance between each group of reference time series data and the time series data to be filled in the time series data matrix, and the weight attenuation coefficient, the correlation between each reference value and each missing value in the corresponding reference time series data is calculated to obtain the correlation coefficient between each reference value and each missing value in the corresponding reference time series data; Using the correlation coefficients at each time dimension corresponding to each group of the reference time series data and the number of missing values ​​in the time series data to be filled, the attention values ​​of the corresponding reference time series data and the time series data to be filled at each time dimension are obtained; Among them, the attention values ​​of each group of the reference time series data and the time series data to be filled in each time dimension constitute a first time series attention matrix.

6. The method according to any one of claims 1 to 3, wherein the step of inputting the time series data to be predicted into a trained nonlinear prediction model to fill in each missing value to obtain filled time series data for the time series data to be filled comprises: Inputting the time series data to be predicted into a trained nonlinear prediction model, performing feature extraction at different levels through a single multi-frame detector and a skip connection layer of the nonlinear prediction model to obtain a multi-scale feature map; Performing semantic information fusion on the multi-scale feature map, inputting the fused feature map into the spatial pyramid pooling layer of the nonlinear prediction model for processing, and obtaining a target feature map of a fixed scale; The target feature map is input into the fully connected layer and the activation function layer of the nonlinear prediction model for processing to obtain the filled time series data for the time series data to be filled.

7. The method according to claim 6, wherein: The parameters of the nonlinear prediction model or the structure of the spatial pyramid pooling layer corresponding to different business scenarios are different; The nonlinear prediction model is a nonlinear prediction model trained for the business scenario; Alternatively, in response to an optimization request for the nonlinear prediction model, the parameters of the nonlinear prediction model or the structure of the spatial pyramid pooling layer are adjusted to obtain an optimized nonlinear prediction model, so that the time series data to be predicted is input into the optimized nonlinear prediction model, and the filling time series data for the time series data to be filled is updated.

8. According to the method according to any one of claims 1 to 3, the training process of the nonlinear prediction model comprises: Acquire multiple training samples and multiple training reference samples corresponding to each of the training samples from multiple historical time series data collected for the business scenario; Extract missing values ​​from each training sample to obtain corresponding missing training samples and training labels; Obtaining a similarity training attention matrix and a temporal training attention matrix between each of the missing training samples and the plurality of training reference samples corresponding to the missing training sample; Obtain corresponding missing samples to be predicted according to each of the missing training samples and the similar training attention matrix and the temporal training attention matrix corresponding to the missing training samples; Inputting the missing samples to be predicted into the initial nonlinear prediction model to fill in each missing value, and obtaining prediction filling samples for the corresponding training missing samples; Obtaining the mean absolute percentage error between each predicted filling value in the predicted filling sample and the training label of the corresponding training missing sample; The initial nonlinear prediction model is trained by minimizing the mean absolute percentage error to obtain a nonlinear prediction model for the business scenario.

9. According to the method according to any one of claims 1 to 3, the step of obtaining the time series data to be filled for the business scenario and multiple groups of reference time series data of the same type as the time series data to be filled comprises: Obtain each set of original time series data for business scenarios; Determine the time series data to be filled in each group of original time series data, and clustering features of missing values ​​in the time series data to be filled; According to the clustering features, clustering the groups of original time series data is performed to determine the cluster clusters where the time series data to be filled is located; A plurality of groups of reference time series data for the time series data to be filled are obtained from the original time series data included in the clusters.

10. An electronic device, comprising: at least one memory and at least one processor, wherein: The memory is used to store a plurality of computer program instructions; The processor is used to load and execute the computer program instructions to implement the following steps: Acquire time series data to be filled for a business scenario, and multiple groups of reference time series data of the same type as the time series data to be filled; there is a missing value in at least one time dimension in the time series data to be filled; Calculate similarity between the time series data to be filled and the multiple groups of reference time series data respectively to obtain a first similarity attention matrix between the multiple groups of reference time series data and the time series data to be filled; According to the similarity between each group of the reference time series data and the time series data to be filled, the correlation between each reference value and each missing value in the corresponding reference time series data is calculated to obtain a first time series attention matrix between the multiple groups of reference time series data and the time series data to be filled; Obtaining the time series data to be predicted according to the first similarity attention matrix, the first time series attention matrix and the time series data to be filled; The time series data to be predicted is input into a trained nonlinear prediction model to fill in each of the missing values, so as to obtain filled time series data for the time series data to be filled.

Images