Time sequence smoothing method and device based on master data, equipment and storage medium

Abstract

The invention provides a time sequence smoothing method and device based on master data, equipment and a storage medium, and relates to the technical field of data processing. The method comprises the steps that to-be-repaired time series data are acquired, main data associated with the to-be-repaired time series data are acquired, and the main data at least comprise evaluation data used for evaluating the to-be-repaired time series data; based on the evaluation data, smoothing the to-be-repaired time sequence data, and determining smooth sequence data meeting a target condition; and based on the smooth sequence data meeting the target condition, repairing the to-be-repaired time sequence data to obtain repaired target time sequence data. The embodiment of the invention is used for solving the defects of low accuracy and large noise of the traditional time sequence recovery technology in the prior art, the accuracy of repairing the time sequence data with defects can be improved, and the noise can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a time series smoothing method, device, equipment and storage medium based on master data. Background Art

[0002] In recent years, the development of industrial Internet of Things has been very rapid, and a large number of sensors have been applied to production and life scenarios. With the popularization of various smart devices, a large amount of real-time data has been generated from family life to industrial production. These data are presented in the form of time series, recording various information such as device status, sensor readings, user behavior, etc., reflecting the changes of an observed entity within a certain time range. With the popularization of IoT devices and the continuous increase in the frequency of sensor acquisition, the speed of generating time series data has shown an explosive growth trend.

[0003] However, due to a series of reasons such as sensor failure, network delay, read and write errors, etc., time series data in real life often contains dirty data, that is, there are problems with incomplete or inconsistent data. Incompleteness refers to missing values ​​in the data, and inconsistency refers to the difference between the collected data and the theoretical value. For example, for an engine, its instantaneous torque and instantaneous speed determine the instantaneous fuel consumption rate, and the collected data may be affected by factors such as the environment and deviate, thus inconsistent with the theoretical value. Using and analyzing this data may have a negative impact on researchers, and even ultimately lead to completely wrong conclusions. Therefore, before using the collected time series data, it is necessary to process the data.

[0004] Master data is often used for data processing in relational data scenarios. Master data refers to consistent and shared business entities within an enterprise. It is usually widely used in various business scenarios of the enterprise and is the cornerstone for describing core entities. Master data has the characteristics of globality, sharing, and stability, and can be shared and reused across systems, processes, and departments. In recent years, with the development of master data management (MDM) technology, master data has received more and more attention. For example, e-commerce platforms rely on master data to manage key information related to users and products, and use master data to clean and maintain relational data of business transactions. Unfortunately, there is still no relevant research on master data management in time series scenarios.

[0005] Traditional time series cleaning methods cannot accurately repair the data and may introduce new errors and increase noise, thus affecting the accuracy of downstream tasks such as time series forecasting, anomaly detection, clustering, etc. Summary of the invention

[0006] The present invention provides a time series smoothing method, device, equipment and storage medium based on master data, which are used to solve the problems in the prior art that the collected time series data cannot be accurately further applied due to inconsistency or incompleteness of the collected time series data, and the traditional time series recovery technology has low accuracy and relatively large noise. The present invention achieves that the time series data to be repaired is repaired by generating smoothed sequence data that meets consistency constraints and smoothness constraints and has the lowest generation cost, thereby improving the accuracy of the repair and reducing the noise.

[0007] In a first aspect, the present invention provides a time series smoothing method based on master data, comprising the following steps: Acquire time series data to be repaired, and acquire master data associated with the time series data to be repaired, wherein the master data at least includes evaluation data for evaluating the time series data to be repaired; Based on the evaluation data, smoothing the time series data to be repaired is performed to determine smoothed series data that meets the target condition; Based on the smoothed sequence data that meets the target condition, the time series data to be repaired is repaired to obtain the repaired target time series data; Among them, the target condition is that the smoothed sequence data satisfies the preset constraint condition and the preset cost condition, the constraint condition is that the smoothed sequence data satisfies the consistency constraint and the smoothness constraint at the same time, and the cost condition is the condition for determining the minimum smoothing cost from multiple smoothing costs generated when smoothing the time series data to be repaired.

[0008] Preferably, according to a time series smoothing method based on master data provided by the present invention, The consistency constraint is that the smoothed sequence data meets the requirements of the master data, and the requirements of the master data include data length requirements, data format requirements, and data time requirements; The smoothness constraint is that the time difference between at least two of the smoothed sequence data that are adjacent in time is less than or equal to a smoothing window, and the smoothing window is a moving window set when smoothing the time series data to be repaired.

[0009] Preferably, according to a time series smoothing method based on master data provided by the present invention, the step of determining that two smoothed series data are temporally adjacent specifically includes: Obtaining a timestamp corresponding to each smoothed sequence data; Calculate the timestamp difference between the timestamp of the current smooth sequence data and the timestamp of the adjacent smooth sequence data; the adjacent smooth sequence data is determined according to the time series order of the smooth sequence data set and the timestamp of the current smooth sequence data; the smooth sequence data set is a set composed of a plurality of smooth sequence data arranged according to the order of the timestamp of each smooth sequence data; Compare the timestamp difference with a preset time interval threshold; When the timestamp difference is less than or equal to the preset time interval threshold, it is determined that the current smoothed sequence data and the adjacent smoothed sequence data are temporally adjacent.

[0010] Preferably, according to a time series smoothing method based on master data provided by the present invention, after the step of comparing the timestamp difference with a preset time interval threshold, the method further comprises: When the timestamp difference is greater than the preset time interval threshold, it is determined that the current smoothed sequence data is not temporally adjacent to the adjacent smoothed sequence data.

[0011] Preferably, according to a time series smoothing method based on master data provided by the present invention, the formula of the smoothing cost is: In the formula, is the time series data to be repaired, To smooth the sequence data, It is the distance measure between the time series data to be repaired and the smoothed series data.

[0012] Preferably, according to a time series smoothing method based on master data provided by the present invention, the smoothing window comprises at least a first time window; The step of smoothing the time series data to be repaired based on the evaluation data to determine smoothed series data that meets the target condition includes: According to the timestamp of each of the time series data to be repaired and the smoothing window, determining whether each of the time series data to be repaired is within the first time window; If yes, each of the time series data to be repaired in the first time window is used as initial node data for smoothing to determine the first smoothed series data that meets the target condition; If not, based on the mapping relationship between the time series data to be repaired that is not in the first time window and the master data, determine a candidate sequence set from the evaluation data, and use the candidate sequence set to smooth the time series data to be repaired that is not in the first time window to generate second smoothed sequence data that meets the target condition; Based on the first smoothed sequence data and the second smoothed sequence data, smoothed sequence data satisfying the target condition is determined.

[0013] Preferably, according to a time series smoothing method based on primary data provided by the present invention, the step of smoothing the time series data to be repaired that is not within the first time window using the candidate sequence set to generate second smoothed sequence data that meets the target condition includes: Determine a neighboring window of each of the time series data to be repaired that is not within the first time window; The candidate sequence data in the candidate sequence set are used to perform smoothness verification processing on each of the time series data to be repaired in the adjacent window, and smooth processing is performed on the time series data to be repaired that passes the smoothness verification to generate the second smoothed sequence data.

[0014] Preferably, according to a time series smoothing method based on master data provided by the present invention, each of the time series data to be repaired in the first time window is used as initial node data for smoothing, and the first smoothed series data satisfying the target condition is determined, including: Randomly determine simulated time series data from a plurality of the time series data to be repaired that are not in the first time window, wherein a timestamp of the simulated time series data is greater than a timestamp of any of the time series data to be repaired in the first time window; Determine a smoothing cost generated when each of the time series data to be repaired in the first time window is used as initial node data and each of the simulated time series data is used as termination node data for smoothing; Determine the initial node data corresponding to the smoothing process with the minimum smoothing cost; The time series data to be repaired corresponding to the initial node data with the minimum smoothing cost is used as the initial node data, and a plurality of the time series data to be repaired within the first time window are smoothed to generate the first smoothed series data.

[0015] In a second aspect, the present invention further provides a time series smoothing device based on master data, comprising the following modules: An acquisition module, used for acquiring time series data to be repaired, and acquiring master data associated with the time series data to be repaired, wherein the master data at least includes evaluation data for evaluating the time series data to be repaired; A determination module, configured to perform smoothing processing on the time series data to be repaired based on the evaluation data, and determine smoothed series data that meets the target condition; A repair module is used to repair the time series data to be repaired based on the smoothed sequence data that meets the target condition to obtain the repaired target time series data; wherein the target condition is that the smoothed sequence data satisfies a preset constraint condition and a preset cost condition, the constraint condition is that the smoothed sequence data satisfies both a consistency constraint and a smoothness constraint, and the cost condition is a condition for determining a minimum smoothing cost from a plurality of smoothing costs generated when smoothing the time series data to be repaired.

[0016] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any of the above-mentioned time series smoothing methods based on primary data.

[0017] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described time series smoothing methods based on primary data.

[0018] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned time series smoothing methods based on primary data.

[0019] The present invention provides a method, device, equipment and storage medium for smoothing a time series based on master data, which obtains time series data to be repaired and master data associated with the time series data to be repaired, wherein the master data at least includes evaluation data for evaluating the time series data to be repaired; based on the evaluation data, the time series data to be repaired is smoothed to determine smoothed series data that meets the target condition; based on the smoothed series data that meets the target condition, the time series data to be repaired is repaired to obtain the repaired target time series data; wherein the target condition is that the smoothed series data satisfies a preset constraint condition and a preset cost condition, the constraint condition is that the smoothed series data satisfies a consistency constraint and a smoothness constraint at the same time, and the cost condition is that when the time series data to be repaired is smoothed, the minimum smoothing cost is determined from multiple smoothing costs generated. It is used to solve the defects in the prior art that the collected time series data cannot be accurately further applied due to inconsistency or incompleteness of the collected time series data, and the traditional time series recovery technology has low accuracy and relatively large noise, so as to improve the accuracy of repairing defective time series data and reduce noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a flow chart of a time series smoothing method based on master data provided by the present invention.

[0022] Figure 2 It is a structural schematic diagram of a time series smoothing device based on master data provided by the present invention.

[0023] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] First, several nouns involved in the present invention are analyzed: Master data: Master data is the core business entity data shared across departments and systems within an enterprise, with high stability and accuracy requirements.

[0026] Time series data: Time series data is data collected at different time points, used to describe the changes of phenomena over time. Time series data refers to data collected at different time points, used to reflect the state or degree of change of a thing, phenomenon, etc. over time. This type of data can be quarterly data, monthly data, etc., and can be subdivided into various types such as stationary process and detrended stationary process.

[0027] Smoothing: Smoothing is a statistical method that reduces noise or random fluctuations by applying a mathematical algorithm to the raw data, thus making the data more stable and easier to analyze.

[0028] The main types of smoothing include: Simple Moving Average (SMA): Smooths data by calculating the average of consecutive subsets. This method is suitable for time series without obvious trends and seasonality.

[0029] Weighted Moving Average (WMA): Similar to the SMA, but gives higher weight to more recent observations to reflect their greater influence on the current value.

[0030] Exponential smoothing: gives more weight to recent observations, with the weight decaying exponentially over time.

[0031] Kalman filter: A recursive algorithm for estimating the unseen portion of observed data in a dynamic system. It is particularly well suited for processing data with missing values.

[0032] In the related art, there are at least the following technical problems: Table 1 below shows the vehicle data recorded by the onboard sensor of a mixer truck at a collection frequency of 60 Hz, including information such as longitude and latitude coordinates, engine speed, engine torque, and fuel consumption rate.

[0033] Table 1

[0034] However, due to a series of reasons such as sensor failure, network delay, read and write errors, time series data in real life often contains dirty data, that is, there are problems with incomplete or inconsistent data. Incomplete means that there are missing values ​​in the data, such as some attribute values ​​of data 4 and data 5 in Table 1 are empty.

[0035] Inconsistency refers to the difference between the collected data and the theoretical value. For an engine, its instantaneous torque and instantaneous speed determine the instantaneous fuel consumption rate, and the collected data may be affected by environmental factors and deviate from the theoretical value. For example, data 3 (third row data) and data 7 (seventh row data) in Table 1 have the same torque and speed, but different fuel consumption rates. Using and analyzing these data may have a negative impact on researchers, and may even lead to completely wrong conclusions. Therefore, before using the collected time series data, it is necessary to clean the data.

[0036] Master data is often used for data cleansing in relational data scenarios. Master data refers to consistent and shared business entities within an enterprise. It is usually widely used in various business scenarios of the enterprise and is the cornerstone for describing core entities. Master data has the characteristics of globality, sharing, and stability, and can be shared and reused across systems, processes, and departments. In recent years, with the development of master data management (MDM) technology, master data has received more and more attention. For example, e-commerce platforms rely on master data to manage key information related to users and products, and use master data to clean and maintain relational data of business transactions.

[0037] However, there is still no research on master data management in time series scenarios in the prior art. The traditional average smoothing algorithm will ignore the important information of the original time series scenario, while the exponential smoothing algorithm will increase the noise and have low accuracy in processing the time series data to be repaired.

[0038] Combine the following Figure 1-Figure 3 The present invention describes a time series smoothing method, device, equipment and storage medium based on master data, which is used to solve the problem in the prior art that the collected time series data cannot be accurately further applied due to inconsistency or incompleteness of the collected time series data, and the traditional time series recovery technology has low accuracy and relatively large noise. The method achieves that the time series data to be repaired is repaired by generating smoothed series data that meets consistency constraints and smoothness constraints and has the lowest generation cost, thereby improving the accuracy of the repair and reducing the noise.

[0039] Figure 1 is a flow chart of a time series smoothing method based on master data provided by the present invention, such as Figure 1 As shown, the method may include but is not limited to steps S100 to S300: S100, acquiring time series data to be repaired, and acquiring master data associated with the time series data to be repaired, wherein the master data at least includes evaluation data for evaluating the time series data to be repaired; S200, based on the evaluation data, smoothing the time series data to be repaired to determine smoothed series data that meets the target condition; S300, based on the smoothed sequence data satisfying the target condition, repair the time series data to be repaired to obtain repaired target time series data.

[0040] In step S100 of some embodiments, time series data to be repaired is obtained, and master data associated with the time series data to be repaired is obtained.

[0041] It should be noted that the master data at least includes evaluation data used to evaluate the time series data to be repaired.

[0042] In order to more clearly and completely explain and illustrate the time series smoothing method based on master data provided by the present invention, the vehicle data recorded by the on-board sensor of a mixer truck at a collection frequency of 60 Hz is taken as an example, including information such as longitude and latitude coordinates, engine speed, engine torque, and fuel consumption rate.

[0043] For example, the time series data to be repaired includes 7 rows of data in Table 1 above, each row of data represents a time series data to be repaired, and each time series data to be repaired corresponds to a time value.

[0044] Obtain master data associated with the time series data to be repaired, where master data refers to key business data that is reused in multiple systems and needs to be highly consistent. In an embodiment of the present invention, for example, the time series data to be repaired is collected vehicle time series data, and the master data associated with the vehicle time series data can be management data within the automobile company, as well as some sharable data, and these shared data are used to guide and evaluate the survival of business data such as vehicle time series data.

[0045] In an embodiment of the present invention, the master data may include evaluation data, such as vehicle information, vehicle usage time information, and some standard parameter information of the vehicle, such as vehicle engine information, vehicle spark plug information, etc. Based on these evaluation data information with standard significance, data supplement or data description can be performed on the time series data to be repaired.

[0046] Master Data is the core business entity data shared across departments and systems within an enterprise, and has high stability and accuracy requirements.

[0047] Master data management is a vital part of enterprise information management. It involves data standardization, integration and sharing to ensure the consistency and accuracy of data between different systems.

[0048] In step S200 of some embodiments, the time series data to be repaired is smoothed based on the evaluation data to determine smoothed series data that meets the target condition.

[0049] Firstly, it should be noted that the target condition is that the smoothed sequence data satisfies the preset constraint condition and the preset cost condition.

[0050] Furthermore, the constraint condition is that the smoothed sequence data satisfies both the consistency constraint and the smoothness constraint.

[0051] In some embodiments of the present invention, the consistency constraint is that the smoothed sequence data meets the requirements of the master data, and the requirements of the master data include data length requirements, data format requirements, and data time requirements.

[0052] The smoothness constraint is that the time difference between at least two of the smoothed sequence data that are adjacent in time is less than or equal to a smoothing window, and the smoothing window is a moving window set when smoothing the time series data to be repaired.

[0053] In some embodiments of the present invention, it is considered that the time series data to be repaired is usually read directly from the database. Therefore, in the embodiments of the present invention, it is expected to perform data cleaning efficiently in a stream manner. The smoothness constraint S considers tuples that are close in time, and the timestamp difference is no greater than .

[0054] Therefore, for each tuple in the time series data to be repaired, that is, each time series data to be repaired, we consider a size of window.

[0055] In the formula, For a length of The time series window, also known as the smoothing window, includes arrive All data points of is the window length, Used to describe definitions.

[0056] Furthermore, the consistency constraint It ensures that each tuple of any time series data meets the requirements of the main data, while the smoothness constraint requires that tuples within a time window are close to each other, so the tuples of the smoothed series data in the expected window are also neighbors in the main data.

[0057] Furthermore, the cost condition is a condition for determining a minimum smoothing cost from a plurality of smoothing costs generated when smoothing the time series data to be repaired.

[0058] Define the smoothing cost, Assume that the time series to be repaired is , the formula for smoothing cost is: In the formula, is the time series data to be repaired, To smooth the sequence data, It is the distance measure between the time series data to be repaired and the smoothed series data.

[0059] In some embodiments of the present invention, the time series data to be repaired is defined as and the master data is , through the flat processing is to determine a Smoothed series data , the determined smoothed sequence data must satisfy the smoothing cost Minimum, and at the same time satisfy the consistency constraint and smoothness constraints ,Right now In the formula, is the smoothing cost, Y is the main data, is the window length, and Used to describe the definition, which can be understood as any two data points in the window. for timestamp, for timestamp, is the smoothness threshold.

[0060] Furthermore, the smoothness threshold is used to determine whether the distance between two data points satisfies the smoothness constraint, that is, when the distance between the two data points is less than or equal to the smoothness threshold, it is determined that the distance between the two data points satisfies the smoothness constraint; when the distance between the two data points is greater than the smoothness threshold, it is determined that the distance between the two data points does not satisfy the smoothness constraint.

[0061] Furthermore, it can be understood that the smoothness constraint is that the time difference between at least two of the smoothed sequence data that are adjacent in time is less than or equal to a smoothing window, and the smoothing window is a moving window set when smoothing the time series data to be repaired.

[0062] For example, Table 1 above is the time series data to be repaired, and the moving window value of the smoothing window is 3. Then, when smoothing the first row of time series data to be repaired in Table 1, the first, second, and third rows of time series data to be repaired are selected as the time series data to be repaired in the first smoothing window.

[0063] When smoothing the time series data to be repaired in the second row in Table 1, the time series data to be repaired in the second row, the third row, and the fourth row are selected as the time series data to be repaired in the second smoothing window.

[0064] When smoothing the time series data to be repaired in the third row in Table 1, the time series data to be repaired in the third row, the fourth row, and the fifth row are selected as the time series data to be repaired in the third smoothing window.

[0065] Of course, the moving window value of the smoothing window can be set arbitrarily according to the needs, and can also be 4. Then, when smoothing the time series data to be repaired in the third row of Table 1, the time series data to be repaired in the third row, fourth row, fifth row, and sixth row are selected as the time series data to be repaired in the third smoothing window.

[0066] Furthermore, the smoothness constraint is that the time difference between at least two smoothed sequence data that are temporally adjacent is less than or equal to the smoothing window, and the specific step of determining whether two smoothed sequence data are temporally adjacent may be to first obtain the timestamp corresponding to each of the smoothed sequence data, and then calculate the timestamp difference between the timestamp of the current smoothed sequence data and the timestamp of the adjacent smoothed sequence data.

[0067] The timestamp difference is compared with a preset time interval threshold.

[0068] When the timestamp difference is less than or equal to the preset time interval threshold, it is determined that the current smoothed sequence data and the adjacent smoothed sequence data are temporally adjacent.

[0069] When the timestamp difference is greater than the preset time interval threshold, it is determined that the current smoothed sequence data is not temporally adjacent to the adjacent smoothed sequence data.

[0070] Furthermore, the adjacent smoothed sequence data are determined according to the time series order of the smoothed sequence data set and the timestamp of the current smoothed sequence data.

[0071] The smoothed sequence data set is a set composed of a plurality of smoothed sequence data arranged in the order of the timestamps of each smoothed sequence data.

[0072] It can be understood that, for example, the smoothed sequence data 1 corresponds to a timestamp 1 of 10:16:15, and the fuel consumption information and engine speed information of the smoothed sequence data 1 are not repeated here. Only the timestamp information of the smoothed sequence data is illustrated here to explain the proximity of time.

[0073] Smoothed sequence data 2, corresponding to timestamp 2 is 10:18:15. Smoothed sequence data 3, corresponding to timestamp 3 is 10:19:15. Smoothed sequence data 4, corresponding to timestamp 4 is 10:40:15.

[0074] The smoothed sequence data set consists of smoothed sequence data 1, smoothed sequence data 2, smoothed sequence data 3, and smoothed sequence data 4.

[0075] Assuming that the current smoothed sequence data is smoothed sequence data 3, then according to the time series order of the smoothed sequence data set and the timestamp of the current smoothed sequence data 3, it can be determined that smoothed sequence data 2 and smoothed sequence data 4 are adjacent smoothed sequence data.

[0076] The timestamp difference 1 between the timestamp of the current smoothed sequence data 3 and the timestamp of the adjacent smoothed sequence data 2 is calculated.

[0077] At the same time, the timestamp difference 2 between the timestamp of the current smoothed sequence data 3 and the timestamp of the adjacent smoothed sequence data 4 is calculated.

[0078] Timestamp difference 1 is 1 minute, and timestamp difference 2 is 21 minutes.

[0079] The timestamp difference is compared with a preset time interval threshold.

[0080] Assume that the preset time interval threshold is 2 minutes.

[0081] By comparing the timestamp difference 1 and the timestamp difference 2 with the preset time interval threshold respectively.

[0082] When the timestamp difference 1 is less than or equal to the preset time interval threshold, it is determined that the current smoothed sequence data 3 and the adjacent smoothed sequence data 2 are temporally adjacent.

[0083] When the timestamp difference 2 is greater than the preset time interval threshold, it is determined that the current smoothed sequence data 3 and the adjacent smoothed sequence data 4 are not temporally adjacent.

[0084] Furthermore, after determining that the current smoothed sequence data 3 is temporally adjacent to the adjacent smoothed sequence data 2, it is determined whether the smoothed sequence data 1 adjacent to the smoothed sequence data 2 is also temporally adjacent to the current smoothed sequence data 3, until adjacent smoothed sequence data that is not temporally adjacent is determined.

[0085] In some embodiments of the present invention, the smoothing window includes at least a first time window.

[0086] In some embodiments of the present invention, the first time window In the formula, is the first time window, X is the time series data to be repaired, is the window length, and For a time interval less than Two data points.

[0087] Table 1 above is an example. The moving window value of the smoothing window is 3. When smoothing the first row of time series data to be repaired in Table 1, the first, second, and third rows of time series data to be repaired are selected as the time series data to be repaired in the first smoothing window.

[0088] The first time window includes the first, second, and third rows of time series data to be repaired.

[0089] The step of smoothing the time series data to be repaired based on the evaluation data to determine smoothed series data that meets the target condition includes: According to the timestamp of each of the time series data to be repaired and the smoothing window, determining whether each of the time series data to be repaired is within the first time window; If yes, each of the time series data to be repaired in the first time window is used as initial node data for smoothing to determine the first smoothed series data that meets the target condition; If not, based on the mapping relationship between the time series data to be repaired that is not in the first time window and the master data, determine a candidate sequence set from the evaluation data, and use the candidate sequence set to smooth the time series data to be repaired that is not in the first time window to generate second smoothed sequence data that meets the target condition; Based on the first smoothed sequence data and the second smoothed sequence data, smoothed sequence data satisfying the target condition is determined.

[0090] It can be understood that, based on the timestamp of each time series data to be repaired and the smoothing window, it is judged whether each time series data to be repaired is within the first time window. The timestamp of the time series data to be repaired within the first time window is related to the moving window value of the smoothing window. For example, if the moving window value is 4, then there are 4 time series data to be repaired within the first time window. According to the timestamp of each time series data to be repaired, multiple time series data to be repaired are sorted in the order of the timestamps, and the first 4 time series data to be repaired within the first time window can be determined.

[0091] For each time series data to be repaired, it is determined whether it is within the first time window.

[0092] When it is determined that each time series data to be repaired is within the first time window, each time series data to be repaired within the first time window is used as initial node data for smoothing to determine first smoothed series data that meets the target condition.

[0093] The simulated time series data is randomly determined from a plurality of the to-be-repaired time series data that are not within the first time window.

[0094] Furthermore, the timestamp of the simulated time series data is greater than the timestamp of any of the time series data to be repaired within the first time window.

[0095] For example, the simulated time series data is the seventh row of data in Table 1, and the time series data to be repaired in the first time window is the first, second, and third rows of data in Table 1. Then the timestamps corresponding to the simulated time series data are greater than the timestamps of the first, second, and third rows.

[0096] Determine each of the to-be-repaired time series data in the first time window as initial node data and the simulation time series data as termination node data, and generate a smoothing cost when performing smoothing processing.

[0097] For example, the time series data to be repaired in the first row is the initial node, and the time series data to be repaired in the seventh row is the termination node data, and its smoothing cost is determined to be F1.

[0098] The time series data to be repaired in the second row is the initial node, and the time series data to be repaired in the seventh row is the termination node data, and its smoothing cost is determined to be F2.

[0099] The time series data to be repaired in the third row is the initial node, and the time series data to be repaired in the seventh row is the termination node data, and its smoothing cost is determined to be F3.

[0100] By comparing F1, F2, and F3, for example, if F2 is smaller than F1, and F1 is smaller than F3, it can be determined that the smoothing cost of F2 is the smallest.

[0101] Then, the initial node data corresponding to the smoothing processing with the smallest smoothing cost F2 is determined, and the time series data to be repaired corresponding to the initial node data with the smallest smoothing cost is used as the initial node data, that is, the time series data to be repaired in the second row is used as the initial node data, and the multiple time series data to be repaired in the first time window are smoothed to generate the first smoothed sequence data. In this way, the cold start problem can be solved and the first smoothed sequence data in the first time window can be prevented from generating higher noise points.

[0102] In some embodiments of the present invention, the step of smoothing the time series data to be repaired that is not within the first time window using the candidate sequence set to generate second smoothed sequence data that meets the target condition includes: Determine the adjacent windows of each of the time series data to be repaired that are not in the first time window, and then obtain the time series data to be repaired in the adjacent windows. Finally, use the candidate sequence data in the candidate sequence set to perform smoothness verification processing on each of the time series data to be repaired in the adjacent windows, and perform smoothing processing on the time series data to be repaired that pass the smoothness verification to generate the second smoothed sequence data. In this way, it is possible to avoid generating the second smoothed sequence data based on those defective time series data to be repaired, thereby improving the accuracy of the second smoothed sequence data.

[0103] In step S300 of some embodiments, the time series data to be repaired is repaired based on the smoothed series data that meets the target condition to obtain repaired target time series data.

[0104] It can be understood that after executing step S200, the specific execution step may be to repair the time series data to be repaired by using the smoothed sequence data that meets the target condition to obtain the repaired target time series data.

[0105] For example, in the case where the fuel consumption rate information in the 7th row and the 3rd row in Table 1 is inconsistent, based on the smoothed sequence data that meets the target conditions, it can be predicted that the fuel consumption information in the 7th row is erroneous information. Therefore, the fuel consumption information in the 3rd row is used to repair and update the information in the 7th row to obtain the repaired target time series data.

[0106] For example, when the fuel consumption rate information of the 4th row in Table 1 is missing, based on the smoothed sequence data that meets the target conditions, it can be predicted that the fuel consumption rate information of the 4th row is 35.1. Then the missing fuel consumption rate information of the 4th row is updated to obtain the repaired target time series data.

[0107] Furthermore, time series prediction, anomaly detection, clustering, etc. can be performed based on the repaired target time series data to verify the accuracy of the repair. When the repair accuracy is greater than or equal to the preset accuracy threshold, the repair can be considered successful, otherwise, the repair is performed again.

[0108] The present invention provides a method, device, equipment and storage medium for smoothing a time series based on master data, which obtains time series data to be repaired and master data associated with the time series data to be repaired, wherein the master data at least includes evaluation data for evaluating the time series data to be repaired; based on the evaluation data, the time series data to be repaired is smoothed to determine smoothed series data that meets the target condition; based on the smoothed series data that meets the target condition, the time series data to be repaired is repaired to obtain the repaired target time series data; wherein the target condition is that the smoothed series data satisfies a preset constraint condition and a preset cost condition, the constraint condition is that the smoothed series data satisfies a consistency constraint and a smoothness constraint at the same time, and the cost condition is that when the time series data to be repaired is smoothed, the minimum smoothing cost is determined from multiple smoothing costs generated. It is used to solve the defects in the prior art that the collected time series data cannot be accurately further applied due to inconsistency or incompleteness of the collected time series data, and the traditional time series recovery technology has low accuracy and relatively large noise, so as to improve the accuracy of repairing defective time series data and reduce noise.

[0109] The time series smoothing device based on master data provided by the present invention is described below. The time series smoothing device based on master data described below and the time series smoothing method based on master data described above can correspond to each other.

[0110] like Figure 2 FIG. 1 is a schematic diagram of the structure of a time series smoothing device based on master data provided by the present invention, wherein the time series smoothing device based on master data comprises the following modules: An acquisition module 210, configured to acquire time series data to be repaired, and to acquire master data associated with the time series data to be repaired, wherein the master data at least includes evaluation data for evaluating the time series data to be repaired; A determination module 220 is used to perform smoothing processing on the time series data to be repaired based on the evaluation data to determine smoothed series data that meets the target condition; The repair module 230 is used to repair the time series data to be repaired based on the smoothed sequence data that meets the target condition to obtain the repaired target time series data; wherein the target condition is that the smoothed sequence data satisfies a preset constraint condition and a preset cost condition, the constraint condition is that the smoothed sequence data satisfies both a consistency constraint and a smoothness constraint, and the cost condition is a condition for determining the minimum smoothing cost from multiple smoothing costs generated when smoothing the time series data to be repaired.

[0111] Preferably, the time series smoothing device based on master data provided by the present invention is further used for the consistency constraint that the smoothed sequence data meets the requirements of the master data, and the requirements of the master data include data length requirements, data format requirements, and data time requirements; The smoothness constraint is that the time difference between at least two of the smoothed sequence data that are adjacent in time is less than or equal to a smoothing window, and the smoothing window is a moving window set when smoothing the time series data to be repaired.

[0112] Preferably, according to a time series smoothing device based on master data provided by the present invention, determining that two smoothed series data are temporally adjacent is specifically used to obtain a timestamp corresponding to each of the smoothed series data; Calculate the timestamp difference between the timestamp of the current smooth sequence data and the timestamp of the adjacent smooth sequence data; the adjacent smooth sequence data is determined according to the time series order of the smooth sequence data set and the timestamp of the current smooth sequence data; the smooth sequence data set is a set composed of a plurality of smooth sequence data arranged according to the order of the timestamp of each smooth sequence data; Compare the timestamp difference with a preset time interval threshold; When the timestamp difference is less than or equal to the preset time interval threshold, it is determined that the current smoothed sequence data and the adjacent smoothed sequence data are temporally adjacent.

[0113] Preferably, a time series smoothing device based on master data provided by the present invention is further configured to determine that the current smoothed sequence data is not temporally adjacent to adjacent smoothed sequence data when the timestamp difference is greater than the preset time interval threshold.

[0114] Preferably, according to a time series smoothing device based on master data provided by the present invention, the formula for the smoothing cost is: In the formula, is the time series data to be repaired, To smooth the sequence data, It is the distance measure between the time series data to be repaired and the smoothed series data.

[0115] Preferably, according to a time series smoothing device based on master data provided by the present invention, The smoothing window includes at least a first time window; The determination module 220 is further specifically used to determine whether each of the time series data to be repaired is within the first time window according to the timestamp of each of the time series data to be repaired and the smoothing window; If yes, each of the time series data to be repaired in the first time window is used as initial node data for smoothing to determine the first smoothed series data that meets the target condition; If not, based on the mapping relationship between the time series data to be repaired that is not in the first time window and the master data, determine a candidate sequence set from the evaluation data, and use the candidate sequence set to smooth the time series data to be repaired that is not in the first time window to generate second smoothed sequence data that meets the target condition; Based on the first smoothed sequence data and the second smoothed sequence data, smoothed sequence data satisfying the target condition is determined.

[0116] The determination module 220 is further specifically configured to determine an adjacent window of each of the time series data to be repaired that is not within the first time window; The candidate sequence data in the candidate sequence set are used to perform smoothness verification processing on each of the time series data to be repaired in the adjacent window, and smooth processing is performed on the time series data to be repaired that passes the smoothness verification to generate the second smoothed sequence data.

[0117] The determination module 220 is further specifically configured to randomly determine simulated time series data from a plurality of the time series data to be repaired that are not in the first time window, wherein a timestamp of the simulated time series data is greater than a timestamp of any of the time series data to be repaired in the first time window; Determine a smoothing cost generated when each of the time series data to be repaired in the first time window is used as initial node data and each of the simulated time series data is used as termination node data for smoothing; Determine the initial node data corresponding to the smoothing process with the minimum smoothing cost; The time series data to be repaired corresponding to the initial node data with the minimum smoothing cost is used as the initial node data, and a plurality of the time series data to be repaired within the first time window are smoothed to generate the first smoothed series data.

[0118] The present invention provides a method, device, equipment and storage medium for smoothing a time series based on master data, which obtains time series data to be repaired and master data associated with the time series data to be repaired, wherein the master data at least includes evaluation data for evaluating the time series data to be repaired; based on the evaluation data, the time series data to be repaired is smoothed to determine smoothed series data that meets the target condition; based on the smoothed series data that meets the target condition, the time series data to be repaired is repaired to obtain the repaired target time series data; wherein the target condition is that the smoothed series data satisfies a preset constraint condition and a preset cost condition, the constraint condition is that the smoothed series data satisfies a consistency constraint and a smoothness constraint at the same time, and the cost condition is that when the time series data to be repaired is smoothed, the minimum smoothing cost is determined from multiple smoothing costs generated. It is used to solve the defects in the prior art that the collected time series data cannot be accurately further applied due to inconsistency or incompleteness of the collected time series data, and the traditional time series recovery technology has low accuracy and relatively large noise, so as to improve the accuracy of repairing defective time series data and reduce noise.

[0119] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310 , a communications interface 320 , a memory 330 and a communication bus 340 , wherein the processor 310 , the communications interface 320 , and the memory 330 communicate with each other via the communication bus 340 . The processor 310 can call the logic instructions in the memory 330 to execute a time series smoothing method based on master data, which includes: obtaining time series data to be repaired, and obtaining master data associated with the time series data to be repaired, wherein the master data at least includes evaluation data for evaluating the time series data to be repaired; based on the evaluation data, smoothing the time series data to be repaired to determine smoothed sequence data that meets the target condition; based on the smoothed sequence data that meets the target condition, repairing the time series data to be repaired to obtain the repaired target time series data; wherein the target condition is that the smoothed sequence data satisfies a preset constraint condition and a preset cost condition, the constraint condition is that the smoothed sequence data satisfies a consistency constraint and a smoothness constraint at the same time, and the cost condition is that when the time series data to be repaired is smoothed, a minimum smoothing cost is determined from multiple smoothing costs generated.

[0120] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0121] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the time series smoothing method based on master data provided by the above methods, the method including: obtaining time series data to be repaired, and obtaining master data associated with the time series data to be repaired, the master data at least including evaluation data for evaluating the time series data to be repaired; based on the evaluation data, smoothing the time series data to be repaired to determine smoothed sequence data that meets a target condition; based on the smoothed sequence data that meets the target condition, repairing the time series data to be repaired to obtain the repaired target time series data; wherein the target condition is that the smoothed sequence data satisfies a preset constraint condition and a preset cost condition, the constraint condition is that the smoothed sequence data simultaneously satisfies a consistency constraint and a smoothness constraint, and the cost condition is when the time series data to be repaired is smoothed, the condition for determining the minimum smoothing cost from multiple smoothing costs generated.

[0122] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the time series smoothing method based on master data provided by the above-mentioned methods, the method comprising: obtaining time series data to be repaired, and obtaining master data associated with the time series data to be repaired, the master data at least comprising evaluation data for evaluating the time series data to be repaired; based on the evaluation data, smoothing the time series data to be repaired to determine smoothed sequence data that meets a target condition; based on the smoothed sequence data that meets the target condition, repairing the time series data to be repaired to obtain repaired target time series data; wherein the target condition is that the smoothed sequence data satisfies a preset constraint condition and a preset cost condition, the constraint condition is that the smoothed sequence data simultaneously satisfies a consistency constraint and a smoothness constraint, and the cost condition is that when the time series data to be repaired is smoothed, a minimum smoothing cost is determined from multiple smoothing costs generated.

[0123] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A time series smoothing method based on master data, characterized in that: include: Acquire time series data to be repaired, and acquire master data associated with the time series data to be repaired, wherein the master data at least includes evaluation data for evaluating the time series data to be repaired; Based on the evaluation data, smoothing the time series data to be repaired is performed to determine smoothed series data that meets the target condition; Based on the smoothed sequence data that meets the target condition, the time series data to be repaired is repaired to obtain the repaired target time series data; Among them, the target condition is that the smoothed sequence data satisfies the preset constraint condition and the preset cost condition, the constraint condition is that the smoothed sequence data satisfies the consistency constraint and the smoothness constraint at the same time, and the cost condition is the condition for determining the minimum smoothing cost from multiple smoothing costs generated when smoothing the time series data to be repaired.

2. The time series smoothing method based on master data according to claim 1 is characterized in that: The consistency constraint is that the smoothed sequence data meets the requirements of the master data, and the requirements of the master data include data length requirements, data format requirements, and data time requirements; The smoothness constraint is that the time difference between at least two of the smoothed sequence data that are adjacent in time is less than or equal to a smoothing window, and the smoothing window is a moving window set when smoothing the time series data to be repaired.

3. The time series smoothing method based on master data according to claim 2 is characterized in that: The step of determining that two smoothed sequence data are temporally adjacent specifically comprises: Obtaining a timestamp corresponding to each smoothed sequence data; Calculate the timestamp difference between the timestamp of the current smooth sequence data and the timestamp of the adjacent smooth sequence data; the adjacent smooth sequence data is determined according to the time series order of the smooth sequence data set and the timestamp of the current smooth sequence data; the smooth sequence data set is a set composed of a plurality of smooth sequence data arranged according to the order of the timestamp of each smooth sequence data; Compare the timestamp difference with a preset time interval threshold; When the timestamp difference is less than or equal to the preset time interval threshold, it is determined that the current smoothed sequence data and the adjacent smoothed sequence data are temporally adjacent.

4. The time series smoothing method based on master data according to claim 3 is characterized in that: After the step of comparing the timestamp difference with a preset time interval threshold, the method further includes: When the timestamp difference is greater than the preset time interval threshold, it is determined that the current smoothed sequence data is not temporally adjacent to the adjacent smoothed sequence data.

5. The time series smoothing method based on master data according to claim 1 is characterized in that: The formula for the smoothing cost is: In the formula, is the time series data to be repaired, To smooth the sequence data, It is the distance measure between the time series data to be repaired and the smoothed series data.

6. The time series smoothing method based on master data according to claim 3 is characterized in that: The smoothing window includes at least a first time window; The step of smoothing the time series data to be repaired based on the evaluation data to determine smoothed series data that meets the target condition includes: According to the timestamp of each of the time series data to be repaired and the smoothing window, determining whether each of the time series data to be repaired is within the first time window; If yes, each of the time series data to be repaired in the first time window is used as initial node data for smoothing to determine the first smoothed series data that meets the target condition; If not, based on the mapping relationship between the time series data to be repaired that is not in the first time window and the master data, determine a candidate sequence set from the evaluation data, and use the candidate sequence set to smooth the time series data to be repaired that is not in the first time window to generate second smoothed sequence data that meets the target condition; Based on the first smoothed sequence data and the second smoothed sequence data, smoothed sequence data satisfying the target condition is determined.

7. The time series smoothing method based on master data according to claim 6 is characterized in that: The step of smoothing the time series data to be repaired that is not within the first time window by using the candidate sequence set to generate second smoothed sequence data that meets the target condition includes: Determine a neighboring window of each of the time series data to be repaired that is not within the first time window; The candidate sequence data in the candidate sequence set are used to perform smoothness verification processing on each of the time series data to be repaired in the adjacent window, and smooth processing is performed on the time series data to be repaired that passes the smoothness verification to generate the second smoothed sequence data.

8. The time series smoothing method based on master data according to claim 6 is characterized in that: The step of taking each of the time series data to be repaired in the first time window as initial node data for smoothing to determine the first smoothed series data that meets the target condition includes: Randomly determine simulated time series data from a plurality of the time series data to be repaired that are not in the first time window, wherein a timestamp of the simulated time series data is greater than a timestamp of any of the time series data to be repaired in the first time window; Determine a smoothing cost generated when each of the time series data to be repaired in the first time window is used as initial node data and each of the simulated time series data is used as termination node data for smoothing; Determine the initial node data corresponding to the smoothing process with the minimum smoothing cost; The time series data to be repaired corresponding to the initial node data with the minimum smoothing cost is used as the initial node data, and a plurality of the time series data to be repaired within the first time window are smoothed to generate the first smoothed series data.

9. A time series smoothing device based on master data, characterized in that: include: An acquisition module, used for acquiring time series data to be repaired, and acquiring master data associated with the time series data to be repaired, wherein the master data at least includes evaluation data for evaluating the time series data to be repaired; A determination module, configured to perform smoothing processing on the time series data to be repaired based on the evaluation data, and determine smoothed series data that meets the target condition; A repair module is used to repair the time series data to be repaired based on the smoothed sequence data that meets the target condition to obtain the repaired target time series data; wherein the target condition is that the smoothed sequence data satisfies a preset constraint condition and a preset cost condition, the constraint condition is that the smoothed sequence data satisfies both a consistency constraint and a smoothness constraint, and the cost condition is a condition for determining a minimum smoothing cost from a plurality of smoothing costs generated when smoothing the time series data to be repaired.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the time series smoothing method based on master data as claimed in any one of claims 1 to 8 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the time series smoothing method based on master data as claimed in any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the time series smoothing method based on master data as claimed in any one of claims 1 to 8 is implemented.

Images