Computer-implemented method for determining data synchronization status between a source database and a target database
Patent Information
- Application Number
- JP2024513121
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-08-27
- Filing Date
- 2022-08-29
- Publication Date
- 2025-09-08
AI Technical Summary
Current methods for determining data synchronization status between legacy and advanced databases are inefficient and prone to errors, leading to asynchronous conditions that are difficult to detect and correct.
A computer-implemented method using source and target maps to structure time-dependent data into frames, followed by determining hash values for these frames to compare and assess synchronization status, allowing for efficient and less computationally expensive synchronization assessment.
This method enables effective and efficient determination of data synchronization between legacy and advanced databases by reducing the need for direct comparison of large data sets, thereby minimizing computational costs and errors.
Smart Images

Figure 00000020_0000 
Figure 00000020_0001 
Figure 00000020_0002
Abstract
Description
[Technical field]
[0001] FIELD OF THEINVENTION The present invention relates to a computer-implemented method, a computational framework, a database system and a computer program product for determining a data synchronization status between a source database and a target database. [Background technology]
[0002] 2. Background of the Invention In current industrial production processes, a number of sensors are provided that monitor the production of products in an industrial plant. The data generated by the number of sensors is typically stored in the form of time series datasets in a process database system, sometimes called a historian, of the industrial plant. Such time series datasets are each associated with a tag that can be considered as an identifier for the time series dataset and that contains a series of time-dependent data values, e.g. measurements of one or more sensors of the industrial plant. Such time series datasets often contain time-dependent data values for many years during which the sensors are measuring parameters of the industrial production plant. Furthermore, it is very common that the process database system in which the time series datasets are stored is not updated to the current technology level, but continues to operate at the current technology level as a legacy system. However, nowadays it is often desirable to use the data provided in the time series datasets in other contexts, e.g. for further analysis of data in higher ranking computational systems, such as big data mining or enterprise control systems. For this purpose, the time series datasets are retrieved from the process database system as a source database and copied to a target database, e.g. a cloud-based database system or a technologically advanced database system that allows access to the time series datasets for the respective desired application. However, copying such huge amounts of data in a continuous manner, especially considering possible constraints of the target database system, may result in failures in the copy procedure, resulting in errors in the copied data set on the target database. In the long run, such failures and errors may lead to a state in which the source and target databases are out of sync. Such an out-of-sync state is, of course, undesirable.However, a simple comparison of the time-dependent data sets stored in the target database and the source database one by one to determine the data synchronization state between the database systems is very inefficient or even impossible, taking into account the technical constraints of the source database, for example due to the fact that it has to be carried out at a time scale that may be unacceptable only in exceptional circumstances or at a lower technical level. Therefore, today, in this context, the data synchronization state of the respective target database and source database is often not known or is only determined if there is non-negligible actual evidence that the two databases are not synchronized. It would therefore be advantageous if a method was provided that allows determining the data synchronization state between the source database and the target database in a more effective and computationally less expensive way in the context of a time series data set corresponding to an industrial plant. Summary of the Invention [Problem to be solved by the invention]
[0003] Summary of the Invention The object of the present invention is to provide a computer-implemented method, a computational framework, a database system and a computer program product that allows for determining the data synchronization status between a process database system of an industrial plant as a source database and a target database in a more efficient and computationally inexpensive manner.Furthermore, the object of the present invention is to enable the determination of the continuous data synchronization status of such databases, when the source database refers to a legacy system and the target database refers to a technologically superior system. [Means for solving the problem]
[0004] In a first aspect of the present invention, a computer implemented method for determining a data synchronization status between a source database and a target database is provided, the target database being configured to store a copy of past time series data sets stored in a source database of an industrial plant, each time series data set being associated with a respective tag and including a respective series of time dependent data values, the method comprising: i) providing a source map for the source database, the source map indicating a structuring of at least a portion of the time dependent data values stored in the source database into a plurality of source data frames, the time dependent data values associated with the data frames being represented by a) respective tags associated with the respective time series data sets; and b) respective start times and respective end times of the time dependent data values of each time series data set associated with the respective tags. the source data frames correspond to the target data frames; ii) providing a target map for the target database, the target map indicating a structuring of at least a portion of the time-dependent data values stored in the target database into a plurality of target data frames, the target map corresponding to the source map such that the structuring of the target database corresponds to the structuring of the source database, whereby the source data frames correspond to the target data frames, respectively; iii) determining hash values for the source data frames and the target data frames, the hash values being sensitive to the time-dependent data values associated with the respective data frames; and iv) determining a data synchronization state by comparing the hash values of the target data frames with the hash values of the corresponding source data frames.
[0005] Since the source database and the target database are respectively structured into a plurality of source data frames and target data frames, hash values are determined for the source data frames and the target data frames, and the data synchronization state is based on a comparison of the hash values, a plurality of time-dependent data values can be compared simultaneously, such that determining the data synchronization state between the source database and the target database is less time-consuming. Furthermore, since a huge amount of time-dependent data values can be compared based on a relatively small number of hash values, the method is computationally less expensive and more effective, such that implementing the method does not overload legacy systems.
[0006] The time series data sets stored in the source database, which may be considered as a process database system of an industrial plant, may refer to any series of time-dependent data values associated with a respective tag. Preferably, each time series data set includes time-dependent data values that refer to measurements of sensors provided in the industrial plant for monitoring the production process of the industrial plant. Thus, each time series data set may also be considered as control data utilized in monitoring and controlling the industrial plant. In a preferred embodiment, the tag associated with the time-dependent data values may indicate the identity of a sensor in the industrial plant that provided the respective time-dependent data value. For example, the time series data set may refer to a time series of temperature measurements provided by a temperature sensor in a chemical reaction during the production of a particular product. The temperature sensor may, for example, be stored in the process database system in association with a tag indicating the identity of the temperature sensor and thus adapted to provide a temperature measurement every few seconds generating the respective time series data set. However, in other embodiments, the series of time-dependent data values may refer to data values measured by multiple sensors, not just one sensor. In this case, the tag associated with the time-dependent data values may indicate multiple sensors or may be completely independent of the source of the time-dependent data values.
[0007] In general, a time series data set includes, in addition to the time dependent data values, time stamps associated with the time dependent data values to indicate the time at which the time dependent data values were measured. Furthermore, optionally, the time series data set may further include a quality value associated with each time dependent data value of the time series data set, the quality value may indicate the quality of the measurement of the respective time dependent data value or any event related to the measurement of the time dependent data value. Without limiting the generality or scope of the present teachings, in an embodiment, a time series data set may refer to an ordered insertion time series data set defined by the most recently inserted, i.e. stored, time dependent data values associated with the time series data set, which are time dependent data values associated with a recent time stamp compared to all other time stamps associated with already stored time dependent data values. Thus, an ordered insertion time series data set may be considered to refer to a time series data set in which all time dependent data values are stored sequentially, i.e. in the order of the associated time stamps. Thus, the most recent time dependent data value is stored after the most recently inserted time dependent data value associated with a time stamp indicating that a measurement was performed before the most recent time dependent data value.
[0008] The time series datasets stored in the source database of the industrial plant are copied continuously, at predefined time intervals or at least occasionally to a target database that may be, for example, technically advanced with respect to the source database or that at least allows further access to the time series datasets for, for example, technically advanced applications. Generally, for time series datasets for which a data synchronization state has been determined, the copying process has already been completed, so the time series datasets stored in the source database and the target database may be considered as past time series datasets.
[0009] The method includes providing a source map for the source database. Generally, providing the source map may include generating the source map. However, the source map may also already be stored in the respective storages, such that providing may include retrieving the source map from the respective storages. For example, the source map may be a source map already used in determining a previous data synchronization state of the source database and the target database. The source map indicates a structuring of at least a portion of the time-dependent data values stored in the source database into a plurality of source data frames. In particular, the structuring refers to an association of the time-dependent data values stored in the source database with the respective source data frames. The time-dependent data values associated with the data frames are defined by a) respective tags associated with the respective time-series data sets, and b) respective start times and respective end times of the time-dependent data values of the respective time-series data sets associated with the respective tags.
[0010] The method further includes providing a target map for the target database. Generally, again in this case, providing the target map may also include generating the target map. However, the target map may also already be stored in the respective storage, such that providing may include retrieving the target map from the respective storage. For example, the target map may be a target map already used in determining a previous data synchronization state of the source database and the target database. The target map also indicates a structuring of at least a portion of the time-dependent data values stored in the target database into a plurality of target data frames. Generally, in the following, when the term "data frame" is used without being preceded by "target" or "source", the respective description applies to both the target data frame and the source data frame.
[0011] The structuring may be considered as an association of time-dependent data values with their respective target data frames. In general, the data frames indicated by the target or source map are defined by the time-dependent data values associated with the respective data frames, which are defined by the time-dependent data set to which they belong, i.e. the tag, and further by the timestamps associated with the time-dependent data values, which may be indicated by the respective start time and the respective end time. Thus, data frames such as the target and source data frames may be considered as a grouping of certain time-dependent data values under the same identity, i.e. the respective data frames. The source and target maps may refer to any kind of data structure that allows identifying the time-dependent data values associated with the respective data frames. For example, the source or target map may refer to a list or table that lists all the time-dependent data values that point to the respective data frames. However, the source or target map may also refer to a more abstract data structure, e.g. a data structure that defines the boundaries of the data frames in an abstract data coordinate system in which the time-dependent data values may be considered to be located, e.g. based on their timestamps and their respective associated tags. Providing the source and target maps may also be considered as providing the source and target data frames.
[0012] In general, a target map corresponds to a source map. In particular, a target map corresponds to a source map if the structuring of the target database corresponds to the structuring of the source database, such that the source data frames correspond to the target data frames, respectively. In one embodiment, the structuring of the target database corresponds to the structuring of the source database if, in a synchronized state, each target data frame is defined by the same time-series data values as the corresponding source data frame, respectively.
[0013] In an embodiment, the source and target maps, and thus the respective data frames, are preferably optimized with respect to retrieval of the time series data values of the respective data frames from the source and / or target databases. In particular, when the source database refers to a legacy system, the size of the respective data frames, i.e. the amount of data associated with the data frames, preferably takes into account the computational constraints of the legacy source database. For example, the source and target maps may be generated based on the data retrieval resource consumption and speed of the known source database. Additionally or alternatively, the data density of the respective time-dependent data sets, i.e. the amount of time-dependent data values stored per unit time, may be taken into account when generating the source and target maps. In a preferred embodiment, optimizing with respect to retrieval of the time-dependent data values of the respective data frames from the source and / or target databases comprises performing a test data extraction, which refers to extracting a predetermined amount of any data from the source and / or target databases, and based on the test data extraction, the responsiveness and data density of the respective databases may be determined. The optimization may be performed by solving the respective optimization problem with respect to predetermined boundary conditions. The boundary conditions preferably refer to a minimum and maximum number of data points per data frame and / or a maximum number of tags per data frame. For example, the number of data points per data frame can be set to be between 50,000 and 5,000,000 data points per data frame, and the maximum number of tags can be set to 3,000. In general, these boundary conditions can be determined heuristically, in particular based on experience with the respective database system. The source and target maps can then be determined by solving an optimization problem with respect to these boundary conditions. Furthermore, in the event of a failure during the processing of the method, for example due to an overload of one of the databases, the respective optimizations are preferably modified by modifying the respective boundary conditions. This modification can be performed until an accuracy criterion is met, for example until the number of failed requests per time falls below a predefined threshold.The individual boundary conditions thus found for each database can then be stored and used in any further optimizations for determining the source and target maps.
[0014] The method further includes determining hash values for the source data frames and the target data frames. In particular, a hash value is preferably determined for each source data frame and each target data frame defined by the source map and the target map, respectively. However, in some embodiments, it may be technically reasonable to determine hash values for only a portion of the source data frames and the target data frames, for example, data synchronization states may need to be determined for only a portion of the source database and the target database. In this case, hash values are determined for corresponding source data frames and target data frames.
[0015] Generally, a hash value refers to a value returned by applying a hash function to a data value. The hash function has the characteristic of mapping data values of any size to data values of a fixed size. Preferably, the hash value is determined based on a predefined hash function. In particular, the hash value is determined to be sensitive to the time-dependent data values associated with the respective data frame. Thus, the hash value is indicative of the time-dependent data values associated with the respective data frame. Preferably, the utilized hash function maps the time-dependent data values associated with the respective data frame to only one hash value. However, the hash function may also be adapted to map the time-dependent data values of the respective data frame to more than one hash value, preferably a hash value smaller than the time-dependent data values associated with the data frame. Since the source and target maps define the respective target and source data frames, the target and source data maps are preferably utilized to identify the time-dependent data values of the respective data frames on the respective databases in order to determine the hash value. However, in other embodiments, the data frames defined by the source and target data maps may already be retrieved from the respective databases and temporarily stored, for example in a temporary storage, before being provided for determining the hash value.
[0016] In a preferred embodiment, the hash value refers to an overall hash value and is determined based on the individual hash values determined for each respective time-dependent data value of the data frame. In particular, a first hash function may be utilized to map the time-dependent data values of the data frame to a hash value. In particular, the individual hash values are preferably determined based on the time, value and / or quality associated with the time-dependent data values. For example, the time associated with the time-dependent data values may refer to a timestamp associated with the time-dependent data values. The quality may refer to a quality value, optionally also associated with the time-dependent data values. The term "value" in this case also refers to the value of the time-dependent data values themselves. The overall hash value can then be determined based on the individual hash values, for example by again utilizing a second hash function that maps the individual hash values to an overall hash value. However, in a preferred embodiment, the overall hash value is determined based on the individual hash values by utilizing a Merkle tree.
[0017] The method then includes determining a data synchronization state by comparing the hash value of the target data frame with the hash value of the corresponding source data frame. If the hash value refers to a global hash value, the data synchronization state is determined by comparing the global hash value of the target data frame with the global hash value of the corresponding source data frame. In particular, the comparing includes determining whether the hash value of the target data frame and the hash value of the corresponding source data frame are the same. Since the hash values of the data frames indicate the time-dependent data values associated with the respective data frames, the hash values of the target data frame and the source data frame being the same is a direct indication that all the time-dependent data values associated with the respective data frames are also the same, i.e., in sync. On the other hand, different hash values are a direct indication that the time-dependent data values of the target data frame and the corresponding source data frame are not in sync, e.g., indicating that one or more time-dependent data values are missing on the target database or an incorrect value of the time-dependent data value has been copied to the target database.
[0018] The data synchronization state may then be determined based on the results of all comparisons performed so far. The data synchronization state may refer to a simple statement that the data of the target database and the source database are in sync or not. However, the data synchronization state may also be determined to indicate the level of synchronization, if there is synchronization between the source database and the target database. For example, the respective synchronization level may be provided based on the number of data frames for which different hash values have been determined. Furthermore, the data synchronization state may simply be considered as an internal value or information utilized for further processing. For example, further processing may be initiated based on whether one or more data frames are in sync. Thus, the data synchronization state may, for example, be provided to a user or may be utilized as internal information that is not directly accessible to the user.
[0019] In one embodiment, data frames are associated with time-dependent data values of different respective time-series data sets, defined by a) different respective tags and b) the same respective start time and respective end time. In particular, it is preferred that different respective tags associated with a data frame refer to subsequent tags. Thus, in this case, a data frame may be defined in a data space spanned in one dimension by tags and in another dimension by the timeline of the time-series data set by defining a tag range and a time range for the respective data frame. Then, all time-dependent data values that are within the defined tag range and time range are associated with the respective data frame.
[0020] In one embodiment, if during the determination of the data synchronization state, it is determined that at least one hash value of the target data frame differs from the hash value of the corresponding source data frame, the method further includes: i) generating a sub-source map for the source database, the sub-source map indicating a structuring of time-dependent data values of the source data frame into sub-source data frames; ii) generating a sub-target map for the target database, the sub-target map indicating a structuring of time-dependent data values of the corresponding target data frame into sub-target data frames, the sub-source map corresponding to the sub-target map; iii) determining sub-hash values for the sub-source data frames and the sub-target data frames, the sub-hash values being sensitive to the time-dependent data values associated with the respective sub-data frames; and iv) determining one or more sub-data frames that include different time-dependent data values by comparing the sub-hash values of the sub-target data frames with the sub-hash values of the corresponding sub-source data frames, the recursion being performed until one or more time-dependent data values causing respective differences between the hash values of the target data frame and the source data frame can be identified.
[0021] In particular, the recursive generation of the sub-source map, the sub-target map, the determination of the sub-hash value and the comparison of the sub-hash values refer to subsequent simultaneous iterations of the method defined above, utilizing as the basis for the generation of the respective sub-maps the portions of the target database and the source database for which the out-of-sync state has been determined in the previous step of the recursion. In particular, in the above-mentioned recursive method, the sub-data frame for which the out-of-sync state has been determined may form the basis of the next recursion level for generating the respective sub-sub-maps of the sub-data frame. In the following, to avoid problems of terminology, the addition "sub" is always used to indicate that the respective function is utilized at any one of the lower recursion levels following the first determination of the data synchronization state. In general, however, the recursion may include any number of iterations until an interruption criterion is met. In particular, the interruption criterion refers to making it possible to identify the respective time-dependent data values that cause the desynchronization of the database. For example, if in one of the sub-steps of the recursion a data frame that is out-of-sync is identified and that data frame is associated with a number of time-dependent data values below a predefined threshold, the recursion may be interrupted. The threshold value may be determined, for example, based on the knowledge that for this respective number of time-dependent data values, a fast single-value comparison can be performed that allows for directly identifying the time-dependent data value that causes each difference. However, in some embodiments, the recursion may also be performed until only one time-dependent data value is associated with the data frame, in which case the final recursion step refers to a direct comparison of the time-dependent data values.
[0022] In one embodiment, the method further includes updating the target database with the identified time-dependent data values such that the target database and the source database are synchronized.
[0023] In one embodiment, the source map and the target map each refer to a key-value map, providing a mapping between a) storage locations of corresponding data frames on the source database and the target database, respectively, and b) respective hash values of the corresponding data frames. Preferably, the comparison of the hash values of the target data frames with the hash values of the corresponding source data frames is based on the key-value mapping of the source map and the target map, respectively.
[0024] In a further aspect of the invention, a computational framework for determining a data synchronization status between a source database and a target database is presented, the target database being configured to store copies of past time series datasets stored in a source database of an industrial plant, each time series dataset being associated with a respective tag and including a respective series of time dependent data values, the framework comprising: i) a source map providing unit for providing a source map for the source database, the source map indicating a structuring of at least a portion of the time dependent data values stored in the source database into a plurality of source data frames, the time dependent data values associated with the data frames being defined by a) respective tags associated with the respective time series datasets, and b) respective start times and respective end times of the time dependent data values of each time series dataset associated with the respective tags. the source map corresponding to the structuring of the target database corresponds to the structuring of the source database, whereby the source data frames correspond respectively to the target data frames; iii) a hash value determining unit for determining hash values for the source data frames and the target data frames, where the hash values are sensitive to the time-dependent data values associated with the respective data frames; and iv) a synchronization state determining unit for determining a data synchronization state by comparing a hash value of a target data frame with a hash value of a corresponding source data frame.
[0025] In a further aspect of the present invention, a database system is presented, the system comprising: i) a source database provided at an industrial plant and configured to store a time series dataset of the industrial plant; ii) a target database configured to store a copy of a historical time series dataset stored in the source database of the industrial plant; and iii) a computational framework as described above.
[0026] In a further aspect of the present invention, a computer program product for determining a data synchronization status between a source database and a target database is presented, the computer program product comprising program code means for causing a computational framework as described above to perform a method as described above.
[0027] It is to be understood that the method as described above, the computational framework as described above, the database system as described above and the computer program product as described above have similar and / or identical preferred embodiments, in particular as defined in the dependent claims.
[0028] It is to be understood that a preferred embodiment of the invention can be any combination of the dependent claims or the above-mentioned embodiments with the respective independent claim.
[0029] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief description of the drawings]
[0030] [Figure 1] 1 illustrates, in a schematic and exemplary manner, one embodiment of a database system including a computational framework for determining a data synchronization status between a source database and a target database. [Diagram 2] 1 is a flow chart illustrating an exemplary embodiment of a method for determining a data synchronization status between a source database and a target database. [Diagram 3]1 illustrates, in a schematic and exemplary manner, the general principle of one embodiment of a method for determining a data synchronization status between a target database and a source database. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0031] Detailed Description of the Preferred Embodiments FIG. 1 illustrates, in a schematic and exemplary manner, a database system 100 including a source database 121 provided in an industrial plant 120, a target database 131 provided anywhere, for example in the cloud 130, and a computational framework 110 for determining a data synchronization status between the source database 121 and the target database 131.
[0032] The source database 121 is provided in the industrial plant 120 and is configured to store the time series dataset of the industrial plant 120. In general, the industrial plant 120 may refer to any technology platform used for industrial purposes. The industrial purpose may be the manufacturing or processing of one or more industrial products, i.e., a manufacturing process or processing performed by the industrial plant. For example, the industrial purpose may refer to the production of a product. The product may be any physical product, such as, for example, chemical, biological, pharmaceutical, food, beverage, textile, metal, plastic, or semiconductor. Additionally or alternatively, the product may also be a service product, including electricity, heating, air conditioning, waste treatment such as recycling, chemical processing such as disintegration or melting, or incineration. Accordingly, the industrial plant 120 may be one or more of a chemical plant, a process plant, a pharmaceutical plant, a fossil fuel processing facility such as an oil well and / or a natural gas valve, a refinery, a petrochemical plant, a fractionation plant, and the like. The industrial plant 120 may also be either a distillery, an incinerator, or a power plant. The industrial plant 120 may also be any combination of any of the examples given above.
[0033] To carry out the production process, the industrial plant 120 includes a technology infrastructure that is controllable by, for example, control parameters implemented in the technology infrastructure by a process control system 122. The technology infrastructure may include equipment or process units such as heat exchangers, towers such as fractionation towers, furnaces, reaction chambers, catalytic cracking units, storage tanks, settlers, pipelines, stacks, filters, valves, actuators, transducers, circuit breakers, machinery, such as turbines, generators, grinders, compressors, fans, pumps, rotating heavy equipment such as motors, etc. Furthermore, the industrial plant 120 typically includes a number of sensors that allow the operating parameters of the technology infrastructure to be measured. The measured operating parameters are stored by the process control system 122 in a source database 121, which may also be considered as a process database system of the industrial plant 120. Furthermore, the operating parameters may also be utilized by the process control system 122 for controlling the production process in the industrial plant 120. The operating parameters measured by the sensors may relate to various process parameters and / or parameters related to equipment or process units. For example, sensors may be used to measure process parameters such as flow rate in a pipeline, fluid level in a tank, temperature of a furnace, chemical composition of gas, and some sensors may be used to measure turbine vibration, fan speed, valve opening, corrosion of a pipeline, voltage across a transformer, and the like. The differences between these sensors may be based not only on the parameter they sense, but also on the sensing principle they employ. Some examples of sensors based on the parameter they sense may include temperature sensors, pressure sensors, radiation sensors such as light sensors, flow sensors, vibration sensors, displacement sensors, and chemical sensors that detect certain substances, such as gases, and the like. Examples of sensors employing different sensing principles may be impedance sensors such as piezoelectric sensors, piezoresistive sensors, thermocouples, capacitance sensors, and resistance sensors, and the like. Sensors generally measure operational parameters in the form of time-dependent data values, i.e. data values associated with a particular time at which they are measured by the sensor.These time-dependent data values measured by the sensors are typically stored in the source database 121 in the form of a time series data set.
[0034] The source database 121 may refer, for example, to a storage including dedicated hardware and / or software for storing time series datasets. However, the source database 121 may also refer to any other computing system allowing general storage or specifically storage of time series datasets. Each time series dataset stored in the source database 121 is associated with a tag that may be considered not only as an identifier of the respective time series dataset, but also, optionally, as an identifier of one or more sensors at which the time-dependent data values of the respective time series dataset are measured. Furthermore, each respective time series dataset associated with a respective tag includes time-dependent data values stored together with a timestamp indicating the time at which the respective time-dependent data value was measured. Optionally, each time series dataset may also include a quality value for each time-dependent data value indicating the quality of the measurement of the respective time-dependent data value. Thus, the source database 121 preferably stores, for each sensor of the industrial plant 120, a respective time series dataset that is continuously updated each time a new time-dependent data value is measured by the respective sensor.
[0035] Typically, the industrial plant 120 may be integrated into an enterprise control system for managing and controlling the production carried out by the industrial plant 120. For the management and control of the industrial plant 120, it is often desirable to retrieve measurement data provided by sensors from the source database 121 and store the retrieved measurement data in the target database 131. In particular, the target database 131 may be selected to allow further processing of the measurement data, i.e. the time series data sets, by other dedicated and more complex management and control systems, such as enterprise control systems. In particular, in many practical applications, the computational frameworks for managing, controlling and retrieving the time series data sets on the source database 121 refer to legacy systems that are not adapted for more complex processing and analysis of the respective data. In this way, the periodic or continuous copying of the time series data sets stored in the source database 121 to the target database 131, which is adapted to not only allow access to but also preferably support more complex data analysis and management tools, such as artificial intelligence-based management tools or big data analysis algorithms, allows further advantageous processing of the time series data sets provided by the industrial plant 120. For example, the target database 131 may be provided in a cloud-based environment that allows access from multiple locations, computers or workspaces. However, for further use and processing of the copied time series data set on the target database 131, it would be advantageous to be able to guarantee that the copied time series data set is the same as the time series data set originally stored on the source database 121, i.e., to know the data synchronization status between the target database 131 and the source database 121.
[0036] As such, the database system 100 includes a computation framework 110 for determining a data synchronization state between the source database 121 and the target database 131. The computation framework 110 includes a source map providing unit 111, a target map providing unit 112, a hash value determining unit 113, and a synchronization state determining unit 114. Optionally, the computation framework 110 may further include an input unit 115, such as a keyboard, a touch screen, a computer mouse, a microphone, etc., that allows a user to access the computation framework 110 and provide input to the computation framework 110. Furthermore, the computation framework 110 may also include an output unit 116 for providing output to a user, such as, for example, a display, a light unit, a speaker, etc.
[0037] The source map providing unit 111 is adapted to provide a source map of the source database 121. The source map indicates a structuring of at least a portion of the time-dependent data values stored in the source database 121 into a plurality of source data frames. In general, the data frames may be defined by the time-dependent data values associated with the data frames. The time-dependent data values may be generally defined by a) respective tags associated with respective time-dependent data sets of the time-dependent data values, and b) respective time frames, e.g. start times and end times, of the time-dependent data values of respective time-series data sets associated with respective tags. Thus, the data frames may also be defined by the tags and time frames of the time-dependent data values associated with the data frames. The source map may then be adapted to include, for example, a mapping of the time-dependent data values to respective data frames, where the mapping may be indicated by referring to the tags and respective time frames of the respective time-dependent data values. Thus, the structuring of the source database 121 into a plurality of data frames may also be considered as a form of grouping of the time-dependent data values stored in the source database 121. In particular, in this aspect, the group of time dependent data values refers to a data frame, and the members of each group of time dependent data values are associated with the same data frame. Preferably, a common data frame is associated with time dependent data values of different respective time series data sets defined by different respective tags and the same respective start times and respective end times. More preferably, the different tags refer to subsequent tags.
[0038] A schematic and exemplary illustration of this principle is provided in Fig. 3. In Fig. 3, time-dependent data values, such as time-dependent data value 311, are stored in a source database, shown as a schematic representation 310. The representation 310 of the source database 121 shows the time-dependent data values stored in the source database 121 in the form of a two-dimensional diagram, where a first dimension 301 refers to a timeline indicating the time associated with each time-dependent data value, and a second dimension 302 refers to the tag with which each time-dependent data value is associated. In general, the tag may refer to any number or other index, but in this example the tags are numbered sequentially starting from 1 and continuing up to the tag number represented by N. In this representation 310 of the source database 121, the time-dependent data values are structured into four data frames 313, 314, 315, 316. In this respect, the data frames can be easily defined by the tags associated with the data frames and the respective time intervals associated with the data frames, where the time intervals can be provided in the form of start and end times. In this manner, the time-dependent data values of the time series data set associated with each data frame 313, 314, 315, 316 are clearly defined.
[0039] The target map providing unit 112 is adapted to provide a target map for the target database 131, applying the same principles as described above with respect to the source map. In particular, the target map is provided such that the structuring of the target database 131 corresponds to that of the source database 121, whereby the source data frames correspond respectively to the target data frames. Or, defined otherwise, the time-dependent data values stored in the target database 131 and the source database 121 are grouped into data frames such that always the same time-dependent data values are grouped into the respective source and target data frames. Thus, when this type of structuring is utilized, in a synchronized state, each target data frame is defined respectively by the same time series data values as the corresponding source data frame.
[0040] This general principle of the invention is also further illustrated in Figure 3 when comparing representation 310 of source database 121 with representation 320 of target database 131, showing that when source database 121 and target database 131 are in sync, each source data frame 313, 314, 315, 316 contains a corresponding target data frame 323, 324, 325, 326 that contains the same time dependent data values. For example, the time dependent data value 311 associated with source data frame 314 is the same as the time dependent data value 321 associated with target data frame 324.
[0041] The hash value determination unit 113 is adapted to determine hash values for the source and target data frames, in particular for all source and target data frames. However, the hash value determination unit 123 may also be adapted to determine hash values only for a portion of the target and source data frames or only for certain source and target data frames. In general, hash values are determined by the hash value determination unit 113 for corresponding source and target data frames. The hash values are determined to be sensitive to the time-dependent data values associated with the respective data frames. In general, the hash values are determined by applying a hash function to one or more data values. For example, a hash function may be applied to all time-dependent data values of the respective data frames, the applied hash function mapping the time-dependent data values to one or more hash values. In a preferred embodiment, the hash value determination unit 113 is adapted to first apply a hash function to each time-dependent data value of the data frame, its associated timestamp and optionally its quality value, such that an individual hash value is determined for each time-dependent data value of the data frame. In a next step, the hash value determination unit 113 is then preferably adapted to apply a further mapping function, such as a further hash function or preferably a Merkle tree, to all the individual hash values of the data frames to determine an overall hash value of the respective data frame. The overall hash value is then taken as the hash value of the respective data frame. Applying a hash function has the effect that the amount of data that has to be compared to identify a data synchronization state can be significantly reduced.
[0042] The synchronization state determination unit 114 is adapted to determine a data synchronization state between the source database 121 and the target database 131 by comparing the hash value of the target data frame with the hash value of the corresponding source data frame, in particular by comparing the hash value of each target data frame with the hash value of each corresponding source data frame for which the hash value is being determined. Since the hash values are sensitive to the time-dependent data values, equal hash values indicate that the respective data frames to which the hash values belong contain the same time-dependent data value and are therefore synchronized. On the other hand, different hash values indicate a difference in the time-dependent data value between the respective data frames and thus indicate an out-of-sync state between the source database 121 and the target database 131. An example where the time-dependent data values are different is also provided in FIG. 3. In particular, the source data frame 313 of the source database representation 310 contains a time-dependent data value 312, while the corresponding target data frame 323 is missing this time-dependent data value, as indicated by the reference symbol 322. Therefore, the hash values of these two data frames are expected to be different.
[0043] To determine the synchronization between the source database 121 and the target database 131, the synchronization state determination unit 114 may be adapted to apply the above principles to the complete database, i.e., all time-dependent data values, but the synchronization state determination unit 114 may also be adapted to determine the data synchronization state of only a part of the database, in which case the above principles are applied only to this part of the database. In general, the data synchronization state may refer to information indicating whether the source database and the target database or the source database and the part of the target database are synchronized or not. However, the data synchronization state may also indicate the level of synchronization. For example, it may indicate the amount of difference of the time-dependent data values, and thus provide an indication of the quality of the copy process between the source database and the target database. Thus, the data synchronization state may be provided to a user, for example, by means of the output unit 116, to inform the user regarding the data synchronization state. However, the data synchronization state may also be used only as internal information, for example, to determine whether the target database needs to be synchronized with the source database.
[0044] In a preferred embodiment, when the synchronization state determination unit 114 determines different hash values for the target data frame and the source data frame and thus determines the data out-of-sync state, the computation framework 110 may be adapted to apply again the same principles as described above for each identified data frame as part of the database. In particular, the principles are preferably applied recursively. For example, each identified data frame may then be considered as part of the database, and for the source data frame, again a source map may be generated, which in this case may be considered as a sub-source map, in terms of the principles as described above. Furthermore, a target map may again be generated for the target database and each target data frame, which in this case may be considered as referring to a sub-target map, in terms of the principles as described above. In particular, the sub-source map again corresponds to the sub-target map of the respective data frame. The data frames defined by the sub-target map and the sub-source map may then be considered as sub-data frames. Again, as also described above, for example, the hash value determination unit 113 may be adapted to determine a hash value for each sub-target data frame and sub-source data frame. These hash values may then be considered as sub-hash values. Thereafter, the synchronization state determination unit 114 may again be adapted to compare the sub-hash values of each corresponding sub-data frame.
[0045] Fig. 2 shows, in a schematic and exemplary manner, a flow chart of a method for determining a data synchronization state between a source database and a target database. In particular, the method 200 can be implemented according to the principles already described above, for example by the system described with respect to Fig. 1. In particular, the method 200 comprises a step 210 of providing a source map for the source database. The source map, as already described above, indicates a structuring of at least a portion of the time-dependent data values stored in the source database into a plurality of source data frames. In particular, the time-dependent data values associated with the data frames are defined by respective tags associated with respective time-series data sets and respective start times and respective end times of the time-dependent data values of the respective time-series data sets associated with the respective tags. Furthermore, the method 200 comprises a step 220 of providing a target map for the target database. The target map indicates a structuring of at least a portion of the time-dependent data values stored in the target database into a plurality of target data frames. As already described in detail also with respect to Fig. 1, the target map corresponds to the source map such that the structuring of the target database corresponds to the structuring of the source database. Thus, the source data frames respectively correspond to the target data frames. In general, the step 210 of providing a target map and the step 220 of providing a source map can each be performed in any order, for example, instead of providing the source map first, the target map can be provided first. Furthermore, the source map and the target map can be provided simultaneously, or even generated simultaneously, as long as the above defined conditions are met. In a subsequent step 230, the method 200 comprises determining hash values for the source data frame and the target data frame, the hash values being sensitive to time-dependent data values associated with the respective data frames. Furthermore, in a step 240 of the method 200, a data synchronization state is determined by comparing the hash value of the target data frame with the hash value of the corresponding source data frame.
[0046] Further, optionally, when it is determined that at least one hash value of the target data frame is different from the hash value of the corresponding source data frame, the method 200 may optionally include recursively repeating steps 210-240 based on the determined target data frame and source data frame. In particular, in this case, providing a source map and a target map may refer to providing a sub-source map and a sub-target map for each data frame. Furthermore, determining a hash value in this case may also refer to determining a sub-hash value, and then determining a data synchronization state by comparing the sub-hash values, thereby enabling further determining which time-dependent data values cause differences between the target database and the source database. In particular, the recursion defined above may be performed until one or more time-dependent data values causing differences between the target database and the source database are identified.
[0047] In the following, a more detailed embodiment of the invention mentioned above is described. In this embodiment, a hash function hash_vqt(v,q,time) maps a data-triple, e.g., time, quality and the data value itself, optionally associated with each time-dependent data value, to a pseudo-unique hash value. This function can be implemented, for example, by computing a string hash of a serialized data-triple object based on the data value optionally associated with the time-dependent data value. Furthermore, a hash function hash_tag(tagid,starttime,endtime) can be defined that generally maps a value defined by a tag identifier, a start time and an end time to a pseudo-unique hash value. For example, this function can internally use a function like hash_vqt defined above and a Merkle tree to compute the overall hash value. Furthermore, when more than one tag is assumed to be implemented, and especially when hash values are assumed to be determined for multiple tags, a function hash_block(tagset,starttime,endtime) can be defined that maps a value defined by a set of tags, a start time and an end time to a pseudo-unique hash value. For example, this function can internally use the function hash_tag defined above and a Merkle tree to calculate the overall block hash value.
[0048] Based on these functions, an embodiment of the computer-implemented method defined above can be provided. In particular, in a first step, a source map and a target map can be generated or provided. Based on the source and target maps, the source and target data frames can be identified, for example, by "splitting" the source and target databases into data blocks, i.e. groups of time-dependent data values defined by data frames, respectively. Preferably, the data blocks have a size, i.e. amount of data, that allows utilizing database queries that are small enough to process the time-dependent data values on both databases. In particular, constraints on the size of the data blocks, and therefore the size of the data frames, can be taken into account with respect to processing data by potential legacy source databases. A hash value can then be determined for each data frame of the source and target databases using the hash_block function defined above. If a comparison of the hash values of the corresponding target and source data frames indicates a difference in the data values associated with the respective data frames, the respective data frames can be subdivided into smaller data blocks, for example based on the respective sub-maps. The algorithm described above can then be performed recursively on the subdivided data blocks until a time-dependent data value causing a mismatch is identified. If no hash value differences are found during a comparison of the hash values of the data frames, then the portions of the database whose hash values were compared are fully synchronized.
[0049] In practice, when starting to recursively subdivide a data block to find the root cause of the mismatch in the hash of the data block, there will be a turning point where it is easier to directly compare the time-dependent data values stored in the source and target databases and, optionally, the data associated with the time-dependent data values, instead of performing further recursions. Therefore, for example during a test run of the algorithm, it is preferable to predetermine the number of time-dependent data values at this turning point as a recursion threshold, and implement this recursion threshold as a recursion interruption criterion.
[0050] There are several strategies for computing the optimal data frame structure for each database, i.e. for generating optimal source and target maps. For example, it is preferable to find a compromise between the query count, i.e. the amount of data retrieved during a query to retrieve time-dependent data values associated with the data frame, database resource consumption and speed. Preferably, for this optimization, the data density of each tag, i.e. in this case the count of data points in the database per time unit, is tracked.
[0051] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.
[0052] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.
[0053] A single unit or device may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
[0054] The steps performed by one or more units or devices, such as providing a source map, providing a target map, determining a hash value or comparing hash values, etc. may be performed by any number of other units or devices. These steps may be implemented as program code means of a computer program and / or as dedicated hardware.
[0055] The computer program product may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, may be supplied together with or as part of other hardware, but may also be distributed in other forms, for example via the Internet or other wired / wireless communication systems.
[0056] Any unit described herein may be a processing unit that is part of a computing system. A processing unit may include a general-purpose processor, but may also include a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any other dedicated circuit. Any memory may be a physical system memory, which may be volatile, non-volatile, or some combination of the two. The term "memory" may include a computer-readable storage medium, such as a non-volatile mass storage. If the computing system is distributed, the processing and / or storage capabilities may also be distributed. A computing system may include multiple structures as "executable components." The term "executable components" is a well-understood structure in the computing arts as a structure that may be software, hardware, or a combination thereof. For example, when implemented in software, one skilled in the art will understand that the executable component structure may include software objects, routines, methods, etc. that may be executed on the computing system. This may include both executable components in the heap of the computing system or executable components on a computer-readable storage medium. The executable component structure may reside on a computer-readable medium that, when interpreted by one or more processors, e.g., processor threads, of the computing system, causes the computing system to perform certain functions. Such structures may be directly computer readable by a processor, such as where the executable components are binary, or may be interpretable and / or compilable, e.g., in either a single step or multiple steps, to generate such binary which is directly interpretable by a processor. In other examples, the structures may be hard-coded or hard-wired logic gates that are implemented exclusively or nearly exclusively in hardware, such as in a field programmable gate array (FPGA), application specific integrated circuit (ASIC), or other dedicated circuitry.Thus, the term "executable components" is a term for structures well understood by those of ordinary skill in the computing arts, whether implemented in software, hardware, or a combination thereof. Any embodiment herein is described with reference to acts performed by one or more processing units of a computing system. When such acts are implemented in software, one or more processors direct the operation of the computing system in response to executing the computer-executable instructions that make up the executable components. A computing system may also include communication channels that allow the computing system to communicate with other computing systems, for example, over a network. A "network" is defined as one or more data links that allow the transmission of electronic data between computing systems and / or modules and / or other electronic devices. When information is transferred or provided to a computing system via a network or another communication connection, for example, either hardwired, wireless, or a combination of hardwired and wireless, the computing system properly regards the connection as a transmission medium. A transmission medium may include a network and / or data links that can be used to carry desired program code means in the form of computer-executable instructions or data structures and can be accessed by a general-purpose or special-purpose computing system or combination thereof. Although not all computing systems require a user interface, in some embodiments a computing system includes a user interface system for use in interfacing with a user. The user interface serves as an input or output mechanism to a user, for example, via a display.
[0057] Those skilled in the art will appreciate that the present invention may be implemented in a networked computing environment having many types of computing system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, cell phones, PDAs, pagers, routers, switches, data centers, wearable devices such as glasses, etc. The present invention may also be implemented in a distributed system environment where tasks are performed by both local and remote computing systems that are linked, for example, by either hardwired data links, wireless data links, or a combination of hardwired and wireless data links over a network. In a distributed system environment, program modules may be located in both local and remote memory storage devices.
[0058] Those skilled in the art will appreciate that the present invention may also be implemented in a cloud computing environment. A cloud computing environment may be distributed, but this is not required. When distributed, a cloud computing environment may be distributed internationally within an organization and / or may have components held across multiple organizations. For purposes of this specification and the following claims, "cloud computing" is defined as a model that enables on-demand network access to a shared pool of configurable computing resources, e.g., networks, servers, storage, applications and services. The definition of "cloud computing" is not limited to any of the many other advantages that may be obtained when such a model is deployed. The computing system of the figure includes various components or functional blocks that may implement various embodiments disclosed herein, as described. The various components or functional blocks may be implemented on a local computing system or may be implemented on a distributed computing system that includes elements that reside in the cloud or implement aspects of cloud computing. The various components or functional blocks may be implemented as software, hardware or a combination of software and hardware. The computing system shown in the figure may include more or fewer components than those shown in the figure and may combine some of the components as circumstances permit.
[0059] Any reference signs in the claims shall not be construed as limiting the scope.
[0060] The present invention relates to a method for determining a synchronization state between a source database and a target database, wherein a dataset is associated with a tag and comprises a series of data values, in one step a source map is provided which shows an organization of values stored in the source database into data frames, the values associated with the frames being defined by tags associated with the dataset, in one step a target map is provided for the target database, the target map corresponding to the source map, in one step hash values are determined for the data frames, the hash values being sensitive to the data values associated with each data frame, and in one step a data synchronization state is determined by comparing the hash values.
Claims
1. 1. A computer-implemented method (200) for determining a data synchronization status between a source database (121) and a target database (131), wherein the target database (131) is configured to store copies of historical time series data sets stored in the source database (121) of an industrial plant (120), each time series data set being associated with a respective tag and including a respective series of time-dependent data values, the method (200) comprising: providing (210) a source map for the source database (121), the source map indicating a structuring of at least some of the time-dependent data values stored in the source database (121) into a plurality of source data frames (313, 314, 315, 316), the time-dependent data values associated with a data frame being defined by a) respective tags associated with respective time-series data sets, and b) respective start times and respective end times of the time-dependent data values of the respective time-series data sets associated with the respective tags; providing (220) a target map for the target database (131), the target map indicating a structuring of at least some of the time-dependent data values stored in the target database (131) into a plurality of target data frames (323, 324, 325, 326), the target map corresponding to the source map such that the structuring of the target database (131) corresponds to the structuring of the source database (121), whereby the source data frames (313, 314, 315, 316) correspond to the target data frames (323, 324, 325, 326), respectively; determining (230) hash values for the source data frames (313, 314, 315, 316) and the target data frames (323, 324, 325, 326), the hash values being sensitive to the time-dependent data values associated with each of the data frames; determining (240) the data synchronization state by comparing the hash values of the target data frames (323, 324, 325, 326) with the hash values of the corresponding source data frames (313, 314, 315, 316); A method (200) comprising:
2. 2. The method (200) of claim 1, wherein the data frames are associated with time-dependent data values of different respective time-series data sets defined by a) different respective tags and b) the same respective start time and respective end time.
3. 3. The method (200) of claim 1 or 2, wherein the structuring of the target database (131) corresponds to the structuring of the source database (121) when, in a synchronized state, each target data frame (323, 324, 325, 326) is defined by the same time series data values as the corresponding source data frame (313, 314, 315, 316).
4. The method (200) of claim 1, wherein the hash value is determined based on a predefined hash function.
5. 2. The method (200) of claim 1, wherein the hash value refers to an overall hash value and is determined based on individual hash values determined for each time-dependent data value of each of the data frames.
6. 6. The method (200) of claim 5, wherein the overall hash value is determined based on the individual hash values using a Merkle tree.
7. 7. The method (200) of claim 5 or 6, wherein the individual hash values are determined based on a time, a value and / or a quality associated with the time-dependent data values.
8. If during the determination of the data synchronization state it is determined that at least one hash value of the target data frame (323) is different from the hash value of the corresponding source data frame (313), then recursively: - generating a sub-source map for said source database (121), said sub-source map indicating the structuring of said time-dependent data values of said source data frames (313) into sub-source data frames; - generating a sub-target map for said target database (131), said sub-target map indicating a structuring of said time-dependent data values of said corresponding target data frame (323) into sub-target data frames, said sub-source map corresponding to said sub-target map; - determining sub-hash values for the sub-source data frames and the sub-target data frames, the sub-hash values being sensitive to the time-dependent data values associated with each sub-data frame; 2. The method (200) of claim 1, further comprising: determining one or more sub-data frames that include different time-dependent data values by comparing the sub-hash values of the sub-target data frames with the sub-hash values of the corresponding sub-source data frames, wherein the recursion is performed until the one or more time-dependent data values that cause respective differences between the hash values of the target data frame (323) and the source data frame (313) can be identified.
9. 9. The method (200) of claim 8, further comprising updating the target database (131) with the identified time-dependent data values so that the target database (131) and the source database (121) are synchronized.
10. 2. The method (200) of claim 1, wherein the source map and the target map each refer to a key-value map, providing a mapping between a) the storage locations of the corresponding data frames on the source database (121) and the target database (131), respectively, and b) the hash values of each of the corresponding data frames.
11. 11. The method (200) of claim 10, wherein the comparison of the hash values of the target data frames (323, 324, 325, 326) with the hash values of the corresponding source data frames (313, 314, 315, 316) is based on the key-value mapping of the source map and the target map, respectively.
12. 1. A computational framework (110) for determining a data synchronization status between a source database (121) and a target database (131), wherein the target database (131) is configured to store copies of past time series data sets stored in the source database (121) of an industrial plant (120), each time series data set being associated with a respective tag and including a respective series of time-dependent data values, the computational framework (110) comprising: a source map providing unit (111) for providing a source map for said source database (121), said source map indicating a structuring of at least some of the time dependent data values stored in said source database (121) into a plurality of source data frames (313, 314, 315, 316), said time dependent data values associated with a data frame being defined by a) respective tags associated with respective time series data sets, and b) respective start times and respective end times of the time dependent data values of said respective time series data sets associated with said respective tags; a target map providing unit (112) for providing a target map for the target database (131), the target map indicating a structuring of at least some of the time-dependent data values stored in the target database (131) into a plurality of target data frames (323, 324, 325, 326), the target map corresponding to the source map such that the structuring of the target database (131) corresponds to the structuring of the source database (121), whereby the source data frames (313, 314, 315, 316) correspond to the target data frames (323, 324, 325, 326), respectively; a hash value determination unit (113) for determining hash values for the source data frames (313, 314, 315, 316) and the target data frames (323, 324, 325, 326), the hash values being sensitive to the time-dependent data values associated with each of the data frames; a synchronization state determination unit (114) for determining the data synchronization state by comparing the hash values of the target data frames (323, 324, 325, 326) with the hash values of the corresponding source data frames (313, 314, 315, 316); A computational framework (110) comprising:
13. 1. A database system comprising: a source database (121) provided in an industrial plant (120) and adapted to store time series data sets of said industrial plant (120); a target database (131) configured to store a copy of the historical time series data sets stored in the source database (121) of the industrial plant (120); a computational framework (110) according to claim 12; A database system including
14. 13. A computer program for determining a data synchronization status between a source database (121) and a target database (131), the computer program comprising program code for causing a computational framework (110) according to claim 12 to perform the method according to claim 1.