Version management method and device of data source

Abstract

The invention provides a data source version management method and device.The method comprises the steps that a snapshot version library is maintained, the snapshot version library is used for recording related information of table snapshot files of at least two data sources, and the table snapshot files are historical full-data snapshot files of corresponding tables; and under the condition that historical table data of a specified table at specified historical time needs to be acquired, performing version backtracking according to related information of the table snapshot file of the specified table in the snapshot version library. According to the method, the snapshot version library of the table snapshot file is maintained in advance, so that tracking and backtracking of the specified version and the specified data source can be efficiently, accurately and quickly realized based on the snapshot version library. In addition, the size of the table and the updating frequency of the table are analyzed by monitoring a data source log, and the snapshot collection period is intelligently adjusted.

Description

Technical Field

[0001] The present application relates to the technical field of data version management, and in particular to a method and device for data source version management. Background Art

[0002] As digital transformation accelerates, the most important source of customer and operational data is shifting significantly from structured data in business systems housed in data centers to semi-structured and unstructured data from various sources outside the data center. As a result of this shift, data consumers will increasingly need to access data residing in data lakes for analysis. Some powerful data analysis tools provide direct access to data lakes using Structured Query Language (SQL) query engines and semantic layers that provide exploratory analysis capabilities. However, there are situations where users need to backtrack or advance to a specified version of data from multiple sources. One way to deal with this situation is to re-enter the data and re-execute all previous actions, which leads to inefficiency and data errors caused by misoperation. Therefore, how to manage data versions from multiple data sources is a very important and meaningful topic.

[0003] With reference to U.S. patent application US202217936086, the application provides a structured data version management method, apparatus and related equipment applied to the field of information technology (IT). Among them, the method mainly includes: obtaining a first version of structured data, wherein the structured data includes attributes of one or more elements and values ​​corresponding to the attributes; according to the structured data of the first version and the structured data of the previous version, obtaining differential data related to the first version, wherein the differential data represents the difference between the structured data of the first version and the structured data of the previous version; and, recording the differential data corresponding to the first version. In this method, changes in structured data can be recorded in an incremental storage mode, so that users can easily view direct iterative changes of different versions.

[0004] The above-mentioned existing technologies mainly focus on storing incremental data from a single source and only record the difference data between versions. They do not solve the problem of efficiently tracing back or moving to a specified version of data from multiple sources, and do not consider changes in data schema. Summary of the Invention

[0005] At least one embodiment of the present application provides a method and apparatus for managing data source versions, which can efficiently, accurately, and quickly track and trace a specified version and a specified data source.

[0006] According to one aspect of the present application, at least one embodiment provides a data source version management method, which is applied to a data analysis system including at least two data sources, the method comprising:

[0007] Maintaining a snapshot version library, wherein the snapshot version library is used to record relevant information of table snapshot files of the at least two data sources, wherein the table snapshot files are historical full-data snapshot files of the corresponding tables;

[0008] When it is necessary to obtain historical table data of a specified table at a specified historical time, version backtracking is performed based on relevant information of the table snapshot file of the specified table in the snapshot version library.

[0009] Optionally, maintain a snapshot repository, including:

[0010] After the table data of any table of any data source is updated, full data snapshot processing is performed on the table to generate a table snapshot file of the table data, and the data source name of the data source, the table name of the table, the snapshot generation time and the address information of the table snapshot file are saved in the snapshot version library.

[0011] Optionally, the data source name of the data source, the table name of the table, the snapshot generation time, and the address information of the table snapshot file are stored in the snapshot version library, including:

[0012] The data source name of the data source, the table name and the snapshot generation time are used as the key, and the address information of the snapshot file of the table is used as the value.<key,value> The snapshot version is stored in the form of

[0013] Optionally, performing version backtracking based on relevant information of the table snapshot file of the specified table in the snapshot version repository includes:

[0014] If there is no relevant information of a table snapshot file of the specified table later than the specified historical time in the snapshot version library, obtain the first table snapshot file before the specified historical time and closest to the specified historical time to obtain historical table data of the specified table at the specified historical time;

[0015] If relevant information of a table snapshot file of the specified table that is later than the specified historical time exists in the snapshot version library, a second table snapshot file that is after the specified historical time and closest to the specified historical time is obtained to obtain the historical table data of the specified table at the specified historical time.

[0016] Optionally, maintain a snapshot repository, including:

[0017] For each table of each data source, according to the snapshot collection period of the table, periodically perform full data snapshot processing on the table, generate a table snapshot file of the table data, and save the data source name of the data source, the table name, the snapshot generation time and the address information of the table snapshot file in the snapshot version library;

[0018] The method further comprises:

[0019] According to the log of the task executed by the data analysis system, a data source update record is maintained. The data source update record records the update information of each table of each data source. The update information includes the number of times the table data of the table is updated in each unit time and the file address information of the corresponding updated data.

[0020] Optionally, the feature is that the data source name of the data source, the table name of the table, the snapshot generation time, and the address information of the table snapshot file are stored in the snapshot version library, including:

[0021] The data source name of the data source and the table name are used as the key, the unit time corresponding to the snapshot generation time is used as the nested key nest_key, and the address information of the table snapshot file is used as the nested value nest_value. <key,list[<nest_key,nest_value> ]> in the snapshot repository.

[0022] Optionally, performing version backtracking based on relevant information of the table snapshot file of the specified table in the snapshot version repository includes:

[0023] Obtain a snapshot file of the third table that is not later than the specified historical time and is closest to the specified historical time;

[0024] According to the data source update record, the updated data of the specified table between the third table snapshot file and the specified historical time is determined, and the third table snapshot file is updated according to the updated data of the specified table to obtain the historical table data of the specified table at the specified historical time.

[0025] Optionally, the above method further includes:

[0026] Set an initial snapshot collection cycle for each table in each data source in advance;

[0027] For each table in each data source, at the end of the current snapshot collection period corresponding to the table, obtain the table data size, the number of table data updates and the table data update size per unit time in the current snapshot collection period, and the number of customer queries in the current snapshot collection period;

[0028] A new snapshot collection cycle is determined based on the table data scale of the table, the number of table data updates and the table data update scale of the table per unit time in the current snapshot collection cycle, and the number of customer queries in the current snapshot collection cycle. The snapshot collection cycle of the table is updated using the new snapshot collection cycle.

[0029] Optionally, a new snapshot collection period is determined based on the table data size of the table, the number of table data updates and the table data update size per unit time in the current snapshot collection period, and the number of customer queries in the current snapshot collection period, including:

[0030] If the number of table data updates or customer queries for the table in the current snapshot collection period is 0, a new snapshot collection period that is not less than the current snapshot sampling period is determined;

[0031] If the number of table data updates for the table in the current snapshot collection cycle is not zero or the number of customer queries is not zero, calculate the weighted number of table data updates for the table in each unit time in the current snapshot collection cycle and the average update size of the table data each time in the current snapshot collection cycle; calculate the table data update amount per unit time based on the weighted number of table data updates and the update size of the table data each time in the current snapshot collection cycle; calculate a new snapshot collection cycle based on the current snapshot collection cycle, the table data size of the table, the table data update amount per unit time, and the number of customer queries in the current snapshot collection cycle.

[0032] Optionally, calculating the weighted number of table data updates of the table within each unit time in the current snapshot collection cycle includes: calculating the weighted number of table data updates of the table within each unit time in the current snapshot collection cycle according to the number of table data updates of the table within each unit time in the current snapshot collection cycle and a weight corresponding to each unit time, wherein the weight corresponding to the unit time is negatively correlated with the duration between the unit time and the end time of the current snapshot collection cycle;

[0033] Moreover, when calculating a new snapshot collection cycle based on the current snapshot collection cycle, the table data scale of the table, the table data update amount within the unit time, and the number of customer queries in the current snapshot collection cycle, the new snapshot collection cycle is negatively correlated with the table data update amount within the unit time and the number of customer queries in the current snapshot collection cycle.

[0034] According to another aspect of the present application, at least one embodiment provides a data source version management device, which is applied to a data analysis system including at least two data sources, and the device includes:

[0035] A version maintenance module is used to maintain a snapshot version library, wherein the snapshot version library is used to record relevant information of the table snapshot files of the at least two data sources, wherein the table snapshot files are historical full-data snapshot files of the corresponding tables;

[0036] The version backtracking module is used to perform version backtracking according to the relevant information of the table snapshot file of the specified table in the snapshot version library when it is necessary to obtain historical table data of the specified table at a specified historical time.

[0037] Optionally, the version maintenance module is further used to:

[0038] After the table data of any table of any data source is updated, full data snapshot processing is performed on the table to generate a table snapshot file of the table data, and the data source name of the data source, the table name of the table, the snapshot generation time and the address information of the table snapshot file are saved in the snapshot version library.

[0039] Optionally, the version maintenance module is further used to:

[0040] The data source name of the data source, the table name and the snapshot generation time are used as the key, and the address information of the snapshot file of the table is used as the value.<key,value> The snapshot version is stored in the form of

[0041] Optionally, the version backtracking module is further configured to:

[0042] If there is no relevant information of a table snapshot file of the specified table later than the specified historical time in the snapshot version library, obtain the first table snapshot file before the specified historical time and closest to the specified historical time to obtain historical table data of the specified table at the specified historical time;

[0043] If relevant information of a table snapshot file of the specified table that is later than the specified historical time exists in the snapshot version library, a second table snapshot file that is after the specified historical time and closest to the specified historical time is obtained to obtain the historical table data of the specified table at the specified historical time.

[0044] Optionally, the version maintenance module is further used to:

[0045] For each table of each data source, according to the snapshot collection period of the table, periodically perform full data snapshot processing on the table, generate a table snapshot file of the table data, and save the data source name of the data source, the table name, the snapshot generation time and the address information of the table snapshot file in the snapshot version library;

[0046] The device further comprises:

[0047] The data source update record module is used to maintain a data source update record based on the log of task execution of the data analysis system. The data source update record records the update information of each table of each data source. The update information includes the number of times the table data of the table is updated in each unit time and the file address information of the corresponding updated data.

[0048] Optionally, the version maintenance module is further used to:

[0049] The data source name of the data source and the table name are used as the key, the unit time corresponding to the snapshot generation time is used as the nested key nest_key, and the address information of the table snapshot file is used as the nested value nest_value. <key,list[<nest_key,nest_value> ]> in the snapshot repository.

[0050] Optionally, the version backtracking module is further configured to:

[0051] Obtain a snapshot file of the third table that is not later than the specified historical time and is closest to the specified historical time;

[0052] According to the data source update record, the updated data of the specified table between the third table snapshot file and the specified historical time is determined, and the third table snapshot file is updated according to the updated data of the specified table to obtain the historical table data of the specified table at the specified historical time.

[0053] Optionally, the above device further includes:

[0054] Periodic maintenance module for:

[0055] Set an initial snapshot collection cycle for each table in each data source in advance;

[0056] For each table in each data source, at the end of the current snapshot collection period corresponding to the table, obtain the table data size, the number of table data updates and the table data update size per unit time in the current snapshot collection period, and the number of customer queries in the current snapshot collection period;

[0057] A new snapshot collection cycle is determined based on the table data scale of the table, the number of table data updates and the table data update scale of the table per unit time in the current snapshot collection cycle, and the number of customer queries in the current snapshot collection cycle. The snapshot collection cycle of the table is updated using the new snapshot collection cycle.

[0058] Optionally, the periodic maintenance module is further configured to:

[0059] If the number of table data updates or customer queries for the table in the current snapshot collection period is 0, a new snapshot collection period that is not less than the current snapshot sampling period is determined;

[0060] If the number of table data updates for the table in the current snapshot collection cycle is not zero or the number of customer queries is not zero, calculate the weighted number of table data updates for the table in each unit time in the current snapshot collection cycle and the average update size of the table data each time in the current snapshot collection cycle; calculate the table data update amount per unit time based on the weighted number of table data updates and the update size of the table data each time in the current snapshot collection cycle; calculate a new snapshot collection cycle based on the current snapshot collection cycle, the table data size of the table, the table data update amount per unit time, and the number of customer queries in the current snapshot collection cycle.

[0061] Optionally, the periodic maintenance module is further configured to:

[0062] Calculating the weighted number of table data updates of the table within each unit time in the current snapshot collection cycle, including: calculating the weighted number of table data updates of the table within each unit time in the current snapshot collection cycle according to the number of table data updates of the table within each unit time in the current snapshot collection cycle and a weight corresponding to each unit time, wherein the weight corresponding to the unit time is negatively correlated with the length of time between the unit time and the end time of the current snapshot collection cycle;

[0063] Moreover, when calculating a new snapshot collection cycle based on the current snapshot collection cycle, the table data scale of the table, the table data update amount within the unit time, and the number of customer queries in the current snapshot collection cycle, the new snapshot collection cycle is negatively correlated with the table data update amount within the unit time and the number of customer queries in the current snapshot collection cycle.

[0064] According to another aspect of the present application, at least one embodiment provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the steps of the method described above are implemented.

[0065] Compared to existing technologies, the data source version management method and device provided in this embodiment of the application pre-maintains a snapshot version library of table snapshot files. This allows for efficient, accurate, and rapid tracking and backtracking of specified versions and data sources based on the snapshot version library. Furthermore, this embodiment of the application monitors data source logs to analyze table size and update frequency, and intelligently adjusts the snapshot collection cycle. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0067] Figure 1 A schematic diagram of an application scenario of the data source version management method according to an embodiment of the present application;

[0068] Figure 2 A schematic diagram of a flow chart of a method for managing a data source version according to an embodiment of the present application;

[0069] Figure 3 This is an example flow chart of a method for managing data source versions according to an embodiment of the present application;

[0070] Figure 4 A schematic diagram of the structure of a data source version management device according to an embodiment of the present application;

[0071] Figure 5 This is another structural diagram of the version management device for a data source according to an embodiment of the present application. DETAILED DESCRIPTION

[0072] The following describes exemplary embodiments of the present application in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0073] It should be understood that references throughout this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic associated with the embodiment is included in at least one embodiment of the present application. Therefore, the appearance of "in one embodiment" or "in an embodiment" throughout this specification does not necessarily refer to the same embodiment. Furthermore, these particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. The terms "first," "second," and so on, used in the specification and claims of this application are used to distinguish similar items and are not necessarily used to describe a particular order or sequential sequence. It should be understood that such usage is interchangeable where appropriate, such that the embodiments of the present application described herein can, for example, be implemented in an order other than that illustrated or described herein. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus. The term "and / or" used in the specification and claims refers to at least one of the connected items.

[0074] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0075] The following description provides examples and does not limit the scope, applicability, or configuration set forth in the claims. Changes may be made to the function and arrangement of the elements discussed without departing from the spirit and scope of this disclosure. The various examples may appropriately omit, substitute, or add various procedures or components. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0076] In order to improve the efficiency of tracking and backtracking a specified version and a specified data source, the embodiment of the present application provides a data source version management method, which is applied to a data analysis system including at least two data sources. Figure 1This is a schematic diagram of a scenario that can be applied to an embodiment of the present application. The data analysis system includes five data sources, namely data sources based on MySQL, Comma-Separated Values ​​(CSV), Hadoop Distributed File System (HDFS), Hive, MongoDB, and other technologies. The above is only an example of the data sources that can be applied to this application and is not intended to limit this application. For example, the data analysis system can also include more data sources, such as S3 data sources, or include fewer data sources.

[0077] like Figure 2 As shown, the version management method of the data source of the embodiment of the present application includes:

[0078] Step 21 : Maintain a snapshot version library, wherein the snapshot version library is used to record relevant information of the table snapshot files of the at least two data sources, wherein the table snapshot files are historical full-data snapshot files of the corresponding tables.

[0079] Here, the relevant information of the table snapshot file includes the data source name of the data source, the table name, the snapshot generation time and the address information of the table snapshot file. The table snapshot file is a full data snapshot file of the table, that is, it includes the complete data of the table at the time of snapshot collection.

[0080] Step 22: When it is necessary to obtain historical table data of a specified table at a specified historical time, version backtracking is performed according to the relevant information of the table snapshot file of the specified table in the snapshot version library.

[0081] Here, when historical table data of a table of a certain data source at a certain historical time is needed, an embodiment of the present application can search the snapshot version library for the table snapshot file of the table of the data source closest to the historical time, and then perform version backtracking based on the found table snapshot file.

[0082] Through the above method, the embodiment of the present application pre-maintains a snapshot version library of the table snapshot file, so that the specified version and the specified data source can be tracked and traced back efficiently, accurately and quickly based on the snapshot version library.

[0083] As a first implementation method, the embodiment of the present application can monitor the data analysis system in real time. When the data in the data source table changes, a snapshot is generated to save all the data in the updated table.

[0084] At this point, in step 21, after the table data of any table in any data source is updated, a full data snapshot can be performed on the table to generate a table snapshot file for the table data. The data source name of the data source, the table name of the table, the snapshot generation time, and the address of the table snapshot file can be saved in the snapshot version repository. For example, by monitoring the log of the task execution (job execution log) of the data analysis system, it can be determined whether the table data has been updated. When an update occurs, a table snapshot file for the table data is generated.

[0085] Specifically, the data source name of the data source, the table name and the snapshot generation time can be used as the key, and the address information of the table snapshot file can be used as the value.<key,value> For example, suppose the table energy_gc in the MySQL database was updated three times at 11:00, 15:00, and 16:00 on 23 / 7 / 10. After each update, all data in this table is exported and saved to the specified address, and then<mysql_energy_gc_23071011,data_file1> 、<mysql_energy_gc_23071015,data_file2> 、<mysql_energy_gc_23071016,data_file3> Save to the cache database (snapshot version library). Here, data_file1 to 3 represent the address information of the table snapshot files updated three times.

[0086] In step 22, the embodiment of the present application can obtain a first table snapshot file before the specified historical time and closest to the specified historical time when there is no relevant information of a table snapshot file of the specified table later than the specified historical time in the snapshot version library, and obtain the historical table data of the specified table at the specified historical time; and obtain a second table snapshot file after the specified historical time and closest to the specified historical time when there is relevant information of a table snapshot file of the specified table later than the specified historical time in the snapshot version library, and obtain the historical table data of the specified table at the specified historical time.

[0087] As a second implementation method, the embodiment of the present application can perform snapshot processing based on the snapshot collection cycle of each table of each data source to reduce the pressure of snapshot files on storage and memory, and better balance cluster resources and version backtracking efficiency.

[0088] At this time, in step 21, a snapshot collection cycle can be maintained for each table of each data source in advance. Then, for each table of each data source, according to the snapshot collection cycle of the table, full data snapshot processing is periodically performed on the table to generate a table snapshot file of the table data, and the data source name of the data source, the table name of the table, the snapshot generation time and the address information of the table snapshot file are saved in the snapshot version library. Specifically, the data source name of the data source and the table name of the table can be used as the keyword key, the unit time corresponding to the snapshot generation time can be used as the nested keyword nest_key, and the address information of the table snapshot file can be used as the nested value nest_value. <key,list[<nest_key,nest_value> ]> in the form of . The above-mentioned unit time can be several hours, one or more natural days, one week (Monday to Sunday) or multiple weeks, etc. Because there may be multiple tables for full data snapshots in the same unit time (such as one day), the above-mentioned method performs nested processing when setting the value to avoid conflicts when maintaining the snapshot version library. In addition, the embodiment of the present application can also maintain a data source update record based on the log of the task executed by the data analysis system, and the data source update record records the update information of each table of each data source, and the update information includes the number of times the table data of the table is updated in each unit time and the file address information of the corresponding updated data.

[0089] In step 22, the embodiment of the present application can obtain a third table snapshot file that is no later than the specified historical time and closest to the specified historical time. Then, based on the data source update record, the updated data of the specified table between the third table snapshot file and the specified historical time is determined, and the third table snapshot file is updated based on the updated data of the specified table to obtain the historical table data of the specified table at the specified historical time.

[0090] To address the need for monitoring and fine-grained management of data and schema changes for each table from each data source, an embodiment of the present application intelligently adjusts the snapshot acquisition cycle to record the algorithm of the current data source (snapshot) by monitoring the data source log and analyzing the table size and update frequency of the table. This enables efficient, accurate, and rapid tracking and backtracking of specified versions and specified data sources.

[0091] Specifically, in the second implementation method described above, an initial snapshot collection period can be set in advance for each table of each data source. Subsequently, the snapshot collection period of each table can be updated according to specific circumstances. For example, for each table of each data source, when the current snapshot collection period corresponding to the table expires, the table data size of the table, the number of table data updates and the table data update scale of the table per unit time in the current snapshot collection period, and the number of customer queries in the current snapshot collection period can be obtained. Then, based on the table data size of the table, the number of table data updates and the table data update scale of the table per unit time in the current snapshot collection period, and the number of customer queries in the current snapshot collection period, a new snapshot collection period is determined, and the snapshot collection period of the table is updated using the new snapshot collection period.

[0092] Specifically, when determining a new snapshot collection cycle:

[0093] (1) If the number of table data updates for the table in the current snapshot acquisition cycle is 0 or the number of customer queries is 0, a new snapshot acquisition cycle that is not less than the current snapshot sampling cycle can be determined. Here, the number of table data updates is 0 or the number of customer queries is 0, indicating that the update frequency of the table data is low or the customer's attention to the table data is low. In this case, the current sampling cycle can be maintained unchanged, and the snapshot sampling cycle of the table does not need to be updated. Alternatively, the embodiment of the present application can also appropriately increase the length of the current sampling cycle on the basis of the current snapshot sampling cycle to obtain a new snapshot acquisition cycle that is greater than the current snapshot sampling cycle, thereby reducing the acquisition frequency of the table snapshot when the update frequency of the table data is low or the customer's attention to the table data is low, so as to reduce the consumption of storage and memory resources by the snapshot file.

[0094] (2) If the number of table data updates of the table in the current snapshot acquisition cycle is not zero or the number of customer queries is not zero, the weighted number of table data updates of the table in each unit time in the current snapshot acquisition cycle and the average update scale of the table data in each unit time in the current snapshot acquisition cycle can be calculated; the table data update amount in unit time is calculated based on the weighted number of table data updates and the update scale of the table data in each unit time in the current snapshot acquisition cycle; a new snapshot acquisition cycle is calculated based on the current snapshot acquisition cycle, the table data scale of the table, the table data update amount in the unit time, and the number of customer queries in the current snapshot acquisition cycle.

[0095] When calculating the weighted number of table data updates for the table within each unit time in the current snapshot collection cycle, the weighted number of table data updates for the table within each unit time in the current snapshot collection cycle can be calculated based on the number of table data updates for the table within each unit time in the current snapshot collection cycle and the weight corresponding to each unit time, wherein the weight corresponding to the unit time is negatively correlated with the duration between the unit time and the end time of the current snapshot collection cycle. In addition, when calculating a new snapshot collection cycle based on the current snapshot collection cycle, the table data size of the table, the amount of table data updates within the unit time, and the number of customer queries within the current snapshot collection cycle, the new snapshot collection cycle is negatively correlated with the amount of table data updates within the unit time and the number of customer queries within the current snapshot collection cycle.

[0096] To better understand the above embodiments, Figure 3 The second implementation method is described below through a more specific example process. The example process mainly includes:

[0097] 1) According to the data source configuration file, add a data source and set a snapshot period for each table of the data source and save it to the snapshot collection period database.

[0098] 2) Monitor the job logs executed by the data analysis system. If the table data is updated, the update times and update operations are recorded in the matrix file corresponding to the matrix in the data source update record matrix. If there is no update, "none" is recorded.

[0099] Here, the real-time monitoring data analysis system records the updated data and saves it to a specified file when the data source table data is updated. A two-dimensional matrix is ​​generated based on the change history of all data source table records. The horizontal rows of the matrix are the date, the vertical columns are the data source_table name, and the intersection of the rows and columns is the number of changes that occurred in the table on that day and the change record file address (.sql). Assume that the data analysis system has N data sources and each workshop has K i (i=1..N) tables record the updated data of the data table for L years (such as 2020-2023), then this matrix is:

[0100] Row: R = L * 365

[0101] Column: C = sum(K i )i=(1..N)

[0102] The size of this dictionary matrix is ​​[R,C].

[0103] Table 1 provides an example of the update record matrix for this data source:

[0104] Table 1

[0105]

[0106] 3) Read the snapshot collection period database, perform full data snapshot processing on the table whose snapshot collection period expires, and save it to the snapshot version library.

[0107] 4) When the current snapshot collection period of a table expires, the data source is read to update the record matrix. If the table has not been updated during the current snapshot collection period, the snapshot collection period of the table will not be updated. Of course, the snapshot collection period of the table can also be extended as needed.

[0108] 5) Monitor the execution job log of the data analysis system. If no client connects to operate this table during the current snapshot collection period, the snapshot collection period will not be updated. If there is a connection, the total number of rows in the table, the number of client connections, the number of table changes, the number of table change rows, and the time when the changes occurred during this snapshot collection period are used as adjustment indicators for the snapshot collection period.

[0109] 6) Calculate a new snapshot period for this table based on the metrics obtained in 5) and save it to the snapshot collection period database.

[0110] 7) In order to obtain the table data of a specified version of a table on a specified date, obtain the table snapshot version (the most recent version) that is before the specified date and closest to the specified date from the snapshot version library.

[0111] 8) Read the data source update record matrix, obtain all data changes that occurred in the table between the specified date and the latest version, reflect all the data changes in the latest version obtained in 7), and obtain the historical data of the specified version of the table.

[0112] To track and manage data source versions, this example sets a default snapshot collection period for each table in each data source, sets the data source table name ds_tname as the key, and saves the data to the cache database cacheDB1 (the snapshot collection period database) with snap_period as the value.

[0113] When the snapshot collection period arrives, the new snapshot collection period is recalculated and adjusted based on the number of data changes, number of customer queries, data scale, update scale, etc. If there are no customers connected during the snapshot collection period, the period remains unchanged, indicating that the customer has no data analysis needs, is not sensitive to the corresponding version changes, and has no urgency for the real-time version backtracking. Assume the following parameter definitions:

[0114] Current snapshot collection period: cur_snap_period;

[0115] Current snapshot collection date: cur_snap_date;

[0116] The total number of rows in the data table to be adjusted: ds_tname_rows;

[0117] Total number of updates within the snapshot collection: total_upd_num;

[0118] The number of rows updated each time in the data table to be adjusted: ds_tname_upd_rows[h], h = (1..total_upd_num);

[0119] The specific time when each change occurs during the snapshot collection period, upd_date(i), where i = (1, 2…total_upd_num);

[0120] Number of changes per day within the snapshot collection period, upd_date_num(j), j = (1, 2…cur_snap_period);

[0121] The number of client connections to the data table to be adjusted during the snapshot collection period: conn_num;

[0122] One way to calculate the snapshot collection period is as follows:

[0123] A: Calculate the number of changes that occurred during this collection period using weighted average:

[0124] ave_upd_num=

[0125] ∑[(cur_snap_period–cur_snap_date-upd_date(i))*upd_date_num(k))] / cur_snap_period

[0126] B: Calculate the number of rows updated each time:

[0127] steps_per_upd=∑(ds_tname_upd_rows[h]) / total_upd_num

[0128] C: Calculate the number of updated rows in the data table every day:

[0129] steps_per_day=steps_per_upd*ave_upd_num

[0130] D: Calculate the new full data snapshot collection period:

[0131] new_snap_period=[cur_snap_period*(ds_tname_rows / conn_num)] / steps_per_day

[0132] E: In the cache database cacheDB1, update the corresponding value of this table with the new snapshot period new_snap_period (the longest period is no more than 6 months and the shortest period is no less than 1 day).

[0133] F: Save the current snapshot of all data in this data table, set the current date as key, and the full data snapshot file address as value.<k,v> Save to the cache database cacheDB2 (snapshot database).

[0134] For example, assume the energy database table gc in MySQL contains 4000 rows. The default full-data snapshot collection period is 7 days. By reading the data source and updating the record matrix, a total of 12 changes are made during this period. The specific time and number of rows are as follows: Day 1: 2 times, 520 rows; Day 2: 1 time, 100 rows; Day 3: 5 times, 920 rows; Day 6: 1 time, 250 rows; Day 7: 3 times, 160 rows. The number of client connections accessing this table, as obtained through job execution and logs in the monitoring system, is 5. Therefore, the new full-data snapshot collection period is:

[0135] Number of changes = (1*2+2*1+3*5+6*1+7*3) / 7 = 6.43 times / day, number of rows updated each time = (520+100+920+250+160) / 12 = 162.5 rows / time.

[0136] Daily update volume = 162.5*6.43 = 1044.6 rows / day, new snapshot collection period = 7*(4000 / 5) / 1044.6 = 5.36, which is 5 days after rounding up.

[0137] The above example is used for version management:

[0138] All tables of all data sources in the data analysis system are intelligently set with snapshot collection cycles and saved, because multiple tables may take full data snapshots on the same day.

[0139] Set Key = data source_table name ds_tname;

[0140] To avoid conflicts when setting the value, nesting is performed, where:

[0141] Set nest_key = snapshot_date;

[0142] Set nest_value = full data snapshot file address snap_path;

[0143] Manage the version of table data saved in the cache database (snapshot repository) in the form of <key, list[<nest_key, nest_value>]>.

[0144] When performing version backtracking in the above example, read the cache database cacheDB2 (snapshot repository):

[0145] 1) Use the data source and table name as the key to obtain the list of all full data snapshot versions of this table v_lst = List[<nest_key, nest_value>].

[0146] 2) Use the specified historical data date as the key, sort the version list sort(v_lst), and obtain the version closest to the specified date.

[0147] 3) Obtain the full data snapshot file address according to the closest date nest_key

[0148] 4) Read the data source update record matrix and update until the specified date

[0149] 5) Obtain the historical data of the specified version and then perform analysis

[0150] Example: The table gc in the energy library of the data source Mysql currently has 4 versions, which are managed in the form of <mysql_energy_gc, list[<2023050>1, v1>, <2023052>1, v2>, <2023060>4, v3>, <2023061>8, v4>]>. Assume that now it is necessary to obtain the data of the gc table version with the date of May 24, 2023, then:

[0151] 1) Sort the version list and obtain the version v2 closest to the specified date 20230524.

[0152] 2) Read the change history of the data source update record matrix table gc, and perform update operations based on the data in version v2 until the specified date 0524.

[0153] 3) Obtain the table data version of the specified date and then perform subsequent data analysis.

[0154] In the above example, according to the update frequency and scale of each table, the snapshot period of the full table data is intelligently adjusted. While making full use of the advantages of the Kubernetes platform, it manages the dataset version more efficiently and refinedly. At the same time, the intelligent adjustment makes the snapshot period more reasonable and reduces the pressure on computing / storage resources, and also enables customers to locate and backtrack the historical version dataset more quickly.

[0155] Please refer to Figure 4The embodiment of the present application provides a structure of a data source version management device, which is applied to a data analysis system including at least two data sources, including:

[0156] A version maintenance module 41 is configured to maintain a snapshot version library, wherein the snapshot version library is configured to record relevant information of table snapshot files of the at least two data sources, wherein the table snapshot files are historical full-data snapshot files of the corresponding tables;

[0157] The version backtracking module 42 is configured to perform version backtracking based on the relevant information of the table snapshot file of the specified table in the snapshot version library when it is necessary to obtain historical table data of the specified table at a specified historical time.

[0158] Optionally, the version maintenance module is further used to:

[0159] After the table data of any table of any data source is updated, full data snapshot processing is performed on the table to generate a table snapshot file of the table data, and the data source name of the data source, the table name of the table, the snapshot generation time and the address information of the table snapshot file are saved in the snapshot version library.

[0160] Optionally, the version maintenance module is further used to:

[0161] The data source name of the data source, the table name and the snapshot generation time are used as the key, and the address information of the snapshot file of the table is used as the value.<key,value> The snapshot version is stored in the form of

[0162] Optionally, the version backtracking module is further configured to:

[0163] If there is no relevant information of a table snapshot file of the specified table later than the specified historical time in the snapshot version library, obtain the first table snapshot file before the specified historical time and closest to the specified historical time to obtain historical table data of the specified table at the specified historical time;

[0164] If relevant information of a table snapshot file of the specified table that is later than the specified historical time exists in the snapshot version library, a second table snapshot file that is after the specified historical time and closest to the specified historical time is obtained to obtain the historical table data of the specified table at the specified historical time.

[0165] Optionally, the version maintenance module is further used to:

[0166] For each table of each data source, according to the snapshot collection period of the table, periodically perform full data snapshot processing on the table, generate a table snapshot file of the table data, and save the data source name of the data source, the table name, the snapshot generation time and the address information of the table snapshot file in the snapshot version library;

[0167] The device further comprises:

[0168] The data source update record module is used to maintain a data source update record based on the log of task execution of the data analysis system. The data source update record records the update information of each table of each data source. The update information includes the number of times the table data of the table is updated in each unit time and the file address information of the corresponding updated data.

[0169] Optionally, the version maintenance module is further used to:

[0170] The data source name of the data source and the table name are used as the key, the unit time corresponding to the snapshot generation time is used as the nested key nest_key, and the address information of the table snapshot file is used as the nested value nest_value. <key,list[<nest_key,nest_value> ]> in the snapshot repository.

[0171] Optionally, the version backtracking module is further configured to:

[0172] Obtain a snapshot file of the third table that is not later than the specified historical time and is closest to the specified historical time;

[0173] According to the data source update record, the updated data of the specified table between the third table snapshot file and the specified historical time is determined, and the third table snapshot file is updated according to the updated data of the specified table to obtain the historical table data of the specified table at the specified historical time.

[0174] Optionally, the above device further includes:

[0175] Periodic maintenance module for:

[0176] Set an initial snapshot collection cycle for each table in each data source in advance;

[0177] For each table in each data source, at the end of the current snapshot collection period corresponding to the table, obtain the table data size, the number of table data updates and the table data update size per unit time in the current snapshot collection period, and the number of customer queries in the current snapshot collection period;

[0178] A new snapshot collection cycle is determined based on the table data scale of the table, the number of table data updates and the table data update scale of the table per unit time in the current snapshot collection cycle, and the number of customer queries in the current snapshot collection cycle. The snapshot collection cycle of the table is updated using the new snapshot collection cycle.

[0179] Optionally, the periodic maintenance module is further configured to:

[0180] If the number of table data updates or customer queries for the table in the current snapshot collection period is 0, a new snapshot collection period that is not less than the current snapshot sampling period is determined;

[0181] If the number of table data updates for the table in the current snapshot collection cycle is not zero or the number of customer queries is not zero, calculate the weighted number of table data updates for the table in each unit time in the current snapshot collection cycle and the average update size of the table data each time in the current snapshot collection cycle; calculate the table data update amount per unit time based on the weighted number of table data updates and the update size of the table data each time in the current snapshot collection cycle; calculate a new snapshot collection cycle based on the current snapshot collection cycle, the table data size of the table, the table data update amount per unit time, and the number of customer queries in the current snapshot collection cycle.

[0182] Optionally, the periodic maintenance module is further configured to:

[0183] Calculating the weighted number of table data updates of the table within each unit time in the current snapshot collection cycle, including: calculating the weighted number of table data updates of the table within each unit time in the current snapshot collection cycle according to the number of table data updates of the table within each unit time in the current snapshot collection cycle and a weight corresponding to each unit time, wherein the weight corresponding to the unit time is negatively correlated with the length of time between the unit time and the end time of the current snapshot collection cycle;

[0184] Moreover, when calculating a new snapshot collection cycle based on the current snapshot collection cycle, the table data scale of the table, the table data update amount within the unit time, and the number of customer queries in the current snapshot collection cycle, the new snapshot collection cycle is negatively correlated with the table data update amount within the unit time and the number of customer queries in the current snapshot collection cycle.

[0185] It should be noted that the various systems provided in the above embodiments are devices corresponding to the above-mentioned data source version management method. The implementation methods in the above-mentioned embodiments are applicable to the embodiments of the device and can achieve the same technical effects. The above-mentioned device provided in the embodiments of the present application can implement all the method steps implemented in the above-mentioned method embodiments and can achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as those in the method embodiments will not be described in detail here.

[0186] Please refer to Figure 5 , a structural diagram of another data source version management device provided in an embodiment of the present application, the device includes: a processor 501, a transceiver 502, a memory 503, a user interface 504 and a bus interface.

[0187] In the embodiment of the present application, the device further includes: a program stored in the memory 503 and executable on the processor 501 .

[0188] The transceiver 502 is configured to send and receive data under the control of the processor;

[0189] The processor 501 is configured to read the computer program in the memory and perform the following operations:

[0190] Maintaining a snapshot version library, wherein the snapshot version library is used to record relevant information of table snapshot files of the at least two data sources, wherein the table snapshot files are historical full-data snapshot files of the corresponding tables;

[0191] When it is necessary to obtain historical table data of a specified table at a specified historical time, version backtracking is performed based on relevant information of the table snapshot file of the specified table in the snapshot version library.

[0192] It is understandable that in the embodiment of the present application, when the computer program is executed by the processor 501, each process of the above-mentioned data source version management method embodiment can be implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0193] exist Figure 5In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 501 and memory represented by memory 503. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 502 may be a plurality of components, i.e., a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. For different user devices, the user interface 504 may also be an interface capable of connecting external or internal devices as required, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, etc.

[0194] The processor 501 is responsible for managing the bus architecture and general processing, and the memory 503 can store data used by the processor 501 when performing operations.

[0195] It should be noted that the device in this embodiment is a device corresponding to the version management method of the above-mentioned data source, and the implementation methods in the above-mentioned embodiments are all applicable to the embodiments of this device and can achieve the same technical effects. In this device, the transceiver 502 and the memory 503, as well as the transceiver 502 and the processor 501 can be connected to each other through a bus interface. The functions of the processor 501 can also be implemented by the transceiver 502, and the functions of the transceiver 502 can also be implemented by the processor 501. It should be noted that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment and can achieve the same technical effects. The parts and beneficial effects that are the same as those in the method embodiment will not be specifically described here.

[0196] In some embodiments of the present application, a computer-readable storage medium is further provided, on which a program is stored. When the program is executed by a processor, the following steps are implemented:

[0197] Maintaining a snapshot version library, wherein the snapshot version library is used to record relevant information of table snapshot files of the at least two data sources, wherein the table snapshot files are historical full-data snapshot files of the corresponding tables;

[0198] When it is necessary to obtain historical table data of a specified table at a specified historical time, version backtracking is performed based on relevant information of the table snapshot file of the specified table in the snapshot version library.

[0199] When the program is executed by the processor, it can implement all the implementation methods of the version management method of the above-mentioned data source and achieve the same technical effect. To avoid repetition, it will not be described here.

[0200] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0201] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0202] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0203] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0204] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0205] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0206] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data source version management method, applied to a data analysis system including at least two data sources, characterized in that: The method comprises: Maintaining a snapshot version library, the snapshot version library is used to record relevant information of the table snapshot files of the at least two data sources, the table snapshot files are historical full data snapshot files of the corresponding tables; When it is necessary to obtain historical table data of a specified table at a specified historical time, version backtracking is performed according to relevant information of the table snapshot file of the specified table in the snapshot version library.

2. The method according to claim 1, characterized in that Maintain a snapshot repository, including: After the table data of any table of any data source is updated, full data snapshot processing is performed on the table to generate a table snapshot file of the table data, and the data source name of the data source, the table name of the table, the snapshot generation time and the address information of the table snapshot file are saved in the snapshot version library.

3. The method according to claim 2, characterized in that The data source name of the data source, the table name of the table, the snapshot generation time and the address information of the table snapshot file are stored in the snapshot version library, including: The data source name of the data source, the table name of the table, and the snapshot generation time are used as the key, and the address information of the table snapshot file is used as the value.<key,value> is saved in the snapshot repository.

4. The method according to claim 1, characterized in that The performing version backtracking according to the relevant information of the table snapshot file of the specified table in the snapshot version library includes: If there is no relevant information of a table snapshot file of the specified table later than the specified historical time in the snapshot version library, obtain the first table snapshot file before the specified historical time and closest to the specified historical time to obtain historical table data of the specified table at the specified historical time; When there is relevant information of a table snapshot file of the specified table later than the specified historical time in the snapshot version library, a second table snapshot file after the specified historical time and closest to the specified historical time is obtained to obtain the historical table data of the specified table at the specified historical time.

5. The method according to claim 1, characterized in that Maintain a snapshot repository, including: For each table of each data source, according to the snapshot collection period of the table, periodically perform full data snapshot processing on the table, generate a table snapshot file of the table data, and save the data source name of the data source, the table name of the table, the snapshot generation time and the address information of the table snapshot file in the snapshot version library; The method further comprises: According to the log of the task executed by the data analysis system, a data source update record is maintained, in which the update information of each table of each data source is recorded. The update information includes the number of times the table data of the table is updated in each unit time and the file address information of the corresponding updated data.

6. The method according to claim 5, characterized in that The data source name of the data source, the table name of the table, the snapshot generation time and the address information of the table snapshot file are stored in the snapshot version library, including: The data source name of the data source and the table name are used as the keyword key, the unit time corresponding to the snapshot generation time is used as the nested keyword nest_key, and the address information of the table snapshot file is used as the nested value nest_value. <key,list[<nest_key,nest_value> ]> in the snapshot repository.

7. The method according to claim 5, characterized in that The performing version backtracking according to the relevant information of the table snapshot file of the specified table in the snapshot version library includes: Obtain a snapshot file of the third table that is not later than the specified historical time and is closest to the specified historical time; According to the data source update record, the update data of the designated table between the third table snapshot file and the designated historical time is determined, and the third table snapshot file is updated according to the update data of the designated table to obtain the historical table data of the designated table at the designated historical time.

8. The method according to claim 5, characterized in that Also includes: Set an initial snapshot collection cycle for each table of each data source in advance; For each table of each data source, when the current snapshot collection period corresponding to the table ends, obtain the table data scale of the table, the number of table data updates and the table data update scale of the table in each unit time in the current snapshot collection period, and the number of customer queries in the current snapshot collection period; A new snapshot collection cycle is determined based on the table data scale of the table, the number of table data updates and the table data update scale of the table per unit time in the current snapshot collection cycle, and the number of customer queries in the current snapshot collection cycle. The snapshot collection cycle of the table is updated using the new snapshot collection cycle.

9. The method according to claim 8, characterized in that According to the table data scale of the table, the number of table data updates and the table data update scale of the table per unit time in the current snapshot collection cycle, and the number of customer queries in the current snapshot collection cycle, a new snapshot collection cycle is determined, including: When the number of table data updates of the table is 0 or the number of customer queries is 0 in the current snapshot collection period, a new snapshot collection period that is not less than the current snapshot sampling period is determined; When the number of table data updates of the table is not 0 or the number of customer queries is not 0 in the current snapshot collection cycle, calculate the weighted number of table data updates of the table in each unit time in the current snapshot collection cycle and the average update scale of the table data each time in the current snapshot collection cycle; calculate the table data update amount per unit time according to the weighted number of table data updates and the update scale of the table data each time in the current snapshot collection cycle; calculate a new snapshot collection cycle according to the current snapshot collection cycle, the table data scale of the table, the table data update amount per unit time, and the number of customer queries in the current snapshot collection cycle.

10. The method according to claim 9, characterized in that Calculating the weighted update times of table data of the table in each unit time in the current snapshot collection cycle, including: calculating the weighted update times of table data of the table in each unit time in the current snapshot collection cycle according to the update times of table data of the table in each unit time in the current snapshot collection cycle and the weight corresponding to each unit time, wherein the weight corresponding to the unit time is negatively correlated with the duration between the unit time and the end time of the current snapshot collection cycle; Moreover, when a new snapshot collection cycle is calculated based on the current snapshot collection cycle, the table data scale of the table, the table data update amount within the unit time, and the number of customer queries in the current snapshot collection cycle, the new snapshot collection cycle is negatively correlated with the table data update amount within the unit time and the number of customer queries in the current snapshot collection cycle.

11. A data source version management device, applied to a data analysis system including at least two data sources, characterized in that: The device comprises: A version maintenance module, used to maintain a snapshot version library, wherein the snapshot version library is used to record relevant information of table snapshot files of the at least two data sources, wherein the table snapshot files are historical full-data snapshot files of the corresponding tables; The version backtracking module is used to perform version backtracking according to the relevant information of the table snapshot file of the specified table in the snapshot version library when it is necessary to obtain the historical table data of the specified table at a specified historical time.

12. The device according to claim 11, characterized in that The version maintenance module is also used for: After the table data of any table of any data source is updated, full data snapshot processing is performed on the table to generate a table snapshot file of the table data, and the data source name of the data source, the table name of the table, the snapshot generation time and the address information of the table snapshot file are saved in the snapshot version library.

13. The device according to claim 11, characterized in that The version backtracking module is also used to: If there is no relevant information of a table snapshot file of the specified table later than the specified historical time in the snapshot version library, obtain the first table snapshot file before the specified historical time and closest to the specified historical time to obtain historical table data of the specified table at the specified historical time; When there is relevant information of a table snapshot file of the specified table later than the specified historical time in the snapshot version library, a second table snapshot file after the specified historical time and closest to the specified historical time is obtained to obtain the historical table data of the specified table at the specified historical time.

14. The device according to claim 11, characterized in that The version maintenance module is also used for: For each table of each data source, according to the snapshot collection period of the table, periodically perform full data snapshot processing on the table, generate a table snapshot file of the table data, and save the data source name of the data source, the table name of the table, the snapshot generation time and the address information of the table snapshot file in the snapshot version library; The device also includes: The data source update record module is used to maintain a data source update record based on the log of the task executed by the data analysis system. The data source update record records the update information of each table of each data source. The update information includes the number of times the table data of the table is updated in each unit time and the file address information of the corresponding updated data.

15. The device according to claim 14, characterized in that The version backtracking module is also used to: Obtain a snapshot file of the third table that is not later than the specified historical time and is closest to the specified historical time; According to the data source update record, the update data of the designated table between the third table snapshot file and the designated historical time is determined, and the third table snapshot file is updated according to the update data of the designated table to obtain the historical table data of the designated table at the designated historical time.

16. The device according to claim 14, characterized in that Also includes: Periodic maintenance module for: Set an initial snapshot collection cycle for each table of each data source in advance; For each table of each data source, when the current snapshot collection period corresponding to the table ends, obtain the table data scale of the table, the number of table data updates and the table data update scale of the table in each unit time in the current snapshot collection period, and the number of customer queries in the current snapshot collection period; A new snapshot collection cycle is determined based on the table data scale of the table, the number of table data updates and the table data update scale of the table per unit time in the current snapshot collection cycle, and the number of customer queries in the current snapshot collection cycle. The snapshot collection cycle of the table is updated using the new snapshot collection cycle.

17. The device according to claim 16, characterized in that The periodic maintenance module is also used for: When the number of table data updates of the table is 0 or the number of customer queries is 0 in the current snapshot collection period, a new snapshot collection period that is not less than the current snapshot sampling period is determined; When the number of table data updates of the table is not 0 or the number of customer queries is not 0 in the current snapshot collection cycle, calculate the weighted number of table data updates of the table in each unit time in the current snapshot collection cycle and the average update scale of the table data each time in the current snapshot collection cycle; calculate the table data update amount per unit time according to the weighted number of table data updates and the update scale of the table data each time in the current snapshot collection cycle; calculate a new snapshot collection cycle according to the current snapshot collection cycle, the table data scale of the table, the table data update amount per unit time, and the number of customer queries in the current snapshot collection cycle.

18. The device according to claim 17, characterized in that The periodic maintenance module is also used for: Calculating the weighted update times of table data of the table in each unit time in the current snapshot collection cycle, including: calculating the weighted update times of table data of the table in each unit time in the current snapshot collection cycle according to the update times of table data of the table in each unit time in the current snapshot collection cycle and the weight corresponding to each unit time, wherein the weight corresponding to the unit time is negatively correlated with the duration between the unit time and the end time of the current snapshot collection cycle; Moreover, when a new snapshot collection cycle is calculated based on the current snapshot collection cycle, the table data scale of the table, the table data update amount within the unit time, and the number of customer queries in the current snapshot collection cycle, the new snapshot collection cycle is negatively correlated with the table data update amount within the unit time and the number of customer queries in the current snapshot collection cycle.

Images