Dynamic and automated management methods, products, media and equipment for the entire data life cycle

Abstract

The present invention discloses a method, product, medium and equipment for dynamic and automated management of data throughout its life cycle, and relates to the field of big data storage. The present invention first obtains company server resource information and database attribute information and calculates the optimal file size for data storage; parses field mapping relationships and table building attributes from user requirements through a natural language parsing model; generates corresponding table building scripts based on table building attributes using table building syntax; creates a connection between the script and the target server based on database information to execute the generated script; after completing script execution, obtains a list of daily data changes based on real-time data monitoring and data prediction; calculates the optimal partitioning strategy for the table based on the optimal file size and the daily data change list; updates scripts and data based on the optimal partitioning strategy for the table; and regularly cleans up expired data based on the storage duration in the table building attributes. The present invention can simplify the data management process, reduce labor costs and the risk of intermediate demand transmission.

Description

Technical Field

[0001] The present invention relates to the field of big data storage technology, and in particular to a method, product, medium and equipment for dynamic and automated management of data throughout its entire life cycle. Background Art

[0002] At present, companies involved in big data storage projects often use manual processing methods to manage the data lifecycle. When business needs arise, designers first generate table field designs based on business logic, and then hand over the table field designs to developers, who create corresponding tables based on database types; operation and maintenance personnel increase data monitoring of the table, and determine whether the partition needs to be adjusted after the data has been stored for a long time; after determining that the partition needs to be adjusted, the developer is notified, and the developer rebuilds the partition after analysis. In addition, operation and maintenance personnel are required to monitor the number of days the data is retained (usually 365 days), and if there is overdue data, it is deleted. It can be seen that the full lifecycle management of data requires the collaboration of the three departments of design, development, and operation and maintenance, with high labor costs, complex processes, and the risk of intermediate demand transmission. Summary of the invention

[0003] The purpose of the present invention is to provide a method, product, medium and equipment for dynamic and automated management of data throughout its entire life cycle, so as to simplify the data management process and reduce labor costs and the risk of intermediate demand transmission.

[0004] To achieve the above object, the present invention provides the following solution.

[0005] In one aspect, the present invention provides a method for dynamic and automated management of data throughout its life cycle, comprising:

[0006] Obtain the company's server resource information and database attribute information through the file parsing function; the server resource information includes the number of servers and the number of cores, memory size and hard disk size of each server; the database attribute information includes the number of replicas, partition strategy and database characteristics; the partition strategy includes no partition, monthly partition, weekly partition and daily partition; the database characteristics include the number of namenode unit management files, namenode memory size, datanode number, datanode corresponding block default size and datanode maximum file size;

[0007] Calculate the optimal file size for data storage based on server resource information and database attribute information;

[0008] The field mapping relationship and table creation attributes are parsed from user requirements through a natural language parsing model; the field mapping relationship includes field name, type, description, and whether it can be empty; the table creation attributes include database name, table name, partition key, storage duration, and database information; the database information includes the IP, port, user name, and password corresponding to the database;

[0009] Based on the table creation attributes, the corresponding table creation script is generated using the table creation syntax;

[0010] Create a connection between the script and the target server based on the database information to execute the generated script;

[0011] After the script is executed, a list of daily data changes is obtained based on real-time data monitoring and data prediction;

[0012] Calculate the optimal partitioning strategy for the table based on the optimal file size and the daily list of data changes;

[0013] Update scripts and data based on the optimal partitioning strategy for the table;

[0014] Periodically clean up expired data based on the storage duration in the table creation properties.

[0015] Optionally, the calculating the optimal file size for data storage based on the server resource information and the database attribute information specifically includes:

[0016] Determine the default usage rate, hardware conditions, and process busyness of the server based on the server resource information, and use the formula Calculate system resource usage;

[0017] Based on database attribute information, according to the formula Calculate minimum file size;

[0018] Based on system resource usage and minimum file size, according to the formula Calculate the optimal file size for data storage; where Indicates taking the maximum value.

[0019] Optionally, obtaining a daily data change list based on real-time data monitoring and data prediction specifically includes:

[0020] Use a time series database to monitor and record changes in the amount of data in each table in real time;

[0021] After completing a cycle of monitoring, the data volume changes of each table are recorded as historical data;

[0022] Through the time series prediction algorithm, the data volume of subsequent dates is predicted based on historical data to obtain the predicted data;

[0023] Build a list of daily data changes based on historical data and forecast data.

[0024] Optionally, calculating the optimal partitioning strategy of the table based on the optimal file size and the daily data change list specifically includes:

[0025] Determine the average daily number of orders corresponding to the third quartile based on the daily data change list, and use the formula Calculate the average daily storage size;

[0026] Based on the optimal file size and average daily storage size, according to the formula Calculate the partition coefficient;

[0027] Determine the optimal partitioning strategy for the table based on the partition coefficient.

[0028] Optionally, determining the optimal partitioning strategy of the table according to the partition coefficient specifically includes:

[0029] When the partition coefficient is greater than 40, the optimal partition strategy for the table is determined to be no partition;

[0030] When the partition coefficient is between 10 and 40, the optimal partition strategy for the table is determined to be monthly partitioning;

[0031] When the partition coefficient is between 2 and 10, the optimal partition strategy for the table is determined to be weekly partitioning;

[0032] When the partition coefficient is between 0.1 and 2, the optimal partition strategy for the table is determined to be daily partitioning;

[0033] When the partition coefficient is less than 0.1, partitions are split according to the business system.

[0034] Optionally, updating the script and data according to the optimal partitioning strategy of the table specifically includes:

[0035] When the optimal partitioning strategy of the table is inconsistent with the existing partitioning strategy, modify the corresponding partition fields to update the script;

[0036] Connect to the database, rename the history table to the backup table, and leave the original table name empty;

[0037] Create a new table according to the new script. The new table still uses the original table name. Import the data in the backup table into the new table to complete the data update.

[0038] Optionally, the periodic cleaning of expired data according to the storage duration in the table creation attribute specifically includes:

[0039] Calculate the earliest date corresponding to the data in each table based on the storage duration;

[0040] Read the actual partition of the corresponding table in the database;

[0041] Determine whether there is data in the actual partition date that exceeds the earliest date. If so, delete it.

[0042] On the other hand, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for dynamic and automated management of data throughout its life cycle.

[0043] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for dynamic and automated management of data throughout its entire life cycle.

[0044] On the other hand, the present invention also provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for dynamic and automated management of data throughout its entire life cycle.

[0045] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0046] Traditional data storage related functions are often implemented manually, while the method of the present invention automatically integrates the contents manually implemented by multiple departments, reducing the complex process that originally required the collaboration of the three departments of design, development, and operation and maintenance to direct configuration by the design department, simplifying the data management process and reducing the risk of intermediate demand transmission; and the execution of the entire method only requires the design department to perform preliminary configuration and can be automatically executed, without the participation of the development and operation and maintenance departments, thereby greatly reducing labor costs. In addition, the method of the present invention can realize automated table creation, data monitoring and analysis, dynamic partition adjustment, and automated cleaning of expired data, improving the stability of the storage system, reducing operation and maintenance risks, and providing better data storage services. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A flow chart of the data full life cycle dynamic automation management method provided by the present invention;

[0049] Figure 2 A schematic diagram of each stage of the execution process of the data full life cycle dynamic automation management method provided by the present invention;

[0050] Figure 3Schematic diagram of determining the optimal partitioning strategy for a table based on the partition coefficient. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The present invention aims to provide a method, product, medium and equipment for dynamic and automated management of data throughout its entire life cycle, so as to simplify the data management process and reduce labor costs and the risk of intermediate demand transmission.

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] Figure 1 The flowchart of the data full life cycle dynamic automatic management method provided by the present invention is as follows: Figure 2 This is a schematic diagram of each stage of the execution process of the data full life cycle dynamic automation management method provided by the present invention. Figure 1 and Figure 2 The present invention provides a method for dynamic and automated management of data throughout its life cycle, comprising steps 1 to 9.

[0055] Step 1: Obtain company server resource information and database attribute information through file parsing function.

[0056] The method of the present invention is applicable to the application scenario of adding a new data source, involving the creation of tables corresponding to the data source, partition optimization and expired data cleaning, combining the input data characteristics to perform data partitioning and structure grouping, and combining the automatic grouping method to complete the automatic partition construction. For example: previously accessed a certain Dong and a certain cat order data, and now needs to add a certain Duoduo order data. After completing the corresponding field combing, the corresponding table is automatically created, the partition is optimized and the expired data is cleaned.

[0057] Specifically, in the information acquisition stage, the company's server resource information and database attribute information are obtained from the information provided by the user through the file parsing function. Taking Hadoop big data as an example, the storage service involved is HDFS. As shown in Table 1, the server resource information obtained includes the number of servers and the number of cores, memory size and hard disk size of each server; the database attribute information obtained includes the number of replicas, partition strategy and database characteristics; the partition strategy includes no partition, monthly partition, weekly partition and daily partition strategies; the database characteristics specifically include the number of namenode unit management files, namenode memory size, datanode number, datanode corresponding block default size and datanode maximum file size.

[0058] Table 1 Server resource information and database attribute information

[0059]

[0060] Namenode is the namespace that manages the file system. Datanode is the core component that actually stores data blocks in Hadoop Distributed File System (HDFS). Block is the default and most basic storage unit of HDFS.

[0061] Step 2: Calculate the optimal file size for data storage based on server resource information and database attribute information.

[0062] Usually, data reading time = file retrieval time + single file content query time. Among them, the file retrieval efficiency depends on the memory usage efficiency. Theoretically, the larger the file, the lower the memory usage rate, and the shorter the file retrieval time. However, as the file size increases, the time for content query in a single file will increase, so it is necessary to find a critical section to improve the efficiency of single file content query as much as possible while making full use of memory resources. Therefore, in the optimal storage parameter calculation stage, the present invention calculates the optimal solution for data storage based on the acquired server resource information and database attribute information, that is, calculates the optimal file size for data storage, because the file size is proportional to the file retrieval time and inversely proportional to the single file content query time.

[0063] To ensure the stable operation of the server, its kernel, memory, and hard disk will not be fully loaded, but it needs to be comprehensively considered based on the server's own hardware conditions and the number of deployed services. The present invention first determines the server's default usage rate, hardware conditions, and process busyness based on the server resource information, and then calculates the system resource usage rate according to formula (1):

[0064] (1)

[0065] The default usage rate is generally 80%. Hardware conditions refer to whether it is a shared server or an independent server, and consider key parameters such as the main frequency. Process busyness refers to whether there is a mixed deployment of multiple services, and a larger margin needs to be considered.

[0066] Based on the obtained database attribute information, the minimum file size can be calculated while ensuring system stability:

[0067] (2)

[0068] The optimal file size is calculated based on the above minimum file size and the system's minimum storage size:

[0069] (3)

[0070] in, The storage multiple is 1 by default. The calculated optimal file size is stored as the system optimal storage parameter in the system database corresponding to the method of the present invention.

[0071] For example, according to the above formulas (2) and (3), the current minimum file size and the current optimal file size can be calculated in real time:

[0072] ;

[0073] .

[0074] Step 3: Use the natural language parsing model to parse the field mapping relationship and table creation attributes from user requirements.

[0075] The reason why it is necessary to build a table is that the existing data management manages structured data based on the table structure, and the present invention is also an optimization method for managing data based on the table. Therefore, in the information acquisition stage, it is necessary to first obtain user requirements, and then automatically parse them through the natural language parsing model, parse the field mapping relationship and table building attributes from the user requirements and store them in the system database.

[0076] As shown in Table 2, the obtained field mapping relationship is information about the fields in the table, including field name, type, description, whether it can be empty, etc. As shown in Table 3, the obtained table creation attributes are information about the table itself, including database name, table name, partition key, storage duration, and database information, etc. The database information specifically includes the IP, port, user name, and password corresponding to the database.

[0077] Table 2 Field mapping relationship

[0078]

[0079] Table 3 Table creation properties

[0080]

[0081] For example, if a user wants to store a batch of user age information, then it is necessary to confirm the relevant field mapping relationship, one is the user identity identifier, namely user_id, the second is the user age, and the third is the partition field. In the table creation attributes, the database name and table name are required, and they are named here according to the conventional naming logic. Table partitioning is a feasible solution to isolate data and optimize queries, which can solve the problem of more data in a single table. After the table is partitioned, the table is still a complete table logically, but the data in the table is stored in multiple table spaces (physical files), so that when querying data, it is not necessary to scan the entire table every time, thereby improving query performance. The present invention defines the partition key as st_date, and splits the data according to the partition key, so that a large data storage unit can be split into multiple small units in terms of content. The storage duration is used to control the amount of data and prevent disk overflow.

[0082] Step 4: Based on the table creation properties, use the table creation syntax to generate the corresponding table creation script.

[0083] In the script generation phase, based on the above table creation properties, the table creation script is constructed according to the table creation syntax of the corresponding type of database. Since the description here is based on Hadoop, it is implemented in the language of hivesql. Then, the script information is stored in the system database.

[0084] For example, based on the above user age record information, the corresponding table creation script can be generated as follows:

[0085]

[0086] Step 5: Create a connection between the script and the target server based on the database information to execute the generated script.

[0087] During the script execution phase, a database connection is first established. Specifically, a connection is created between the execution script and the target server based on the IP, host, user, and password in the table creation properties, which is used to execute the script generated in step 4 above.

[0088] Step 6: After the script is executed, a list of daily data changes is obtained based on real-time data monitoring and data prediction.

[0089] A time series database is mainly used to process data with time tags (data that changes in the order of time, i.e., time serialization). Data with time tags is also called time series data. In the real-time data monitoring stage, the present invention is based on a time series database. After the script is executed, data monitoring is performed on each table to record changes in the amount of data in each table. Specifically, table data counting is implemented through code and written into a time series database to record changes in the amount of table data. The present invention uses a time series database to record changes in the amount of table data. Compared with traditional relational databases, it has better performance, real-time performance, and compression efficiency.

[0090] The time series prediction algorithm is used to predict and analyze future change trends based on the time series of historical data. In the data prediction stage, after the real-time data monitoring completes a cycle of monitoring, the data volume changes of each table are recorded as historical data. Then, through various time series prediction algorithms such as ARIMA, the data volume of subsequent dates is predicted based on the historical data to obtain predicted data. The historical data and predicted data constitute a daily data change list.

[0091] Taking the ARIMA time series forecasting algorithm as an example, the implementation logic is as follows: 1) Load data: obtain historical data by reading the time series database; 2) Divide the historical data into training set and test set; 3) Use the training set to train the ARIMA model and predict the test set; 4) After the training is completed, use the trained ARIMA model to obtain the forecast data for subsequent dates.

[0092] For example, the daily change list of order volume data of a comprehensive fast-moving consumer goods brand is shown in Table 4 below. The data before April 9 are real historical data, and the data after that are predicted data.

[0093] Table 4 Daily changes in order volume data for a comprehensive fast-moving consumer goods brand

[0094]

[0095] Step 7: Calculate the optimal partitioning strategy for the table based on the optimal file size and the list of daily data changes.

[0096] In the partition calculation phase, the optimal partition strategy for the table is calculated based on the system optimal storage parameters in step 2 and the daily data change list in step 6. There are two major types of partition strategies: one is horizontal partitioning based on the storage time, such as day, week, month, year, etc.; the other is vertical partitioning based on business scenarios, such as stores, products or business lines, etc. The present invention adopts the first type of partition strategy.

[0097] Specifically, first sort the daily granularity real historical data of the most recent cycle and the daily granularity forecast data of the next month in the daily data change list, then take the third quartile of the sorted data to obtain the daily average number of orders corresponding to the third quartile. Then calculate the daily average storage size according to formula (4):

[0098] (4)

[0099] Based on the optimal file size and the average daily storage size, the partition coefficient is calculated according to formula (5):

[0100] (5)

[0101] The default business coefficient is 1.

[0102] Based on the above partition coefficients, we can determine whether the partition selection corresponding to the table is reasonable and determine the optimal partition strategy for the table. Figure 3 As shown in the figure, the process of determining the optimal partitioning strategy for the table according to the partition coefficient is as follows: when the partition coefficient is greater than 40, it means that the data volume is relatively small, and the optimal partitioning strategy for the table is determined to be no partition; when the partition coefficient is between 10 and 40, it means that the data size of a month is close to the optimal partition data size, and the optimal partitioning strategy for the table is determined to be monthly partition; when the partition coefficient is between 2 and 10, it means that the data size of a week is close to the optimal partition data size, and the optimal partitioning strategy for the table is determined to be weekly partition; when the partition coefficient is between 0.1 and 2, it means that the data size of a single day is close to the optimal partition data size, and the optimal partitioning strategy for the table is determined to be daily partition; when the partition coefficient is less than 0.1, it means that the data size of a single day is much larger than the optimal partition data size, and partition splitting is required according to the business system.

[0103] For example, for the order volume of a comprehensive fast-moving consumer goods brand shown in Table 4, the average daily storage size corresponding to 10,000 orders calculated according to formula (4) is 1.45 MB:

[0104]

[0105] The partition coefficient calculated according to formula (5) is:

[0106]

[0107] Based on the above partition coefficients, it can be determined that the most suitable partitioning strategy (that is, the optimal partitioning strategy for the table) is weekly partitioning.

[0108] When the partition coefficient is a critical value, such as 10, the optimal partitioning strategy for the table can be either monthly partitioning or weekly partitioning.

[0109] Step 8: Update scripts and data based on the optimal partitioning strategy for the table.

[0110] In the script update phase, based on the table creation properties in step 3 and the script generated in step 4, if the optimal partitioning strategy of the table is inconsistent with the existing partitioning strategy, modify the corresponding partitioning field to update the script. For example, according to the above calculation, it can be confirmed that the optimal partitioning strategy of the table is weekly partitioning, and the original default partitioning strategy is daily partitioning, then modify the previous st_date to st_week.

[0111] Before modification (here named Table A for distinction):

[0112]

[0113] After modification (here named Table B for distinction):

[0114]

[0115] During the data update phase, historical data needs to be placed in a new table. The specific process is: 1) Connect to the database; 2) Rename historical table A to backup table A', that is, rename table A and leave the original table name empty; 3) Create a new table B based on the new script. Note that the new table B still uses the original table name. You only need to import the data in backup table A' into the new table B to complete the data update.

[0116] Step 9: Regularly clean up expired data based on the storage duration in the table creation properties.

[0117] During the data cleaning phase, expired data is cleaned up regularly according to the storage duration in the table creation attributes. The specific process is as follows: 1) First, the storage duration in the table creation attributes output in step 3 is read from the system database, such as 90 days in the example in Table 3. If not, the default is 365 days; 2) The earliest date corresponding to the data in each table is calculated according to the storage duration; 3) The actual partition of the corresponding table in the database is read; 4) Whether there is data exceeding the earliest date in the actual partition date; 5) If so, delete it.

[0118] The method of the present invention connects three independent functions of table creation, partition reconstruction and overdue data deletion in series, simplifies the data management process, and reduces the risk of intermediate demand transmission. Previously, designers were required to pass the design content to developers for implementation, and they needed to interact with operation and maintenance personnel during system operation. Now, only designers are required to complete it independently. Therefore, the method of the present invention reduces labor costs. Subsequently, only designers are required to perform configuration, and development and operation and maintenance do not need to participate. Through the method of the present invention, the efficient use of servers can be guaranteed, the stability of the storage system can be improved, the operation and maintenance risks can be reduced, and better data storage services can be provided.

[0119] In some embodiments, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for dynamic and automated management of data throughout its life cycle.

[0120] In some embodiments, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for dynamic and automated management of data throughout its entire life cycle.

[0121] In some embodiments, the present invention also provides a computer device, which includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the dynamic and automated management method of the data life cycle can be realized.

[0122] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for dynamic and automated management of data throughout its life cycle, characterized in that: include: Obtain the company's server resource information and database attribute information through the file parsing function; the server resource information includes the number of servers and the number of cores, memory size and hard disk size of each server; the database attribute information includes the number of replicas, partition strategy and database characteristics; the partition strategy includes no partition, monthly partition, weekly partition and daily partition; the database characteristics include the number of namenode unit management files, namenode memory size, datanode number, datanode corresponding block default size and datanode maximum file size; Calculate the optimal file size for data storage based on server resource information and database attribute information; The calculating the optimal file size for data storage based on the server resource information and the database attribute information specifically includes: Determine the default usage rate, hardware conditions and process busyness of the server based on the server resource information, and calculate the system resource usage rate according to the formula system resource usage rate = default usage rate * hardware conditions * process busyness; Based on database attribute information, according to the formula Calculate minimum file size; Based on the system resource usage and the minimum file size, the optimal file size for data storage is calculated according to the formula optimal file size = storage multiple * max (minimum storage size, minimum file size) * system resource usage; where max() means taking the maximum value; The field mapping relationship and table creation attributes are parsed from user requirements through a natural language parsing model; the field mapping relationship includes field name, type, description, and whether it can be empty; the table creation attributes include database name, table name, partition key, storage duration, and database information; the database information includes the IP, port, user name, and password corresponding to the database; Based on the table creation attributes, the corresponding table creation script is generated using the table creation syntax; Create a connection between the script and the target server based on the database information to execute the generated script; After the script is executed, a list of daily data changes is obtained based on real-time data monitoring and data prediction; Calculate the optimal partitioning strategy for the table based on the optimal file size and the daily list of data changes; The optimal partitioning strategy for the table is calculated based on the optimal file size and the daily data change list, specifically including: Determine the average daily number of orders corresponding to the third quartile based on the daily data change list, and calculate the average daily storage size based on the formula: average daily storage size = average daily number of orders corresponding to the third quartile (10,000) * storage size corresponding to 10,000 orders; Based on the optimal file size and average daily storage size, according to the formula Calculate the partition coefficient; Determine the optimal partitioning strategy for the table based on the partition coefficient; Update scripts and data based on the optimal partitioning strategy for the table; The script and data update according to the optimal partitioning strategy of the table specifically includes: When the optimal partitioning strategy of the table is inconsistent with the existing partitioning strategy, modify the corresponding partition fields to update the script; Connect to the database, rename the history table to the backup table, and leave the original table name empty; Create a new table based on the new script. The new table still uses the original table name. Import the data in the backup table into the new table to complete the data update. Periodically clean up expired data based on the storage duration in the table creation properties.

2. The method for dynamic and automated management of data throughout its life cycle according to claim 1, characterized in that: The daily data change list obtained based on real-time data monitoring and data prediction specifically includes: Use a time series database to monitor and record changes in the amount of data in each table in real time; After completing a cycle of monitoring, the data volume changes of each table are recorded as historical data; Through the time series prediction algorithm, the data volume of subsequent dates is predicted based on historical data to obtain the predicted data; Build a list of daily data changes based on historical data and forecast data.

3. The method for dynamic and automated management of data throughout its life cycle according to claim 1, characterized in that: Determining the optimal partition strategy of the table according to the partition coefficient specifically includes: When the partition coefficient is greater than 40, the optimal partition strategy for the table is determined to be no partition; When the partition coefficient is between 10 and 40, the optimal partition strategy for the table is determined to be monthly partitioning; When the partition coefficient is between 2 and 10, the optimal partition strategy for the table is determined to be weekly partitioning; When the partition coefficient is between 0.1 and 2, the optimal partition strategy for the table is determined to be daily partitioning; When the partition coefficient is less than 0.1, partitions are split according to the business system.

4. The method for dynamic and automated management of data throughout its life cycle according to claim 1, characterized in that: The periodic cleaning of expired data according to the storage duration in the table creation attribute specifically includes: Calculate the earliest date corresponding to the data in each table based on the storage duration; Read the actual partition of the corresponding table in the database; Determine whether there is data in the actual partition date that exceeds the earliest date. If so, delete it.

5. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for dynamic and automated management of data throughout its life cycle as described in any one of claims 1 to 4 is implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for dynamic and automated management of data throughout its life cycle as described in any one of claims 1 to 4 is implemented.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for dynamic and automated management of the entire life cycle of data as described in any one of claims 1 to 4.

Images