Data backup and cleaning method and device based on data change perception

Abstract

The invention relates to the field of database backup, and discloses a data backup and cleaning method and device based on data change perception, and the key point of the technical scheme is that the method comprises the following steps: S1, in response to a backup task, executing backup environment initialization, establishing target database connection and managing data connection of databases, creating a backup directory in the management database; s2, acquiring historical backup data from the management database and performing information verification, and executing S3 after verification is passed; s3, acquiring a target data object from the target database, comparing the target data object with historical backup data, and if the data is changed, backing up the changed data to a backup directory; s4, after backup is completed, data cleaning configuration is obtained, whether historical backup data needs to be cleaned or not is judged, if yes, data cleaning is executed, and minimum data backup is executed based on automatic sensing of data changes; flexible configuration of data expiration strategies is realized, and automatic cleaning of expired data can be realized.

Description

Technical Field

[0001] The present invention relates to the field of database backup, and more specifically, to a data backup and cleaning method and device based on data change perception. Background Art

[0002] As an information system storage device, the database provides business-oriented data storage and access capabilities. It is always the core of the information system. In the real business environment, measures such as backup and expired data cleanup are needed to ensure the continuity and stability of information system services.

[0003] Database backup is to store the data and information in the database to another file or medium to prevent data loss or damage. In the actual production process, the lack of backup may cause production failures or even bankruptcy of enterprises. Therefore, backup is a necessary management measure for databases, especially in financial fields such as banks and securities.

[0004] The expired data cleanup measure is to clean up the data that is no longer needed during the actual business. As the information system stores non-critical data such as transaction records, operation records, and cash flow. Long-term operation will cause the total amount of database data to continue to accumulate. Especially when there are massive users and high concurrent access to the transaction system, the total amount of data increases very rapidly. The accumulation of massive data and the increasing pressure on the database will lead to performance bottlenecks in the database on the one hand; on the other hand, it will also consume more storage, computing and other resources, increasing operating costs. Therefore, in actual production, enterprises will use various methods to clean up the database regularly or irregularly to control the total amount of the database.

[0005] Although database backup and expired data are both necessary means of database management, when enterprises actually carry out corresponding work, due to the limitations of existing technologies and solutions, they will design and build database backups separately and perform data cleanup when necessary. However, in the actual production environment, it is found that they are interdependent. Specifically, (1) If the information system does not perform effective data cleanup, the database backup will increase the backup time and backup storage costs due to the increase in the total amount of data; (2) The expired data cleanup operation may cause significant changes to the actual production data after the operation. It may be uncertain whether a backup is needed, so a backup operation must be performed before the cleanup can be performed. (3) Backup and cleanup are independent of each other. On the one hand, there is the possibility of duplicate backup, and on the other hand, it increases the difficulty of data cleanup.

[0006] In addition, traditional backup can be divided into physical backup and logical backup according to the technical route. Physical backup can be further divided into full backup and incremental backup. However, in actual production, duplicate backup and recovery efficiency are key issues in the field of database backup.

[0007] Under the premise of well-configured incremental and full backup strategies, physical backup can solve the problem of duplicate backups, but it cannot solve the problem of recovery efficiency in specific scenarios. For example, for a 1TB MySQL database, a full plus incremental backup method is used. If a 1MB table needs to be restored, then the initial full backup needs to be restored, followed by the incremental backup, before the 1MB table can be accessed.

[0008] Traditional logical backups are difficult to solve the problem of duplicate backups. For example, for a 1TB MySQL database, the 1TB of data needs to be accessed every day during backup. This problem is particularly prominent when the total amount of data in the information system is large and the amount of changes is small. Summary of the invention

[0009] The purpose of the present invention is to provide a data backup and cleanup method and device based on data change perception, based on automatic perception of data changes, to perform minimized data backup; to achieve flexible configuration of data expiration policies, and to achieve automatic cleanup of expired data.

[0010] The above technical objectives of the present invention are achieved through the following technical solutions: A data backup and cleaning method based on data change perception, comprising the following steps:

[0011] S1. In response to the backup task, perform backup environment initialization, establish a target database connection, a management database data connection, and create a backup directory in the management database;

[0012] S2, obtain historical backup data from the management database and perform information verification, and execute S3 after the verification passes;

[0013] S3. Obtain the target data object from the target database and compare it with the historical backup data. If the data has changed, back up the changed data to the backup directory.

[0014] S4. After the backup is completed, obtain the data cleaning configuration and determine whether the historical backup data needs to be cleaned up. If so, perform data cleaning.

[0015] As a preferred technical solution of the present invention, S1 includes:

[0016] S1-1, initializing environment variables, the environment variable initialization comprising: confirming the environment variable information when the backup operation is executed;

[0017] S1-2, initializing the management database connection, wherein the management database connection initialization includes: connecting to a database storing backup configuration and operation status data according to environment variable information;

[0018] S1-3, target database connection initialization The target database connection initialization includes: connecting to the target database according to the environment variable information;

[0019] S1-4, the backup thread and its parameter initialization includes: starting a corresponding number of backup threads according to the device load parameters, and setting backup thread parameters, and after the backup thread is started, obtaining environment variable information, managing database connections and target database connections;

[0020] S1-5, the backup directory initialization includes: selecting a backup directory location and creating a backup directory.

[0021] As a preferred technical solution of the present invention, the environment variable information includes a backup basic directory, backup management system connection information, a current timestamp, a backup unique identifier, and backup system log information;

[0022] The device load parameters include CPU usage, IO usage, and IO latency; the minimum number of backup threads is 2, and the maximum number of backup threads is CPU core number - 2; the parameters of the backup threads include data consistency read parameters, transaction log location read parameters, and enable parallel read parameters;

[0023] The permissions of the backup directory are to allow backup of program location and protocol data;

[0024] Execute exception handling steps when the management database connection fails, the target database connection fails, the backup thread and its parameter initialization fails, or the backup directory initialization fails.

[0025] As a preferred technical solution of the present invention, the exception handling step includes outputting an exception log, recovering environment variable information, and issuing an alarm. The alarm methods include: SMS, email, and Webhook. After the alarm is completed, the backup task is exited.

[0026] As a preferred technical solution of the present invention, S2 includes:

[0027] S2-1, obtaining the host identification information, wherein the host identification information includes the database cluster name, server address, and host name;

[0028] S2-2, querying the backup information in the management database through the local identification information, wherein the backup information includes the backup basic information and the backup characteristic information;

[0029] S2-3. Analyze the basic information of historical backup.

[0030] S2-4, parsing historical backup feature information, where the backup feature information is a backup unique identifier of data that has been backed up or needs to be backed up;

[0031] S2-5. Verify the backup set according to the historical backup basic information and the historical backup characteristic information;

[0032] S2-6. If the verification succeeds, execute S3. If the verification fails, update the backup status and execute the exception handling steps.

[0033] As a preferred technical solution of the present invention, the backup basic information includes: database name, backup table name, backup file name, backup size, backup file storage path, backup start time and completion time;

[0034] The backup feature information includes: table structure information, table record row number, index maximum / minimum record value, table calculation code;

[0035] The backup status includes tracking and managing the accuracy, success or failure of data backup;

[0036] The process of verifying a backup set is as follows: calculating the file verification data storage pointer, obtaining the backup set header information, obtaining the backup set footer information, generating a backup calculation code, and comparing the calculation codes. If the comparison results are consistent, the verification succeeds; if the comparison results are inconsistent, the verification fails.

[0037] As a preferred technical solution of the present invention, S3 includes the following steps:

[0038] S3-1, obtain the data object from the target database, and compare its feature data with the historical backup information;

[0039] S3-2, if there is any data change, back up the data object to the management database;

[0040] The data backup process is:

[0041] The basic information of the table storing the data object,

[0042] Stores table characteristic information of data objects,

[0043] Slice the data object according to the table characteristic information to obtain data slices;

[0044] The data slices are delivered to each backup thread for storage until all backup threads are executed and the backup task is marked as completed.

[0045] As a preferred technical solution of the present invention, the content of the feature data comparison includes table structure information, maximum and minimum index records, number of table record rows, and calculation code;

[0046] The basic table information includes database name, table name, table type, and character set;

[0047] The table characteristic information includes table structure information, index head and tail values, number of data rows, and data calculation code;

[0048] The slicing process is as follows: according to the number of data rows and the number of backup threads, the data object is divided into data slices with the target number of data rows, and according to the number of data rows, the detailed information of each data slice is read from the index. The detailed information of the data slice includes the table name, index column, maximum value, minimum value, and number of rows.

[0049] As a preferred technical solution of the present invention, S4 includes:

[0050] S4-1. Obtain data cleaning configuration, where the data cleaning configuration is a configuration for managing a database and is used to store a list of tables to be cleaned, a cleaning method, and a strategy;

[0051] S4-2, determine the table type and calculate the cleaning time of each type of data;

[0052] S4-3, determining whether the cleaning time of each type of data has arrived, and if so, obtaining the cleaning method of each type of data;

[0053] S4-4. Generate cleaning statements for each type of data according to the cleaning method;

[0054] S4-5. Execute the cleanup statements one by one. If the execution fails, execute the exception handling steps.

[0055] A data backup and cleanup device based on data change perception is characterized in that it includes a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor implements the above method when executing the computer program.

[0056] In summary, the present invention has the following beneficial effects: based on automatic perception of data changes, minimized data backup is performed; flexible configuration of data expiration policies is achieved without the need for information systems to publish versions; automatic cleanup of expired data can be achieved; backup status identification before data cleanup is achieved, and no backup operations need to be performed while ensuring data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of the present invention;

[0058] Figure 2 is a flow chart of initializing the backup environment of the present invention;

[0059] Figure 3 It is a historical backup data acquisition and verification flow chart of the present invention;

[0060] Figure 4 It is a backup set verification flow chart of the present invention;

[0061] Figure 5 is a data change perception flow chart of the present invention;

[0062] Figure 6 is a characteristic data comparison flow chart of the present invention;

[0063] Figure 7 is a backup execution flow chart of the present invention;

[0064] Figure 8 It is a data cleaning flow chart of the present invention. DETAILED DESCRIPTION

[0065] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0066] like Figure 1 As shown, the present invention provides a data backup and cleaning method based on data change perception, comprising the following steps:

[0067] S1. In response to the backup task, perform backup environment initialization, establish a target database connection, a management database data connection, and create a backup directory in the management database;

[0068] like Figure 2 As shown, S1 includes:

[0069] S1-1, initializing environment variables, the environment variable initialization includes: confirming the environment variable information when the backup operation is executed to represent the backup set information;

[0070] The environment variable information includes the backup basic directory, backup management system connection information, current timestamp, backup unique identifier, and backup system log information. The backup unique identifier ensures the uniqueness of the database backup set.

[0071] S1-2, management database connection initialization, management database connection initialization includes: according to the environment variable information, connecting to the database storing the backup configuration and operation status data;

[0072] S1-3. Target database connection initialization Target database connection initialization includes: connecting to the target database according to environment variable information;

[0073] S1-4, initializing the backup thread and its parameters includes: starting a corresponding number of backup threads according to the device load parameters, and setting the backup thread parameters. After the backup thread is started, it obtains the environment variable information, manages the database connection, and the target database connection;

[0074] Backup threads refer to the threads used when performing backup tasks. Device load parameters include CPU usage, IO usage, and IO latency. The minimum number of backup threads is 2, and the maximum number of backup threads is the number of CPU cores minus 2. Using multiple threads to perform backup operations can improve backup efficiency and speed.

[0075] The parameters of the backup thread include data consistency read parameters, transaction log position read parameters, and parallel read enable parameters.

[0076] S1-5, initialization of the backup directory includes: selecting the backup directory location and creating the backup directory. The permissions of the backup directory are to allow the backup of program locations and protocol data;

[0077] When selecting a backup directory location, you need to confirm that it has sufficient storage space and is easily accessible as the backup directory. Use file system commands, such as mkdir, to create the backup directory in the selected location and ensure that the backup directory's permissions are set to allow the backup program to access and write data. This usually involves setting the correct user and group ownership, as well as appropriate read and write permissions. Finally, check whether the backup directory has been successfully created and verify that its permissions are set correctly. If all operations are executed successfully, the backup environment is initialized.

[0078] During the execution of S1-1 to S1-5, the exception handling steps are executed when the management database connection fails, the target database connection fails, the backup thread and its parameter initialization fails, and the backup directory initialization fails.

[0079] S2, such as Figure 3 As shown, historical backup data is obtained from the management database and information verification is performed. After the verification passes, S3 is executed;

[0080] Historical backup data is obtained from the management database, and then the existing backup files are verified based on the backup information.

[0081] S2 includes:

[0082] S2-1. Obtain the local identification information. The host identification information includes the database cluster name, server address, and host name. Since the management database stores the backup information of all target databases, it is necessary to obtain the local identification information to confirm the target.

[0083] S2-2, query the backup information in the management database through the local identification information;

[0084] The backup information includes backup basic information and backup characteristic information;

[0085] S2-3. Analyze the basic information of historical backups. The basic information includes: database name, backup table name, backup file name, backup size, backup file storage path, backup start time and completion time, including the entire backup process and status;

[0086] S2-4. Analyze historical backup feature information. The backup feature information is a unique identifier of the backup data that has been backed up or needs to be backed up. The backup feature information includes: table structure information, table record row number, index maximum / minimum record value, and table calculation code.

[0087] For example, for the same table, if the table structure and data do not change, then its characteristic data will not change; conversely, if the table structure and data change, the characteristic data will be different.

[0088] S2-5. Verify the backup set according to the historical backup basic information and the historical backup characteristic information;

[0089] like Figure 4 As shown, the process of verifying the backup set is: calculating the file verification data storage pointer, obtaining the backup set header information, obtaining the backup set tail information, generating a backup calculation code, comparing the calculation code, and comparing the currently generated calculation code with the previously stored calculation code. If the comparison results are consistent, it means that the file is not damaged, that is, the verification is successful. If the comparison results are inconsistent, it means that the historical backup set has been damaged and changed, and other abnormalities have occurred, that is, the verification has failed.

[0090] S2-6. If the verification succeeds, execute S3. If the verification fails, update the backup status, which includes tracking and managing the accuracy, success or failure of data backup. At this time, since the backup set verification fails, the corresponding backup status is updated to backup set verification failure, and the exception handling steps are executed.

[0091] S3, such as Figure 5 As shown, the target data object is obtained from the target database and compared with the historical backup data. If the data has changed, the changed data is backed up in the backup directory;

[0092] S3 includes the following steps:

[0093] S3-1, obtain the data object from the target database and compare its feature data with the historical backup information, such as Figure 6 As shown;

[0094] The content of feature data comparison includes table structure information, maximum and minimum index records, number of table record rows, and calculation code; the acquisition of table feature data is divided into different levels, which help to confirm the table feature data more intelligently, from shallow to deep, and at the fastest speed.

[0095] The method of acquiring features in the present invention is divided into four levels:

[0096] Level 1, comparison table structure information:

[0097] Feature content: mainly includes column name, column type, column length, character set, encoding, index structure, number of columns, number of indexes

[0098] The execution method is: obtain the data table definition statement, calculate the MD5 of the content in step A, and compare the MD5 value of the history table to confirm whether they are the same.

[0099] Level 2, compare the maximum / minimum records of the index:

[0100] Feature content: The index itself does not directly store the value of the "maximum" or "minimum" record, but you can use the characteristics of the index to write a query statement to directly access the leftmost and rightmost ends of the index to access the maximum / minimum record of the index.

[0101] The execution method is: determine whether there is a primary key index, if not, directly return different; query the maximum / minimum record on the primary key index record. Then use PRIKEY, table name, primary key column name, maximum value, and minimum value as input items, calculate its feature code through the MD5 algorithm; compare the MD5 value of the history table to confirm whether it is the same.

[0102] Level 3, compare the number of rows in the data table:

[0103] Feature content: When migrating data, synchronizing, verifying backups, or simply monitoring data changes, counting the total number of rows is a common method. However, if you count the number of rows directly without considering the cost of database execution, it will lead to a waste of time and database resources. Therefore, this method will identify the primary key and index structure, and then count the number.

[0104] Calculation method: Determine whether there is a primary key index. If so, obtain the primary key column, then count the total number of rows based on the column, then use the table name, PRIKEY, primary key column name, and number of rows as input items, calculate its signature through MD5, and finally execute step C. If there is no primary key column, determine whether there is a non-empty index column. If so, obtain the index column, then count the total number of rows based on the column, then use the table name, index name, index column name, and number of rows as input items, calculate its signature through MD5, and if there is no index column, directly return different; compare the MD5 values ​​of the history table to confirm whether they are the same.

[0105] Level 4, calculation code for comparison data:

[0106] Characteristic content: The calculation code of data is an aggregation operation of all data contents through an algorithm, and a string is used to uniquely identify the data of the two-dimensional table structure. After calculation, any two pieces of data can only get the same calculation code if the content is exactly the same, otherwise the calculation code will be different.

[0107] Calculation method: Slice the data, support configurable number of rows and slices, such as this example uses 10,000 rows and is divided into 1,000 slices; according to the obtained slice content, concatenate each row of data according to the specified delimiter, such as this example uses "|+|"; then concatenate each data row of the slice according to the specified delimiter, such as this example uses "|-|"; use the table name and slice number as input items, generate a 32-bit identifier through the MD5 algorithm, and then use the first 16-bit string as the slice identifier header and the last 16-bit string as the slice identifier tail. Finally, connect the slice header, slice content, and slice tail as input, and obtain the final calculation code through bytecode conversion and CRC32 algorithm; after calculating the calculation codes of all slices, compare the MD5 values ​​of the corresponding slices in the history table to confirm whether they are the same.

[0108] S3-2, if there is any data change, back up the data object to the management database;

[0109] like Figure 7 As shown in the figure, the process of data backup is:

[0110] The basic information of the table storing the data object includes database name, table name, table type, and character set;

[0111] The table characteristic information of the stored data object includes table structure information, index header and tail values, number of data rows, and data calculation code;

[0112] Slice the data object according to the table characteristic information to obtain data slices;

[0113] The slicing process is as follows: according to the number of data rows and the number of backup threads, the data object is divided into data slices with the target number of data rows, and according to the number of data rows, the detailed information of each data slice is read from the index. The detailed information of the data slice includes the table name, index column, maximum value, minimum value, and number of rows.

[0114] For example: Based on the stored table feature information, such as the table's primary key, index structure, and data rows, combined with the number of parallel threads, the number of data rows that each slice should contain is generated according to the number of data rows / number of parallel threads. The head and tail values ​​of each slice are read from the index based on the number of rows. Finally, the table name, index column, minimum value, maximum value, number of rows, and other information of each slice are obtained.

[0115] The slice information is delivered concurrently to each backup thread, and each backup thread will execute the data query task and synchronously update the backup task status to "in progress". During the backup process, the output results will be encoded and converted to binary through the pipeline, and then stored. After the backup is completed, the backup thread will be updated to "completed".

[0116] Check and update the backup task status. After the backup task is started, each slice performs normal data backup. Then the main thread monitors the execution status of the backup threads. After all backup threads are completed, the entire backup task is marked as completed and the overall backup status is updated.

[0117] S4, such as Figure 8 As shown, after the backup is completed, the data cleaning configuration is obtained to determine whether the historical backup data needs to be cleaned up. If so, data cleaning is performed.

[0118] Expired data cleaning refers to the process of identifying and deleting data that has lost its use value or exceeded the preset validity period. Its purpose is to optimize the data storage structure, reduce unnecessary storage space usage, and improve the efficiency and accuracy of data processing. Since this method provides a set of data and backup overall solutions, the data cleaning task will only be triggered after the data backup is completed, and the data that is no longer needed in the actual business will be cleaned up.

[0119] S4 includes:

[0120] S4-1. Obtain data cleaning configuration. Data cleaning configuration is a configuration for managing the database. Data cleaning configuration is used to store a list of tables to be cleaned, cleaning methods, and policies.

[0121] As an embodiment of the present invention, the main fields and descriptions of the data cleaning configuration are shown in Table 1 below:

[0122] Table 1

[0123]

[0124] S4-2, determine the table type and calculate the cleaning time of each type of data;

[0125] From the data cleaning configuration table, only the database name and table name of the table are configured, and no detailed cleaning strategy is configured. Therefore, it is necessary to automatically generate a default configuration to simplify the user's usage threshold. The specific strategy is shown in Table 2 below:

[0126] Table 2

[0127]

[0128] S4-3, determining whether the cleaning time of each type of data has arrived, and if so, obtaining the cleaning method of each type of data;

[0129] When the above steps are completed, the data cleaning scope is confirmed. For the truncate table operation, directly concatenate the table name, which is the final cleaning statement; for the truncate partition cleaning method, when only a single partition is involved, directly concatenate the table name and partition name, which is the final cleaning statement; for the truncate partition cleaning method, when multiple partitions are involved, all partition names need to be convenient, the final partition list needs to be confirmed, and then the partition cleaning statement is generated; for the delete cleaning method, the primary key and unique index key values ​​need to be confirmed according to the cleaning conditions, and then only the data that meets both the unique index and the cleaning conditions is cleaned row by row.

[0130] S4-4. Generate cleaning statements for each type of data according to the cleaning method;

[0131] S4-5. Execute the cleanup statements one by one. If the execution fails, execute the exception handling steps.

[0132] According to the generated cleanup statements, the statements are obtained one by one and are located in the target database center. When the execution is completed, for the statements of the truncate operation, the execution status of the SQL statement is updated to success after the execution is successful, and then the next statement is executed. If the execution of the cleanup statement fails, the exception handling step is entered. For the delete statement, the number of successes of the corresponding cleanup range is updated after the execution is successful; if the execution of the cleanup statement fails, the exception handling step is executed.

[0133] In the present invention, there is also an exception handling step, including outputting an exception log, recovering environment variable information, and issuing an alarm. The alarm methods include: SMS, email, Webhook, and the backup task is exited after the alarm is completed.

[0134] Corresponding to the above method, the present invention also provides a data backup and cleanup device based on data change perception, including: a processor and a memory, the memory stores a computer program executable by the processor, and the above method is implemented when the processor executes the computer program.

[0135] The technical advantages of the solution of the present invention are: Automatic data change perception: During the data backup process, the data is docked and scanned. This perception capability can be accurate to the specific data record or field level, and can quickly capture any subtle data changes. Automatic data change perception ensures that the backup system always attempts to discover the changed data with the lowest time cost, resource cost and the lowest impact on the business database, and perform the corresponding backup or cleanup operation.

[0136] Reduce backup costs and improve recovery efficiency: When using a data backup method that automatically perceives data changes, a more accurate backup strategy can be implemented. Only the data that has changed is backed up, rather than backing up the entire database. This greatly reduces the amount of backup data, thereby reducing the consumption of storage resources and significantly reducing backup costs. At the same time, when data recovery is required, since the amount of backup data is small and highly targeted, the recovery process can quickly locate the key data that needs to be restored, reducing the time and complexity of data recovery and improving recovery efficiency. This allows the system to recover to normal more quickly in the face of unexpected situations or data corruption, ensuring business continuity and stability, and reducing losses caused by long data recovery time.

[0137] Backup and cleanup integration: This method combines the two processes of data backup and expired data cleanup. While automatically sensing data changes and backing up, the system also identifies and cleans up expired data according to preset policies. This integrated design avoids the duplication of backup and cleanup operations separately, improving the efficiency and consistency of operations.

[0138] Good compatibility: This method is compatible with various common database systems and can run and be implemented smoothly regardless of the type, version or architecture of the database. It will not cause any interference or conflict with the existing database structure and application system and can coexist harmoniously with other systems and components. Whether it is applied in a newly developed system or adapted to an already launched business system, it can be easily implemented without worrying about additional risks or difficulties caused by compatibility issues.

[0139] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A data backup and cleanup method based on data change perception, characterized by: The steps include: S1. In response to the backup task, perform backup environment initialization, establish a target database connection, a management database data connection, and create a backup directory in the management database; S2, obtain historical backup data from the management database and perform information verification, and execute S3 after the verification passes; S3. Obtain the target data object from the target database and compare it with the historical backup data. If the data has changed, back up the changed data to the backup directory. S4. After the backup is completed, obtain the data cleaning configuration and determine whether the historical backup data needs to be cleaned up. If so, perform data cleaning.

2. According to the data change perception-based data backup and cleanup method of claim 1, it is characterized by: S1 include: S1-1, initializing environment variables, the environment variable initialization comprising: confirming the environment variable information when the backup operation is executed; S1-2, initializing the management database connection, wherein the management database connection initialization includes: connecting to a database storing backup configuration and operation status data according to environment variable information; S1-3, target database connection initialization The target database connection initialization includes: connecting to the target database according to the environment variable information; S1-4, the backup thread and its parameter initialization includes: starting a corresponding number of backup threads according to the device load parameters, and setting backup thread parameters, and after the backup thread is started, obtaining environment variable information, managing database connections and target database connections; S1-5, the backup directory initialization includes: selecting a backup directory location and creating a backup directory.

3. The data backup and cleanup method based on data change perception according to claim 2 is characterized in that: The environment variable information includes the backup basic directory, backup management system connection information, current timestamp, backup unique identifier, and backup system log information; The device load parameters include CPU usage, IO usage, and IO delay; The minimum number of backup threads is 2, and the maximum number of backup threads is CPU core number minus 2; the parameters of the backup threads include data consistency read parameters, transaction log position read parameters, and parallel read enable parameters; The permissions of the backup directory are to allow backup of program location and protocol data; Execute exception handling steps when the management database connection fails, the target database connection fails, the backup thread and its parameter initialization fails, or the backup directory initialization fails.

4. The data backup and cleanup method based on data change perception according to claim 3 is characterized by: The exception handling step includes outputting an exception log, collecting environment variable information, and issuing an alarm. The alarm method includes: SMS, email, and Webhook. After the alarm is completed, the backup task is exited.

5. The data backup and cleanup method based on data change perception according to claim 4 is characterized in that: The S2 includes: S2-1, obtaining the host identification information, wherein the host identification information includes the database cluster name, server address, and host name; S2-2, querying the backup information in the management database through the local identification information, wherein the backup information includes the backup basic information and the backup characteristic information; S2-3. Analyze the basic information of historical backup. S2-4, parsing historical backup feature information, where the backup feature information is a backup unique identifier of data that has been backed up or needs to be backed up; S2-5. Verify the backup set according to the historical backup basic information and the historical backup characteristic information; S2-6. If the verification succeeds, execute S3. If the verification fails, update the backup status and execute the exception handling steps.

6. The data backup and cleanup method based on data change perception according to claim 5 is characterized by: The backup basic information includes: database name, backup table name, backup file name, backup size, backup file storage path, backup start time and completion time; The backup feature information includes: table structure information, table record row number, index maximum / minimum record value, table calculation code; The backup status includes tracking and managing the accuracy, success or failure of data backup; The process of verifying a backup set is as follows: calculating the file verification data storage pointer, obtaining the backup set header information, obtaining the backup set footer information, generating a backup calculation code, and comparing the calculation codes. If the comparison results are consistent, the verification succeeds; if the comparison results are inconsistent, the verification fails.

7. The data backup and cleaning method based on data change perception according to claim 6 is characterized in that: S3 includes the following steps: S3-1, obtain the data object from the target database, and compare its feature data with the historical backup information; S3-2, if there is any data change, back up the data object to the management database; The data backup process is: The basic information of the table storing the data object, Stores table characteristic information of data objects, Slice the data object according to the table characteristic information to obtain data slices; The data slices are delivered to each backup thread for storage until all backup threads are executed and the backup task is marked as completed.

8. The data backup and cleaning method based on data change perception according to claim 7 is characterized in that: the characteristic data The contents of the comparison include table structure information, maximum and minimum index records, number of table record rows, and calculation codes; The basic table information includes database name, table name, table type, and character set; The table characteristic information includes table structure information, index head and tail values, number of data rows, and data calculation code; The slicing process is as follows: according to the number of data rows and the number of backup threads, the data object is divided into data slices with the target number of data rows, and according to the number of data rows, the detailed information of each data slice is read from the index. The detailed information of the data slice includes the table name, index column, maximum value, minimum value, and number of rows.

9. The data backup and cleaning method based on data change perception according to claim 8 is characterized in that: S4 include: S4-1. Obtain data cleaning configuration, where the data cleaning configuration is a configuration for managing a database and is used to store a list of tables to be cleaned, a cleaning method, and a strategy; S4-2, determine the table type and calculate the cleaning time of each type of data; S4-3, determining whether the cleaning time of each type of data has arrived, and if so, obtaining the cleaning method of each type of data; S4-4. Generate cleaning statements for each type of data according to the cleaning method; S4-5. Execute the cleanup statements one by one. If the execution fails, execute the exception handling steps.

10. A data backup and cleaning device based on data change perception, characterized in that: include: A processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

Images