Data processing method and device, equipment and storage medium

Abstract

Embodiments of the present application disclose a data processing method, device and equipment, and a storage medium, wherein the method comprises: when there is a cleaning event of log data of a host database, obtaining a file descriptor of backup log data which is backed up from the host by a backup machine last time, the file descriptor of any log data being used to reflect the order in which any log data is generated, the backup machine constantly backing up log data from the host according to the order in which each log data is generated; determining a reference file descriptor according to the file descriptor of the backup log data, and obtaining a target file descriptor corresponding to target log data to be cleaned based on the reference file descriptor, the target file descriptor being less than or equal to the reference file descriptor; virtually cleaning the target log data based on the target file descriptor, and asynchronously deleting the target log data. The embodiments of the present application can prevent disk jitter caused by log data cleaning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data processing method and device, equipment and storage medium. BACKGROUND

[0002] The log data (may also be referred to as binary log) in MySQL is stored in binary form in the database, the binary log is mainly used for recording the changes of the database, contains all the updated or potentially updated data, is the basis of master-slave replication, and can also be used for data disaster recovery. The so-called data disaster recovery refers to establishing a remote data system, backing up the data in the database to the remote data system, so as to protect the safety of the data and improve the continuous availability of the data.

[0003] The binary log is usually stored in the disk, for relatively busy business, a large number of binary logs may be generated every day, if these log data are not cleaned for a long time, it will cause great waste of disk space, therefore, the cleaning of the binary log data is an important work to ensure the high utilization rate of the disk. The commonly used binary log cleaning method is to set the log expiration time, when the expiration time is reached, the log will be automatically cleaned. The cleaning of the expired log is to synchronously delete the entire log data, if the log data is large, it may cause disk jitter, thereby causing MySQL jitter. Therefore, in the field of MySQL application, how to effectively clean the log data has become one of the hot issues in current research. SUMMARY

[0004] The embodiment of the present application provides a data processing method, device, equipment and storage medium, and the log data is cleaned in the mode of virtual cleaning and asynchronous deletion, so that the disk space jitter caused by synchronous deletion of large log data can be avoided.

[0005] In one aspect, the embodiment of the present application provides a data processing method, comprising:

[0006] When there is a cleaning event of the log data of the host database, the file descriptor of the backup log data of the standby machine last backed up from the host is acquired, a plurality of log data are stored in the host, each log data corresponds to a file descriptor, the file descriptor of any log data is used to reflect the order of generation of the any log data, and the standby machine constantly backs up the log data from the host according to the order of generation of each log data;

[0007] The reference file descriptor is determined according to the file descriptor of the backup log data, and the target file descriptor corresponding to the target log data to be cleaned in the host database is acquired based on the reference file descriptor, and the target file descriptor is less than or equal to the reference file descriptor;

[0008] The target log data is virtually cleaned based on the target file descriptor, and the target log data is asynchronously deleted.

[0009] In one aspect, the embodiment of the present application provides a data processing apparatus, which runs the following units:

[0010] The obtaining unit is used for obtaining a backup log data file descriptor of the host database which is backed up by the standby host last time when there is a cleaning event of the log data of the host database, the host stores a plurality of log data, each log data corresponds to a file descriptor, and the file descriptor corresponding to any log data is used for reflecting the order of generation of any log data, and the standby host constantly backs up the log data from the host according to the order of generation of each log data;

[0011] The determining unit is used for determining a reference file descriptor according to the backup log data file descriptor;

[0012] The obtaining unit is further used for obtaining a target file descriptor corresponding to the log data to be cleaned in the host database based on the reference file descriptor, and the target file descriptor is less than or equal to the reference file descriptor;

[0013] The processing unit is used for virtually cleaning the target log data based on the target file descriptor, and asynchronously deleting the target log data.

[0014] In one embodiment, the processing unit performs the following steps when virtually cleaning the target log data based on the target file descriptor:

[0015] Hard link creation processing is performed for the target file descriptor, a candidate file descriptor corresponding to the target file descriptor is obtained, a corresponding relationship between the candidate file descriptor and the target log data is established, and the target file descriptor is deleted from the host.

[0016] In one embodiment, the processing unit performs the following steps when virtually cleaning the target log data based on the target file descriptor:

[0017] A deleted mark is added to the target file descriptor.

[0018] In one embodiment, the processing unit performs the following steps when asynchronously deleting the target log data:

[0019] The target log data is obtained from the disk of the host according to the candidate file descriptor or the target file descriptor with the deleted mark, and the data content of the target log data is deleted in batches according to a target threshold.

[0020] In one embodiment, the processing unit performs the following steps when deleting the data content of the target log data in batches according to the target threshold:

[0021] The data content of the target log data is split according to the target threshold, and the size of the data content of each batch of to-be-deleted data obtained by the splitting is less than or equal to the target threshold; and the deletion is performed in sequence according to the splitting order of each batch.

[0022] In one embodiment, the cleaning event of the log data in the host refers to that the storage space of the disk is less than a storage threshold, and the host database is in a business low peak period.

[0023] The determining unit is further configured to determine a business type corresponding to the host data, and the obtaining unit is further configured to obtain a throughput threshold corresponding to the business type; and the determining unit is further configured to determine that the host database is in a business low peak period if the throughput of the host database is less than the throughput threshold.

[0024] In one embodiment, the number of the standby machines is at least one, the backup log data includes at least one backup log data which is most recently backed up from the host by the at least one standby machine, and the determining unit performs the following steps when determining the reference file descriptor according to the file descriptors of the backup log data:

[0025] The first file descriptor corresponding to the log data which has been completely backed up in the host is determined based on the file descriptors of the backup log data, the log data which has been completely backed up refers to the log data which is backed up in each of the at least one standby machine, and a file descriptor less than or equal to the first file descriptor is selected as the reference file descriptor, wherein the larger the file descriptor is, the later the order of the log data corresponding to the file descriptor is backed up in each of the at least one standby machine.

[0026] In one embodiment, if the number of the first file descriptors is at least one, the file descriptor less than or equal to the first file descriptor refers to a file descriptor less than or equal to the larger one of at least one file descriptor.

[0027] In one embodiment, the data processing apparatus further includes a sending unit, and the at least one standby machine includes a business slave machine of the host and a data disaster recovery standby machine.

[0028] The sending unit is configured to send, if it is detected that the log data backup event of the data disaster backup machine occurs and the file descriptor corresponding to the log data to be backed up indicated by the log data backup event is less than the reference file descriptor, notification information of backing up the log data to be backed up from the service slave to the data disaster backup machine; the notification information is used to notify the data disaster backup machine that the log data to be backed up has been cleaned from the host, and instruct the data disaster backup machine to obtain the log data to be backed up from the service slave.

[0029] In one aspect, an embodiment of the present application provides a data processing device, characterized in that comprising: a processor adapted to implement one or more computer programs; and a computer storage medium, the computer storage medium storing one or more computer programs, the one or more computer programs being adapted to be loaded and executed by the processor:

[0030] When there is a cleaning event of log data of a host database, a file descriptor of backup log data last backed up from the host by a backup machine is obtained, a plurality of log data are stored in the host, each log data corresponds to a file descriptor, and the file descriptor corresponding to any log data is used to reflect the order of generation of any log data, and the backup machine constantly backs up log data from the host according to the order of generation of each log data;

[0031] A reference file descriptor is determined according to the file descriptor of the backup log data, and a target file descriptor corresponding to target log data to be cleaned is obtained based on the reference file descriptor, the target file descriptor being less than or equal to the reference file descriptor;

[0032] The target log data is virtually cleaned based on the target file descriptor, and the target log data is asynchronously deleted.

[0033] In one aspect, an embodiment of the present application provides a computer storage medium, characterized in that the computer storage medium stores a computer program, and the computer program is executed by a processor to perform the following steps:

[0034] When there is a cleaning event of log data of a host database, a file descriptor of backup log data last backed up from the host by a backup machine is obtained, a plurality of log data are stored in the host, each log data corresponds to a file descriptor, and the file descriptor corresponding to any log data is used to reflect the order of generation of any log data, and the backup machine constantly backs up log data from the host according to the order of generation of each log data;

[0035] A reference file descriptor is determined according to the file descriptor of the backup log data, and a target file descriptor corresponding to target log data to be cleaned is obtained based on the reference file descriptor, the target file descriptor being less than or equal to the reference file descriptor;

[0036] The target log data is virtually cleaned based on the target file descriptor, and the target log data is asynchronously deleted.

[0037] In one aspect, the embodiment of the present application provides a computer program product or a computer program. The computer program product comprises a computer program stored in a computer storage medium. A processor of a data processing device reads the computer program from the computer storage medium. The processor executes the computer program, so that the data processing device executes the following steps:

[0038] When there is a cleaning event of log data in a host database, a file descriptor of backup log data which is last backed up from the host by a standby machine is obtained. The host stores a plurality of log data, each of which corresponds to a file descriptor. The file descriptor of any log data reflects the order in which any log data is generated. The standby machine backs up the log data from the host according to the order in which each log data is generated.

[0039] A reference file descriptor is determined according to the file descriptor of the backup log data, and a target file descriptor corresponding to target log data to be cleaned is obtained based on the reference file descriptor. The target file descriptor is less than or equal to the reference file descriptor.

[0040] The target log data is virtually cleaned based on the target file descriptor, and the target log data is asynchronously deleted.

[0041] In the embodiment of the present application, when there is a cleaning event of log data in a host database, a file descriptor of backup log data which is backed up from the host database by a standby machine is obtained. Further, a reference file descriptor is determined according to the file descriptor of the backup log data, and a target file descriptor less than or equal to the reference file descriptor is selected. Then, target log data indicated by the target file descriptor is virtually cleaned and asynchronously deleted based on the target file descriptor. In the above cleaning process of the log data, the target log data to be cleaned is determined by referring to the backup log data which is already backed up in the standby machine. Thus, the situation that the log data in the standby machine is lost due to the cleaning of the log data in the host database can be avoided. In addition, the target log data is deleted from the host database by using the virtual cleaning and asynchronous deletion, so that the disk jitter caused by the synchronous deletion of the target log data is avoided. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 is a structural schematic diagram of a data management system provided by an embodiment of the present application;

[0044] Figure 2 is a flow schematic diagram of a data processing method provided by an embodiment of the present application;

[0045] Figure 3 is a schematic diagram of virtual cleaning and asynchronous deletion of target log data provided by an embodiment of the present application;

[0046] Figure 4 is a flow schematic diagram of another data processing method provided by an embodiment of the present application;

[0047] Figure 5 is a schematic diagram of determining a reference file descriptor provided by an embodiment of the present application;

[0048] Figure 6 is a structural schematic diagram of a data processing device provided by an embodiment of the present application;

[0049] Figure 7 is a result schematic diagram of a data processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application.

[0051] MySQL is a relational database management system. A relational database saves data in different tables instead of putting all data in one big warehouse, thus increasing speed and providing flexibility. Due to this advantage of the MySQL database, it has become one of the most popular databases.

[0052] The SQL language used by MySQL is the most commonly used standardized language for accessing a database, that is, the access to the MySQL database is implemented based on the statements of the SQL language. The access to the MySQL database can include adding data in the database, deleting data in the database, and modifying data in the database, etc.

[0053] All changes in the MySQL database are recorded in the form of a binary log (hereinafter referred to as log data). The binary log records all operations of performing changes to the MySQL database, and records the time when the statement occurs, the execution time, the operation data, and other additional information, but does not record the SQL statements such as SELECT and SHOW which do not modify data.

[0054] The MySQL binary log is the basis for realizing database master-slave replication, and can also be used for data disaster recovery. The master-slave replication refers to copying data in a host database to a slave machine completely, so as to realize that the host provides a write service and the slave machine provides a read-only service, thereby achieving read-write separation, and also realizing high availability of the database by the slave machine, when the host fails, the slave machine is switched automatically. The data disaster recovery refers to establishing an off-site data system, and backing up data in the host database to the off-site data system, so as to protect the safety of data in the database. In some application systems such as a financial payment system, data disaster recovery is the top priority.

[0055] Based on the foregoing description, the embodiment of the present application provides a data management system, referring to Figure 1 The structure diagram of the data management system provided by the embodiment of the present application is shown in FIG. 1. Figure 1 The data management system shown in the figure can include a host 101, at least one service slave machine 102, and a data disaster recovery standby machine 103 for data disaster recovery. The host 101, the service slave machine 102, and the data disaster recovery standby machine 103 can all be a database server. For the convenience of description, the service slave machine 102 and the data disaster recovery standby machine 103 can be collectively referred to as a standby machine of the host 101. The service slave machine 102 can be a slave machine of downstream services of the host 101, and is mainly used to provide read-only services, thereby realizing read-write separation of the database.

[0056] In an embodiment, data in the database of the host 101 can be asynchronously replicated to the service slave machine 102, that is, the host 101 does not need to keep connected with the slave machine 102 all the time, and the slave machine 102 can even connect the host 101 intermittently through dial-up to replicate data in the database of the host 101. The slave machine 102 can replicate all data in the database or part of data in the database from the host 101. Similarly, the data disaster recovery standby machine 103 can replicate data in the database of the host 101 from the host 101. It should be noted that the host in the embodiment of the present application can refer to a device that needs to clean up log data, and the host can be a real host or a standby machine of a certain host.

[0057] In order to ensure that data in the host 101 and the service slave machine 102 always remain consistent, if data in the host 101 changes, data in the service slave machine 102 also needs to change. As known from the foregoing, the change of data in the database is recorded in the form of a binary log, therefore, the service slave machine 102 needs to backup the binary log in the host, and maintain data in the database of the service slave machine 102 according to the binary log.

[0058] Based on this, the working principle that the business slave 102 copies data from the master 101 can be summarized as follows: the business slave 102 first accurately copies the data on the master 101, and then activates the binary log function on the master 101; then, the business slave 102 periodically connects the master 101 and detects the binary log, if there is new binary log since the last connection, the business slave 102 backs up the new binary log, and maintains the data in the business slave 102 according to the new binary log. As can be seen, the binary log is the basis of the master-slave replication.

[0059] The backup of the data in the database in the data disaster recovery machine 103 is realized by the database snapshot backup and the incremental binary log backup. When the network where the master 101 is located sends an exception, data disaster recovery needs to be used to recover the data. When recovering, the data backup snapshot is first imported into an empty instance, and then the backup binary log is played back to the time point before the failure. For example, the data snapshot backup is performed once a week, for example, the data snapshot backup is performed on the previous Monday, and all changes of the data in the database from the previous Monday to the current time are recorded in the binary log, which is called the incremental binary log backup.

[0060] In an embodiment, the log data generated in the master 101 is usually stored in a disk, and if the log data is not cleaned for a long time, the disk storage space can be tight. Based on this, the embodiment of the present application provides a data processing scheme for cleaning the log data. In a specific implementation, when a cleaning event of the log data of the master database is detected, it is determined which backup log data has been acquired in the backup machine; then, the target log data to be cleaned is determined based on the backup log data acquired in the backup machine; and finally, the target log data to be cleaned is cleaned in a virtual cleaning and asynchronous deletion manner.

[0061] In the embodiment of the present application, when the target log data to be cleaned is determined, the backup log data acquired in the backup machine is used as the basis, so that it can be ensured that the deletion of the target log data to be cleaned has no influence on the maintenance of the database in the backup machine, or in other words, it can be ensured that after the target log data to be cleaned is deleted, the data maintained in the backup machine can still remain the same as the corresponding data of the master database. The commonly used log cleaning method in the prior art can include directly deleting all the log data when a cleaning event of the log data is detected, and does not consider whether the log data to be deleted has an influence on the maintenance of the data in the backup machine. Compared with the prior art, the embodiment of the present application can ensure that the business slave or the data disaster recovery backup machine will not be affected by the log cleaning, and avoid the problem of data loss in the backup machine due to log cleaning.

[0062] In addition, in the embodiment of the present application, when cleaning the target log data to be cleaned, instead of synchronously deleting all data contents of the target log data to be cleaned, a method of first virtually cleaning and then asynchronously deleting is adopted. The virtual cleaning can be understood as first publishing a message that the target log data has been deleted, but at this time the target log data has not been actually deleted. The asynchronous deletion refers to actually cleaning the data contents in the target log data in batches after the message is published. Compared with the method of synchronously and one-time deleting the target log data to be cleaned in the prior art, the embodiment of the present application realizes the smooth deletion of the log data to be cleaned, and avoids disk jitter.

[0063] Based on the above data management system, the embodiment of the present application provides a data processing method, which is described with reference to Figure 2 A flowchart of the data processing method provided by the embodiment of the present application is shown in FIG. 2. Figure 2 The data processing method shown in FIG. 2 can be executed by a host, and specifically can be executed by a processor of the host, Figure 2 The data processing method shown in FIG. 2 can include the following steps:

[0064] Step 201: When there is a cleaning event of log data of a host database, a file descriptor of backup log data last backed up by a standby machine from the host is acquired.

[0065] In one embodiment, as known from the foregoing, the log data is mainly stored in a disk. If the storage space of the disk is small and cannot store other useful data, it is considered to delete a part of the log data in the disk. Based on this, the cleaning event of the log data of the host database can refer to that the storage space of the disk is less than a storage threshold. In other words, if the storage space of the disk is sufficient, the log data does not need to be cleaned, and as many log files as possible are retained; if the disk space is insufficient, the log data starts to be cleaned.

[0066] In other embodiments, in order to reduce the influence of log cleaning on database business, such as the increase of business access time, log cleaning can be selected to be performed in a business low peak period. In this case, the cleaning event of the log data of the host database can refer to that the disk space is less than the storage threshold, and the host database is in a business low peak period. In other words, since the log data cleaning will occupy a certain resource, if the log cleaning is performed in a business peak period, it can cause the increase of access time,

[0067] Optionally, the higher the throughput of the host database is, the more transactions the host database processes, and at this time the host database can be in a business peak period; on the contrary, the lower the throughput of the host database is, the fewer transactions the host database processes, and the host database can be in a business low peak period.

[0068] In one embodiment, the throughput of the host database can be measured by using TPS (Transactions Per Second). The greater the TPS, the greater the throughput of the host database, and vice versa. In the embodiment of the present application, the transaction refers to the operation of adding, deleting or modifying data in the database. Based on this, the embodiment of the present application can set a throughput threshold. If the TPS is less than the throughput threshold, it is determined that the service is in the low peak period. If the TPS is greater than the throughput threshold, it is determined that the service is in the peak period.

[0069] It should be understood that different types of services have different sensitivities to TPS. For example, for service A, TPS reaching 1000 indicates that the service has reached the peak period, but for service B, TPS reaching 3000 indicates that the service has reached the peak period. In the embodiment of the present application, the throughput threshold can be dynamically adjusted, and different thresholds can be set for different types of database services. In a specific implementation, the database administrator (DBA) can be intelligently recommended to set appropriate throughput thresholds for different databases according to the TPS monitoring data. The database administrator is a general term for artificial work related to the management and maintenance of the database management system (DBMS), which belongs to a branch of operation and maintenance engineers, and is mainly responsible for the whole life cycle management of the business database from design, test to deployment.

[0070] Based on the above description, the host can determine whether the host database is in the low peak period by determining the type of service corresponding to the host database, obtaining the throughput threshold corresponding to the type of service, and determining that the host database is in the low peak period if the throughput of the host database is less than the throughput threshold.

[0071] In order to ensure that the log data in the standby machine is not lost, the embodiment of the present application will obtain the backup log data of the last backup from the host before starting the log cleaning, and determine which log data in the host can be cleaned according to the backup log data.

[0072] In a specific implementation, each log data in the host corresponds to a file descriptor, and the file descriptor corresponding to any log data is equivalent to the number corresponding to the log data, according to which the corresponding log data can be found in the disk. The file descriptor corresponding to any log data can be used to reflect the order in which the log data is generated, for example, the file descriptor corresponding to the first generated log data can be represented as binlog01, and the file descriptor corresponding to the second generated log data can be represented as binlog02. The backup machine continuously backs up log data from the host according to the order in which the log data is generated, for example, the backup machine first backs up the log data with the file descriptor binlog01 from the host, and then backs up the log data with the file descriptor binlog02 from the host.

[0073] When there is a cleaning event for the log data of the host database, the file descriptor corresponding to the backup log data last backed up by the backup machine from the host is obtained, and then in step S202, the reference file descriptor is determined according to the file descriptor of the backup log data, and then the target file descriptor less than or equal to the reference file descriptor is selected from the host, and the target log data corresponding to the target file descriptor is selected as the log data to be cleaned.

[0074] Here, how to obtain the file descriptor corresponding to the backup log data last backed up by the backup machine from the host is emphasized, and how to determine the reference file descriptor and how to select the target file descriptor will be specifically introduced in step S202.

[0075] In an embodiment, the number of backup machines corresponding to the host can be at least one, and the backup log data can include at least one backup log data last backed up by at least one backup machine from the host. Specifically, the at least one backup machine can include one or more service slaves and one or more data disaster recovery backup machines corresponding to the downstream business of the host, and then the backup log data can include the log data last backed up by the service slave from the host, and / or one or more log data last backed up by the data disaster recovery backup machine from the host.

[0076] In this case, the number of file descriptors of the backup log data obtained in step S201 can be one or more, for example, the backup machines of the host include a service slave and a data disaster recovery, the file descriptor corresponding to the backup log data last backed up by the service slave from the host is binlog02, and the file descriptor corresponding to the log data last backed up by the data disaster recovery backup machine from the host is binlog03.

[0077] Step S202, determining the reference file descriptor according to the file descriptor of the backup log data, and obtaining the target file descriptor corresponding to the target log data to be cleaned based on the reference file descriptor.

[0078] The reference file descriptor refers to the maximum value of the file descriptors corresponding to the log data allowed to be cleaned in the host, in other words, the log data in the host whose file descriptors are less than or equal to the reference file descriptor can be deleted.

[0079] The target file descriptor refers to the file descriptor less than or equal to the reference file descriptor among the multiple file descriptors included in the host. For example, if the reference descriptor is binlog02, then the file descriptor binlog01 and the file descriptor binlog02 can be regarded as the target file descriptor.

[0080] In an embodiment, the number of the standby machines is at least one, the backup log data includes at least one backup log data which is backed up from the host by the at least one standby machine last time, and the reference file descriptor can be determined according to the first file descriptor corresponding to the log data which has been completely backed up among the multiple log data included in the host. The log data which has been completely backed up refers to the log data which is backed up in each of the at least one standby machine. For example, the log data A has been backed up by the business slave machine and the data disaster recovery standby machine, so the log data A is the log data which has been completely backed up. For another example, the log data B has been backed up by the business slave machine but not by the data disaster recovery standby machine, so the log data B is not the log data which has been completely backed up.

[0081] In a specific implementation, if the number of the standby machines is at least one, the backup log data includes at least one backup log data which is backed up from the host by the at least one standby machine last time, and the reference data descriptor is determined according to the file descriptor of the backup log data, which can include: determining the first file descriptor corresponding to the log data which has been completely backed up in the host based on the file descriptor of the backup log data. The log data which has been completely backed up refers to the log data which is backed up in each of the at least one standby machine. Selecting the file descriptor less than or equal to the first file descriptor as the reference file descriptor, wherein the larger the file descriptor is, the later the order of the log data corresponding to the file descriptor is backed up by each of the at least one standby machine.

[0082] It should be understood that the backup of the host is constantly backing up the log data from the host according to the order of the generation of each log data, that is, the earlier the generated log data is, the earlier the backup machine backs up, and then the file descriptor of the backup log data backed up by the backup machine last time represents the largest log data backed up from the host, for example, the file descriptor of the log data backed up by a backup machine from the host last time is binlog03, which means that the largest log data backed up from the host in the backup machine is the log data indicated by binlog03, that is, the log data corresponding to the file descriptor less than or equal to binlog03 has been backed up by the backup machine.

[0083] Therefore, determining the first file descriptor corresponding to the log data completely backed up in the host according to the file descriptor of the backup log data can include: taking the smaller one of the at least one file descriptor of the backup log data as the largest file descriptor corresponding to the log data completely backed up in the host; and determining the file descriptor less than or equal to the largest file descriptor as the first file descriptor corresponding to the log data completely backed up in the host. For example, the file descriptor of the backup log data backed up by the data disaster backup machine from the host last time is binlog03, and the file descriptor of the backup log data backed up by the data disaster backup machine from the host last time is binlog03, and the first file descriptor corresponding to the log data completely backed up in the host is binlog01 and binlog02, binlog02 is the largest one of the log data completely backed up, and is also the smallest one of the file descriptors corresponding to the backup log data backed up by different backup machines last time.

[0084] It should be understood that the number of first file descriptors can be one or more. If the number of first file descriptors is one, then the file descriptor less than or equal to the first file descriptor can refer to the file descriptor less than or equal to the first file descriptor. For example, the first file descriptor is binlog01, and then the file descriptor less than or equal to the first file descriptor includes binlog01. If the number of first file descriptors is more than one, then the file descriptor less than or equal to the first file descriptor can refer to the file descriptor less than or equal to the larger one of the first file descriptors. For example, the first file descriptors include binlog01 and binlog02, and then the file descriptor less than or equal to the first file descriptor is binlog01 and binlog02.

[0085] In one embodiment, as known from the foregoing, the standby machine can include a service slave machine and a data disaster recovery standby machine, and the backup log data includes at least one backup log data that the standby machine has backed up from the host machine last time. It can be understood that the backup log data mentioned above can refer to the log data that the service slave machine has backed up from the host machine last time; or the backup log data can refer to the log data that the data disaster recovery standby machine has backed up from the host machine last time; or the backup log data can refer to the log data that the service slave machine has backed up from the host machine last time and the log data that the data disaster recovery standby machine has backed up from the host machine last time.

[0086] It should be understood that if the backup log data mentioned above is the log data that the service slave machine or the data disaster recovery standby machine has backed up from the host machine last time, it indicates that only the log data that the service slave machine or the data disaster recovery standby machine has backed up last time is traversed when the reference file descriptor is determined, or in other words, the reference file descriptor at this time only represents the log data that has been completely backed up in the service slave machine or the data disaster recovery standby machine. In this way, the number of the log data that needs to be traversed in the standby machine last time can be reduced, and the log cleaning progress can be accelerated.

[0087] However, there is a problem in this way: if only the file descriptor of the backup log data that the service slave machine has backed up last time is traversed when the reference file descriptor is determined, there can be a part of the log data that has been backed up in the service slave machine and has not been backed up in the data disaster recovery standby machine. In order to solve this problem, the data disaster recovery device can interact with the service slave machine to obtain the log data that has not been backed up in the data disaster recovery standby machine from the standby machine.

[0088] Similarly, if only the file descriptor of the backup log data that the data disaster recovery standby machine has backed up is traversed when the reference file descriptor is determined, there can be a part of the log data that has been backed up in the data disaster recovery standby machine and has not been backed up in the service slave machine. At this time, the service slave machine can communicate with the data disaster recovery standby machine to obtain the log data that has not been backed up in the service slave machine from the data disaster recovery device.

[0089] For example, the backup log data includes the log data that the service slave machine has backed up from the host database, and after the asynchronous deletion processing of the target log data is performed, the method further includes: if a log data backup event of the data disaster recovery standby machine is detected, and the file descriptor corresponding to the log data to be backed up indicated by the log data backup event is less than the reference file descriptor, sending notification information of backing up the log data to be backed up from the service slave machine to the data disaster recovery standby machine; the notification information is used to notify the data disaster recovery standby machine that the log data to be backed up has been cleaned from the host, and instruct the data disaster recovery standby machine to obtain the log data to be backed up from the service slave machine.

[0090] In other embodiments, if the number of the backup machines is at least one, the backup log data includes at least one backup log data backed up by at least one backup machine from the master machine; and in order to clean up as much log data as possible in one log data cleaning process, the reference file descriptor can also be determined according to the maximum of the file descriptors corresponding to the backup log data that has been backed up in the backup machines.

[0091] In a specific implementation, determining the reference file descriptor according to the file descriptor of the backup log data includes: obtaining the file descriptor corresponding to each backup log data in the at least one backup log data; selecting the maximum file descriptor from the file descriptors corresponding to each backup log data; and taking the file descriptor less than or equal to the maximum file descriptor as the reference file descriptor. For example, assuming that the backup log data includes backup log data A and backup log data B, the file descriptor corresponding to the backup log data A is binlog01, and the file descriptor corresponding to the backup log data B is binlog02, then binlog02 is the maximum file descriptor, and the file descriptor less than or equal to binlog02 is taken as the reference file descriptor.

[0092] In this case, there can be backup log data that has been backed up in one backup machine but not in other backup machines, and at this time, the backup log data in each backup machine can be ensured to be the same through mutual interaction between the backup machines.

[0093] In step S203, the target log data is virtually cleaned based on the target file descriptor, and the target log data is asynchronously deleted.

[0094] The virtual cleaning of the target log data can mean that only a message that the target log data has been cleaned is sent, but at this time, the target log data is not cleaned and is still stored in the disk; and the asynchronous deletion of the target log data means that after the message that the target log data has been cleaned is sent, the log data in the disk is deleted in batches. The combination of the virtual cleaning and the asynchronous deletion can avoid the influence of the synchronous deletion of a large target log file on the disk IO (disk IO means disk input and output), and can avoid disk jitter.

[0095] Deleting a large target log file at once impacts disk I / O, which is one reason for increased processing time during log cleanup. This invention can effectively avoid this. Another reason for increased processing time is untimely database response. Log cleanup involves database locks, and increased lock wait times lead to increased processing time. Locks are the most commonly used concurrency control mechanism, a primary means of preventing other transactions from accessing and executing resources, and achieving concurrency control. A lock is a request made by a transaction to access a database resource (such as a table or record). Once a transaction acquires the lock, it gains control over the data, and other transactions cannot modify the data until the lock is released. When a transaction is rolled back, the locked resource is released. In simpler terms, when a user locks an object in the database, other users cannot access that object.

[0096] In this embodiment of the invention, deleting target log data in batches can refer to deleting a certain amount of data content from the target log data each time, such as deleting only 20M of data content each time. Each time deletion is performed, the lock used to lock the target log data can be reacquired. Since the amount of data to be deleted each time is small, the deletion can be completed quickly, avoiding the problem of increased business time caused by long-term lock occupation.

[0097] In one embodiment, virtual cleanup of target log data based on the target file descriptor can refer to adding a deleted marker to the target file descriptor. This way, when there is an operation on the target log data based on the target file descriptor, such as copying the target log data, a message will appear indicating that the target log data has been deleted, but the target log data has not actually been deleted, thus achieving virtual cleanup of the target log data.

[0098] In other embodiments, virtual cleanup of target log data based on target file descriptors may further include: creating hard links for the target file descriptors to obtain candidate file descriptors corresponding to the target file descriptors; establishing a correspondence between the candidate file descriptors and the target log data; and deleting the target log file descriptors from the host.

[0099] A hard link (also called a link) refers to creating multiple file descriptors for a single log data entry. A hard link allows multiple file descriptors, whether located in the same directory or not, to simultaneously modify the same log data. In other words, two file descriptors with a hard link point to the same log data, and modifications or other operations can be performed on the same log data using these two file descriptors.

[0100] After the hard link creation processing of the target file descriptor is completed, the target file descriptor and the candidate file descriptor are in a hard link relationship.

[0101] Optionally, after the target log data is virtually cleaned, the target log data can be deleted asynchronously. In an embodiment, the asynchronous deletion of the target log data includes: obtaining the target log data according to the candidate file descriptor or the target file descriptor that has been added with a deletion mark; and deleting the data content of the target log data in batches according to a target threshold.

[0102] The target threshold can be any value, such as 20M, 30M, and the like. Optionally, the deletion of the data content of the target log data in batches according to the target threshold includes: splitting the data content of the target log data according to the target threshold, and the size of the data content of each batch of to-be-deleted data obtained by the splitting is less than or equal to the target threshold; and deleting in sequence according to the splitting order of each batch.

[0103] For example, referring to FIG. 3, which is a schematic diagram of virtual cleaning and asynchronous deletion of target log data according to an embodiment of the present application, assuming that the target log file descriptor is binlog01, the virtual cleaning and asynchronous deletion of the target log file corresponding to the target log file descriptor can include: (1) establishing a hard link binlog01.h of binlog01; (2) deleting the target file descriptor, which can mean deleting the target file descriptor in the database and the target file descriptor in the disk; and (3) asynchronously deleting the target log file in the disk.

[0104] In the embodiment of the present application, when there is a cleaning event of log data in the host database, the file descriptor of the backup log data that has been backed up from the host database by the standby machine is obtained; further, a reference file descriptor is determined according to the descriptor of the backup log data, and a target reference descriptor smaller than or equal to the reference file descriptor is selected, and then the target log data indicated by the target file descriptor is virtually cleaned and deleted asynchronously based on the target file descriptor. In the above cleaning process of the log data, the target log data to be cleaned is determined by referring to the backup log data that has been backed up in the standby machine, so that the situation that the log data in the standby machine is lost due to the cleaning of the log in the host database can be avoided. Moreover, the target log data is deleted from the host database by using the virtual cleaning and asynchronous deletion, so that the disk jitter caused by the synchronous deletion of the target log data is avoided.

[0105] Based on the above data processing method embodiment, another data processing method is provided in the embodiment of the present application. Referring to FIG. 4, which is a schematic diagram of virtual cleaning and asynchronous deletion of target log data according to another embodiment of the present application, assuming that the target log file descriptor is binlog01, the virtual cleaning and asynchronous deletion of the target log file corresponding to the target log file descriptor can include: (1) establishing a hard link binlog01.h of binlog01; (2) deleting the target file descriptor, which can mean deleting the target file descriptor in the database and the target file descriptor in the disk; and (3) asynchronously deleting the target log file in the disk. Figure 4A flowchart of another data processing method provided by the embodiment of the present application is shown. Figure 4 The data processing method shown includes a service slave of the host and a data disaster recovery slave. Figure 3 The data processing method shown can include the following steps:

[0106] In step S401, it is detected whether the storage space of the disk is less than a storage threshold value.

[0107] In step S402, if the storage space of the disk is less than the storage threshold value, the throughput of the host service data is calculated and the throughput threshold value corresponding to the host database is obtained.

[0108] In step S403, it is detected whether the throughput of the host database service is less than the throughput threshold value.

[0109] In step S404, if it is less, the file descriptor of the backup log data backed up from the host last time by the service slave is obtained, and the file descriptor of the backup log data backed up from the host last time by the data disaster recovery slave is obtained.

[0110] In step S405, the smallest file descriptor is selected as the reference file descriptor from the two obtained file descriptors.

[0111] As known from the foregoing, the reference file descriptor refers to the maximum value in the file descriptors corresponding to the log data allowed to be deleted in the host. The smallest value in the file descriptors corresponding to the backup log data backed up last time in each slave is taken as the reference file descriptor, which can ensure that the log data to be deleted in the host is already backed up in each slave.

[0112] The method for cleaning log data in the prior art can include: the first kind, setting the expiration time of the log data, and automatically cleaning the log when it expires; the second kind, directly executing the PURGE tool, which can support cleaning to a certain log data or before a certain time point, and this scheme is generally manually executed or added to the timing task execution. The problem of this is that it cannot accurately determine which log data should be cleaned or which log data generated at which time should be cleaned. If too little is cleaned, the purpose of releasing the storage control of the disk cannot be achieved; if too much is cleaned, the data in the slave can be lost.

[0113] Compared with the above two kinds of prior art, the embodiment of the present application can ensure that the slave can backup all the backup logs in the host, avoiding the loss of log data in the slave.

[0114] For example, referring to Figure 5A schematic diagram for determining a reference file descriptor is provided for the embodiment of the present application. It is assumed that the host comprises 6 log data, and the file descriptors corresponding to each log data are binlog01, binlog02, binlog03, binlog04, binlog05 and binlog06 respectively. It is assumed that the host corresponds to two service slaves and one data disaster recovery machine. It is assumed that the file descriptor corresponding to the backup log data last backed up from the host by the first service slave is binlog06. The file descriptor corresponding to the backup log data last backed up from the host by the second service slave is binlog04. The file descriptor corresponding to the backup log data last backed up from the host by the data disaster recovery machine is binlog05. Among the file descriptors of the backup log data last backed up in each of the above backup machines, the smallest file descriptor is binlog04. Therefore, binlog04 is taken as the reference file data descriptor.

[0115] In step S406, each file descriptor in the host is traversed in sequence. If a file descriptor is less than or equal to the reference file descriptor, the file descriptor is taken as a target file descriptor, and hard link processing is performed on the target file descriptor to obtain a candidate file descriptor corresponding to the target file descriptor.

[0116] In step S407, the target file descriptor is deleted.

[0117] In step S408, the target log file indicated by the target file descriptor is deleted in batches.

[0118] Optionally, in order to avoid the influence of deleting log data on the service and to retain as many log files as possible, after deleting one log data each time, it is determined again whether the storage space of the disk is less than the storage threshold. If yes, step S408 is repeatedly executed. If no, the log deletion operation can be performed.

[0119] Through verification in practice, the data processing method of the embodiment of the present application greatly reduces the influence of log cleaning on the service. The data loss rate of the backup machine due to log cleaning is reduced to 0%, and the invalidation rate of backup data due to log cleaning is reduced to 0%.

[0120] In the embodiment of the application, when the storage space of the disk is less than a storage threshold and the host database is in a business off-peak period, the cleaning of the log data in the host is started; in the cleaning process of the log data, the file descriptors corresponding to the log data backed up in each backup machine are obtained, one of the multiple file descriptors is selected as the largest file descriptor corresponding to the multiple log data allowed to be deleted in the host, and then all the log data smaller than the largest file descriptor in the host are virtually cleaned and asynchronously deleted in batches, so that the cleaned log data in the host has been backed up in all backup machines, the data loss in the backup machine can be effectively avoided, and the batch deletion of the log data to be cleaned can avoid the disk jitter caused by the deletion of large log data at one time.

[0121] It should be noted that, Figure 2 and Figure 4 The data processing method described in the above embodiments is implemented outside the MyQSL database, and if self-research is needed in the future, the above data processing method can be implemented in the MyQSL.

[0122] Based on the above data processing method, the embodiment of the application provides a data processing device. Referring to Figure 6 , a result schematic diagram of a data processing device provided by the embodiment of the application. Figure 6 The data processing device shown in the figure can run the following units:

[0123] The acquisition unit 601 is configured to acquire a file descriptor of backup log data of a backup machine last time backed up from a host when there is a cleaning event of log data of the host database, the host stores multiple log data, each log data corresponds to a file descriptor, the file descriptor of any log data is used to reflect the order of generation of the any log data, and the backup machine constantly backs up log data from the host according to the order of generation of each log data;

[0124] The determination unit 602 is configured to determine a reference file descriptor according to the file descriptor of the backup log data;

[0125] The acquisition unit 601 is further configured to acquire a target file descriptor corresponding to the log data to be cleaned in the host database based on the reference file descriptor, the target file descriptor is less than or equal to the reference file descriptor;

[0126] The processing unit 603 is configured to virtually clean the target log data based on the target file descriptor and asynchronously delete the target log data.

[0127] In one embodiment, the processing unit 603 performs the following steps when performing virtual cleaning of the target log data based on the target file descriptor:

[0128] In one embodiment, the processing unit 603 performs the following steps when performing virtual cleaning of the target log data based on the target file descriptor:

[0129] In one embodiment, the processing unit 603 performs the following steps when performing virtual cleaning of the target log data based on the target file descriptor:

[0130] In one embodiment, the processing unit 603 performs the following steps when performing asynchronous deletion processing of the target log data:

[0131] In one embodiment, the processing unit 603 performs the following steps when performing asynchronous deletion processing of the target log data:

[0132] In one embodiment, the processing unit 603 performs the following steps when performing asynchronous deletion processing of the target log data:

[0133] In one embodiment, the processing unit 603 performs the following steps when performing asynchronous deletion processing of the target log data:

[0134] In one embodiment, the cleaning event of the log data in the host refers to that the storage space of the disk is less than a storage threshold, and the host database is in a business low peak period; the determining unit 602 is further configured to determine a business type corresponding to the host database; the obtaining unit 601 is further configured to obtain a throughput threshold corresponding to the business type; and the determining unit 602 is further configured to determine that the host database is in a business low peak period if the throughput of the host database is less than the throughput threshold.

[0135] In one embodiment, the number of standby machines is at least one, the backup log data includes at least one backup log data that is most recently backed up by at least one standby machine from the host, and the determining unit 602 performs the following steps when determining a reference file descriptor according to the file descriptor of the backup log data:

[0136] Based on the file descriptor of the backup log data, determine the first file descriptor corresponding to the log data that has been fully backed up in the host, wherein the fully backed-up log data refers to the log data that has been backed up in each of the at least one standby machine;

[0137] A file descriptor less than or equal to the first file descriptor is selected as the reference file descriptor, wherein the log data corresponding to the larger file descriptor is backed up later by each of the at least one standby machine.

[0138] In one embodiment, if the number of the first file descriptors is at least one, then a file descriptor less than or equal to the first file descriptor refers to a file descriptor less than or equal to the larger of at least one file descriptor.

[0139] In one embodiment, Figure 6 The data processing apparatus shown further includes a sending unit 604; the at least one backup machine includes the service slave machine of the host and a data disaster recovery backup machine. If the backup log data includes the log data last backed up by the service slave machine from the host database, then the sending unit 604 is used for:

[0140] If a log data backup event is detected on the data disaster recovery backup machine, and the file descriptor corresponding to the log data to be backed up indicated by the log data backup event is smaller than the reference file descriptor, then a notification message is sent to the data disaster recovery backup machine to back up the log data to be backed up from the service slave machine; the notification message is used to notify the data disaster recovery backup machine that the log data to be backed up has been cleaned up from the host machine, and to instruct the data disaster recovery backup machine to obtain the log data to be backed up from the service slave machine.

[0141] According to one embodiment of the present invention, Figure 2 and Figure 4 The data processing method shown can involve various steps that can be derived from... Figure 6 This is performed by each unit in the data processing apparatus shown. For example, Figure 2 The aforementioned step S201 can be performed by Figure 6 The acquisition unit 601 in the data processing device shown is responsible for executing step S202, which can be performed by... Figure 6 The data processing device shown executes step S203 by acquiring unit 601 and determining unit 602. Figure 6 The data processing unit 603 in the shown data processing apparatus executes the operation; for example, Figure 4 In the data processing method shown, steps S401, S403, and steps S405-S408 can be derived from... Figure 6 The processing unit 603 in the data processing device shown executes step S402, which can be performed by...Figure 6 The determining unit 602 and the obtaining unit 601 in the data processing apparatus are configured to perform the step S404. Figure 6 The obtaining unit 601 in the data processing apparatus is configured to perform the step S404.

[0142] According to another embodiment of the present application, Figure 6 The units in the data processing apparatus shown in the present application can be combined into one or several other units respectively or all, or some of the units can be further split into a plurality of units with smaller functions to form, which can achieve the same operation without affecting the technical effects of the embodiments of the present application. The above units are divided based on logical functions, and in actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, based on the information sharing apparatus, other units can also be included, and in actual applications, these functions can also be assisted by other units, and can be implemented by multiple units.

[0143] According to another embodiment of the present application, a computer program (including program codes) for performing the steps of the corresponding method shown in the present application can be run on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a random access memory (RAM), a read-only memory (ROM), etc. Figure 2 and Figure 3 The data processing apparatus shown in the present application is constructed by a computer program (including program codes) for performing the steps of the corresponding method shown in the present application, and the data processing method of the embodiments of the present application is implemented. The computer program can be recorded on a computer readable storage medium, loaded into the above computing device through the computer readable storage medium, and run in the computing device. Figure 6 The data processing apparatus shown in the present application is constructed by a computer program (including program codes) for performing the steps of the corresponding method shown in the present application, and the data processing method of the embodiments of the present application is implemented. The computer program can be recorded on a computer readable storage medium, loaded into the above computing device through the computer readable storage medium, and run in the computing device.

[0144] In the embodiments of the present application, when there is a cleaning event of the log data in the host database, the obtaining unit 601 obtains the file descriptor of the backup log data which has been backed up from the host database by the standby machine; further, the determining unit 602 determines a reference file descriptor according to the descriptor of the backup log data, and selects a target reference descriptor which is less than or equal to the reference file descriptor, and then the processing unit 603 performs virtual cleaning and asynchronous deletion processing on the target log data indicated by the target file descriptor based on the target file descriptor. In the above cleaning process of the log data, the target log data to be cleaned is determined by referring to the backup log data which has been backed up in the standby machine, so that the situation that the log data in the standby machine is lost due to the cleaning of the log in the host database can be avoided. Moreover, the target log data is deleted from the host database by using the virtual cleaning and asynchronous deletion mode, so that the disk jitter caused by synchronous deletion of the target log data is avoided.

[0145] Based on the method embodiments and the device embodiments described above, the embodiments of the present application further provide a data processing device, which can correspond to the host described above. Referring to Figure 7 A structural schematic diagram of a data processing device provided by the embodiments of the present application is shown. Figure 7 The data processing device shown can at least include a processor 701, an input interface 702, an output interface 703 and a computer storage medium 704. The processor 701, the input interface 702, the output interface 703 and the computer storage medium 704 can be connected through a bus or other means.

[0146] The computer storage medium 704 can be stored in the memory of the data processing device, and is used to store a computer program. The processor 801 is used to execute the computer program stored in the computer storage medium 704. The processor 701 (or CPU (Central Processing Unit, central processor)) is the computing core and control core of the data processing device, which is suitable for implementing one or more computer programs, and is particularly suitable for loading and executing:

[0147] When there is a cleaning event of the log data of the host database, a file descriptor of backup log data which is most recently backed up by the standby machine from the host is acquired, a plurality of log data are stored in the host, each log data corresponds to a file descriptor, the file descriptor corresponding to any log data is used to reflect the order in which the any log data is generated, the standby machine continuously backs up log data from the host according to the order in which each log data is generated; a reference file descriptor is determined according to the file descriptor of the backup log data, and a target file descriptor corresponding to target log data to be cleaned in the host database is acquired based on the reference file descriptor, the target file descriptor is less than or equal to the reference file descriptor; the target log data is virtually cleaned based on the target file descriptor, and the target log data is asynchronously deleted.

[0148] In the embodiment of the present application, when there is a cleaning event of log data in the host database, the file descriptor of the backup log data which has been backed up from the host database by the backup host is acquired; further, the reference file descriptor is determined according to the descriptor of the backup log data, and the target reference descriptor smaller than or equal to the reference file descriptor is selected, and then the target log data indicated by the target file descriptor is virtually cleaned and asynchronously deleted based on the target file descriptor. In the above cleaning process of the log data, the target log data to be cleaned is determined by referring to the backup log data which has been backed up by the backup host, so that the situation that the log data in the backup host is lost due to the cleaning of the log in the host database can be avoided. Moreover, the target log data is deleted from the host database by using the virtual cleaning and asynchronous deletion manner, so that the disk jitter caused by the synchronous deletion of the target log data can be avoided.

[0149] The embodiment of the present application further provides a computer storage medium (Memory), which is a memory device in a data processing device, and is used for storing programs and data. It can be understood that the computer storage medium herein can include the built-in storage medium of the data processing device, and of course can also include the extended storage medium supported by the data processing device. The computer storage medium provides a storage space, and the storage space stores the operating system of the data processing device. Moreover, one or more computer programs suitable for being loaded and executed by the processor 701 are also stored in the storage space. It should be noted that the computer storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory; and optionally can also be at least one computer storage medium located away from the aforementioned processor.

[0150] In one embodiment, the computer storage medium can be loaded and executed by the processor 701 to implement the corresponding steps of the data processing method shown in the above Figure 2 and Figure 4 In the specific implementation, the one or more computer programs in the computer storage medium are loaded and executed by the processor 701 to implement the following steps:

[0151] When there is a log data cleaning event for a host database, a file descriptor of backup log data that a backup machine last backed up from the host is obtained, the host stores a plurality of log data, each log data corresponds to a file descriptor, the file descriptor of any log data reflects the order in which the log data is generated, and the backup machine continuously backs up log data from the host according to the order in which each log data is generated; a reference file descriptor is determined according to the file descriptor of the backup log data, and a target file descriptor corresponding to target log data to be cleaned in the host database is obtained based on the reference file descriptor, the target file descriptor being less than or equal to the reference file descriptor; the target log data is virtually cleaned based on the target file descriptor, and the target log data is asynchronously deleted.

[0152] In one embodiment, when the processor 701 virtually cleans the target log data based on the target file descriptor, the following steps are performed:

[0153] A hard link creation process is performed for the target file descriptor to obtain a candidate file descriptor corresponding to the target file descriptor; a correspondence between the candidate file descriptor and the target log data is established, and the target file descriptor is deleted from the host.

[0154] In one embodiment, when the processor 701 virtually cleans the target log data based on the target file descriptor, the following steps are performed: a deleted flag is added to the target file descriptor.

[0155] In one embodiment, when the processor 701 asynchronously deletes the target log data, the following steps are performed:

[0156] The target log data is obtained from the disk of the host according to the candidate file descriptor or the target file descriptor with the added deleted flag; and the data content of the target log data is deleted in batches according to a target threshold.

[0157] In one embodiment, when the processor 701 deletes the data content of the target log data in batches according to a target threshold, the following steps are performed: the data content of the target log data is split according to the target threshold, and the size of the data content of each batch of data to be deleted obtained by the splitting is less than or equal to the target threshold; and the deletion is performed in sequence according to the splitting order of each batch.

[0158] In one embodiment, the log data cleaning event for the host is that the storage space of the disk is less than a storage threshold, and the host database is in a business low peak period; the processor 701 is further configured to:

[0159] determining a service type corresponding to the host database; obtaining a throughput threshold corresponding to the service type; and determining that the host database is in a service off-peak period if a throughput of the host database is less than the throughput threshold.

[0160] In one embodiment, the number of the standby machines is at least one, the backup log data comprises at least one backup log data which is most recently backed up from the host by each of the at least one standby machine, and the processor 701 performs the following steps when determining the reference file descriptor according to the file descriptors of the backup log data:

[0161] determining, based on the file descriptors of the backup log data, a first file descriptor corresponding to log data which has been completely backed up in the host, the log data which has been completely backed up being log data which is backed up in each of the at least one standby machine;

[0162] selecting, as the reference file descriptor, a file descriptor which is less than or equal to the first file descriptor, wherein the greater the file descriptor, the later the order of the corresponding log data being backed up in each of the at least one standby machine.

[0163] In one embodiment, if the number of the first file descriptors is at least one, the file descriptor which is less than or equal to the first file descriptor refers to a file descriptor which is less than or equal to a larger one of the at least one file descriptor.

[0164] In one embodiment, the at least one standby machine comprises a service slave machine of the host and a data disaster recovery standby machine, and if the backup log data comprises log data which is last backed up from the host database by the service slave machine, the processor 701 further performs the following steps after the asynchronous deletion of the target log data:

[0165] if a log data backup event of the data disaster recovery standby machine is detected and a file descriptor corresponding to log data to be backed up indicated by the log data backup event is less than the reference file descriptor, sending, to the data disaster recovery standby machine, notification information of backing up the log data to be backed up from the service slave machine; the notification information is used to notify the data disaster recovery standby machine that the log data to be backed up has been cleaned up from the host and instruct the data disaster recovery standby machine to acquire the log data to be backed up from the service slave machine.

[0166] In the embodiment of the present application, when there is a cleaning event for log data in the host database, a file descriptor of backup log data which has been backed up from the host database by the standby machine is acquired; further, a reference file descriptor is determined according to the descriptor of the backup log data, and a target reference descriptor smaller than or equal to the reference file descriptor is selected, and then the target log data indicated by the target file descriptor is virtually cleaned and asynchronously deleted based on the target file descriptor. In the above cleaning process of the log data, the target log data to be cleaned is determined by referring to the backup log data which has been backed up by the standby machine, so that the situation that the log data in the standby machine is lost due to the cleaning of the log data in the host database can be avoided. Moreover, the target log data is deleted from the host database by using the virtual cleaning and asynchronous deletion manner, so that the disk jitter caused by the synchronous deletion of the target log data can be avoided.

[0167] According to an aspect of the present application, the embodiment of the present application further provides a computer product or a computer program, the computer product comprises a computer program, and the computer program is stored in a computer readable storage medium. The processor 701 reads the computer program from the computer readable storage medium, and the processor 701 executes the computer program, so that the terminal executes the data processing method shown in the method of claim 1, and specifically: Figure 2 and Figure 4

[0168] When there is a cleaning event for log data in the host database, a file descriptor of backup log data which has been backed up from the host database by the standby machine is acquired; further, a reference file descriptor is determined according to the descriptor of the backup log data, and a target reference descriptor smaller than or equal to the reference file descriptor is selected, and then the target log data indicated by the target file descriptor is virtually cleaned and asynchronously deleted based on the target file descriptor. In the above cleaning process of the log data, the target log data to be cleaned is determined by referring to the backup log data which has been backed up by the standby machine, so that the situation that the log data in the standby machine is lost due to the cleaning of the log data in the host database can be avoided. Moreover, the target log data is deleted from the host database by using the virtual cleaning and asynchronous deletion manner, so that the disk jitter caused by the synchronous deletion of the target log data can be avoided.

[0169] ​In the embodiment of the present application, when there is a cleaning event of log data in the host database, the file descriptor of the backup log data which has been backed up from the host database by the standby machine is acquired; further, the reference file descriptor is determined according to the descriptor of the backup log data, and the target reference descriptor smaller than or equal to the reference file descriptor is selected, and then the target log data indicated by the target file descriptor is virtually cleaned and asynchronously deleted based on the target file descriptor. In the above cleaning process of the log data, the target log data to be cleaned is determined by referring to the backup log data which has been backed up in the standby machine, so that the situation that the log data in the standby machine is lost due to the log cleaning in the host database can be avoided. Moreover, the target log data is deleted from the host database by using the virtual cleaning and asynchronous deletion mode, so that the disk jitter caused by the synchronous deletion of the target log data is avoided.

Claims

1. A data processing method, characterized in that, include: When there is a cleanup event for the log data of the host database, the file descriptor of the backup log data most recently backed up by the standby machine from the host is obtained. The host stores multiple log data, each log data corresponds to a file descriptor, and the file descriptor corresponding to the log data is used to reflect the order in which the log data is generated. The standby machine continuously backs up log data from the host according to the order in which each log data is generated. A reference file descriptor is determined based on the file descriptor of the backup log data, and a target file descriptor corresponding to the target log data to be cleaned is obtained based on the reference file descriptor, wherein the target file descriptor is less than or equal to the reference file descriptor; The target log data is virtually cleaned up based on the target file descriptor, and the target log data is deleted asynchronously. The virtual cleanup of the target log data based on the target file descriptor includes: creating hard links for the target file descriptor to obtain candidate file descriptors corresponding to the target file descriptor; establishing a correspondence between the candidate file descriptors and the target log data; and deleting the target file descriptor from the host; or... The virtual cleanup of the target log data based on the target file descriptor includes: adding a deleted marker to the target file descriptor.

2. The method as described in claim 1, characterized in that, The asynchronous deletion process for the target log data includes: Based on the candidate file descriptor or the target file descriptor with the deletion mark added, obtain the target log data from the disk of the host; The target log data is deleted in batches according to the target threshold.

3. The method as described in claim 2, characterized in that, The step of deleting the data content of the target log data in batches according to the target threshold includes: The data content of the target log data is split according to the target threshold, and the size of the data content of each batch of data to be deleted is less than or equal to the target threshold. Delete them sequentially according to the splitting order of each batch.

4. The method as described in claim 1, characterized in that, The event of cleaning up log data in the host refers to: the disk storage space is less than the storage threshold, and the host database is in a low-traffic period; the method further includes: Determine the service type corresponding to the host database; Obtain the throughput threshold corresponding to the service type; If the throughput of the host database is less than the throughput threshold, then the host database is determined to be in a low-traffic period.

5. The method as described in claim 1, characterized in that, The number of standby machines is at least one, and the backup log data includes at least one backup log data from the most recent backup of the host machine by at least one standby machine. The step of determining a reference file descriptor based on the file descriptor of the backup log data includes: Based on the file descriptor of the backup log data, determine the first file descriptor corresponding to the log data that has been fully backed up in the host, wherein the fully backed-up log data refers to the log data that has been backed up in each of the at least one standby machine; A file descriptor less than or equal to the first file descriptor is selected as the reference file descriptor, wherein the log data corresponding to the larger file descriptor is backed up later by each of the at least one standby machine.

6. The method as described in claim 5, characterized in that, If the number of the first file descriptors is at least one, then a file descriptor less than or equal to the first file descriptor refers to a file descriptor less than or equal to the larger of at least one file descriptor.

7. The method as described in claim 5, characterized in that, The at least one backup machine includes the service slave machine of the host and the data disaster recovery backup machine. If the backup log data includes the log data last backed up by the service slave machine from the host database, then after the asynchronous deletion of the target log data, the method further includes: If a log data backup event of the data disaster recovery backup machine is detected, and the file descriptor corresponding to the log data to be backed up indicated by the log data backup event is smaller than the reference file descriptor, then a notification message is sent to the data disaster recovery backup machine to back up the log data to be backed up from the service slave machine. The notification information is used to inform the data disaster recovery backup machine that the log data to be backed up has been cleaned up from the host, and to instruct the data disaster recovery backup machine to obtain the log data to be backed up from the service slave machine.

8. A data processing apparatus, characterized in that, include: The acquisition unit is used to acquire the file descriptor of the backup log data that the standby machine has most recently backed up from the host when there is a cleanup event of log data in the host database. The host stores multiple log data, each log data corresponds to a file descriptor, and the file descriptor corresponding to the log data is used to reflect the order in which the log data is generated. The standby machine continuously backs up log data from the host according to the order in which each log data is generated. The determining unit is configured to determine a reference file descriptor based on the file descriptor of the backup log data; The acquisition unit is further configured to acquire, based on the reference file descriptor, the target file descriptor corresponding to the target log data to be cleaned in the host database, wherein the target file descriptor is less than or equal to the reference file descriptor; The processing unit is used to perform virtual cleanup of the target log data based on the target file descriptor, and to asynchronously delete the target log data; When the processing unit performs virtual cleanup on the target log data based on the target file descriptor, it executes the following steps: performing hard link creation processing on the target file descriptor to obtain candidate file descriptors corresponding to the target file descriptor; establishing a correspondence between the candidate file descriptors and the target log data; and deleting the target file descriptor from the host. Alternatively, perform the following steps: add a deleted marker to the target file descriptor.

9. A data processing device, characterized in that, include: A processor is used to implement one or more computer programs; as well as A computer storage medium storing one or more computer programs, said one or more computer programs being adapted to be loaded by said processor and executed as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, is used to perform the data processing method as described in any one of claims 1-7.

Images