Data cleaning method and device, computer device and storage medium

Abstract

The application relates to a data cleaning method and device, computer equipment, a storage medium and a computer program product, and relates to the technical field of cloud computing. The method comprises the following steps: determining a cleaning cache space corresponding to each to-be-cleaned data table according to a data type corresponding to each to-be-cleaned data table, and storing index information of each to-be-cleaned data table in the corresponding cleaning cache space; determining a target cleaning cache space from a plurality of cleaning cache spaces, deleting data in a to-be-cleaned data table corresponding to the index information in the database based on the index information in the target cleaning cache space, and determining the target cleaning cache space from the remaining cleaning cache spaces, jumping to the step of deleting data in a to-be-cleaned data table corresponding to the index information in the database based on the index information in the target cleaning cache space until there is no target cleaning cache space. The method can improve the data cleaning efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing, and in particular to a data cleaning method and device, a computer device, a storage medium and a computer program product. BACKGROUND

[0002] In a distributed use scenario, data grows explosively, and more and more data brings great challenges to a storage system. When the amount of data reaches a certain degree, the performance of writing and reading data will be seriously affected, and a read-write performance bottleneck will occur, which will further affect the system stability and user experience. Therefore, data cleaning needs to be performed on data that is not used or has low use value.

[0003] Data cleaning refers to deleting data from a persistent device (generally a database). A traditional data cleaning method is to directly connect a database to initiate a cleaning request.

[0004] However, in a massive data scenario, the traditional data cleaning method cannot clean all the data to be cleaned within an effective time, resulting in cleaning timeout, deletion failure, and affecting the stable operation of a distributed system. Therefore, the traditional data cleaning method has low cleaning efficiency. SUMMARY

[0005] Therefore, it is necessary to provide a data cleaning method, device, computer device, computer readable storage medium and computer program product capable of improving cleaning efficiency to solve the above technical problems.

[0006] In a first aspect, the present application provides a data cleaning method. The method comprises:

[0007] When a preset cleaning condition is reached, classifying the data tables to be cleaned according to a preset classification standard to obtain data types corresponding to each of the data tables to be cleaned;

[0008] According to the data types corresponding to each of the data tables to be cleaned, determining a cleaning cache space corresponding to each of the data tables to be cleaned, and storing index information of each of the data tables to be cleaned in the corresponding cleaning cache space;

[0009] Determining a target cleaning cache space from the plurality of cleaning cache spaces, deleting data in a data table to be cleaned corresponding to index information in the target cleaning cache space based on the index information, and determining a target cleaning cache space from the remaining cleaning cache spaces, jumping to the step of deleting data in a data table to be cleaned corresponding to index information in the target cleaning cache space based on the index information until there is no target cleaning cache space.

[0010] In one of the embodiments, the classification criteria is data volume, the data types include large data tables, medium data tables and small data tables; the determining of the cleaning cache space corresponding to each of the to-be-cleaned data tables according to the data type corresponding to each of the to-be-cleaned data tables includes:

[0011] For each to-be-cleaned data table of the data type of large data table, the cleaning cache space corresponding to the to-be-cleaned data table is determined as a separate cleaning cache space;

[0012] In the case that the data table quantity of the to-be-cleaned data table of the data type of medium data table exceeds a preset quantity threshold, the space quantity of the cleaning cache space of the to-be-cleaned data of the data type of medium data table is calculated according to the data table quantity and the quantity threshold;

[0013] According to a preset allocation rule, the to-be-cleaned data table of the data type of medium data table is allocated to the space quantity of cleaning cache spaces;

[0014] The to-be-cleaned data table of the data type of small data table is allocated to a separate cleaning cache space.

[0015] In one of the embodiments, the determining of the target cleaning cache space from the plurality of cleaning cache spaces includes:

[0016] For each cleaning cache space, the highest cleaning level in the cleaning levels of the to-be-cleaned data tables corresponding to the cleaning cache space is taken as the cleaning level of the cleaning cache space;

[0017] Based on the cleaning levels of the cleaning cache spaces, the target cleaning cache space is determined from the plurality of cleaning cache spaces.

[0018] In one of the embodiments, the deleting of the data in the to-be-cleaned data table corresponding to the index information in the database based on the index information in the target cleaning cache space includes:

[0019] A data cleaning time range is obtained;

[0020] According to the data cleaning time range, target index information of to-be-cleaned data in the to-be-cleaned data table is filtered from the index information in the target cleaning cache space;

[0021] The to-be-cleaned data corresponding to the target index information in the database is deleted.

[0022] In one of the embodiments, the method is applied to a distributed system, the distributed system includes a plurality of databases; the deleting of the data in the to-be-cleaned data table corresponding to the target index information in the database includes:

[0023] resolving an address of a database corresponding to the target index information;

[0024] If the database corresponding to the target index information is of different database types, then for each database type of the database corresponding to the target index information, the address of the database of the database type, the data table name of the data table to be cleaned, and the data cleaning time range are used to form cleaning information of the database type.

[0025] The cleaning information of each database type is respectively sent to the database of each database type; the cleaning information is used to instruct the database to perform deletion processing on the data to be cleaned.

[0026] In one embodiment, the deletion processing on the data in the data table to be cleaned corresponding to the index information in the database includes:

[0027] A target database group is obtained, and it is determined whether the database corresponding to the index information belongs to the target database group.

[0028] If the database corresponding to the index information belongs to the target database group, then the deletion processing on the data in the data table to be cleaned corresponding to the index information in the database is performed.

[0029] In one embodiment, the method further includes:

[0030] The alive detection request is sent according to a preset alive detection frequency, and node information corresponding to the alive detection request is received; the node information includes node state information.

[0031] If the node information indicates that there is a faulty node, then the cleaning information is stopped from being sent to the faulty node.

[0032] If the node information indicates that there is a new node, then the new node is added to the target database group.

[0033] In one embodiment, the method further includes:

[0034] If the cleaning of the data table to be cleaned fails, then when a preset retry time is reached, the data to be cleaned is cleaned again.

[0035] In a second aspect, the application also provides a data cleaning device. The device includes:

[0036] A classification module is configured to classify the data table to be cleaned according to a preset classification standard when a preset cleaning condition is reached, so as to obtain a data type corresponding to each data table to be cleaned.

[0037] determining, according to a data type corresponding to each of the to-be-cleaned data tables, a cleaning cache space corresponding to each of the to-be-cleaned data tables, and storing index information of each of the to-be-cleaned data tables in the corresponding cleaning cache space;

[0038] cleaning, from the plurality of cleaning cache spaces, a target cleaning cache space, and based on index information in the target cleaning cache space, performing deletion processing on data in a to-be-cleaned data table corresponding to the index information in the database, and jumping to the step of performing deletion processing on data in a to-be-cleaned data table corresponding to the index information in the database based on index information in a target cleaning cache space, until there is no target cleaning cache space.

[0039] In one of the embodiments, the classification standard is data volume, and the data types include large data tables, medium data tables, and small data tables; and the determining module is specifically configured to:

[0040] For each to-be-cleaned data table of the large data table type, the cleaning cache space corresponding to the to-be-cleaned data table is determined as a separate cleaning cache space;

[0041] In a case where the number of data tables of the to-be-cleaned data tables of the medium data table type exceeds a preset number threshold, the number of cleaning cache spaces for the to-be-cleaned data of the medium data table type is calculated according to the number of data tables and the number threshold;

[0042] According to a preset allocation rule, the to-be-cleaned data tables of the medium data table type are allocated to the number of cleaning cache spaces;

[0043] The to-be-cleaned data tables of the small data table type are allocated to a separate cleaning cache space.

[0044] In one of the embodiments, the cleaning module is specifically configured to:

[0045] For each cleaning cache space, the highest cleaning level in the cleaning levels of the to-be-cleaned data tables corresponding to the cleaning cache space is taken as the cleaning level of the cleaning cache space;

[0046] Based on the cleaning levels of the cleaning cache spaces, a target cleaning cache space is determined from the plurality of cleaning cache spaces.

[0047] In one of the embodiments, the cleaning module is specifically configured to:

[0048] obtaining a data cleaning time range;

[0049] According to the data cleaning time range, target index information of the to-be-cleaned data in the target cleaning cache space is screened from index information in the target cleaning cache space.

[0050] The to-be-cleaned data corresponding to the target index information in the database is deleted.

[0051] In one embodiment, the apparatus is applied to a distributed system including a plurality of databases; and the cleaning module is specifically configured to:

[0052] An address of a database corresponding to the target index information is parsed.

[0053] If the database corresponding to the target index information is of different database types, for each database type of the database corresponding to the target index information, the address of the database of the database type, the data table name of the to-be-cleaned data table, and the data cleaning time range are combined to form cleaning information of the database type.

[0054] The cleaning information of each database type is respectively sent to the database of each database type; and the cleaning information is used to instruct the database to delete the to-be-cleaned data.

[0055] In one embodiment, the cleaning module is specifically configured to:

[0056] A target database group is obtained, and it is determined whether the database corresponding to the index information belongs to the target database group.

[0057] If the database corresponding to the index information belongs to the target database group, data in the to-be-cleaned data table corresponding to the index information in the database is deleted.

[0058] In one embodiment, the apparatus further includes:

[0059] The transceiving module is configured to send a live detection request according to a preset sniffing frequency, and receive node information corresponding to the live detection request; the node information includes node state information.

[0060] The stop sending module is configured to stop sending cleaning information to a faulty node if the node information indicates that there is the faulty node.

[0061] The adding module is configured to add a new node to the target database group if the node information indicates that there is the new node.

[0062] In one embodiment, the apparatus further includes:

[0063] The retry module is configured to, if the cleaning of the to-be-cleaned data table fails, re-clean the to-be-cleaned data when a preset retry time is reached.

[0064] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the first aspect when executing the computer program.

[0065] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the first aspect.

[0066] In a fifth aspect, the present application further provides a computer program product. The computer program product comprises a computer program, and the computer program is executed by a processor to implement the steps of the first aspect.

[0067] The data cleaning method, device, computer device, storage medium and computer program product described above, when a preset cleaning condition is reached, classify the to-be-cleaned data tables according to a preset classification standard to obtain the data types corresponding to the to-be-cleaned data tables; determine the cleaning cache spaces corresponding to the to-be-cleaned data tables according to the data types corresponding to the to-be-cleaned data tables, and store the index information of the to-be-cleaned data tables in the corresponding cleaning cache spaces; determine a target cleaning cache space from the plurality of cleaning cache spaces, delete the data in the to-be-cleaned data table corresponding to the index information in the database based on the index information in the target cleaning cache space, and determine a target cleaning cache space from the remaining cleaning cache spaces, jump to the step of deleting the data in the to-be-cleaned data table corresponding to the index information in the database based on the index information in the target cleaning cache space until there is no target cleaning cache space. In this way, the to-be-cleaned data tables are classified, and the index information of to-be-cleaned data tables of different types is divided into different cleaning cache spaces, and then the to-be-cleaned data tables corresponding to different cleaning cache spaces are cleaned at different times, realizing the peak-shaving cleaning of to-be-cleaned data tables corresponding to different cleaning cache spaces, which can avoid the timeout of the cleaning of large amounts of to-be-cleaned data tables and the failure of data cleaning caused by the occupation of resources by other threads when large amounts of to-be-cleaned data tables are cleaned at the same time, so that in the scenario of massive data, the to-be-cleaned data can be cleaned within an effective time, and the cleaning efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 A flowchart of a data cleaning method in an embodiment;

[0069] Figure 2 a flowchart of a process for determining a cleaning cache space corresponding to each to-be-cleaned data table in the database in one embodiment;

[0070] Figure 3 a flowchart of a process for determining a target cleaning cache space from a plurality of cleaning cache spaces in one embodiment;

[0071] Figure 4 a flowchart of a process for deleting data in a to-be-cleaned data table corresponding to index information in the database in one embodiment;

[0072] Figure 5 a flowchart of a process for deleting data in a to-be-cleaned data table corresponding to target index information in the database in one embodiment;

[0073] Figure 6 a flowchart of a process for deleting data in a to-be-cleaned data table corresponding to index information in the database in another embodiment;

[0074] Figure 7 a flowchart of a process for a data cleaning method in another embodiment;

[0075] Figure 8 a block diagram of a structure of a data cleaning apparatus in one embodiment;

[0076] Figure 9 a block diagram of an internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0077] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0078] In one embodiment, as shown in FIG. 1, a data cleaning apparatus 100 includes a data cleaning module 110 and a database 120. Figure 1As shown, a data cleaning method is provided. The embodiment is exemplified by the method applied to a terminal. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction of the terminal and the server. The method can be applied to a distributed system. The distributed system can be implemented through a cloud computing platform. For example, the distributed system can be implemented through Platform as a Service (PaaS). The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server can be implemented by an independent server or a server cluster composed of multiple servers. In the embodiment, the method includes the following steps:

[0079] In step 101, when a preset cleaning condition is reached, the terminal classifies the to-be-cleaned data table according to a preset classification standard, to obtain a data type corresponding to each to-be-cleaned data table.

[0080] In the embodiment of the application, the to-be-cleaned data table is a data table in which to-be-cleaned data is located. The to-be-cleaned data table can be a single table or a set. The to-be-cleaned data can be data that is not used or data that has low use value. The classification standard can be various classification standards that can classify data, which are not limited in the application. For example, the classification standard can be data quantity size and data content identification. In the case where the classification standard is data content identification, the data type can include index type, system type, and link type.

[0081] When the preset cleaning condition is reached, the terminal classifies the to-be-cleaned data table according to the preset classification standard, to obtain a data type corresponding to each to-be-cleaned data table.

[0082] In one example, when a preset cleaning time is reached, the terminal determines that the preset cleaning condition is reached. The cleaning time can be a time when system resources are idle. For example, the cleaning time can be a night time. The cleaning time can be a certain fixed time in a period. For example, the cleaning time can be 3 a.m. every day.

[0083] In one example, when a cleaning request is received, the terminal determines that the preset cleaning condition is reached.

[0084] In one example, the terminal can update the to-be-cleaned data table in real time through a dynamic configuration device.

[0085] In step 102, according to the data type corresponding to each to-be-cleaned data table, a terminal determines a cleaning cache space corresponding to each to-be-cleaned data table, and stores index information of each to-be-cleaned data table in the corresponding cleaning cache space.

[0086] In the embodiment of the present application, the terminal determines the cleaning cache space corresponding to each to-be-cleaned data table according to the data type corresponding to each to-be-cleaned data table. Then, the terminal stores the index information of each to-be-cleaned data table in the corresponding cleaning cache space. The cleaning cache space is used to store the index information of the to-be-cleaned data table.

[0087] In one example, after the classification of the to-be-cleaned data table is completed, the terminal can perform compliance verification on the format of the to-be-cleaned data table to obtain a verification result of the to-be-cleaned data table. Specifically, the terminal can check whether the table name of the to-be-cleaned data table conforms to the naming specification. If the verification result of the to-be-cleaned data table indicates that the verification is passed, the terminal determines the cleaning cache space corresponding to the to-be-cleaned data table according to the data type corresponding to the to-be-cleaned data table. If the verification result of the to-be-cleaned data table indicates that the verification is not passed, the terminal records the to-be-cleaned data table to an alarm log, or the terminal generates alarm information based on the to-be-cleaned data table and sends the alarm information.

[0088] In step 103, a target cleaning cache space is determined from the plurality of cleaning cache spaces, data in a to-be-cleaned data table corresponding to index information in a database is deleted based on the index information in the target cleaning cache space, and a target cleaning cache space is determined from the remaining cleaning cache spaces, and the step of deleting data in a to-be-cleaned data table corresponding to index information in a database based on the index information in the target cleaning cache space is jumped to until there is no target cleaning cache space.

[0089] In the embodiment of the present application, the terminal determines the target cleaning cache space from the plurality of cleaning cache spaces. Then, the terminal deletes data in a to-be-cleaned data table corresponding to index information in a database based on the index information in the target cleaning cache space. Then, the terminal determines the target cleaning cache space from the remaining cleaning cache spaces. Then, the terminal jumps to the step of deleting data in a to-be-cleaned data table corresponding to index information in a database based on the index information in the target cleaning cache space until there is no target cleaning cache space.

[0090] In one example, the terminal randomly determines the target cleaning cache space from the plurality of cleaning cache spaces.

[0091] In one example, the terminal obtains the cleaning levels of each to-be-cleaned data table corresponding to the index information in the target cleaning cache space. Then, the terminal queries the cleaning order of each to-be-cleaned data table according to the cleaning level of each to-be-cleaned data table in the mapping relationship between the preset cleaning level and the cleaning order. Then, the terminal deletes the data in the to-be-cleaned data table corresponding to the index information in the database according to the cleaning order of each to-be-cleaned data table.

[0092] In the above data cleaning method, when the preset cleaning condition is reached, the to-be-cleaned data tables are classified according to the preset classification standard to obtain the data types corresponding to each to-be-cleaned data table; the cleaning cache spaces corresponding to each to-be-cleaned data table are determined according to the data types corresponding to each to-be-cleaned data table, and the index information of each to-be-cleaned data table is stored in the corresponding cleaning cache space; the target cleaning cache space is determined from the plurality of cleaning cache spaces, the data in the to-be-cleaned data table corresponding to the index information in the database is deleted based on the index information in the target cleaning cache space, and the target cleaning cache space is determined from the remaining cleaning cache spaces, and the step of deleting the data in the to-be-cleaned data table corresponding to the index information in the database based on the index information in the target cleaning cache space is jumped to until there is no target cleaning cache space. In this way, the to-be-cleaned data tables are classified, and the index information of to-be-cleaned data tables of different types is divided into different cleaning cache spaces, and the to-be-cleaned data tables corresponding to different cleaning cache spaces are cleaned at different times, realizing peak-shaving cleaning of to-be-cleaned data tables corresponding to different cleaning cache spaces, which can avoid large amount of to-be-cleaned data table cleaning timeout caused by other threads occupying resources while large amount of to-be-cleaned data tables are cleaned, and data cleaning failure, so that in the scenario of massive data, the to-be-cleaned data can be cleaned in an effective time, and the cleaning efficiency is improved. Moreover, the index information of to-be-cleaned data tables of different types is divided into different cleaning cache spaces, the cleaning cache space is reduced, the index tree in a single cleaning cache space is simple, the index tree updating time is shortened in the process of data cleaning, and the cleaning efficiency can be further improved.

[0093] In one embodiment, the classification standard is data volume, and the data types include large data tables, medium data tables and small data tables. As shown in FIG. 2, the specific process of determining the cleaning cache space corresponding to each to-be-cleaned data table according to the data type corresponding to each to-be-cleaned data table includes the following steps: Figure 2

[0094] Step 201, for each to-be-cleaned data table of the data type of large data table, determining that the cleaning cache space corresponding to the to-be-cleaned data table is a separate cleaning cache space.

[0095] ​In the embodiments of the present application, if the data amount of the to-be-cleaned data table is greater than the preset first data amount threshold, the terminal determines that the data type of the to-be-cleaned data table is a large data table. If the data amount of the to-be-cleaned data table is less than the preset second data amount threshold, the terminal determines that the data type of the to-be-cleaned data table is a small data table. If the data amount of the to-be-cleaned data table is greater than the preset second data amount threshold and less than the preset first data amount threshold, the terminal determines that the data type of the to-be-cleaned data table is a medium data table. The first data amount threshold is greater than the second data amount threshold. The first data amount threshold and the second data amount threshold can be different orders of magnitude. For example, the first data amount threshold can be 1T, and the second data amount threshold can be 1G.

[0096] For each to-be-cleaned data table of the data type of a large data table, the terminal determines that the cleaning cache space corresponding to the to-be-cleaned data table is a separate cleaning cache space. Specifically, the terminal allocates different cleaning cache spaces for each to-be-cleaned data table of the data type of a large data table, and the cleaning cache space only stores the index information of one to-be-cleaned data table.

[0097] Step 202, in the case that the data table number of the to-be-cleaned data table of the data type of a medium data table exceeds the preset number threshold, the space number of the cleaning cache space of the to-be-cleaned data of the data type of a medium data table is calculated according to the data table number and the number threshold.

[0098] In the embodiments of the present application, the terminal presets the number threshold. Then, the terminal determines whether the data table number of the to-be-cleaned data table of the data type of a medium data table exceeds the preset number threshold. In the case that the data table number of the to-be-cleaned data table of the data type of a medium data table exceeds the preset number threshold, the terminal calculates the space number of the cleaning cache space of the to-be-cleaned data of the data type of a medium data table according to the data table number and the number threshold. Specifically, the terminal divides the data table number by the number threshold, and the quotient obtained is taken as the space number of the cleaning cache space of the to-be-cleaned data of the data type of a medium data table. The number threshold can be the maximum value of the number of to-be-cleaned data tables corresponding to the index information stored in one cleaning cache space in the case of the data type of a medium data table. For example, the number threshold can be 5.

[0099] Step 203, according to the preset allocation rule, the to-be-cleaned data table of the data type of a medium data table is allocated to the cleaning cache space of the space number.

[0100] In the embodiment of the present application, the terminal assigns the to-be-cleaned data table of the medium data table type to the space number of cleaning cache spaces according to the preset assignment rule. Specifically, the terminal takes the quantity threshold as the maximum value of the number of to-be-cleaned data tables corresponding to the index information stored in one cleaning cache space. Then, the terminal assigns the to-be-cleaned data table of the medium data table type to the space number of cleaning cache spaces according to the maximum value.

[0101] In one example, the terminal determines a quantity threshold number of target to-be-cleaned data tables from the to-be-cleaned data tables of the medium data table type. Then, the terminal assigns the quantity threshold number of target to-be-cleaned data tables to a separate cleaning cache space. Then, the terminal determines a quantity threshold number of target to-be-cleaned data tables from the to-be-cleaned data tables of the medium data table type. Then, the terminal jumps to the step of assigning the quantity threshold number of target to-be-cleaned data tables to a separate cleaning cache space until there is no target to-be-cleaned data table.

[0102] Step 204, assigning the to-be-cleaned data table of the small data table type to one separate cleaning cache space.

[0103] In the embodiment of the present application, the terminal assigns the to-be-cleaned data table of the small data table type to one separate cleaning cache space.

[0104] In the data cleaning method, the to-be-cleaned data tables are classified according to data volume, different separate cleaning cache spaces are allocated to each to-be-cleaned data table of the large data table type, different separate cleaning cache spaces are allocated to each quantity threshold to-be-cleaned data table of the medium data table type, and each to-be-cleaned data table of the small data table type is allocated to one cleaning cache space. In this way, different cleaning cache space allocation manners are adopted for to-be-cleaned data tables of different data volumes, the rationality of cleaning cache space allocation is improved, and the cleaning efficiency is further improved. Moreover, different separate cleaning cache spaces are allocated to each to-be-cleaned data table of the large data table type, each to-be-cleaned data table of the large data volume is cleaned in a separate time, the time of cleaning the to-be-cleaned data table is staggered with the time of cleaning other to-be-cleaned data tables, the cleaning timeout of the to-be-cleaned data table of the large data volume caused by the occupation of resources by other threads during the cleaning of the to-be-cleaned data table of the large data volume is further avoided, data cleaning failure is further avoided, the to-be-cleaned data is cleaned in an effective time in the scenario of massive data, and the cleaning efficiency is further improved. Moreover, different separate cleaning cache spaces are allocated to each quantity threshold to-be-cleaned data table of the medium data table type, and each to-be-cleaned data table of the small data table type is allocated to one cleaning cache space, which not only prevents the increase of data cleaning time caused by too many cleaning cache spaces, but also prevents the increase of data cleaning time and cleaning timeout caused by too large cleaning cache spaces, and the cleaning efficiency is further improved.

[0105] In one embodiment, as shown in FIG. 6, the specific process of determining the target cleaning cache space from the plurality of cleaning cache spaces includes the following steps: Figure 3

[0106] Step 301, for each cleaning cache space, the highest cleaning level of the cleaning levels of the to-be-cleaned data tables corresponding to the cleaning cache space is taken as the cleaning level of the cleaning cache space.

[0107] In the embodiment of the present application, for each cleaning cache space, the terminal obtains the cleaning levels of the to-be-cleaned data tables corresponding to the cleaning cache space. Then, the terminal compares the cleaning levels of the to-be-cleaned data tables. Then, the terminal takes the highest cleaning level of the cleaning levels of the to-be-cleaned data tables corresponding to the cleaning cache space as the cleaning level of the cleaning cache space. The cleaning level can be used to represent the priority of cleaning. For example, the cleaning level is 1-5 grades, and the larger the grade value is, the lower the cleaning level is.

[0108] In one example, the terminal can update the cleaning level of the to-be-cleaned data table in real time through a dynamic configuration device.

[0109] ​At step 302, the target cleaning cache space is determined from the plurality of cleaning cache spaces based on the cleaning levels of the respective cleaning cache spaces.

[0110] In the embodiments of the present application, the terminal determines the target cleaning cache space from the plurality of cleaning cache spaces according to the cleaning levels of the respective cleaning cache spaces. Specifically, the terminal compares the cleaning levels of the respective cleaning cache spaces. Then, the terminal takes the cleaning cache space with the largest cleaning level as the target cleaning cache space.

[0111] In the above data cleaning method, for each cleaning cache space, the highest cleaning level in the cleaning levels of the respective to-be-cleaned data tables corresponding to the cleaning cache space is taken as the cleaning level of the cleaning cache space; and the target cleaning cache space is determined from the plurality of cleaning cache spaces based on the cleaning levels of the respective cleaning cache spaces. In this way, the highest cleaning level of the to-be-cleaned data tables corresponding to the cleaning cache space is taken as the cleaning level of the cleaning cache space, and the cleaning order of the cleaning cache space is determined according to the cleaning level of the cleaning cache space, thereby realizing the cleaning of the data in the to-be-cleaned data tables corresponding to the cleaning cache space, improving the accuracy of the cleaning order of the cleaning cache space, and cleaning the data that needs to be cleaned more in time, and improving the rationality of data cleaning.

[0112] In one embodiment, as shown in FIG. 4, the specific process of deleting the data in the to-be-cleaned data table corresponding to the index information in the database based on the index information in the target cleaning cache space includes the following steps. Figure 4

[0113] At step 401, the data cleaning time range is obtained.

[0114] In the embodiments of the present application, the terminal obtains the data cleaning time range of each to-be-cleaned data table. The data cleaning time range is used to indicate the time range in which the data needs to be cleaned. The data cleaning time ranges of the respective to-be-cleaned data tables can be the same or different.

[0115] In one example, the terminal can obtain the default configuration from the configuration file to obtain the data cleaning time range.

[0116] In one example, the terminal can obtain the real-time adjustment value from the dynamic configuration device. If the real-time adjustment value is empty or the dynamic configuration is abnormal, the terminal uses the default configuration obtained from the configuration file to obtain the data cleaning time range. In this way, the double obtaining mechanism is used to obtain the data cleaning time range, ensuring the validity of the cleaning date.

[0117] At step 402, the target index information of the to-be-cleaned data in the to-be-cleaned data table is filtered in the index information in the target cleaning cache space according to the data cleaning time range. ​

[0118] In the embodiment of the present application, the terminal filters the target index information of the to-be-cleaned data in the to-be-cleaned data table from the index information in the target cleaning cache space according to the data cleaning time range. The target index information is the index information of the to-be-cleaned data within the cleaning time range. Specifically, the terminal takes the index information of the to-be-cleaned data within the cleaning time range as the target index information of the to-be-cleaned data in the to-be-cleaned data table from the index information in the target cleaning cache space.

[0119] In step 403, the to-be-cleaned data corresponding to the target index information in the database is deleted.

[0120] In the embodiment of the present application, the terminal deletes the to-be-cleaned data corresponding to the target index information in the database.

[0121] In one example, the terminal stores the target index information obtained by filtering into a temporary cache list. Then, the terminal obtains the target index information in the temporary cache list. Next, the terminal deletes the to-be-cleaned data corresponding to the target index information in the database.

[0122] In the above data cleaning method, the data cleaning time range is obtained; the target index information of the to-be-cleaned data in the to-be-cleaned data table is filtered from the index information in the target cleaning cache space according to the data cleaning time range; and the to-be-cleaned data corresponding to the target index information in the database is deleted. In this way, the target index information of the to-be-cleaned data within the data cleaning time range in the to-be-cleaned data table is filtered from the index information in the target cleaning cache space, and only the to-be-cleaned data corresponding to the target index information in the database is deleted, that is, only the to-be-cleaned data within the specified time range is cleaned, avoiding the data outside the specified time range from being mistakenly cleaned, and improving the accuracy of data cleaning.

[0123] In one embodiment, the data cleaning method is applied to a distributed system including a plurality of databases. As shown in Figure 5 The specific process of deleting the data in the to-be-cleaned data table corresponding to the target index information in the database includes the following steps:

[0124] In step 501, the address of the database corresponding to the target index information is parsed.

[0125] In the embodiment of the present application, the terminal parses the address of the database corresponding to the target index information. The address of the database can include the IP and port of the database. The address of the database can be the address of the client node of the database.

[0126] In step 502, if the database corresponding to the target index information is of different database types, the terminal constructs, for each database type of the database corresponding to the target index information, the address of the database of the database type, the data table name of the data table to be cleaned, and the data cleaning time range, to form the cleaning information of the database type.

[0127] In the embodiment of the present application, if the database corresponding to the target index information is of different database types, the terminal constructs, for each database type of the database corresponding to the target index information, the address of the database of the database type, the data table name of the data table to be cleaned, and the data cleaning time range, to form the cleaning information of the database type.

[0128] In one embodiment, the cleaning information is a cleaning link. The format of the cleaning link is: protocol: / / ip:port.data table name.data cleaning time range, such as http: / / 127.0.01:8080.data-to-clean.2022-08-08.

[0129] In step 503, the cleaning information of each database type is sent to the database of each database type, respectively.

[0130] The cleaning information is used to instruct the database to perform deletion processing on the data to be cleaned.

[0131] In the embodiment of the present application, the terminal sends the cleaning information of each database type to the database of each database type, respectively.

[0132] In one example, the terminal sends the cleaning information of each database type to the client of the database of each database type, respectively. Then, the client of each database forwards the cleaning information to the data end of the database. Then, the data end of the database completes the cleaning request. For example, the data end of the database can execute the actual cleaning action by using open source HttpDelete.

[0133] In the data cleaning method, the data cleaning method is applied to a distributed system, the distributed system includes a plurality of databases, and an address of a database corresponding to target index information is parsed; if the database corresponding to the target index information is of different database types, for each database type of the database corresponding to the target index information, the address of the database of the database type, the data table name of the data table to be cleaned, and the data cleaning time range are combined to form cleaning information of the database type; and the cleaning information of each database type is sent to the database of each database type. In this way, in the case that the data in the databases of a plurality of database types in a distributed system all need to be cleaned, the cleaning information of each database type formed by the parsed address of the database, the data table name of the data table to be cleaned, and the data cleaning time range is sent to the database of each database type, the concurrent processing of data cleaning is implemented by multiplexing databases of different types, and the efficiency of data cleaning is further improved.

[0134] In one embodiment, as shown in FIG. 6, the specific process of deleting data in the data table to be cleaned corresponding to the index information in the database includes the following steps. Figure 6

[0135] Step 601: Obtain a target database group, and determine whether the database corresponding to the index information belongs to the target database group.

[0136] In the embodiment of the present application, the terminal obtains a target database group. Then, the terminal determines whether the database corresponding to the index information belongs to the target database group. The target database group is a group of databases that are specified to need to be cleaned. The target database group can be one or more databases, or one or more groups of databases. The default value of the target database group is all databases.

[0137] In one example, the terminal receives target database group information. The target database group information is used to indicate the target database group.

[0138] Step 602: If the database corresponding to the index information belongs to the target database group, delete data in the data table to be cleaned corresponding to the index information in the database.

[0139] In the embodiment of the present application, if the database corresponding to the index information belongs to the target database group, the terminal deletes data in the data table to be cleaned corresponding to the index information in the database. If the database corresponding to the index information does not belong to the target database group, the terminal deletes the index information from the cleaning cache space.

[0140] ​In the data cleaning method, the target database group is acquired, and it is judged whether the database corresponding to the index information belongs to the target database group; if the database corresponding to the index information belongs to the target database group, the data in the data table to be cleaned corresponding to the index information in the database is deleted. In this way, before the data to be cleaned in the database is deleted, it is judged whether the database corresponding to the index information belongs to the target database group, and only the data in the database belonging to the target database group is deleted, so that the target database group needing data cleaning is specified, data cleaning caused by incorrect information in the data cleaning process is prevented, the accuracy of data cleaning is further improved, and the stable operation of the cleaned data system is ensured.

[0141] In one embodiment, as shown in Figure 7 The data cleaning method further includes the following steps:

[0142] In step 701, a live detection request is sent according to a preset sniffing frequency, and node information corresponding to the live detection request is received.

[0143] The node information includes node state information.

[0144] In the embodiment of the application, the terminal sends a live detection request according to a preset sniffing frequency. At the same time, the terminal receives node information corresponding to the live detection request. The sniffing frequency is the frequency of sending the live detection request. The live detection request is used to sniff the situation of the nodes in the distributed system, and at least includes the number of nodes and the running state of the nodes.

[0145] In step 702, if the node information indicates that there is a faulty node, the terminal stops sending cleaning information to the faulty node.

[0146] In the embodiment of the application, if the node information indicates that there is a faulty node, the terminal stops sending cleaning information to the faulty node. In one example, the terminal generates node fault alarm information according to the faulty node. Then, the terminal sends the node fault alarm information to the user. In this way, the database node abnormal alarm information is triggered to the user in time, so that the user can perceive the fault in the first time and handle the fault in time, thereby ensuring the stable operation of the system and the efficiency of data cleaning.

[0147] In step 703, if the node information indicates that there is a new node, the new node is added to the target database group.

[0148] In the embodiment of the application, the terminal compares the node information received this time with the node information received last time to obtain a comparison result. If the comparison result indicates that there is a new node, the terminal adds the new node to the target database group.

[0149] In the data cleaning method, the alive detection request is sent according to the preset sniffing frequency, and the node information corresponding to the alive detection request is received; if the node information indicates that there is a faulty node, the cleaning information is stopped from being sent to the faulty node; and if the node information indicates that there is an added node, the added node is added to the target database group. In this way, once the system is started, the alive detection request is sent according to the sniffing frequency throughout the life cycle, and the cleaning information is stopped from being sent to the faulty node in a timely manner according to the received node information, so that in the data cleaning scene of the group-level database in which multiple databases work together, the situation that the data cannot be cleaned due to the failure of a certain database and the waste of system resources caused by the attempt to connect the database is avoided, and the stability and efficiency of data cleaning are further improved. In addition, the added node is added to the target database group in a timely manner, so that the node added after expansion can be automatically included in the cleaning range without stopping, for example, if there are originally 9 nodes and 3 nodes are temporarily added due to promotion, the 3 added nodes are automatically included in the group to be cleaned by the method, so that real-time cleaning without stopping and zero operation is realized, and the efficiency of data cleaning is further improved.

[0150] In one embodiment, the data cleaning method further includes the following steps: if the cleaning of the to-be-cleaned data table fails, the to-be-cleaned data is cleaned again when the preset retry time is reached.

[0151] In the embodiments of the present application, if the cleaning of the to-be-cleaned data table fails, the terminal cleans the to-be-cleaned data again when the preset retry time is reached. The retry time is the time interval from the failure of the to-be-cleaned data table to the cleaning again.

[0152] In one example, while the terminal cleans the to-be-cleaned data again, the terminal updates the number of repeated cleanings of the to-be-cleaned data. If the number of repeated cleanings of the to-be-cleaned data is greater than the preset cleaning number threshold, the terminal sends a cleaning link to the user. The user can clean the data in the to-be-cleaned data table through the cleaning link.

[0153] In the data cleaning method, if the cleaning of the to-be-cleaned data table fails, the to-be-cleaned data is cleaned again when the preset retry time is reached. In this way, only when it is detected that there is cleaning failure, the retry mechanism is triggered, and the to-be-cleaned data is cleaned again when the preset retry time is reached, so that the reliability of data cleaning is improved while the waste of resources is avoided.

[0154] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0155] Based on the same inventive concept, the embodiments of the present application also provide a data cleaning device for implementing the above-mentioned data cleaning method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more data cleaning device embodiments provided below can refer to the limitations of the data cleaning method described above, which will not be repeated here.

[0156] In one embodiment, as shown in Figure 8 A data cleaning device 800 is provided, comprising a classification module 810, a determination module 820 and a cleaning module 830, wherein:

[0157] The classification module 810 is configured to classify the to-be-cleaned data tables according to a preset classification standard when a preset cleaning condition is reached, to obtain data types corresponding to each of the to-be-cleaned data tables.

[0158] The determination module 820 is configured to determine a cleaning cache space corresponding to each of the to-be-cleaned data tables according to the data types corresponding to each of the to-be-cleaned data tables, and store index information of each of the to-be-cleaned data tables in the corresponding cleaning cache space.

[0159] The cleaning module 830 is configured to determine a target cleaning cache space from a plurality of cleaning cache spaces, perform deletion processing on data in a to-be-cleaned data table corresponding to index information in the target cleaning cache space based on the index information, and determine a target cleaning cache space from the remaining cleaning cache spaces, jump to the step of performing deletion processing on data in a to-be-cleaned data table corresponding to the index information in the target cleaning cache space based on the index information, until there is no target cleaning cache space.

[0160] Optionally, the classification standard is data volume, and the data types include large data tables, medium data tables and small data tables. The determination module 820 is specifically configured to:

[0161] For each to-be-cleaned data table of the data type of large data table, determine the cleaning cache space corresponding to the to-be-cleaned data table as a separate cleaning cache space;

[0162] In a case where the data table quantity of the to-be-cleaned data table of the data type of medium data table exceeds a preset quantity threshold, calculate the space quantity of the cleaning cache space of the to-be-cleaned data of the data type of medium data table according to the data table quantity and the quantity threshold;

[0163] According to a preset allocation rule, allocate the to-be-cleaned data table of the data type of medium data table to the space quantity of cleaning cache spaces;

[0164] Allocate the to-be-cleaned data table of the data type of small data table to a separate cleaning cache space.

[0165] Optionally, the cleaning module 830 is specifically configured to:

[0166] For each cleaning cache space, take the highest cleaning level in the cleaning levels of the to-be-cleaned data tables corresponding to the cleaning cache space as the cleaning level of the cleaning cache space;

[0167] Based on the cleaning levels of the cleaning cache spaces, determine a target cleaning cache space from the plurality of cleaning cache spaces.

[0168] Optionally, the cleaning module 830 is specifically configured to:

[0169] Obtain a data cleaning time range;

[0170] According to the data cleaning time range, filter target index information of to-be-cleaned data in the to-be-cleaned data table from the index information in the target cleaning cache space;

[0171] Perform a deletion process on the to-be-cleaned data corresponding to the target index information in the database.

[0172] Optionally, the apparatus is applied to a distributed system, the distributed system includes a plurality of databases; and the cleaning module 830 is specifically configured to:

[0173] Parse an address of the database corresponding to the target index information;

[0174] If the database corresponding to the target index information is of different database types, for each database type of the database corresponding to the target index information, construct the address of the database of the database type, the data table name of the to-be-cleaned data table, and the data cleaning time range into cleaning information of the database type;

[0175] The cleaning information of each database type is sent to the database of each database type respectively, and the cleaning information is used to instruct the database to delete the to-be-cleaned data.

[0176] Optionally, the cleaning module 830 is specifically used for:

[0177] The target database group is acquired, and it is determined whether the database corresponding to the index information belongs to the target database group.

[0178] If the database corresponding to the index information belongs to the target database group, the data in the to-be-cleaned data table corresponding to the index information in the database is deleted.

[0179] Optionally, the apparatus 800 further includes:

[0180] The transceiving module is used to send a live detection request according to a preset sniffing frequency, and receive node information corresponding to the live detection request; the node information includes node state information.

[0181] The stop sending module is used to stop sending cleaning information to a fault node if the node information indicates that there is the fault node.

[0182] The adding module is used to add a new node to the target database group if the node information indicates that there is the new node.

[0183] Optionally, the apparatus 800 further includes:

[0184] The retry module is used to re-clean the to-be-cleaned data when a preset retry time is reached if the to-be-cleaned data table cleaning fails.

[0185] Each module in the data cleaning apparatus can be realized by software, hardware and a combination thereof in whole or in part. Each module can be embedded in or independent of a processor in a computer device in a hardware form, or can be stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0186] In one embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in Figure 9The computer device shown in the figure includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be realized through WIFI, mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by the processor to realize a data cleaning method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0187] Those skilled in the art can understand that, Figure 9 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0188] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in each of the above method embodiments.

[0189] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in each of the above method embodiments.

[0190] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the steps in each of the above method embodiments.

[0191] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of the country and region.

[0192] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of each method can be included. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0193] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0194] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A data cleaning method, characterized by, The method comprises: when a preset cleaning condition is reached, classifying the to-be-cleaned data tables according to a preset classification standard to obtain data types corresponding to each of the to-be-cleaned data tables; determining a cleaning cache space corresponding to each of the to-be-cleaned data tables according to the data type corresponding to each of the to-be-cleaned data tables, and storing index information of each of the to-be-cleaned data tables in the corresponding cleaning cache space; determining a target cleaning cache space from the plurality of cleaning cache spaces, and performing deletion processing on data in a to-be-cleaned data table corresponding to index information in the target cleaning cache space in the database based on the index information, and determining a target cleaning cache space from the remaining cleaning cache spaces, and jumping to the step of performing deletion processing on data in a to-be-cleaned data table corresponding to index information in the target cleaning cache space in the database based on the index information until there is no target cleaning cache space; the classification standard is data volume, the data types include large data tables, medium data tables and small data tables, and the determination of the cleaning cache space corresponding to each of the to-be-cleaned data tables according to the data type corresponding to each of the to-be-cleaned data tables comprises: for each to-be-cleaned data table of the data type of a large data table, determining that the cleaning cache space corresponding to the to-be-cleaned data table is a separate cleaning cache space; when the number of to-be-cleaned data tables of the data type of a medium data table exceeds a preset number threshold, calculating the number of cleaning cache spaces of the to-be-cleaned data of the data type of a medium data table according to the number of data tables and the number threshold; allocating the to-be-cleaned data tables of the data type of a medium data table to the number of cleaning cache spaces according to a preset allocation rule; allocating a to-be-cleaned data table of the data type of a small data table to a separate cleaning cache space.

2. The method of claim 1, wherein, The determination of the target cleaning cache space from the plurality of cleaning cache spaces comprises: for each cleaning cache space, taking the highest cleaning level in the cleaning levels of each to-be-cleaned data table corresponding to the cleaning cache space as the cleaning level of the cleaning cache space; determining the target cleaning cache space from the plurality of cleaning cache spaces based on the cleaning levels of the cleaning cache spaces.

3. The method of claim 1, wherein, The deletion processing on data in a to-be-cleaned data table corresponding to index information in the target cleaning cache space in the database based on the index information in the target cleaning cache space comprises: obtaining a data cleaning time range; filtering target index information of to-be-cleaned data in the to-be-cleaned data table from the index information in the target cleaning cache space according to the data cleaning time range; performing deletion processing on to-be-cleaned data corresponding to the target index information in the database.

4. The method of claim 3, wherein, The method is applied to a distributed system, the distributed system comprises a plurality of databases, and the deletion processing on data in a to-be-cleaned data table corresponding to target index information in the database comprises: parsing an address of a database corresponding to the target index information; If the database corresponding to the target index information is of different database types, for each database type of the database corresponding to the target index information, the address of the database of the database type, the data table name of the data table to be cleaned, and the data cleaning time range constitute the cleaning information of the database type; The cleaning information of each database type is respectively sent to the database of each database type; the cleaning information is used to instruct the database to delete the data to be cleaned.

5. The method of claim 1, wherein, The method further comprises: If the database corresponding to the target index information belongs to the target database group, the data in the data table to be cleaned corresponding to the index information in the database is deleted. The method further comprises:

6. The method of claim 5, wherein, The alive detection request is sent at a preset alive detection frequency, and node information corresponding to the alive detection request is received; the node information comprises node state information; If the node information indicates that there is a faulty node, the cleaning information is stopped from being sent to the faulty node; If the node information indicates that there is a new node, the new node is added to the target database group. The method further comprises:

7. The method of claim 1, wherein, If the data table to be cleaned fails to be cleaned, the data to be cleaned is cleaned again when a preset retry time is reached. The apparatus comprises:

8. A data cleansing apparatus, characterized by, The classification module is configured to classify the data tables to be cleaned according to a preset classification standard when a preset cleaning condition is reached, to obtain the data types corresponding to each data table to be cleaned; The determination module is configured to determine the cleaning cache space corresponding to each data table to be cleaned according to the data types corresponding to each data table to be cleaned, and store the index information of each data table to be cleaned in the corresponding cleaning cache space; The cleaning module is configured to determine a target cleaning cache space from a plurality of cleaning cache spaces, delete the data in the data table to be cleaned corresponding to the index information in the database based on the index information in the target cleaning cache space, and determine a target cleaning cache space from the remaining cleaning cache spaces, and jump to the step of deleting the data in the data table to be cleaned corresponding to the index information in the database based on the index information in the target cleaning cache space, until there is no target cleaning cache space. The classification standard is data volume, and the data types comprise large data tables, medium data tables, and small data tables; the determination module is specifically configured to: For each data table to be cleaned of the data type of large data tables, the cleaning cache space corresponding to the data table to be cleaned is determined as a separate cleaning cache space; In a case where the number of data tables to be cleaned of the data type of medium data tables exceeds a preset number threshold, the number of cleaning cache spaces of the data type of medium data tables is calculated according to the number of data tables and the number threshold. ​ According to a preset allocation rule, the to-be-cleaned data table of the medium data table type is allocated to the space number of cleaning cache spaces; The to-be-cleaned data table of the small data table type is allocated to a single cleaning cache space. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

Images