Method, apparatus, device, medium and product for data cleaning

Abstract

The application relates to the technical field of big data, and particularly discloses a data cleaning method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: obtaining a historical cleaning cutoff time for cleaned data from historical cleaning parameters; determining a current cleaning start time for to-be-cleaned data based on the historical cleaning cutoff time; determining current cleaning parameters based on the historical cleaning parameters and the current cleaning start time; the current cleaning parameters comprise a plurality of sub cleaning parameters; determining to-be-cleaned partitions corresponding to each sub cleaning parameter according to a partition identifier in each sub cleaning parameter; and cleaning to-be-cleaned data in the to-be-cleaned partitions according to the current cleaning start time and the current cleaning days in the sub cleaning parameters. The method can improve the data cleaning efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular to a data cleaning method and device, computer equipment, a storage medium, and a computer program product. BACKGROUND

[0002] A large number of business details are generated by the core business system of a large institution every day. The core business system is a customer system, and storing a large amount of detail data in the core business system undoubtedly reduces the database performance and prolongs the service response time, thereby affecting the customer experience. For example, the detail data of a certain bank within three months is stored in the core business system, the detail data of three months to two years is stored in a query migration system, and the detail data of more than two years is stored in a historical archiving system, so the core business system needs to clean the detail data outside the three months.

[0003] In the prior art, the detail data that needs to be cleaned in the core business system is mainly cleaned by using a batch program and manually configuring cleaning parameters, which results in low configuration efficiency of the parameters and low cleaning efficiency of the data. SUMMARY

[0004] Therefore, it is necessary to provide a data cleaning method and device, computer equipment, a computer readable storage medium, and a computer program product that can improve the data cleaning efficiency.

[0005] In a first aspect, the present application provides a data cleaning method, which comprises:

[0006] obtaining a historical cleaning cutoff time for cleaned data from historical cleaning parameters;

[0007] determining a current cleaning start time for to-be-cleaned data based on the historical cleaning cutoff time;

[0008] determining current cleaning parameters based on the historical cleaning parameters and the current cleaning start time; the current cleaning parameters comprise a plurality of sub cleaning parameters;

[0009] determining to-be-cleaned partitions corresponding to each of the sub cleaning parameters according to a partition identifier in each of the sub cleaning parameters;

[0010] cleaning to-be-cleaned data in the to-be-cleaned partitions according to the current cleaning start time and the current cleaning number of days in the sub cleaning parameters.

[0011] In one of the embodiments, the cleaning to-be-cleaned data in the to-be-cleaned partitions according to the current cleaning start time and the current cleaning number of days in the sub cleaning parameters comprises:

[0012] determine a current cleaning deadline for the to-be-cleaned data based on the current cleaning start time and the current cleaning days;

[0013] clean the to-be-cleaned data in the to-be-cleaned partition according to the current cleaning start time and the current cleaning deadline.

[0014] In one of the embodiments, the sub cleaning parameter comprises a cleaning execution time corresponding to the to-be-cleaned data, an execution time node to which the cleaning execution time belongs in a preset period, and a cleaning round value;

[0015] clean the to-be-cleaned data in the to-be-cleaned partition according to the current cleaning start time and the current cleaning days in the sub cleaning parameter, comprising:

[0016] obtain a base value in the configuration parameter;

[0017] determine a remainder between a value corresponding to the execution time node and the base value;

[0018] in a case where the remainder represented by the remainder is same as the cleaning round value, determine the to-be-cleaned data in the to-be-cleaned partition based on the current cleaning start time and the cleaning days in the sub cleaning parameter;

[0019] clean the to-be-cleaned data.

[0020] In one of the embodiments, the cleaning of the to-be-cleaned data comprises:

[0021] in a case where a current time matches the cleaning execution time in the sub cleaning parameter, clean the to-be-cleaned data in the to-be-cleaned partition.

[0022] In one of the embodiments, the cleaning of the to-be-cleaned data in the to-be-cleaned partition according to the current cleaning start time and the current cleaning days in the sub cleaning parameter comprises:

[0023] determine a cleaning execution time for the to-be-cleaned data in the sub cleaning parameter, and a preset special time;

[0024] in a case where the cleaning execution time is different from the preset special time, clean the to-be-cleaned data in the to-be-cleaned partition based on the current cleaning start time and the current cleaning days in the sub cleaning parameter.

[0025] In one of the embodiments, the determination of the to-be-cleaned partition corresponding to each of the sub cleaning parameters comprises:

[0026] determining an identification position where each of the partition identifiers in the current cleaning parameter is located;

[0027] splitting the current cleaning parameter based on the identification positions to obtain a plurality of sub cleaning parameters;

[0028] sending the sub cleaning parameter to a to-be-cleaned partition corresponding to the partition identifier in each of the sub cleaning parameters according to the partition identifier, so that each of the to-be-cleaned partitions performs data cleaning based on the current cleaning start time and the current cleaning days in the received sub cleaning parameter.

[0029] In a second aspect, the present application provides a data cleaning device, which comprises:

[0030] a first time acquisition module configured to acquire a historical cleaning end time for cleaned data from a historical cleaning parameter;

[0031] a second time acquisition module configured to determine a current cleaning start time for to-be-cleaned data based on the historical cleaning end time;

[0032] a parameter determination module configured to determine a current cleaning parameter based on the historical cleaning parameter and the current cleaning start time; the current cleaning parameter comprises a plurality of sub cleaning parameters;

[0033] a partition determination module configured to determine a to-be-cleaned partition corresponding to each of the sub cleaning parameters according to a partition identifier in each of the sub cleaning parameters;

[0034] a data cleaning module configured to clean to-be-cleaned data in the to-be-cleaned partition according to the current cleaning start time and the current cleaning days in the sub cleaning parameter.

[0035] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0036] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0037] In a fifth aspect, the present application provides a computer program product comprising a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0038] The method, device, computer device, storage medium and computer program product for data cleaning can automatically configure the current cleaning parameter, thereby reducing the time for parameter configuration, and can simplify the process of data cleaning. The current cleaning parameter is determined based on the historical cleaning parameter and the current cleaning start time, and the current cleaning start time is determined based on the historical cleaning parameter and the historical cleaning end time of the cleaned data. The cleaning of the to-be-cleaned data in each to-be-cleaned partition is performed based on the current cleaning start time and the current cleaning days in each sub cleaning parameter corresponding to the to-be-cleaned partition, thereby improving the cleaning efficiency of the to-be-cleaned data in the to-be-cleaned partition.

[0039] In addition, the current cleaning end time is determined based on the current cleaning start time and the current cleaning days, and the to-be-cleaned data that needs to be cleaned is determined based on the generation time of the data in the to-be-cleaned partition, thereby avoiding the waste of system resources caused by cleaning the data that does not need to be cleaned. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 An application environment diagram of the method for data cleaning in one embodiment;

[0041] Figure 2 A flowchart of the method for data cleaning in one embodiment;

[0042] Figure 3 A flowchart of the determination of the current end time in one embodiment;

[0043] Figure 4 A flowchart of the determination of the remainder result in one embodiment;

[0044] Figure 5 A flowchart of the method for data cleaning in another embodiment;

[0045] Figure 6 A flowchart of the method for data cleaning in another embodiment;

[0046] Figure 7 A block diagram of the device for data cleaning in one embodiment;

[0047] Figure 8 An internal structure diagram of the computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be given below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0049] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0050] The data cleaning method provided by the embodiments of the present application can be applied to an application environment as shown in Figure 1 . The server 104 communicates with the terminal 102 through the network. The data storage system can store the data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The server 104 obtains the historical cleaning cutoff time for the cleaned data from the historical cleaning parameters. The server 104 determines the current cleaning start time for the to-be-cleaned data according to the historical cleaning cutoff time. The server 104 determines the current cleaning parameters according to the historical cleaning parameters and the current cleaning start time. The current cleaning parameters include a plurality of sub cleaning parameters. The server 104 determines the to-be-cleaned partition corresponding to each sub cleaning parameter according to the partition identifier in each sub cleaning parameter. The server 104 cleans the to-be-cleaned data in the to-be-cleaned partition according to the current cleaning start time and the current cleaning days in the sub cleaning parameter. The terminal 102 can be, but is not limited to, various personal computers, notebook computers, tablet computers, Internet of Things devices, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0051] In one embodiment, as shown in Figure 2 , a data cleaning method is provided. Taking the server in Figure 1 as an example, the method comprises the following steps:

[0052] Step 202, obtaining the historical cleaning cutoff time for the cleaned data from the historical cleaning parameters.

[0053] The historical cleaning parameter refers to the parameter used when the data in the partition database is cleaned last time. The historical cleaning parameter mainly includes partition number, partition identifier, historical cleaning execution time, historical cleaning cutoff time for cleaned data, historical cleaning days, historical cleaning standby value, historical cleaning flag, execution time node of the historical cleaning execution time in the preset period, cleaning execution date of the last data cleaning, historical cleaning start time for cleaned data, and the like. The partition number and the partition identifier are used to distinguish the database of the partition or the agency. For example, the partition number is 036, and the partition identifier is QH, which means the database of Qinghai Branch. The sub-cleaning parameter of the partition number 036 and the partition identifier QH is the parameter used for data cleaning of the partition database of Qinghai Branch. The historical cleaning execution time refers to the date of data cleaning. For example, the data is cleaned on January 1, 2023, and January 1, 2023 is the historical cleaning execution time. The cleaning cutoff time for cleaned data refers to the latest generation time of the cleaned data. For example, the data is cleaned on January 1, 2023, and the latest generation time of the cleaned data is December 1, 2022, and December 1, 2022 is the historical cleaning cutoff time for cleaned data. The historical cleaning days refer to how many days of data are cleaned at a time. For example, the data from December 1, 2022 to December 4, 2022 is cleaned on January 1, 2023, and the interval is 3 days, so the historical cleaning days are 3 days. The historical cleaning flag contains Y and N, N means not to clean the data, and Y means to clean the data. The execution time node of the historical cleaning execution time in the preset period refers to the day of the year when the cleaning time belongs to. For example, the historical cleaning execution time is October 11, 2023, which belongs to the 284th day of 2023, so the 284th day is the execution time node, and the total number of days in 2023 is the length of the preset period.

[0054] The cleaned data refers to the data that originally exists in the partition database but has been cleaned.

[0055] Optionally, the server obtains the historical cleaning cutoff time for cleaned data from the historical cleaning parameter including the partition number, the partition identifier, the historical cleaning execution time, the historical cleaning cutoff time for cleaned data, the historical cleaning days, the historical cleaning standby value, the historical cleaning flag, the execution time node of the historical cleaning execution time in the preset period, the cleaning execution date of the last data cleaning, and the historical cleaning start time for cleaned data.

[0056] In step 204, based on the historical cleaning cutoff time, the current cleaning start time for the to-be-cleaned data is determined.

[0057] The current cleaning start time can be understood as the earliest generation time of the data to be cleaned when the data cleaning is performed this time. For example, when the data cleaning is performed on January 1, 2023, the data to be cleaned generated from December 1 to December 4, 2022 is cleaned, and then December 1, 2022 is the current cleaning start time. The current cleaning start time is determined according to the historical cleaning end time. For example, the historical cleaning end time corresponding to the last data cleaning is January 1, 2023, and then when the data cleaning is performed this time, the current cleaning start time corresponding to the data to be cleaned is also January 1, 2023. The data to be cleaned refers to the data in the partition database that has not been cleaned.

[0058] Optionally, the server determines the current cleaning start time for the data to be cleaned in the partition database according to the historical cleaning end time.

[0059] Step 206, determining the current cleaning parameter based on the historical cleaning parameter and the current cleaning start time. The current cleaning parameter includes a plurality of sub cleaning parameters.

[0060] The current cleaning start time is included in the current cleaning parameter, that is, the historical cleaning end time in the historical cleaning parameter is directly determined as the current cleaning start time in the current cleaning parameter. The current cleaning parameter also includes the current cleaning end time and the current cleaning day number. The current cleaning end time is determined according to the current cleaning parameter and the current cleaning day number.

[0061] The current cleaning day number can be the same as or different from the historical cleaning parameter in the historical cleaning parameter.

[0062] Optionally, after determining the current cleaning start time according to the historical cleaning end time in the historical cleaning parameter, the server determines the current cleaning end time according to the current cleaning start time and the current cleaning day number, and determines the current cleaning parameter according to the historical cleaning parameter and the current cleaning start time. The current cleaning parameter includes a plurality of sub cleaning parameters.

[0063] Step 208, determining the data to be cleaned corresponding to each sub cleaning parameter according to the partition identifier in each sub cleaning parameter.

[0064] The partition identifier is used to distinguish the database of the partition or institution. For example, the partition identifier QH represents Qinghai Branch, the partition identifier GS represents Gansu Branch, and the partition identifier NX represents Ningxia Branch.

[0065] Optionally, the server determines the partition identifier included in each sub-clean parameter, and determines the identifier of each partition, and determines the partition as the to-be-cleaned partition corresponding to the partition identifier in the sub-clean parameter in a case where the identifier of the partition matches the partition identifier in the sub-clean parameter. The matching refers to that the identifier of the partition and the partition identifier in the sub-clean parameter are the same or have the same meaning.

[0066] In step 210, the to-be-cleaned data in the to-be-cleaned partition is cleaned according to the current cleaning start time and the current cleaning days in the sub-clean parameter.

[0067] Optionally, the server determines the current cleaning deadline for the to-be-cleaned data according to the current cleaning start time and the current cleaning days in the sub-clean parameter. The server determines the data generated between the current cleaning start time and the current cleaning deadline as the to-be-cleaned data in the to-be-cleaned partition, and cleans the to-be-cleaned data in the to-be-cleaned partition.

[0068] In the above method for data cleaning, the current cleaning start time for the to-be-cleaned data is determined based on the historical cleaning deadline for the cleaned data in the historical clean parameter, and then the current clean parameter is automatically determined based on the historical clean parameter and the current cleaning start time, so that the data cleaning can be performed according to the automatically generated current clean parameter when the data row cleaning is performed, and the process of data cleaning can be simplified. The to-be-cleaned partition corresponding to each sub-clean parameter is determined according to the partition identifier in each sub-clean parameter, so that the to-be-cleaned data in each to-be-cleaned partition can be cleaned according to the current cleaning start time and the current cleaning days in the sub-clean parameter corresponding to the to-be-cleaned partition, and the cleaning efficiency of the to-be-cleaned data in the to-be-cleaned partition is improved. In addition, the current cleaning deadline is determined according to the current cleaning start time and the current cleaning days, so that the to-be-cleaned data that needs to be cleaned can be determined according to the generation time of the data in the to-be-cleaned partition, and the system resources can be avoided from being wasted by cleaning the data that does not need to be cleaned.

[0069] In one embodiment, as shown in FIG. 2, the to-be-cleaned data in the to-be-cleaned partition is cleaned according to the current cleaning start time and the current cleaning days in the sub-clean parameter, including: Figure 3

[0070] In step 302, the current cleaning deadline for the to-be-cleaned data is determined based on the current cleaning start time and the current cleaning days.

[0071] ​The current cleanup deadline is determined by the sum of the current cleanup start time and the current number of cleanup days in terms of time units. For example, if the current cleanup start time is December 1, 2022, and the current number of cleanup days is 3 days, then the current cleanup deadline is December 1, 2022 plus three days, which is three days after December 1, 2022, meaning the current cleanup deadline is December 4, 2022. Data to be cleaned refers to data that still exists in the partition and has not yet been cleaned.

[0072] The current cleanup start time can be understood as the earliest generation time of the data that needs to be cleaned during this cleanup. For example, if data cleanup is performed on January 1, 2023, and the data generated between December 1 and December 4, 2022 is to be cleaned, then December 1, 2022, is the current cleanup start time. The current cleanup start time is determined based on the historical cleanup end time.

[0073] Optionally, the server determines the current cleanup deadline for the data that has not been cleaned in the partition to be cleaned based on the sum of the current cleanup start time and the current number of cleanup days in time units.

[0074] Step 304: Clean up the data to be cleaned in the partition to be cleaned according to the current cleanup start time and the current cleanup end time.

[0075] Optionally, the server obtains the generation time of each data in the partition to be cleaned, and determines the data whose generation time is between the current cleanup start time and the current cleanup end time as the data to be cleaned in the partition to be cleaned, and then cleans the data to be cleaned.

[0076] In this embodiment, the current cleanup deadline is determined based on the current cleanup start time and the current number of cleanup days. This allows for the identification of data to be cleaned according to the generation time of each data item in the partition to be cleaned, thus avoiding the deletion of unnecessary data and the waste of system resources.

[0077] In one embodiment, such as Figure 4 As shown, the sub-cleanup parameters include the cleanup execution time corresponding to the data to be cleaned, the execution time node within the preset period to which the cleanup execution time belongs, and the cleanup waiting value. Based on the current cleanup start time and current cleanup days in the sub-cleanup parameters, the data to be cleaned in the partition to be cleaned is cleaned, including:

[0078] Step 402: Obtain the base value from the configuration parameters.

[0079] The configuration parameter includes two types of data, one is a cardinal value used to determine the remainder result corresponding to the execution time node, and the other is a cleaning flag including Y and N, N indicating not to clean up data, and Y indicating to clean up data. When the cleaning flag in the configuration parameter is N, the cleaning of data is not performed regardless of whether the data in the current cleaning parameter indicates to clean up data. For example, the current cleaning parameter indicates that the to-be-cleaned data in the to-be-cleaned partition can be cleaned up, but when the cleaning flag in the configuration parameter is N, the to-be-cleaned data in the to-be-cleaned partition is still retained and not cleaned up. The configuration parameter and the current cleaning parameter can exist in the same file.

[0080] Step 404, determining the remainder result between the number corresponding to the execution time node and the cardinal value.

[0081] The number corresponding to the execution time node refers to which day of the preset period the cleaning execution time is in. For example, the cleaning execution time is October 11, 2023, which belongs to the 284th day of 2023, so the 284th day is the execution time node, and the total number of days in 2023 is the length of the preset period.

[0082] The remainder result refers to the remainder result of the number corresponding to the execution time node divided by the cardinal value. For example, the number corresponding to the execution time node is 284, and the cardinal value is 001, so the number represented in the remainder result is 000.

[0083] Optionally, after the server determines the number corresponding to the execution time node and the cardinal value in the configuration parameter, the server determines the number represented in the remainder result according to the remainder result of the number corresponding to the execution time node divided by the cardinal value.

[0084] Step 406, in the case that the remainder represented by the remainder result is the same as the cleaning round value, determining the to-be-cleaned data in the to-be-cleaned partition based on the current cleaning start time and the cleaning days in the sub-cleaning parameter.

[0085] The to-be-cleaned data is determined according to the current cleaning start time and the cleaning days, and the current cleaning end time determined by the current cleaning start time. For example, the current cleaning start time is December 1, 2022, and the current cleaning end time is December 4, 2022, so the data generated in the to-be-cleaned partition between December 1, 2022 and December 4, 2022 are to-be-cleaned data. The current cleaning end time is determined according to the sum of the current cleaning start time and the current cleaning days in time units.

[0086] Optionally, the server determines a current cleaning end time according to the current cleaning start time and the cleaning days in a case where the remainder represented by the remainder result is the same as the cleaning waiting value. The server determines data generated in the to-be-cleaned partition and having a generation time between the current cleaning start time and the current cleaning end time as the to-be-cleaned data.

[0087] In step 408, the to-be-cleaned data is cleaned.

[0088] In step 409, the to-be-cleaned data in the to-be-cleaned partition is retained without performing data cleaning in a case where the remainder represented by the remainder result is different from the cleaning waiting value.

[0089] In the embodiment, by determining the remainder result between the value corresponding to the time node of performing and the base value in the configuration parameter, and determining the to-be-cleaned data in the to-be-cleaned partition according to the current cleaning start time and the cleaning days in the sub cleaning parameter in a case where the remainder represented by the remainder result is the same as the cleaning waiting value, the data is cleaned, so that the cleaning operation of the cleaning execution data in the cleaning that is not needed to be performed can be avoided, and the waste of cleaning resources of the system can be reduced, and the utilization rate of the cleaning resources of the system can be improved.

[0090] In one embodiment, cleaning the to-be-cleaned data comprises:

[0091] In a case where the current time matches the cleaning execution time in the sub cleaning parameter, the to-be-cleaned data in the to-be-cleaned partition is cleaned.

[0092] The current time refers to the present time. For example, today is January 1, 2023, and the cleaning execution time in the sub cleaning parameter is also January 1, 2023, which means that the current time matches the cleaning execution time.

[0093] Optionally, the server determines the current time, and in a case where it is detected that the current time matches the cleaning execution time in the sub cleaning parameter, the to-be-cleaned data in the to-be-cleaned partition is cleaned.

[0094] In one of the embodiments, in a case where the current time does not match the cleaning execution time in the sub cleaning parameter, the cleaning of the to-be-cleaned data in the to-be-cleaned partition is not performed.

[0095] In the embodiment, by setting that the to-be-cleaned data in the to-be-cleaned partition is cleaned only in a case where the current time matches the cleaning execution time in the sub cleaning parameter, the data processing in the date that does not need to be performed can be avoided, so that the waste of cleaning resources in the system can be reduced, and the utilization rate of the cleaning resources can be improved.

[0096] In one embodiment, as Figure 5As shown, the data to be cleaned in the to-be-cleaned partition is cleaned according to the current cleaning start time and the current cleaning days in the sub-clean parameter, including:

[0097] In step 502, the cleaning execution time for the data to be cleaned in the sub-clean parameter is determined, and a preset special time is determined.

[0098] The cleaning execution time for the data to be cleaned refers to the date of cleaning the data in the database of the to-be-cleaned partition that has not been cleaned. For example, if the data in the database of the to-be-cleaned partition that has not been cleaned is cleaned on January 1, 2023, then January 1, 2023 is the cleaning execution time for the data to be cleaned.

[0099] The preset special time refers to a special date that is set in advance in the server and is expected to generate a large amount of data. For example, during Double Eleven, multiple transaction data will be generated, so during Double Eleven, the cleaning of data will not be performed, and most of the resources of the system will be used for the generation and maintenance of transaction data.

[0100] Optionally, the server determines the cleaning execution time for the data in the database of the to-be-cleaned partition that has not been cleaned from the sub-clean parameter, and obtains the preset special time set in advance.

[0101] In step 504, if the cleaning execution time and the preset special time are different, the data to be cleaned in the to-be-cleaned partition is cleaned based on the current cleaning start time and the current cleaning days in the sub-clean parameter.

[0102] The preset special time is at least one. The difference means that the cleaning execution time and any preset special time are different. For example, the cleaning execution time is March 1, 2023, and there is no time of March 1, 2023 in the preset special time, which means that the cleaning execution time and the preset special time are different.

[0103] Optionally, the service compares the cleaning execution time with the preset special time one by one, and in the case that the cleaning execution time is detected to be different from any preset special time, the server obtains the current cleaning end time determined according to the current cleaning start time and the current cleaning days in the sub-clean parameter, and then cleans and cleans the data in the to-be-cleaned partition whose generation time is between the current cleaning start time and the current cleaning end time.

[0104] In step 505, if the cleaning execution time and the preset special time are the same, the cleaning task for the data to be cleaned in the to-be-cleaned partition is not performed.

[0105] In this embodiment, the cleaning execution time for the data to be cleaned and the preset special time are determined, and the data to be cleaned in the to-be-cleaned partition is cleaned according to the current cleaning start time and the current cleaning days only when there is a difference between the cleaning execution time and the preset special time, so that in the case of a large amount of data generation, the resources in the system are more applied in aspects such as maintaining data and reading data, thereby reducing the risk of database crash.

[0106] In one embodiment, as shown in FIG. 6, the to-be-cleaned partition corresponding to each sub cleaning parameter is determined according to the partition identifier in each sub cleaning parameter, including: Figure 6

[0107] Step 602, determine the identifier position of each partition identifier in the current cleaning parameter.

[0108] The identifier position refers to the position of the partition identifier in the current cleaning parameter. For example, the current cleaning parameter has ten positions in total, and the partition identifier belongs to the parameter at the first position, so the identifier position can be understood as the first position in the current cleaning parameter.

[0109] Step 604, split the current cleaning parameter based on the identifier positions to obtain multiple sub cleaning parameters.

[0110] The splitting refers to extracting each sub cleaning parameter from the current cleaning parameter to form a complete cleaning parameter independently, without changing the state of the current cleaning parameter. For example, the current cleaning parameter has three sub cleaning parameters, and the three sub cleaning parameters are extracted separately according to the positions of the partition identifiers.

[0111] Step 606, send the sub cleaning parameter to the to-be-cleaned partition corresponding to the partition identifier according to the partition identifier in each sub cleaning parameter, so that each to-be-cleaned partition performs data cleaning based on the current cleaning start time and the current cleaning days in the received sub cleaning parameter.

[0112] Optionally, after obtaining the multiple separated sub cleaning parameters, the server distributes each sub cleaning parameter to the to-be-cleaned partition corresponding to the partition identifier in the sub cleaning parameter according to the partition identifier in each sub cleaning parameter and the identifier of each to-be-cleaned partition, and then performs data cleaning based on the current cleaning start time and the current cleaning days in the received sub cleaning parameter.

[0113] ​In this embodiment, the to-be-cleaned partition matching the partition identifier is determined according to the partition identifier in the sub-clean parameter, and the to-be-cleaned data in the partition is cleaned using the sub-clean parameter matching each to-be-cleaned partition, so that each to-be-cleaned partition can simultaneously clean data according to the corresponding sub-clean parameter, thereby improving the efficiency of data cleaning.

[0114] The application further provides an application scenario of the data cleaning method. Specifically, the data cleaning method is applied in the application scenario as follows.

[0115] The server obtains the configuration parameter with the cardinality value 001 and the cleaning flag Y, and also obtains the historical cleaning parameter, which is specifically 035, GS, 20230601, 20230508, 001, 000, N; 036, QH, 20231011, 20230511, 003, 000, Y, 00284, 20230601, 20230508; 037, NX, 20230601, 20230508, 003, 000, N. The first field is the partition number, the second field is the partition identifier, the third field is the cleaning execution time for the cleaned data, the fourth field is the historical cleaning deadline for the cleaned data, the fifth field is the historical cleaning days, the sixth field is the cleaning standby times, the seventh field is the partition cleaning flag, the eighth field is the execution time node to which the cleaning execution time belongs in the preset period, the ninth field is the execution date of the last data cleaning, and the tenth field is the deadline of the last data cleaning or the historical cleaning start time of this cleaning. When generating the current cleaning parameter, since 035, GS, 20230601, 20230508, 001, 000, N and 037, NX, 20230601, 20230508, 003, 000, N indicate that the data is not cleaned, the server does not update these two sub-clean parameters, but only updates the sub-clean parameter 036, QH, 20231011, 20230511, 003, 000, Y, 00284, 20230601, 20230508.

[0116] The server determines the historical cleaning cutoff time 20230511 in the historical cleaning parameter as the content of the tenth field in the current cleaning parameter, that is, determines 20230511 as the current cleaning start time in the current cleaning parameter, takes the historical cleaning days as the current cleaning days, and then obtains the content of the fourth field in the current cleaning parameter according to the sum of the current cleaning days 003 and the current cleaning start time 20230511 in the time unit, that is, determines 20230514 as the current cleaning cutoff time in the current cleaning parameter. The second field in the current cleaning parameter will be updated as the current time changes, that is, from 20231011 in the historical cleaning parameter to 20231012 in the current cleaning parameter, and the eighth field in the current cleaning parameter will also change from 0284 to 0285, the ninth field in the current cleaning parameter is the execution date of the last data cleaning, that is, 20231011, and the contents of the remaining fields remain unchanged. Therefore, the current cleaning parameter is updated to 035, GS, 20230601, 20230508, 001, 000, N; 036, QH, 20231012, 20230514, 003, 000, Y, 0285, 20231011, 20230511; 037, NX, 20230601, 20230508, 003, 000, N.

[0117] The server extracts each sub cleaning parameter in the current cleaning parameter from the current cleaning parameter according to the identification position of each partition identification in the current cleaning parameter, and forms a complete cleaning parameter independently. After obtaining a plurality of separated sub cleaning parameters, the server distributes each sub cleaning parameter to the to-be-cleaned partition corresponding to the partition identification in the sub cleaning parameter according to the partition identification in each sub cleaning parameter and the identification of each to-be-cleaned partition. The server compares the cleaning execution time in each sub cleaning parameter with the preset special time one by one, and detects that the cleaning execution time is different from any preset special time. The server determines the current time, and detects that only the cleaning execution time in the sub cleaning parameter with the partition identification QH matches the current time.

[0118] The server obtains the current cleaning start time and the current cleaning cutoff time from the sub cleaning parameter with the partition identification QH, and then generates data between the current cleaning start time and the current cleaning cutoff time in the to-be-cleaned partition matched with the partition identification QH as to-be-cleaned data for cleaning, and performs cleaning.

[0119] The method of data cleaning in the embodiment can determine the current cleaning start time for the to-be-cleaned data based on the historical cleaning parameter and the historical cleaning end time for the cleaned data, and then automatically determine the current cleaning parameter based on the historical cleaning parameter and the current cleaning start time, so that the cleaning of data can be performed according to the automatically generated current cleaning parameter when the data row cleaning is performed, and the process of data cleaning can be simplified; the to-be-cleaned partition corresponding to each sub cleaning parameter is determined according to the partition identifier in each sub cleaning parameter, so that the cleaning of the to-be-cleaned data in each to-be-cleaned partition can be performed according to the current cleaning start time and the current cleaning days in the sub cleaning parameter corresponding to each to-be-cleaned partition, and the cleaning efficiency of the to-be-cleaned data in the to-be-cleaned partition is improved. In addition, the current cleaning end time is determined according to the current cleaning start time and the current cleaning days, so that the to-be-cleaned data that needs to be cleaned can be determined according to the generation time of the data in the to-be-cleaned partition, so that the system resources can be avoided from being wasted by cleaning the data that does not need to be cleaned.

[0120] It should be understood that, although each step in the flowchart involved in each of the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0121] Based on the same inventive concept, the embodiments of the present application also provide a data cleaning device for implementing the above-mentioned data cleaning method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more data cleaning device embodiments provided below can refer to the limitations of the data cleaning method in the above text, which will not be repeated here.

[0122] In one embodiment, as shown in Figure 7 a data cleaning device 700 is provided, comprising:

[0123] The first time acquisition module 702 is configured to acquire the historical cleaning end time for the cleaned data from the historical cleaning parameter.

[0124] The second time obtaining module 704 is configured to determine a current cleaning start time for the to-be-cleaned data based on the historical cleaning deadline.

[0125] The parameter determining module 706 is configured to determine a current cleaning parameter based on the historical cleaning parameter and the current cleaning start time. The current cleaning parameter includes a plurality of sub cleaning parameters.

[0126] The partition determining module 708 is configured to determine a to-be-cleaned partition corresponding to each sub cleaning parameter according to a partition identifier in the sub cleaning parameter.

[0127] The data cleaning module 710 is configured to clean the to-be-cleaned data in the to-be-cleaned partition according to the current cleaning start time and the current cleaning days in the sub cleaning parameter.

[0128] In one of the embodiments, the data cleaning module is further configured to determine a current cleaning deadline for the to-be-cleaned data based on the current cleaning start time and the current cleaning days, and clean the to-be-cleaned data in the to-be-cleaned partition according to the current cleaning start time and the current cleaning deadline.

[0129] In one of the embodiments, the data cleaning module is further configured to:

[0130] obtain a base value in the configuration parameter, determine a remainder result between a value corresponding to the execution time node and the base value, and determine the to-be-cleaned data in the to-be-cleaned partition based on the current cleaning start time and the cleaning days in the sub cleaning parameter when the remainder represented by the remainder result is the same as the cleaning waiting value, and clean the to-be-cleaned data.

[0131] In one of the embodiments, the data cleaning module is further configured to:

[0132] clean the to-be-cleaned data in the to-be-cleaned partition when the current time matches the cleaning execution time in the sub cleaning parameter.

[0133] In one of the embodiments, the data cleaning module is further configured to:

[0134] determine a cleaning execution time for the to-be-cleaned data in the sub cleaning parameter and a preset special time, and clean the to-be-cleaned data in the to-be-cleaned partition based on the current cleaning start time and the current cleaning days in the sub cleaning parameter when the cleaning execution time is different from the preset special time.

[0135] In one of the embodiments, the partition determining module is further configured to determine an identification position where each partition identifier in the current cleaning parameter is located. The current cleaning parameter is split based on the identification positions to obtain a plurality of sub cleaning parameters. Each sub cleaning parameter is sent to a to-be-cleaned partition corresponding to the partition identifier in the sub cleaning parameter, so that each to-be-cleaned partition performs data cleaning based on the current cleaning start time and the current cleaning days in the received sub cleaning parameter.

[0136] The data cleaning apparatus can perform data cleaning according to the automatically generated current cleaning parameter when performing data row cleaning, and can simplify the data cleaning process. The to-be-cleaned partitions corresponding to each sub cleaning parameter are determined according to the partition identifier in each sub cleaning parameter, so that the cleaning of the to-be-cleaned data in each to-be-cleaned partition is performed according to the current cleaning start time and the current cleaning days in the sub cleaning parameter corresponding to the to-be-cleaned partition, and the cleaning efficiency of the to-be-cleaned data in the to-be-cleaned partition is improved.

[0137] In addition, the current cleaning end time is determined according to the current cleaning start time and the current cleaning days, so that the to-be-cleaned data that needs to be cleaned can be determined according to the generation time of the data in the to-be-cleaned partition, and the system resources can be avoided from being wasted by cleaning the data that does not need to be cleaned.

[0138] The modules in the data cleaning apparatus can be implemented by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in a hardware form, or stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform the operations corresponding to the modules.

[0139] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in FIG. 1. Figure 8As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store historical cleaning parameters, current cleaning start time, current cleaning parameters, partition identification, historical cleaning cutoff time, to-be-cleaned partition and current cleaning days. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to realize a data cleaning method.

[0140] Those skilled in the art can understand that, Figure 8 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or less components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0141] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the data cleaning method of each of the above embodiments.

[0142] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by the processor to implement the steps of the data cleaning method of each of the above embodiments.

[0143] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by the processor to implement the steps of the data cleaning method of each of the above embodiments.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0145] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0146] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A data cleaning method, characterized in that, The method includes: Obtain the historical cleanup deadline for the cleaned data from the historical cleanup parameters; Set the historical cleanup deadline in the historical cleanup parameters as the current cleanup start time in the current cleanup parameters; Based on the historical cleanup parameters and the current cleanup start time, the current cleanup parameters are determined; the current cleanup parameters include multiple sub-cleanup parameters, wherein the sub-cleanup parameters include the cleanup execution time corresponding to the data to be cleaned, the execution time node to which the cleanup execution time belongs in a preset period, and the cleanup waiting value; Based on the partition identifier in each of the sub-cleanup parameters, determine the partition to be cleaned corresponding to each of the sub-cleanup parameters; According to the current cleanup start time and current cleanup days in the sub-cleanup parameters, the data to be cleaned in the partition to be cleaned is cleaned. The step of cleaning the data to be cleaned in the partition to be cleaned according to the current cleanup start time and current cleanup days in the sub-cleanup parameters includes: obtaining a base value in the configuration parameters; determining the remainder between the value corresponding to the execution time node and the base value; if the remainder represented by the remainder is the same as the cleanup waiting value, determining the data to be cleaned in the partition to be cleaned based on the current cleanup start time and cleanup days in the sub-cleanup parameters; and cleaning the data to be cleaned.

2. The method according to claim 1, characterized in that, The step of cleaning up the data to be cleaned in the partition to be cleaned according to the current cleanup start time and current cleanup days in the sub-cleanup parameters includes: Based on the current cleanup start time and the current number of cleanup days, determine the current cleanup deadline for the data to be cleaned; Clean up the data to be cleaned in the partition to be cleaned according to the current cleanup start time and the current cleanup end time.

3. The method according to claim 1, characterized in that, The cleaning of the data to be cleaned includes: If the current time matches the cleanup execution time in the sub-cleanup parameters, the data to be cleaned in the partition to be cleaned is cleaned.

4. The method according to claim 1, characterized in that, The step of cleaning up the data to be cleaned in the partition to be cleaned according to the current cleanup start time and current cleanup days in the sub-cleanup parameters includes: The sub-cleanup parameters are determined, including the cleanup execution time for the data to be cleaned and the preset special time. If there is a difference between the cleanup execution time and the preset special time, the data to be cleaned in the partition to be cleaned is cleaned based on the current cleanup start time and current cleanup days in the sub-cleanup parameters.

5. The method according to claim 1, characterized in that, The step of determining the partition to be cleaned corresponding to each of the sub-cleaning parameters based on the partition identifier in each sub-cleaning parameter includes: Determine the location of each partition identifier in the current cleanup parameters; The current cleanup parameters are split based on the positions of the identified markers to obtain multiple sub-cleanup parameters; According to the partition identifier in each of the sub-cleaning parameters, the sub-cleaning parameters are sent to the partition to be cleaned corresponding to the partition identifier, so that each partition to be cleaned can perform data cleanup based on the current cleanup start time and current cleanup days in the received sub-cleaning parameters.

6. A data cleaning apparatus, characterized in that, The device includes: The first-time acquisition module is used to obtain the historical cleanup deadline for cleaned data from the historical cleanup parameters; The second time acquisition module is used to determine the historical cleanup end time in the historical cleanup parameters as the current cleanup start time in the current cleanup parameters; The parameter determination module is used to determine the current cleanup parameters based on the historical cleanup parameters and the current cleanup start time. The current cleanup parameters include multiple sub-cleanup parameters, wherein the sub-cleanup parameters include the cleanup execution time corresponding to the data to be cleaned, the execution time node to which the cleanup execution time belongs in a preset period, and the cleanup waiting value. The partition determination module is used to determine the partition to be cleaned corresponding to each of the sub-cleaning parameters based on the partition identifier in each sub-cleaning parameter. The data cleaning module is used to clean the data to be cleaned in the partition to be cleaned according to the current cleaning start time and current cleaning days in the sub-cleaning parameters. The step of cleaning the data to be cleaned in the partition to be cleaned according to the current cleaning start time and current cleaning days in the sub-cleaning parameters includes: obtaining a base value in the configuration parameters; determining the remainder between the value corresponding to the execution time node and the base value; if the remainder represented by the remainder is the same as the cleaning waiting value, determining the data to be cleaned in the partition to be cleaned based on the current cleaning start time and cleaning days in the sub-cleaning parameters; and cleaning the data to be cleaned.

7. The apparatus according to claim 6, characterized in that, The data cleaning module is also used for: Based on the current cleanup start time and the current cleanup days, determine the current cleanup deadline for the data to be cleaned; clean the data to be cleaned in the partition to be cleaned according to the current cleanup start time and the current cleanup deadline.

8. The apparatus according to claim 6, characterized in that, The data cleaning module is also used for: If the current time matches the cleanup execution time in the sub-cleanup parameters, the data to be cleaned in the partition to be cleaned is cleaned.

9. The apparatus according to claim 6, characterized in that, The data cleaning module is also used for: The cleaning execution time for the data to be cleaned and the preset special time are determined in the sub-cleaning parameters. If there is a difference between the cleaning execution time and the preset special time, the data to be cleaned in the partition to be cleaned is cleaned based on the current cleaning start time and the current number of cleaning days in the sub-cleaning parameters.

10. The apparatus according to claim 6, characterized in that, The partition determination module is also used for: The identifier position of each partition identifier in the current cleanup parameters is determined; the current cleanup parameters are split based on each identifier position to obtain multiple sub-cleanup parameters; according to the partition identifier in each sub-cleanup parameter, the sub-cleanup parameter is sent to the partition to be cleaned corresponding to the partition identifier, so that each partition to be cleaned can perform data cleanup based on the current cleanup start time and current cleanup days in the received sub-cleanup parameters.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images