Data cleaning method and device, computer device, and storage medium

Abstract

The application discloses a data cleaning method, and applies to the technical field of data processing. The method provided by the application comprises the following steps: obtaining a target data cleaning task of a target database containing a target system and a target data table of the target database; obtaining the number of deployment environments of the target system; splitting the target data cleaning task according to the number of the deployment environments and the hardware resource performance of the deployment environments to obtain a data acquisition task list; executing the data acquisition task according to a preset data acquisition rule to add a to-be-processed data table set; generating a data processing task list according to the target data table in the target data cleaning task; executing the data processing task in the data processing task list, taking the target data table after being processed as a to-be-updated data table; obtaining the to-be-updated data table according to a preset data updating rule, and updating the to-be-updated data table to the deployment environment of the target system.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to data cleaning methods, apparatus, computer equipment and storage media. Background Technology

[0002] In the field of data processing technology, there is a widespread need to clean the data in the target database of a target system, to process historical dirty data or to perform other processing on historical data to meet business needs.

[0003] However, in the execution of existing data cleaning technologies, on the one hand, the reading operation of the source data puts a lot of access pressure on the database, causing database blockage and thus triggering production accidents; on the other hand, the data cleaning process can cause concurrency problems, resulting in repeated reading and processing operations, which in turn leads to waste of resources. Summary of the Invention

[0004] This application provides a data cleaning method, apparatus, computer equipment, and storage medium to solve the problems of database congestion and redundant processing caused by existing data cleaning technologies.

[0005] A first aspect of this application provides a data cleaning method, comprising:

[0006] Obtain the target data cleaning task, wherein the data cleaning task includes the target database of the target system and the target data table of the target database;

[0007] Obtain all deployment environments of the target system, each deployment environment containing the target database and the target data table;

[0008] A data acquisition task list is generated, which is obtained by splitting the target data cleaning task according to the hardware resource performance of the deployment environment.

[0009] According to the preset data acquisition rules, the data acquisition tasks in the data acquisition task list are executed to obtain the target data table, and the obtained target data table is added to the set of data tables to be processed.

[0010] A data processing task list is generated based on the different processing methods applied to the target data table in the target data cleaning task.

[0011] Execute the data processing tasks in the data processing task list, and add the processed target data table as the data table to be updated to the data table set.

[0012] According to a preset data updating rule, the to-be-updated data table in the to-be-updated data table set is acquired, and the to-be-updated data table is updated to all deployment environments of the target system.

[0013] In a second aspect, the present application provides a data cleaning device, comprising:

[0014] A data cleaning task acquisition module is configured to acquire a target data cleaning task, wherein the target data cleaning task comprises a target database of a target system and a target data table of the target database.

[0015] A deployment environment acquisition module is configured to acquire all deployment environments of the target system, wherein each of the deployment environments comprises the target database and the target data table.

[0016] A data acquisition task module is configured to generate a data acquisition task list, wherein the data acquisition task list is obtained by splitting the target data cleaning task according to hardware resource performance of the acquired deployment environments.

[0017] A data acquisition execution module is configured to execute data acquisition tasks in the data acquisition task list according to a preset data acquisition rule to acquire the target data table, and add the acquired target data table to a to-be-processed data table set.

[0018] A data processing task module is configured to generate a data processing task list, wherein the data processing task list is generated according to different processing manners of the target data table in the target data cleaning task.

[0019] A data processing execution module is configured to execute data processing tasks in the data processing task list, and add the processed target data table to a to-be-updated data table set as a to-be-updated data table.

[0020] A data updating module is configured to acquire the to-be-updated data table in the to-be-updated data table set according to a preset data updating rule, and update the to-be-updated data table to all deployment environments of the target system.

[0021] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above data cleaning method.

[0022] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executable on a processor to implement the steps of the above data cleaning method.

[0023] The aforementioned data cleaning method, apparatus, computer equipment, and storage medium acquire target data cleaning tasks containing a target database of a target system and target data tables in the target database; acquire the number of deployment environments of the target system; split the target data cleaning tasks into a data acquisition task list based on the number of deployment environments and the hardware resource performance of the deployment environments; execute the data acquisition tasks according to preset data acquisition rules and add them to a set of data tables to be processed; generate a data processing task list based on the target data tables in the target data cleaning tasks; execute the data processing tasks in the data processing task list, and use the processed target data tables as data tables to be updated; acquire the data tables to be updated according to preset data update rules, and update the data tables to be updated in the deployment environment of the target system. This not only reduces the access pressure on the database during the data cleaning process, but also avoids the problems of database congestion and redundant processing in the data cleaning process, while further improving the efficiency of data cleaning. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of an application environment for a data cleaning method according to an embodiment of this application;

[0026] Figure 2 This is a flowchart of a data cleaning method according to an embodiment of this application;

[0027] Figure 3 This is a schematic diagram of the structure of a data cleaning apparatus in one embodiment of this application;

[0028] Figure 4 This is a schematic diagram of a computer device according to one embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] The data cleaning method provided in this application can be applied to, for example... Figure 1In this application environment, the computer equipment can be, but is not limited to, various personal computers and laptops. The computer equipment can also be a server, which can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This is understandable. Figure 1 The number of computer devices shown is merely illustrative and can be expanded in any number according to actual needs.

[0031] In one embodiment, such as Figure 2 As shown, a data cleaning method is provided, which is applied to... Figure 1 The following steps, S101 to S107, are used as an example of computer equipment in the example:

[0032] S101. Obtain the target data cleaning task, wherein the data cleaning task includes the target database of the target system and the target data table of the target database;

[0033] Specifically, the data cleaning task involves cleaning the data in the target data tables within the target database of the target system. Cleaning refers to the process of identifying incomplete, incorrect, or irrelevant parts of the data, and then replacing, modifying, or deleting the dirty data. Data cleaning addresses various data issues, including but not limited to: data integrity, data validity, data consistency, data uniqueness, and data authority. For example, user data in a user table may lack age information; a user's age may be greater than 150 years old; or the same user may have at least two records in the user information table.

[0034] Furthermore, before acquiring the target data cleaning task, the process includes: acquiring the dataset contained in the target system, which includes the target database and the target data tables of the target database. Then, the dataset is partitioned according to the data update time. Finally, a data operation time index of the partitioned dataset is generated. Partitioning the dataset contained in the target system involves dividing all the data contained in the target system into partitions, which can be managed and accessed independently. Partitioning can improve scalability, reduce contention, and optimize performance. Further, the partitioning operation of the dataset contained in the target system includes, but is not limited to: horizontal partitioning, vertical partitioning, and functional partitioning. Horizontal partitioning is often referred to as sharding. In horizontal partitioning, each partition is an independent data storage, and each partition stores a specific subset of the data. For example, after sorting by time, order data of users older than a preset time is stored in one horizontal partition. Vertical partitioning refers to each partition storing a subset of fields in the data storage. For example, frequently accessed fields in user information are stored in one vertical partition, and other less frequently accessed fields in user information are stored in another vertical partition. Functional partitioning refers to dividing and storing data in the target system according to different functions. For example, in an insurance system, user insurance purchase records are stored in one functional partition, and user accident records are stored in another. Furthermore, it should be noted that horizontal partitioning, vertical partitioning, and functional partitioning can be combined in practice. For example, data can be horizontally partitioned first, and then the subset of horizontally partitioned data can be vertically partitioned. After partitioning all data in the target system, the data query efficiency during the target data cleaning task can be effectively shortened, further improving the execution efficiency of the data cleaning method in this embodiment. The data operation time index is a data structure that pre-sorts the values ​​of one or more columns. By using the data operation time index, the database system can directly locate records that meet the conditions without scanning the entire table, thus improving query speed and consequently improving the execution efficiency of the data cleaning method in this embodiment.

[0035] S102. Obtain all deployment environments of the target system, each deployment environment containing the target database and the target data table.

[0036] Specifically, obtaining the number of all deployment environments of the target system includes: First, obtaining a first number of formal environments and a second number of grayscale environments of the target system, wherein the deployment environments include the formal environments and the grayscale environments. Grayscale environments are a common engineering method in software engineering, and specific technical details will not be elaborated here. Then, a first access frequency is set based on the hardware resource performance of the formal environments, and a second access frequency is set based on the hardware resource performance of the grayscale environments. Because the performance of the hardware resources such as computer devices or computer clusters supporting the formal environments and the grayscale environments differs, different data access frequencies need to be set according to the hardware resource performance of different deployment environments. Generally, the hardware resource performance of the formal environments is higher than that of the grayscale environments, so the first access frequency is generally set to be greater than the second access frequency. Further, historical peak data of the data access peak of the formal environments is obtained, and the first access frequency is divided into at least one first segmented access frequency based on the historical peak data using a preset access frequency segmentation method. The method of dividing the first access frequency into first segmented access frequencies based on the peak historical data effectively ensures that the data acquisition step during the data cleaning method of this embodiment does not affect the user experience of real users in the production environment. For example, if the data acquisition during the data cleaning method coincides with the peak access time of real users in the production environment, it will inevitably lead to insufficient hardware resource performance in the production environment, resulting in poor user experience such as lag and latency for real users accessing the production environment. The aforementioned reasonable allocation of the first access frequency to the production environment based on the peak historical data of the production environment can effectively avoid such problems. Finally, access switches for the production environment and the grayscale environment are configured. The access switches are used to determine whether data in the target data table can be obtained from the production environment or the grayscale environment corresponding to the access switch. Since the formal environment is the deployment environment that directly faces the user group, and the gray environment also has a portion of the user group, the data cleaning task in this embodiment will put some pressure on the formal environment and / or the gray environment when performing data acquisition-related steps. At this time, it is necessary to design an access switch to determine whether the formal environment and / or the gray environment can be accessed. Before the data cleaning task performs data acquisition-related steps, the access switch is set according to the current access pressure of the formal environment and / or the gray environment so as not to put too much pressure on the formal environment and / or the gray environment.

[0037] S103. Generate a data acquisition task list, which is obtained by splitting the target data cleaning task according to the performance of the hardware resources of the deployment environment.

[0038] Specifically, generating the data acquisition task list includes: First, acquiring the hardware performance of the first number of formal environments and the second number of grayscale environments, respectively. Simultaneously, based on historical data acquisition task execution records, the data volume of the target data table included in the target data cleaning task, and preset data acquisition channel setting rules, acquiring the formal environment and / or the grayscale environment that meets the hardware performance requirements as the data acquisition channel environment. The historical data acquisition task execution records contain historical time consumption records for acquiring different data from the formal environment and the grayscale environment. The formal environment or grayscale environment corresponding to the historical time consumption record with the shorter time consumption is selected. It should also be noted that the data in the historical time consumption records and the data in the target data table included in the target data cleaning task originate from the same data source. In addition to referencing the historical time consumption records, it is also necessary to consider the growth rate of the data in the target data table. That is, it is also necessary to consider the amount of data in the target data table at the current time relative to the data volume at the time point of the historical time consumption records. For example, if the amount of data in the target data table at the current time has increased significantly compared to the time point of the historical time consumption records, then the data acquisition channel for the target data table should be switched from the formal environment to the grayscale environment to avoid excessive access pressure on the formal environment. Then, the access switch for the data acquisition channel environment is turned on, and the access switches for other environments not belonging to the data acquisition channel environment, such as the formal environment and / or the grayscale environment, are turned off. The function and beneficial effects of the access switches will not be elaborated here. Simultaneously, the target data cleaning task is divided into corresponding data acquisition sub-tasks according to the preset data acquisition channel setting rules. The number of data acquisition sub-tasks can be determined based on… Figure 1 The system configures the number of data acquisition subtasks based on the remaining hardware resources of the computer equipment. For example, when the computer equipment has abundant hardware resources, a larger number of data acquisition subtasks can be set, while when the computer equipment has scarce hardware resources, a smaller number of data acquisition subtasks can be set. Furthermore, when the computer equipment has severely scarce hardware resources, the step of splitting the target data cleaning task into corresponding data acquisition subtasks according to the preset data acquisition channel setting rules enters a polling state waiting for the computer equipment's hardware resources to recover. Finally, the data acquisition subtasks are associated with the data acquisition channel environment and added to the data processing task list.

[0039] S104. Execute the data acquisition tasks in the data acquisition task list according to the preset data acquisition rules to obtain the target data table, and add the obtained target data table to the set of data tables to be processed.

[0040] Further, the step of executing data acquisition tasks in the data acquisition task list according to preset data acquisition rules to obtain the target data table includes: First, obtaining the earliest data operation time in the data operation time index as the initial time. Using the earliest data operation time in the data operation time index as the initial time means acquiring the data furthest from the current time first. Alternatively, the largest data operation time in the data operation time index can be used as the initial time, so that data acquisition starts from the current time. Different initial time settings affect the order in which data is acquired from the target data table. However, in practical applications, there are scenarios where data needs to be acquired according to time order. For example, only data cleaning may be performed on purchase records from the first three years in a user purchase data table, while purchase records older than three years may not be processed or may undergo other processing. Simultaneously, the initial time is used as the start time of a first time range, and the sum of the start time of the first time range and a preset time span is used as the end time of the first time range. Then, data with data operation times within the first time range is acquired from the target data table. Finally, after each successful data acquisition, the start and end times of the first time range are simultaneously added to the preset time to update the first time. Then, the updated first time range is used to acquire data from the target data table until all data in the target data table has been acquired. That is, after each successful acquisition of data from the target data table according to the first time range, the time span of the first time range is increased or decreased by the preset time to ensure that the data in the target data table acquired later according to the updated first time range is not identical to the data in the target data table acquired earlier according to the unupdated first time range.

[0041] Furthermore, after retrieving data from the target data table within the first time range, the process further includes: first, recording the first time spent retrieving data from the target data table within the first time range. Then, determining whether the first time spent is within a preset time adjustment range; if so, adjusting the preset time span according to preset time adjustment rules. Although the previous steps have improved the efficiency of retrieving data from the target data table through partitioning, data indexing, associating with the data acquisition channel environment, and setting access switches, other problems may still exist in the actual data acquisition process. For example, if the data acquisition channel environment is the formal environment and data is being retrieved from the target data table in the formal environment, a surge in user access to the formal environment may further reduce the remaining hardware resources, leading to an increase in the first time spent retrieving data from the target data table. This indicates that the target database containing the target data table is under certain access pressure. However, it should be noted that data cleaning tasks are generally of lower priority than actual business tasks. In this case, to avoid putting excessive access pressure on the database system and to allow other business tasks access to data, the preset time needs to be further adjusted. Specifically, it is determined whether the first time consumption is within the preset time adjustment range. If it is, the preset time span is adjusted according to the preset time adjustment rules. Furthermore, in addition to increasing the preset time when the first time consumption increases, the preset time can also be decreased when the first time consumption decreases. This is because the access pressure of the target database will not be consistently at its peak, and a decrease in the access pressure of the target database will lead to a decrease in the first time consumption. Thus, by keeping the access pressure of the target database within a reasonable range, the preset time can be reduced, thereby improving the efficiency of data retrieval from the target data table in the target database.

[0042] S105. Generate a data processing task list, which is generated based on the different processing methods of the target data table in the target data cleaning task.

[0043] Unlike traditional data cleaning methods that process each data table individually, this embodiment generates different data processing task lists for different processing methods. Each data processing task list includes at least one target data table used for a specific data cleaning method. Furthermore, the target data table is included in at least one of the data processing task lists. For example, one data processing task in one list might round three-decimal-place numbers in the target data table to two decimal places, while another might convert the format of all date type fields. Since the user data list contains both decimal and date type data fields, it exists in both data processing tasks within the data processing task lists.

[0044] S106. Execute the data processing tasks in the data processing task list, and add the processed target data table as the data table to be updated to the data table set.

[0045] Furthermore, after all data processing tasks in the data processing task list have been executed, the execution results of the data processing tasks are also verified. That is, it is checked whether all the data tables in the set of data tables to be updated have completed the data processing tasks according to the data processing requirements. If they have not been completed or have only been partially completed, the data processing tasks need to be re-executed.

[0046] S107. Obtain the data table to be updated from the set of data tables to be updated according to the preset data update rules, and update the data table to be updated to all deployment environments of the target system.

[0047] The process of updating the data table to be updated to the deployment environment of the target system is similar to the process of obtaining data from the target database in the target data table from the deployment environment of the target system. On the one hand, data is obtained by querying the database, and on the other hand, the updated data is written to the database. Both of these processes put access pressure on the target database. Therefore, the design of the preset data update rule refers to the aforementioned data acquisition design rule to keep the target database within a reasonable access pressure range when updating the data table to be updated to the deployment environment of the target system. Specific technical details will not be elaborated here.

[0048] The data cleaning method provided in this embodiment involves: acquiring target data cleaning tasks that include a target database of a target system and target data tables in the target database; acquiring the number of deployment environments of the target system; splitting the target data cleaning tasks into a data acquisition task list based on the number of deployment environments and their hardware resource performance; executing the data acquisition tasks according to preset data acquisition rules and adding them to a set of data tables to be processed; generating a data processing task list based on the target data tables in the target data cleaning tasks; executing the data processing tasks in the data processing task list, using the processed target data tables as data tables to be updated; acquiring the data tables to be updated according to preset data update rules, and updating the data tables to be updated in the deployment environment of the target system. This method not only reduces the access pressure on the database during data cleaning but also avoids database congestion and redundant processing during the data cleaning process, while further improving the efficiency of data cleaning.

[0049] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0050] In one embodiment, a data cleaning apparatus 100 is provided, which corresponds one-to-one with the data cleaning methods described in the above embodiments. For example... Figure 3 As shown, the data cleaning device 100 includes a data cleaning task acquisition module 11, a deployment environment acquisition module 12, a data acquisition task module 13, a data acquisition execution module 14, a data processing task module 15, a data processing execution module 16, and a data update module 17. Detailed descriptions of each functional module are as follows:

[0051] The data cleaning task acquisition module 11 is used to acquire the target data cleaning task, which includes the target database of the target system and the target data table of the target database.

[0052] The deployment environment acquisition module 12 is used to acquire all deployment environments of the target system, and each deployment environment contains the target database and the target data table;

[0053] The data acquisition task module 13 is used to generate a data acquisition task list, which is obtained by splitting the target data cleaning task according to the hardware resource performance of the deployment environment.

[0054] The data acquisition execution module 14 is used to execute the data acquisition tasks in the data acquisition task list according to the preset data acquisition rules to obtain the target data table, and add the obtained target data table to the set of data tables to be processed;

[0055] Data processing task module 15 is used to generate a data processing task list, which is generated based on different processing methods of the target data table in the target data cleaning task.

[0056] Data processing execution module 16 is used to execute data processing tasks in the data processing task list and add the processed target data table as a data table to be updated to the data table set.

[0057] The data update module 17 is used to obtain the data table to be updated from the set of data tables to be updated according to the preset data update rules, and update the data table to be updated to all deployment environments of the target system.

[0058] Furthermore, the data cleaning task acquisition module 11 also includes:

[0059] The dataset acquisition submodule is used to acquire the dataset contained in the target system, wherein the dataset includes the target database and the target data table of the target database;

[0060] The data partitioning processing submodule is used to partition the dataset according to the data update time;

[0061] The data index generation submodule is used to generate a data operation time index for the dataset after it has been partitioned.

[0062] Furthermore, the deployment environment acquisition module 12 also includes:

[0063] The environment quantity acquisition submodule is used to acquire the first number of formal environments of the target system and the second number of grayscale environments of the target system, wherein the deployment environment includes the formal environment and the grayscale environment;

[0064] The access frequency setting submodule is used to set a first access frequency based on the hardware resource performance of the formal environment and a second access frequency based on the hardware resource performance of the grayscale environment.

[0065] The first access switch setting submodule is used to configure the access switches for the formal environment and the grayscale environment. The access switches are used to determine whether data in the target data table can be obtained from the formal environment or the grayscale environment corresponding to the access switch.

[0066] Furthermore, the data acquisition task module 13 also includes:

[0067] The hardware performance acquisition submodule is used to acquire the hardware performance of the first number of formal environments and the second number of grayscale environments, respectively.

[0068] The data acquisition channel environment submodule is used to acquire the formal environment and / or the grayscale environment that meet the hardware performance requirements as the data acquisition channel environment based on historical data acquisition task execution records, the amount of data in the target data table included in the target data cleaning task, and preset data acquisition channel setting rules.

[0069] The second access switch setting submodule is used to turn on the access switch of the data acquisition channel environment and turn off the access switches of other formal environments and / or grayscale environments that do not belong to the data acquisition channel environment.

[0070] The data cleaning task splitting sub-module is used to split the target data cleaning task into corresponding data acquisition sub-tasks according to the preset data acquisition channel setting rules;

[0071] The task and channel association submodule is used to associate the data acquisition subtask with the data acquisition channel environment and add the data acquisition subtask to the data processing task list.

[0072] Furthermore, the data acquisition and execution module 14 also includes:

[0073] The initial time submodule is used to obtain the earliest data operation time in the data operation time index as the initial time;

[0074] The first time range submodule is used to take the initial time as the start time of the first time range and the sum of the start time of the first time range and the preset time span as the end time of the first time range.

[0075] The first data acquisition submodule is used to acquire data from the target data table whose data operation time is within the first time range;

[0076] The data acquisition loop submodule is used to update the first time range by adding the preset time to the start and end times of the first time range each time data is successfully acquired, and then use the updated first time range to acquire data in the target data table until all data in the target data table has been acquired.

[0077] Furthermore, the first data acquisition submodule also includes:

[0078] The first time consumption recording subunit is used to record the first time consumption of data obtained from the target data table within the first time range;

[0079] The preset time span adjustment subunit is used to determine whether the first time consumption is within the preset time adjustment range. If it is, the preset time span is adjusted according to the preset time adjustment rules.

[0080] The terms "first" and "second" in the above-mentioned modules / units are only used to distinguish different modules / units and are not intended to specify which module / unit has a higher priority or any other limiting meaning. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The module divisions appearing in this application are merely logical divisions; in actual applications, different division methods may be used.

[0081] Specific limitations regarding the data cleaning device can be found in the limitations of the data cleaning method described above, and will not be repeated here. Each module in the aforementioned data cleaning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0082] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data involved in the data cleaning method. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data cleaning method.

[0083] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the data cleaning method described in the above embodiments, for example...Figure 2 The steps S101 to S107 shown, as well as other extensions and related steps of the method, are examples. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit of the data cleaning apparatus in the above embodiments, for example... Figure 3 The functions of modules 11 to 17 are shown. To avoid repetition, they will not be described again here.

[0084] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting various parts of the computer device via various interfaces and lines.

[0085] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, video data, etc.).

[0086] The memory can be integrated into the processor or it can be set up separately from the processor.

[0087] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the steps of the data cleaning method described in the above embodiments, for example... Figure 2 The steps S101 to S107 shown, as well as other extensions and related steps of the method, are considered. Alternatively, when a computer program is executed by a processor, it implements the functions of each module / unit of the data cleaning apparatus in the above embodiments, for example... Figure 3 The functions of modules 11 to 17 are shown. To avoid repetition, they will not be described again here.

[0088] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0089] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0090] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data cleaning method, characterized in that, The method comprises the following steps: obtaining a target data cleaning task, wherein the data cleaning task comprises a target database of a target system and a target data table of the target database; obtaining all deployment environments of the target system, wherein each of the deployment environments comprises the target database and the target data table; generating a data acquisition task list, wherein the data acquisition task list is obtained by splitting the target data cleaning task according to the hardware resource performance of the obtained deployment environments; performing data acquisition tasks in the data acquisition task list according to a preset data acquisition rule to obtain the target data table, and adding the obtained target data table to a set of to-be-processed data tables; generating a data processing task list, wherein the data processing task list is generated according to different processing modes of the target data table in the target data cleaning task; performing data processing tasks in the data processing task list, and adding the processed target data table to a set of to-be-updated data tables as a to-be-updated data table; obtaining the to-be-updated data table in the set of to-be-updated data tables according to a preset data update rule, and updating the to-be-updated data table to all deployment environments of the target system; wherein, obtaining all deployment environments of the target system comprises: obtaining a first environment quantity of a formal environment of the target system and a second environment quantity of a gray environment of the target system, wherein the deployment environments comprise the formal environment and the gray environment; setting a first access frequency according to the hardware resource performance of the formal environment, and setting a second access frequency according to the hardware resource performance of the gray environment; configuring access switches of the formal environment and the gray environment, wherein the access switches are used to determine whether data in the target data table can be obtained from the formal environment or the gray environment corresponding to the access switches; wherein, generating the data acquisition task list comprises: obtaining the hardware performance of the first environment quantity of the formal environment and the hardware performance of the second environment quantity of the gray environment respectively; obtaining the formal environment and / or the gray environment satisfying the hardware performance as a data acquisition channel environment according to historical data acquisition task execution records, the data volume of the target data table contained in the target data cleaning task, and a preset data acquisition channel setting rule; opening the access switches of the data acquisition channel environment, and closing the access switches of the formal environment and / or the gray environment which do not belong to the data acquisition channel environment; splitting the target data cleaning task into data acquisition subtasks according to the preset data acquisition channel setting rule; associating the data acquisition subtasks with the data acquisition channel environment, and adding the data acquisition subtasks to the data processing task list.

2. The data cleaning method of claim 1, wherein, Before obtaining the target data cleaning task, the method further comprises the following steps: obtaining a data set contained in the target system, wherein the data set comprises the target database and the target data table of the target database; performing partition processing on the data set according to data update time. generating a data operation time index of the data set processed by partitioning.

3. The data cleaning method of claim 2, wherein The executing the data acquisition tasks in the data acquisition task list according to the preset data acquisition rule to acquire the target data table comprises: acquiring the earliest data operation time in the data operation time index as an initial time; taking the initial time as the start time of a first time range, and taking the sum of the start time of the first time range and a preset time span as the end time of the first time range; acquiring data with data operation time within the first time range from the target data table; updating the start time and the end time of the first time range by adding the preset time to the start time and the end time of the first time range respectively each time data is successfully acquired, and acquiring data in the target data table by using the updated first time range until all data in the target data table is acquired.

4. The data cleaning method of claim 3, wherein, The method further comprises: recording a first time consumption of acquiring data with data operation time within the first time range from the target data table; judging whether the first time consumption is within a preset time adjustment range, and adjusting the preset time span according to a preset time adjustment rule if the first time consumption is within the preset time adjustment range.

5. A data cleaning apparatus characterized by comprising: The method comprises: a data cleaning task acquisition module configured to acquire a target data cleaning task, the target data cleaning task comprising a target database of a target system and a target data table of the target database; a deployment environment acquisition module configured to acquire all deployment environments of the target system, each of the deployment environments comprising the target database and the target data table; a data acquisition task module configured to generate a data acquisition task list, the data acquisition task list being obtained by splitting the target data cleaning task according to hardware resource performance of the acquired deployment environments; a data acquisition execution module configured to execute data acquisition tasks in the data acquisition task list according to a preset data acquisition rule to acquire the target data table, and add the acquired target data table to a to-be-processed data table set; a data processing task module configured to generate a data processing task list, the data processing task list being generated according to different processing manners of the target data table in the target data cleaning task; a data processing execution module configured to execute data processing tasks in the data processing task list, and add the processed target data table to a to-be-updated data table set as a to-be-updated data table; a data updating module configured to acquire the to-be-updated data table in the to-be-updated data table set according to a preset data updating rule, and update the to-be-updated data table to all deployment environments of the target system. The deployment environment obtaining module further includes: an environment quantity obtaining submodule, configured to obtain a first environment quantity of formal environments of the target system and a second environment quantity of gray-scale environments of the target system, wherein the deployment environment includes the formal environments and the gray-scale environments; an access frequency setting submodule, configured to set a first access frequency according to hardware resource performance of the formal environments and a second access frequency according to hardware resource performance of the gray-scale environments; and a first access switch setting submodule, configured to configure access switches of the formal environments and the gray-scale environments, wherein the access switches are configured to determine whether data in the target data table can be obtained from the formal environments or the gray-scale environments corresponding to the access switches. The data obtaining task module further includes: a hardware performance obtaining submodule, configured to obtain hardware performance of the formal environments of the first environment quantity and hardware performance of the gray-scale environments of the second environment quantity respectively; a data obtaining channel environment submodule, configured to obtain the formal environments and / or the gray-scale environments satisfying hardware performance as data obtaining channel environments according to historical data obtaining task execution records, a data amount of the target data table included in the target data cleaning task, and a preset data obtaining channel setting rule; a second access switch setting submodule, configured to open the access switches of the data obtaining channel environments and close the access switches of the formal environments and / or the gray-scale environments that do not belong to the data obtaining channel environments; a data cleaning task splitting submodule, configured to split the target data cleaning task into data obtaining subtasks according to the preset data obtaining channel setting rule; and a task and channel association submodule, configured to associate the data obtaining subtasks with the data obtaining channel environments and add the data obtaining subtasks to the data processing task list.

6. The data cleaning device according to claim 5, wherein a data set obtaining submodule, configured to obtain a data set included in the target system, wherein the data set includes the target database and a target data table of the target database; a data partition processing submodule, configured to perform partition processing on the data set according to data update time; a data index generating submodule, configured to generate a data operation time index of the data set after being partition processed.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the data cleaning method according to any one of claims 1 to 4.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the steps of the data cleaning method according to any one of claims 1 to 4.

Images