Data cleaning method and device, equipment, storage medium and program product

Abstract

The invention discloses a data cleaning method and device, equipment, a storage medium and a program product. The invention relates to the technical field of data processing. The method comprises the following steps: according to a target cleaning mode matched with each to-be-cleaned data table, determining a candidate server set corresponding to the to-be-cleaned data table from a data cleaning server cluster; according to the current state information of each candidate server in the candidate server set corresponding to each to-be-cleaned data table, determining a target server for performing data cleaning on the to-be-cleaned data table; sending each to-be-cleaned data table to a corresponding target server; and obtaining the cleaned data tables of the target servers, and associating the m cleaned data tables according to association rules matched with the m to-be-cleaned data tables to obtain n associated data tables. According to the data cleaning method, the data cleaning efficiency and the cleaning flexibility are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data cleaning method, device, equipment, storage medium and program product. Background Art

[0002] Data is the foundation and core of big data engineering. When using data for model training or data mining, the quality of data is very important. Data cleaning is a key step to ensure data quality. Data cleaning can find and correct errors in data to reduce noise and redundancy in data and improve data accuracy and reliability.

[0003] Currently, data cleaning can be automated by writing a data cleaning program. For example, if a user needs to deduplicate data, a deduplication program can be written, and then the data to be cleaned can be input into the deduplication program and run to deduplicate data.

[0004] However, in the above process, when performing data cleaning, the user is required to write a corresponding data cleaning program, which results in low cleaning efficiency and low cleaning flexibility. Summary of the invention

[0005] The present application provides a data cleaning method, device, equipment, storage medium and program product to solve the technical problems of low efficiency and low cleaning flexibility in data cleaning in related technologies.

[0006] In a first aspect, the present application provides a data cleaning method, comprising:

[0007] According to the target cleaning method matched with each data table to be cleaned, determine a set of candidate servers corresponding to the data table to be cleaned from the data cleaning server cluster; wherein the servers in the candidate server set can all implement the target cleaning method, the number of the data tables to be cleaned is m, m is an integer greater than 1, and the data cleaning server cluster can implement at least two cleaning methods;

[0008] Determining a target server for data cleaning of each data table to be cleaned according to current status information of each candidate server in a candidate server set corresponding to each data table to be cleaned;

[0009] Sending each of the data tables to be cleaned to a corresponding target server, so that the target server performs data cleaning on the data table to be cleaned according to a target cleaning method corresponding to the data table to be cleaned;

[0010] The cleaned data tables of each target server are obtained, and according to the association rules matching the m data tables to be cleaned, the m cleaned data tables are associated to obtain n associated data tables; wherein n is an integer less than m.

[0011] In a second aspect, the present application provides a data cleaning system, the system comprising:

[0012] A data cleaning trigger device and a data cleaning server cluster connected to the data cleaning trigger device; wherein the data cleaning server cluster can implement at least two cleaning modes;

[0013] The data cleaning trigger device is used in the data cleaning method described in the first aspect above;

[0014] The target server in the data cleaning server cluster is used to receive the corresponding data table to be cleaned, and perform data cleaning on the received data table to be cleaned according to the target cleaning method corresponding to the received data table to be cleaned.

[0015] In a third aspect, the present application provides a data cleaning device, the device comprising:

[0016] A first determination module is used to determine, from a data cleaning server cluster, a set of candidate servers corresponding to each data table to be cleaned according to a target cleaning method matched with the data table to be cleaned; wherein all servers in the candidate server set can implement the target cleaning method, the number of the data tables to be cleaned is m, m is an integer greater than 1, and the data cleaning server cluster can implement at least two cleaning methods;

[0017] A second determination module is used to determine a target server for data cleaning of each data table to be cleaned according to current status information of each candidate server in a candidate server set corresponding to each data table to be cleaned;

[0018] A sending module, used for sending each of the data tables to be cleaned to a corresponding target server, so that the target server performs data cleaning on the data table to be cleaned according to a target cleaning method corresponding to the data table to be cleaned;

[0019] The acquisition association module is used to obtain the cleaned data tables of each target server, and associate the m cleaned data tables according to the association rules matching the m data tables to be cleaned to obtain n associated data tables; wherein n is an integer less than m.

[0020] In a fourth aspect, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a data cleaning method as described in any one of the present applications is implemented.

[0021] In a fifth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, it implements any data cleaning method described in the present application.

[0022] In a sixth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements a data cleaning method as described in any one of the present applications.

[0023] The technical solution provided by the present application includes: determining a set of candidate servers corresponding to the data table to be cleaned from a data cleaning server cluster according to a target cleaning method matched with each data table to be cleaned, wherein the servers in the candidate server set can all implement the target cleaning method, the number of data tables to be cleaned is m, m is an integer greater than 1, and the data cleaning server cluster can implement at least two cleaning methods; determining a target server for data cleaning of the data table to be cleaned according to current status information of each candidate server in the candidate server set corresponding to each data table to be cleaned; sending each data table to be cleaned to the corresponding target server, so that the target server cleans the data table to be cleaned according to the target cleaning method corresponding to the data table to be cleaned; obtaining the cleaned data tables of each target server, and associating the m cleaned data tables according to the association rules matching the m data tables to be cleaned, to obtain n associated data tables, wherein n is an integer less than m. The invention has the following technical effects: on the one hand, it realizes data cleaning of a data table to be cleaned according to a target cleaning method matching the data table to be cleaned, without the need for users to write data cleaning programs, and can flexibly clean according to the target cleaning method, thereby improving cleaning efficiency and cleaning flexibility; on the other hand, distributed data cleaning is realized through target servers in a data cleaning server cluster, thereby further improving cleaning efficiency; on the other hand, the cleaned data tables are associated according to association rules to obtain associated data tables, thereby facilitating subsequent businesses to use these cleaned data tables, thereby improving business operation efficiency and success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A schematic diagram of an application scenario of the data cleaning method provided in an embodiment of the present application;

[0026] Figure 2 A flowchart of a data cleaning method provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a cleaning configuration interface in a data cleaning method provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of determining a target cleaning method in a data cleaning method provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of a candidate server set and a target server in an embodiment of the present application;

[0030] Figure 6 A flowchart of another data cleaning method provided in an embodiment of the present application;

[0031] Figure 7 A flowchart of another data cleaning method provided in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of a data table sequence and a network rate sequence in an embodiment of the present application;

[0033] Fig. 9 A schematic diagram of another data table sequence and a network rate sequence in an embodiment of the present application;

[0034] Fig.10 A flowchart of another data cleaning method provided in an embodiment of the present application;

[0035] Fig.11 A schematic diagram of interaction between target servers in an embodiment of the present application;

[0036] Fig.12 A schematic diagram of the structure of a data cleaning system provided in an embodiment of the present application;

[0037] Fig.13 A schematic diagram of the structure of a data cleaning device provided in an embodiment of the present application;

[0038] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0040] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. At the same time, in the description of this application, the terms "first", "second", "candidate" and "target" are only used to distinguish the description and cannot be understood as indicating or implying relative importance. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. The purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0041] Figure 1 Schematic diagram of an application scenario of the data cleaning method provided in the embodiment of the present application. Figure 1 As shown, the data cleaning method provided in this embodiment can be applied to Figure 1 In the electronic device 11 shown. The electronic device 11 in this embodiment can be a computer device or a user terminal, etc. The user terminal in this embodiment can be a mobile phone, a personal computer device or a tablet computer, etc. The electronic device 11 can be connected to the data cleaning server cluster 12. The data cleaning server cluster 12 includes at least two servers 1201. Figure 1 In the example, the data cleaning server cluster 12 includes five servers 1201. The data cleaning server cluster 12 can implement at least two cleaning methods. The electronic device 11 in this embodiment interacts with the target server in the data cleaning server cluster 12 by executing the data cleaning method provided in this embodiment, and implements data cleaning of the data table to be cleaned according to the target cleaning method matching the data table to be cleaned, without the need for the user to write a data cleaning program, and can implement flexible cleaning according to the target cleaning method, thereby improving cleaning efficiency and cleaning flexibility.

[0042] The data cleaning method of the present application is further described in detail below through several specific embodiments.

[0043] Figure 2 The present invention provides a flow chart of a data cleaning method according to an embodiment of the present invention. The method can be performed by a data cleaning device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. The electronic device here can be Figure 1 For example, the electronic device in this embodiment may be a computer device. Figure 2 As shown, the data cleaning method provided in this embodiment includes the following steps.

[0044] Step 201: According to the target cleaning mode matched with each data table to be cleaned, a set of candidate servers corresponding to the data table to be cleaned is determined from the data cleaning server cluster.

[0045] Among them, all servers in the candidate server set can implement the target cleaning method. The number of data tables to be cleaned is m, and m is an integer greater than 1. The data cleaning server cluster can implement at least two cleaning methods.

[0046] The data table to be cleaned in this embodiment may include the data to be cleaned and the fields corresponding to the data. The data table to be cleaned in this embodiment may be a data table generated during the operation of the business. Exemplarily, the data table to be cleaned in this embodiment may be a data table generated during the operation of financial software.

[0047] In this embodiment, distributed data cleaning can be implemented through a data cleaning server cluster to improve cleaning efficiency. The data cleaning server cluster in this embodiment can implement at least two cleaning methods. Optionally, the cleaning method in this embodiment may include at least two of the following: data verification, data deduplication, data padding, data format conversion, data grouping, data conversion, and data marking.

[0048] In this embodiment, each data table to be cleaned corresponds to a target cleaning method that matches it, so as to achieve personalized data cleaning for each data table to be cleaned. The target cleaning methods corresponding to multiple data tables to be cleaned can be the same, different, or partially the same. This is not limited in this embodiment. It can be understood that the target cleaning method in this embodiment is a cleaning method in at least two cleaning methods that can be implemented by the data cleaning server cluster.

[0049] The data cleaning server cluster in this embodiment includes multiple server sets. Servers in the same server set can implement the same cleaning method.

[0050] In step 201, for each data table to be cleaned, a candidate server set corresponding to the data table to be cleaned can be determined from the data cleaning server cluster according to the target cleaning method matched by the data table to be cleaned. The servers in the candidate server set in this embodiment can all implement the corresponding target cleaning method.

[0051] Optionally, for a data table to be cleaned, there may be multiple corresponding target cleaning methods, and the number of candidate server sets corresponding to the data table to be cleaned is the same as the number of target cleaning methods corresponding to the data table to be cleaned.

[0052] Optionally, the target cleaning modes matched by at least two data tables to be cleaned may be the same, and then the candidate server sets corresponding to the at least two data tables to be cleaned are the same.

[0053] In one implementation, before step 201, the method provided in this embodiment further includes the following steps: receiving a target cleaning method matched with each data table to be cleaned input by a user through a data cleaning configuration interface, wherein different data tables to be cleaned correspond to the same or different target cleaning methods.

[0054] In this implementation, the target cleaning method of each data table to be cleaned is determined by the user according to the needs, so that the data table to be cleaned can be flexibly processed according to the user's needs, thus meeting the actual needs of the user.

[0055] Figure 3 Schematic diagram of a cleaning configuration interface in a data cleaning method provided in an embodiment of the present application. Figure 3 As shown, the name and storage path of each data table to be cleaned are shown in the cleaning configuration interface 31 , and the user can select a target cleaning method that matches each data table to be cleaned through the cleaning method option 301 . Figure 3 In the example, the number of data tables to be cleaned is 3. Assume that for data table 312 to be cleaned, the target cleaning method input by the user that matches the data table 312 to be cleaned is data deduplication; for data table 313 to be cleaned, the target cleaning method input by the user that matches the data table 313 to be cleaned is data filling; for data table 314 to be cleaned, the target cleaning method input by the user that matches the data table 314 to be cleaned is data marking.

[0056] In another implementation, before step 201, the method provided in this embodiment also includes the following steps: obtaining m data tables to be cleaned; inputting each table to be cleaned into a pre-trained data feature analysis model to obtain data features of the data table to be cleaned output by the data feature analysis model; and determining a target cleaning method that matches the data table to be cleaned based on the data features of each data table to be cleaned.

[0057] In this implementation method, a data feature analysis model trained by big data is implemented to determine the data features of the data table to be cleaned, and then based on the data features of the data table to be cleaned, a target cleaning method that matches the data table to be cleaned is determined. The target cleaning method that matches the data table to be cleaned can be accurately and efficiently determined, thereby improving the accuracy of data cleaning.

[0058] Figure 4 A schematic diagram of determining a target cleaning method in a data cleaning method provided in an embodiment of the present application. Figure 4 As shown, each data table to be cleaned is input into the data feature analysis model 41 respectively, and the data features of the data table to be cleaned output by the data feature analysis model 41 are obtained.

[0059] Furthermore, according to the data features of each data table to be cleaned, determining a target cleaning method matching the data table to be cleaned includes at least one of the following seven implementation methods.

[0060] If the data characteristics indicate that there are erroneous data in the data table to be cleaned, the target cleaning method of the data table to be cleaned is determined to be data verification. The purpose of data verification is to correct or delete the erroneous data in the data table to be cleaned. The process of data verification can be: using statistical analysis methods to identify erroneous values ​​or outliers, such as determining values ​​with large deviations through deviation analysis, or identifying values ​​that do not comply with distribution or regression equations, and deleting or correcting the identified erroneous values ​​or outliers.

[0061] If the data characteristics indicate that duplicate data exists in the data table to be cleaned, it is determined that the target cleaning method of the data table to be cleaned is data deduplication. The purpose of data deduplication is to remove duplicate data in the data table to be cleaned. The process of data deduplication can be: detecting and removing duplicate data through sorting and predefined duplicate identification rules. Furthermore, the deleted data can be backed up to ensure the integrity of the original data.

[0062] If the data characteristics indicate that there are incomplete data in the data table to be cleaned, the target cleaning method of the data table to be cleaned is determined to be data filling. The purpose of data filling is to complete the incomplete data in the data table to be cleaned. The process of data filling can be: according to the importance level and missing ratio of the field, decide whether to delete the incomplete records or fill the missing fields.

[0063] If the data characteristics indicate that there is a data format in the data table to be cleaned that does not match the format indicated by the field, then the target cleaning method of the data table to be cleaned is determined to be data format conversion. The purpose of data format conversion is to convert data into data that matches the format indicated by the field corresponding to the data. The process of data format conversion can be: for information collected manually or filled in by users, clean data that does not conform to the prescribed format and content, such as data that does not conform to the format displayed for time, date, value, etc.

[0064] If the data characteristics indicate that there are clusterable data in the data table to be cleaned, then the target cleaning method of the data table to be cleaned is determined to be data grouping. The process of data grouping can be: grouping the same type of data into one group, and finally forming multiple groups of data.

[0065] If the data characteristics indicate that there are data with inconsistent units in the data table to be cleaned, the target cleaning method of the data table to be cleaned is determined to be data conversion. The purpose of data conversion is to convert the units of data in the same field into consistent units.

[0066] If the data characteristics indicate that there is data in the data table to be cleaned that does not match the field, then the target cleaning method of the data table to be cleaned is determined to be data tagging. The purpose of data tagging is to match the data to the corresponding field. The process of data tagging can be: processing the correlation between data to ensure the consistency and accuracy of data.

[0067] Step 202: Determine a target server for data cleaning of each data table to be cleaned according to the current status information of each candidate server in the candidate server set corresponding to each data table to be cleaned.

[0068] In this embodiment, in order to improve the efficiency of the data cleaning process, for each data table to be cleaned, the target server for data cleaning of the data table to be cleaned can be determined from the current status information of each candidate server in the candidate server set corresponding to the data table to be cleaned.

[0069] In one implementation, the current status information of the candidate server may include a network rate parameter, and the candidate server with the largest network rate parameter in the candidate server set corresponding to each data table to be cleaned may be determined as the target server for data cleaning of the data table to be cleaned.

[0070] In another implementation, the current status information of the candidate server may include network security parameters, and the candidate server with the largest network security parameter in the candidate server set corresponding to each data table to be cleaned may be determined as the target server for data cleansing of the data table to be cleaned.

[0071] In another implementation, the current status information of the candidate server may include processing performance parameters, and the candidate server with the largest processing performance parameter in the candidate server set corresponding to each data table to be cleaned may be determined as the target server for data cleaning of the data table to be cleaned.

[0072] It can be understood that in step 202, the target server determined according to the current status information of each candidate server can realize efficient data cleaning, thereby further improving the data cleaning efficiency.

[0073] Step 203: Send each data table to be cleaned to the corresponding target server, so that the target server performs data cleaning on the data table to be cleaned according to the target cleaning method corresponding to the data table to be cleaned.

[0074] After the target server is determined, each data table to be cleaned is sent to the corresponding target server in step 203. The sending method can be wired or wireless, and the specific sending method is determined according to the connection method between the electronic device and the target server.

[0075] After receiving the data table to be cleaned, the target server performs data cleaning on the data table to be cleaned according to the target cleaning method corresponding to the data table to be cleaned, that is, the cleaning method supported by the target server, to obtain a cleaned data table. The target server feeds the cleaned data table back to the electronic device.

[0076] Optionally, in order to improve security, the electronic device may encrypt the data table to be cleaned and send it to the corresponding target server. After receiving the ciphertext of the data table to be cleaned, the target server decrypts it and performs data cleaning.

[0077] It should be noted that, for a data table to be cleaned with multiple corresponding target cleaning methods, multiple target servers are determined, and the data table to be cleaned is sent to the corresponding target servers respectively in step 203. In a scenario where the candidate server sets corresponding to at least two data tables to be cleaned are the same, the target servers corresponding to the at least two data tables to be cleaned may also be the same, and in step 203, the at least two data tables to be cleaned are sent to the same target server.

[0078] Step 204: obtaining the cleaned data tables of each target server, and associating the m cleaned data tables according to the association rules matching the m data tables to be cleaned, to obtain n associated data tables.

[0079] Here, n is an integer less than m.

[0080] After obtaining the cleaned data tables sent by each target server, in order to facilitate the subsequent business to use these cleaned data tables to improve business operation efficiency and success rate, in this embodiment, the m cleaned data tables can be associated according to the association rules matching the m data tables to be cleaned to obtain n associated data tables.

[0081] The association rules in this embodiment are used to indicate the merging rules of m data tables to be cleaned. For example, it is assumed that there are three data tables to be cleaned: data table 31 to be cleaned, data table 32 to be cleaned, and data table 33 to be cleaned. Data table 31 to be cleaned includes field 311 and field 312. Data table 32 to be cleaned includes field 321, field 322, and field 323. Data table 33 to be cleaned includes field 331, field 332, and field 333. Then the association rule can be to merge field 321, field 322, and field 323 in data table 32 to be cleaned into data table 31 to be cleaned, and merge field 321 and field 322 in data table 32 to be cleaned into data table 33 to be cleaned. The cleaned data table corresponding to data table 31 to be cleaned is data table 31a, the cleaned data table corresponding to data table 32 to be cleaned is data table 31a, and the cleaned data table corresponding to data table 33 to be cleaned is data table 33a. The associated data table may be the data table 31a after merging fields and the data table 33a after merging fields.

[0082] Optionally, in this embodiment, the process of associating the m cleaned data tables according to the association rules matching the m to-be-cleaned data tables is the process of field merging.

[0083] In one implementation, before step 204, the data cleaning method provided in this embodiment further includes: receiving association rules input by a user that match the m data tables to be cleaned.

[0084] This implementation method enables the association of cleaned data tables according to user needs, further meets user needs and improves the flexibility of data cleaning.

[0085] In another implementation, before step 204, the data cleaning method provided in this embodiment further includes: extracting fields of m data tables to be cleaned, and determining association rules matching the m data tables to be cleaned according to the fields of the m data tables to be cleaned.

[0086] In this implementation, association rules matching the m data tables to be cleaned can be automatically determined based on the internal connections or characteristic information between the fields of the m data tables to be cleaned. For example, assume that the extracted fields of the data table A to be cleaned include field A1 and field A2, and the fields of the data table B to be cleaned include field B1 and field B2. Based on the characteristics of these four fields, it is analyzed that if these four fields are located in one table, it is more convenient for subsequent business to use the data in the table. Therefore, the determined association rule can be: merge field B1 and field B2 into the data table A to be cleaned.

[0087] The following is a specific example to illustrate the implementation process of the data cleaning method provided in this embodiment. Figure 5Schematic diagram of a candidate server set and a target server in an embodiment of the present application. Figure 5 As shown, it is assumed that there are two data tables to be cleaned: data table 51 to be cleaned and data table 52 to be cleaned. The target cleaning method matched by data table 51 to be cleaned is data deduplication, and the target cleaning method matched by data table 52 to be cleaned is data verification. Execute step 201, and assume that the candidate server set corresponding to data table 51 to be cleaned is determined to be candidate server set 53, and the candidate server set corresponding to data table 52 to be cleaned is determined to be candidate server set 54. Execute step 202, and assume that the target server determined from candidate server set 53 is server 531, and the target server determined from candidate server set 54 is server 541. Execute step 203, send data table 51 to be cleaned to server 531, and send data table 52 to be cleaned to server 541. Execute step 204, obtain the cleaned data table corresponding to data table 51 to be cleaned, and obtain the cleaned data table corresponding to data table 52 to be cleaned. Assuming that the association rule is to merge all fields in the cleaned data table corresponding to the to-be-cleaned data table 52 into the cleaned data table corresponding to the to-be-cleaned data table 51, the finally obtained association data table is the cleaned data table corresponding to the to-be-cleaned data table 51 after merging the fields.

[0088] In the related art, when performing data cleaning, programmers need to clarify the logical relationship between data services and write specific programs. The program reuse rate is low and subsequent maintenance is difficult. It can be seen that the data cleaning method provided in this embodiment decouples the logical relationship between programmers and data services, the program has strong versatility and high reuse rate, and subsequent maintenance is simple. In addition, the data cleaning method provided in this embodiment can flexibly determine the corresponding candidate server set from the data cleaning server cluster according to the target cleaning method of the data table to be cleaned, thereby realizing the free combination of target cleaning methods and having high flexibility during data cleaning.

[0089] The data cleaning method provided in this embodiment includes: determining a candidate server set corresponding to a data table to be cleaned from a data cleaning server cluster according to a target cleaning method matched with each data table to be cleaned, wherein the servers in the candidate server set can all implement the target cleaning method, the number of data tables to be cleaned is m, m is an integer greater than 1, and the data cleaning server cluster can implement at least two cleaning methods; determining a target server for data cleaning of the data table to be cleaned according to current status information of each candidate server in the candidate server set corresponding to each data table to be cleaned; sending each data table to be cleaned to a corresponding target server, so that the target server performs data cleaning on the data table to be cleaned according to the target cleaning method corresponding to the data table to be cleaned; obtaining a cleaned data table of each target server, and according to an association rule matching the m data tables to be cleaned, associating the m cleaned data tables to obtain n associated data tables, wherein n is an integer less than m. The invention has the following technical effects: on the one hand, it realizes data cleaning of a data table to be cleaned according to a target cleaning method matching the data table to be cleaned, without the need for users to write data cleaning programs, and can flexibly clean according to the target cleaning method, thereby improving cleaning efficiency and cleaning flexibility; on the other hand, distributed data cleaning is realized through target servers in a data cleaning server cluster, thereby further improving cleaning efficiency; on the other hand, the cleaned data tables are associated according to association rules to obtain associated data tables, thereby facilitating subsequent businesses to use these cleaned data tables, thereby improving business operation efficiency and success rate.

[0090] Figure 6 A flow chart of another data cleaning method provided in an embodiment of the present application. Figure 2 Based on the illustrated embodiment and various optional implementations, the steps after obtaining the associated data table are described in detail. Figure 6 As shown, the data cleaning method provided in this embodiment includes the following steps.

[0091] Step 601: According to the target cleaning mode matched with each data table to be cleaned, a set of candidate servers corresponding to the data table to be cleaned is determined from the data cleaning server cluster.

[0092] Among them, all servers in the candidate server set can implement the target cleaning method. The number of data tables to be cleaned is m, and m is an integer greater than 1. The data cleaning server cluster can implement at least two cleaning methods.

[0093] Step 602: Determine a target server for data cleaning of each data table to be cleaned according to the current status information of each candidate server in the candidate server set corresponding to each data table to be cleaned.

[0094] Step 603: Send each data table to be cleaned to the corresponding target server, so that the target server performs data cleaning on the data table to be cleaned according to the target cleaning method corresponding to the data table to be cleaned.

[0095] Step 604: Obtain the cleaned data tables of each target server, and associate the m cleaned data tables according to the association rules matching the m data tables to be cleaned, to obtain n associated data tables.

[0096] Here, n is an integer less than m.

[0097] The implementation process and technical principle of step 601 and step 201, step 602 and step 202, step 603 and step 203, and step 604 and step 204 are similar, and will not be repeated here.

[0098] Step 605: Obtain field conversion rules that match the m data tables to be cleaned.

[0099] The field conversion rule is used to indicate that the original fields in the associated data table are converted into standard fields.

[0100] The data cleaning method provided in this embodiment can also convert certain fields in the associated data table after obtaining the associated data table in order to further meet subsequent business needs.

[0101] In step 605, field conversion rules matching the m data tables to be cleaned can be obtained according to actual business needs. The field conversion rules in this embodiment are used to characterize the fields to be converted in the n associated data tables and the mapping relationship between the fields to be converted and the standard fields. The standard fields in this embodiment refer to fields that meet subsequent business needs.

[0102] Step 606: According to the field conversion rule, determine the fields to be converted in the n associated data tables.

[0103] Step 607: According to the field conversion rules, the fields to be converted in the n associated data tables are converted into corresponding standard fields, and the data corresponding to the fields to be converted are converted into data in a data format that conforms to the data format specified by the standard fields corresponding to the fields to be converted, to obtain n target data tables.

[0104] In step 606, the fields to be converted in the n associated data tables may be determined from the field conversion rules.

[0105] In step 607, the fields to be converted in the n associated data tables are converted into corresponding standard fields according to the fields to be converted in the field conversion rules and the mapping relationship between the fields to be converted and the standard fields, thereby obtaining n target data tables. At the same time, in step 607, the data corresponding to the fields to be converted are converted into data in a data format that conforms to the data format specified by the standard fields corresponding to the fields to be converted, so that the fields match the data corresponding to the fields.

[0106] For example, the field conversion rule may be to convert a field "time (seconds)" in the associated data table into a field "time (milliseconds)". In step 606, the field "time (seconds)" to be converted is determined. In step 607, the field "time (seconds)" is converted into "time (milliseconds)", and the data corresponding to "time (seconds)" is converted into data in the data format specified by "time (milliseconds)". Assuming that the data corresponding to "time (seconds)" is 1, the converted data in the data format specified by "time (milliseconds)" is 1000.

[0107] The data cleaning method provided in this embodiment obtains field conversion rules matching m data tables to be cleaned, determines the fields to be converted in n associated data tables according to the field conversion rules, converts the fields to be converted in the n associated data tables into corresponding standard fields according to the field conversion rules, obtains n target data tables, and converts the data corresponding to the fields to be converted into data in a data format that complies with the data format specified by the standard fields corresponding to the fields to be converted, so that the determined target data tables further meet subsequent business needs and improve business operation efficiency and success rate.

[0108] Figure 7 A flowchart of another data cleaning method provided in an embodiment of the present application is provided. Figure 2 , Figure 6 Based on the illustrated embodiment and various optional implementations, the steps of determining the target server are described in detail. Figure 7 As shown, the data cleaning method provided in this embodiment includes the following steps.

[0109] Step 701: According to the target cleaning mode matched with each data table to be cleaned, a set of candidate servers corresponding to the data table to be cleaned is determined from the data cleaning server cluster.

[0110] Among them, all servers in the candidate server set can implement the target cleaning method. The number of data tables to be cleaned is m, and m is an integer greater than 1. The data cleaning server cluster can implement at least two cleaning methods.

[0111] The implementation process and technical principle of step 701 are similar to those of step 201 and will not be repeated here.

[0112] Optionally, the current state information of the candidate server in this embodiment includes: network rate parameters, network security parameters and processing performance parameters. In this implementation, according to the current state information of each candidate server in the candidate server set corresponding to each of the data tables to be cleaned, the implementation method of determining the target server for data cleaning of the data table to be cleaned includes the following steps 702 to 704.

[0113] Step 702: For each data table to be cleaned, determine as a first intermediate server a candidate server whose corresponding network security parameter is greater than a preset network security threshold among the candidate servers in the candidate server set corresponding to the data table to be cleaned.

[0114] In this embodiment, in the process of determining the target server, the candidate servers in the candidate server set are first screened from the security dimension to determine the first intermediate server in the candidate server set that meets the security requirements.

[0115] The network security parameter in this embodiment may be a value obtained by quantifying the network environment of the candidate server. The process of quantifying the network environment of the candidate server may be to input the network environment parameter into the big data model, and to quantify the network security parameter corresponding to the candidate server by the big data model. Alternatively, the process of quantifying the network environment of the candidate server may be a value obtained by quantifying the network environment of the candidate server based on a network security algorithm.

[0116] The preset network security threshold in this embodiment refers to a predetermined minimum quantitative index used to characterize network security. If the network security parameter of a candidate server is less than or equal to the preset network security threshold, it means that the candidate server is in an unsafe network environment and is not suitable for data cleaning.

[0117] Step 703: among the first intermediate servers, the first intermediate server whose corresponding processing performance parameter is greater than a preset processing performance threshold is determined as a second intermediate server.

[0118] After the candidate servers are screened from the security dimension, in step 703, the first intermediate servers are screened from the processing performance dimension to determine a second intermediate server that meets the processing performance requirement among the first intermediate servers.

[0119] Optionally, the processing performance parameter of the first intermediate server in this embodiment may be a value obtained by quantifying at least one of the following: processor utilization, computing speed, memory usage, and remaining memory resources.

[0120] The preset processing performance threshold in this embodiment refers to the pre-determined minimum quantitative index used to characterize the processing performance of the server. If the processing performance parameter of a first intermediate server is less than or equal to the preset processing performance threshold, it means that the processing performance of the first intermediate server is insufficient and is not suitable for data cleaning.

[0121] Step 704: Determine a target server for cleaning the data table to be cleaned according to the network rate parameters of each second intermediate server.

[0122] After the first intermediate servers are screened from the processing performance dimension, in step 704, the second intermediate servers are screened from the network rate dimension to determine the target server in the second intermediate servers that meets the network rate requirement.

[0123] In one implementation, step 704 specifically includes the following steps: determining, among the second intermediate servers, the second intermediate server with the largest corresponding network rate parameter as the target server.

[0124] In this implementation, the candidate server set corresponds to at least one data table to be cleaned, and the target servers corresponding to the at least one data table to be cleaned are the same.

[0125] In another implementation, the first candidate server set corresponds to at least two first data tables to be cleaned. Step 704 specifically includes the following steps 7041 to 7044. In this embodiment, the first data table to be cleaned refers to a data table to be cleaned with the same matching target cleaning method among the m data tables to be cleaned.

[0126] Step 7041: sort the identifiers of at least two first data tables to be cleaned in descending order of data volume to obtain a data table sequence of the data tables to be cleaned.

[0127] Step 7042: sort the identifiers of the second intermediate servers corresponding to the first candidate server set in descending order according to the corresponding network rates, and obtain a network rate sequence of the second intermediate servers corresponding to the first candidate server set.

[0128] Step 7043: If the number r of first data tables to be cleaned is less than or equal to the number t of each second intermediate server corresponding to the first candidate server set, then determine the second intermediate server corresponding to each first data table to be cleaned according to the identifiers of the first r second intermediate servers in the network rate sequence and the data table sequence, and determine the second intermediate server corresponding to the first data table to be cleaned as the target server corresponding to the first data table to be cleaned.

[0129] The position of the first data table to be cleaned in the data table sequence is the same as the position of the second intermediate server corresponding to the first data table to be cleaned in the network rate sequence.

[0130] In step 7043, a one-to-one match is achieved between the identifier of the second intermediate server ranked in the front r in the network rate sequence and the data table sequence, and the effect achieved is: the network rate of the target server corresponding to the first data table to be cleaned with a larger amount of data is larger, so as to further improve the data cleaning efficiency.

[0131] Figure 8 Schematic diagram of a data table sequence and a network rate sequence in an embodiment of the present application. Figure 8 As shown, the number of the first data table to be cleaned is 4, and the number of each second intermediate server corresponding to the first candidate server set is 5. The data table sequence of the data table to be cleaned is (data table to be cleaned 81a, data table to be cleaned 82a, data table to be cleaned 83a, data table to be cleaned 84a). The network rate sequence of each second intermediate server is (second intermediate server 81b, second intermediate server 82b, second intermediate server 83b, second intermediate server 84b, second intermediate server 85b). It is finally determined that the target server corresponding to the data table to be cleaned 81a is the second intermediate server 81b, the target server corresponding to the data table to be cleaned 82a is the second intermediate server 82b, the target server corresponding to the data table to be cleaned 83a is the second intermediate server 83b, and the target server corresponding to the data table to be cleaned 84a is the second intermediate server 84b. It can be seen that the larger the data volume, the greater the network rate of the target server corresponding to the first data table to be cleaned, which further improves the data cleaning efficiency.

[0132] Step 7044: If the number r of first data tables to be cleaned is greater than the number t of second intermediate servers corresponding to the first candidate server set, determine the second intermediate server corresponding to each first data table to be cleaned based on the data table sequence and the network rate sequence, and determine the second intermediate server corresponding to the first data table to be cleaned as the target server corresponding to the first data table to be cleaned.

[0133] Among them, the position of the first data table to be cleaned in the first data table to be cleaned in the data table sequence is the same as the position of the second intermediate server corresponding to the first data table to be cleaned in the network rate sequence. The positions of other first data tables to be cleaned in the first data table to be cleaned except the first data table to be cleaned in the data table sequence are greater than the positions of the second intermediate servers corresponding to other first data tables to be cleaned in the network rate sequence. Each second intermediate server corresponds to at least one first data table to be cleaned.

[0134] In step 7044, a larger number of first data tables to be cleaned are matched with a smaller number of second intermediate servers, and the effect achieved is that the first data table to be cleaned corresponding to the target server with the highest network rate has the largest amount of data, so as to reduce the transmission pressure of other target servers except the target server with the highest network rate, and further improve the data cleaning efficiency.

[0135] Fig. 9 FIG. 1 is a schematic diagram of another data table sequence and network rate sequence in an embodiment of the present application. Fig. 9 As shown, the number of the first data tables to be cleaned is 4, and the number of the second intermediate servers corresponding to the first candidate server set is 3. The data table sequence of the data tables to be cleaned is (data table 91a to be cleaned, data table 92a to be cleaned, data table 93a to be cleaned, data table 94a to be cleaned). The network rate sequence of each second intermediate server is (second intermediate server 91b, second intermediate server 92b, second intermediate server 93b). It is finally determined that the target server corresponding to the data table 91a to be cleaned is the second intermediate server 91b, the target server corresponding to the data table 92a to be cleaned is the second intermediate server 91b, the target server corresponding to the data table 93a to be cleaned is the second intermediate server 92b, and the target server corresponding to the data table 94a to be cleaned is the second intermediate server 93b. It can be seen that the target server with the largest network rate corresponds to the first data table to be cleaned with the largest amount of data, which further improves the data cleaning efficiency.

[0136] Optionally, before step 702, the data cleaning method provided in this embodiment further includes the following steps: obtaining the network security parameters of the candidate server from the network management device of each candidate server; obtaining the processing performance parameters of the candidate server according to the processing log of each candidate server; and determining the network rate parameters of the candidate server according to the interactive data volume and interactive duration of each candidate server. Through this implementation, the current status information of each candidate server can be accurately obtained to improve the accuracy of the target server determined subsequently, and further improve the data cleaning efficiency.

[0137] Step 705: Send each data table to be cleaned to the corresponding target server, so that the target server performs data cleaning on the data table to be cleaned according to the target cleaning method corresponding to the data table to be cleaned.

[0138] Step 706: Obtain the cleaned data tables of each target server, and associate the m cleaned data tables according to the association rules matching the m data tables to be cleaned, to obtain n associated data tables.

[0139] Here, n is an integer less than m.

[0140] The implementation process and technical principle of step 705 and step 203, and step 706 and step 204 are similar, and will not be repeated here.

[0141] The data cleaning method provided in this embodiment determines, among each candidate server, a candidate server whose corresponding network security parameter is greater than a preset network security threshold as a first intermediate server, determines, among each first intermediate server, a first intermediate server whose corresponding processing performance parameter is greater than a preset processing performance threshold as a second intermediate server, and determines the target server according to the network rate parameters of each second intermediate server, thereby achieving comprehensive screening of candidate servers from three dimensions: network security, processing performance, and network rate performance, and finally determining the target server corresponding to the data table to be cleaned, thereby enabling the target server to perform data cleaning safely and efficiently, thereby improving the security of data cleaning and further improving the efficiency of data cleaning.

[0142] Fig.10 A flowchart of another data cleaning method provided in an embodiment of the present application is shown below. Figure 2 , Figure 6 , Figure 7 Based on the illustrated embodiment and various optional implementations, the steps of determining the target server are described in detail. Fig.10 As shown, the data cleaning method provided in this embodiment includes the following steps.

[0143] Step 1001: According to the target cleaning mode matched with each data table to be cleaned, a set of candidate servers corresponding to the data table to be cleaned is determined from a data cleaning server cluster.

[0144] Among them, all servers in the candidate server set can achieve the target cleaning method, the number of data tables to be cleaned is m, m is an integer greater than 1, and the data cleaning server cluster can achieve at least two cleaning methods.

[0145] Step 1002: Determine a target server for data cleaning of each data table to be cleaned according to current status information of each candidate server in a candidate server set corresponding to each data table to be cleaned.

[0146] The implementation process and technical principles of step 1001 and step 201, and step 1002 and step 202 are similar, and will not be repeated here.

[0147] The scenario of this embodiment is a scenario where a data table to be cleaned needs to be cleaned in two ways. In this embodiment, the second data table to be cleaned matches at least two target cleaning ways, and the number of target servers corresponding to the second data table to be cleaned is at least two.

[0148] Step 1003: Determine the interaction indication information of the target server corresponding to the second data table to be cleaned according to the target cleaning mode matched by the second data table to be cleaned.

[0149] In this embodiment, the interaction indication information of the target server corresponding to the second data table to be cleaned can be determined according to the target cleaning method and the mapping relationship between the cleaning method and the interaction indication information. For example, the interaction indication information can indicate: first perform data deduplication in the server set that implements data deduplication, and after deduplication, perform data verification in the server set that implements data verification. Since the target server is a server determined from the candidate server set, the interaction indication information is equivalent to indicating the interaction order between the target servers corresponding to the second data table to be cleaned.

[0150] Step 1004: If the interaction indication information indicates that there is no interaction between the target servers corresponding to the second data table to be cleaned, the second data table to be cleaned is sent to the corresponding target server.

[0151] After receiving the second data table to be cleaned, the target server performs data cleaning on the data table to be cleaned according to the target cleaning method supported by the target server.

[0152] In step 1004, the second data table to be cleaned may be sent to at least two corresponding target servers respectively.

[0153] Step 1005: If the interaction indication information includes the interaction sequence between the target servers corresponding to the second data table to be cleaned, the second data table to be cleaned and the interaction indication information are sent to the first target server indicated in the interaction indication information.

[0154] After the first target server cleans the second data table to be cleaned, it sends the data table after the first cleansing to the second target server indicated in the interaction indication information, and so on, until the last target server indicated in the interaction indication information completes data cleaning for the data after the w-1th cleansing. Wherein, w is the number of target servers corresponding to the second data table to be cleaned indicated in the interaction indication information, and w is an integer greater than 1.

[0155] In step 1005, each target server can transmit the cleaned data table in sequence according to the interaction order, without the need to obtain the cleaned data table through the electronic device, thereby further improving the data cleaning efficiency.

[0156] Fig.11 Schematic diagram of interaction between target servers in the embodiment of the present application. Fig.11As shown, it is assumed that the second data table to be cleaned corresponds to three target servers: target server 111, target server 112, and target server 113. The interaction order indicated in the interaction indication information is target server 111, target server 112, and target server 113. Then target server 111 transmits the data table after the first cleaning to target server 112. Target server 112 completes data cleaning of the data table after the first cleaning, and sends the data table after the second cleaning to target server 113. Target server 113 completes data cleaning of the data table after the second cleaning.

[0157] Step 1006: Obtain the cleaned data tables of each target server, and associate the m cleaned data tables according to the association rules matching the m data tables to be cleaned, to obtain n associated data tables.

[0158] Here, n is an integer less than m.

[0159] It can be understood that, with respect to the implementation of step 1005 , the cleaned data tables of each target server can also be obtained in step 1006 .

[0160] The implementation process and technical principle of step 1006 are similar to those of step 204 and will not be repeated here.

[0161] The data cleaning method provided in this embodiment determines the interaction indication information of the target server corresponding to the second data table to be cleaned according to the target cleaning mode matched by the second data table to be cleaned. If the interaction indication information includes the interaction sequence between the target servers corresponding to the second data table to be cleaned, the second data table to be cleaned and the interaction indication information are sent to the first target server indicated in the interaction indication information, so that each target server can transmit the cleaned data table in sequence according to the interaction sequence, without the need to obtain the cleaned data table through the electronic device, thereby further improving the data cleaning efficiency.

[0162] Fig.12 This is a schematic diagram of the structure of a data cleaning system provided in an embodiment of the present application. Fig.12 As shown, the data cleaning system provided by this embodiment includes: a data cleaning trigger device 121 and a data cleaning server cluster 122 connected to the data cleaning trigger device 121. The data cleaning server cluster 122 can implement at least two cleaning modes.

[0163] The data cleaning trigger device 121 is used to execute the data cleaning method provided by any of the above embodiments.

[0164] The target server 1221 in the data cleaning server cluster 122 is used to receive the corresponding data table to be cleaned, and perform data cleaning on the received data table to be cleaned according to the target cleaning method corresponding to the received data table to be cleaned.

[0165] The data cleaning system provided in the embodiment of the present application can implement the technical solution of the data cleaning method in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0166] Fig.13 A schematic diagram of the structure of a data cleaning device provided in an embodiment of the present application. Fig.13 As shown, the data cleaning device provided in this embodiment includes the following modules: a first determining module 131 , a second determining module 132 , a sending module 133 and an obtaining association module 134 .

[0167] The first determining module 131 is used to determine a set of candidate servers corresponding to each data table to be cleaned from the data cleaning server cluster according to a target cleaning mode matched with the data table to be cleaned.

[0168] Wherein, all servers in the candidate server set can implement the target cleaning method. The number of the data tables to be cleaned is m, where m is an integer greater than 1. The data cleaning server cluster can implement at least two cleaning methods.

[0169] The second determining module 132 is used to determine a target server for data cleaning of each data table to be cleaned according to current status information of each candidate server in a candidate server set corresponding to each data table to be cleaned.

[0170] The sending module 133 is used to send each of the data tables to be cleaned to a corresponding target server, so that the target server performs data cleaning on the data table to be cleaned according to a target cleaning method corresponding to the data table to be cleaned.

[0171] The acquisition association module 134 is used to acquire the cleaned data tables of each target server, and associate the m cleaned data tables according to the association rules matching the m data tables to be cleaned, to obtain n associated data tables.

[0172] Here, n is an integer less than m.

[0173] In one embodiment, the device further includes: a first acquisition module, a third determination module and a conversion module.

[0174] The first acquisition module is used to acquire field conversion rules that match the m data tables to be cleaned. The field conversion rules are used to indicate that the original fields in the associated data table are converted to standard fields. The third determination module is used to determine the fields to be converted in the n associated data tables according to the field conversion rules. The conversion module is used to convert the fields to be converted in the n associated data tables into corresponding standard fields according to the field conversion rules, and convert the data corresponding to the fields to be converted into data that conforms to the data format specified by the standard fields corresponding to the fields to be converted, so as to obtain n target data tables.

[0175] In one embodiment, the device further comprises: a receiving module, configured to receive a target cleaning method matched with each of the data tables to be cleaned input by a user through a data cleaning configuration interface, wherein the target cleaning methods corresponding to different data tables to be cleaned are the same or different.

[0176] In one embodiment, the device further includes: a second acquisition module, a fourth determination module and a fifth determination module.

[0177] The second acquisition module is used to acquire m data tables to be cleaned. The fourth determination module is used to input each of the data tables to be cleaned into a pre-trained data feature analysis model to obtain data features of the data tables to be cleaned output by the data feature analysis model. The fifth determination module is used to determine a target cleaning method that matches the data table to be cleaned based on the data features of each of the data tables to be cleaned.

[0178] In one embodiment, the fifth determination module is specifically used to perform at least one of the following: if the data feature indicates that there is erroneous data in the data table to be cleaned, determining that the target cleaning method of the data table to be cleaned is data verification; if the data feature indicates that there is duplicate data in the data table to be cleaned, determining that the target cleaning method of the data table to be cleaned is data deduplication; if the data feature indicates that there is incomplete data in the data table to be cleaned, determining that the target cleaning method of the data table to be cleaned is data filling; if the data feature indicates that there is a data format that does not match the format indicated by the field in the data table to be cleaned, determining that the target cleaning method of the data table to be cleaned is data format conversion; if the data feature indicates that there is clusterable data in the data table to be cleaned, determining that the target cleaning method of the data table to be cleaned is data grouping; if the data feature indicates that there is data with inconsistent units in the data table to be cleaned, determining that the target cleaning method of the data table to be cleaned is data conversion; if the data feature indicates that there is data in the data table to be cleaned that does not match the field, determining that the target cleaning method of the data table to be cleaned is data marking.

[0179] In one embodiment, the current status information of the candidate server includes: network rate parameters, network security parameters and processing performance parameters. The second determination module 132 is specifically used to: determine the candidate server whose corresponding network security parameter is greater than the preset network security threshold among the candidate servers as the first intermediate server; determine the first intermediate server whose corresponding processing performance parameter is greater than the preset processing performance threshold among the first intermediate servers as the second intermediate server; and determine the target server according to the network rate parameters of the second intermediate servers.

[0180] In one embodiment, the device further includes a third acquisition module, a fourth acquisition module and a sixth determination module.

[0181] The third acquisition module is used to obtain the network security parameters of each candidate server from the network management device of the candidate server. The fourth acquisition module is used to obtain the processing performance parameters of the candidate server according to the processing log of each candidate server. The sixth determination module is used to determine the network rate parameters of the candidate server according to the amount of interaction data and interaction duration of each candidate server.

[0182] In one embodiment, in determining the target server based on the network rate parameters of each of the second intermediate servers, the second determination module 132 is specifically used to: determine the second intermediate server with the largest corresponding network rate parameter among the second intermediate servers as the target server.

[0183] In one embodiment, if the first candidate server set corresponds to at least two first data tables to be cleaned, then in the aspect of determining the target server according to the network rate parameters of each of the second intermediate servers, the second determination module 132 is specifically used to: sort the identifiers of the at least two first data tables to be cleaned in order of data volume from large to small, to obtain a data table sequence of the data tables to be cleaned; sort the identifiers of the second intermediate servers corresponding to the first candidate server set in order of corresponding network rates from large to small, to obtain a network rate sequence of the second intermediate servers corresponding to the first candidate server set; if the number r of the first data tables to be cleaned is less than or equal to the number t of the second intermediate servers corresponding to the first candidate server set, then determine the second intermediate server corresponding to each first data table to be cleaned according to the identifiers of the second intermediate servers ranked first r in the network rate sequence and the data table sequence, and determine the second intermediate server corresponding to the first data table to be cleaned as the target server corresponding to the first data table to be cleaned, wherein the number of the first data tables to be cleaned is less than or equal to the number t of the second intermediate servers corresponding to the first candidate server set. The position of the first data table to be cleaned in the data table sequence is the same as the position of the second intermediate server corresponding to the first data table to be cleaned in the network rate sequence; if the number r of the first data tables to be cleaned is greater than the number t of the second intermediate servers corresponding to the first candidate server set, then according to the data table sequence and the network rate sequence, the second intermediate server corresponding to each first data table to be cleaned is determined, and the second intermediate server corresponding to the first data table to be cleaned is determined as the target server corresponding to the first data table to be cleaned, wherein the position of the first data table to be cleaned in the first data table to be cleaned in the data table sequence is the same as the position of the second intermediate server corresponding to the first data table to be cleaned in the network rate sequence, the positions of the other first data tables to be cleaned in the first data table to be cleaned except the first data table to be cleaned in the data table sequence are greater than the positions of the second intermediate servers corresponding to the other first data tables to be cleaned in the network rate sequence, and each of the second intermediate servers corresponds to at least one first data table to be cleaned.

[0184] In one embodiment, the second data table to be cleaned matches at least two target cleaning methods, and the number of target servers corresponding to the second data table to be cleaned is at least two. The device also includes a seventh determination module, which is used to determine the interaction indication information of the target server corresponding to the second data table to be cleaned according to the target cleaning method matched by the second data table to be cleaned. The sending module 133 is specifically used to: if the interaction indication information indicates that there is no interaction between the target servers corresponding to the second data table to be cleaned, then send the second data table to be cleaned to the corresponding target server; if the interaction indication information includes the interaction sequence between the target servers corresponding to the second data table to be cleaned, then send the second data table to be cleaned and the interaction indication information to the first target server indicated in the interaction indication information, so that after the first target server performs data cleaning on the second data table to be cleaned, the data table after the first cleaning is sent to the second target server indicated in the interaction indication information, and so on, until the last target server indicated in the interaction indication information completes data cleaning for the data after the w-1th cleaning. Wherein, w is the number of target servers corresponding to the second data table to be cleaned indicated in the interaction indication information, and w is an integer greater than 1.

[0185] In one embodiment, the receiving module is further used to receive the association rules input by the user that match the m data tables to be cleaned. Alternatively, the device further includes an eighth determination module, which is used to extract the fields of the m data tables to be cleaned and determine the association rules that match the m data tables to be cleaned based on the fields of the m data tables to be cleaned.

[0186] In one embodiment, the data table to be cleaned is a data table generated during the operation of financial software.

[0187] The data cleaning device provided in the embodiment of the present application can be used to execute the technical solution of the data cleaning method in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0188] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the first determination module 131, the second determination module 132, the sending module 133, and the acquisition association module 134 can be separately established processing elements, or they can be integrated in a chip of the above device. In addition, they can also be stored in the memory of the above device in the form of program code, and a processing element of the above device can call and execute the functions of the first determination module 131, the second determination module 132, the sending module 133, and the acquisition association module 134. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0189] Fig.14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.14 As shown, the electronic device may include: a processor 142 and a memory 143.

[0190] The processor 142 executes the computer execution instructions stored in the memory, so that the processor 142 executes the scheme in the above embodiment. The processor 142 can be a general-purpose processor, including a central processing unit CPU, a network processor (network processor, NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0191] The memory 143 is connected to the processor 142 via a system bus to complete the communication between them. The memory 143 is used to store computer program instructions.

[0192] Optionally, the electronic device may further include a transceiver 141. The transceiver 141 may be used to implement interaction with a target server.

[0193] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the data cleaning device and other computer devices. The memory may include random access memory (RAM) and may also include non-volatile memory.

[0194] An embodiment of the present application also provides a chip for running instructions, which is used to execute the technical solution of the data cleaning method in the above embodiment.

[0195] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution of the data cleaning method of the above embodiment.

[0196] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements a data cleaning method as provided in any embodiment of the present application.

[0197] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present application, and the programming language includes an object-oriented programming language, such as Java, Smalltalk, C++, and also includes a conventional procedural programming language, such as "C" language or similar programming language. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0198] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A data cleaning method, characterized in that: include: According to the target cleaning method matched with each data table to be cleaned, determine a set of candidate servers corresponding to the data table to be cleaned from the data cleaning server cluster; wherein the servers in the candidate server set can all implement the target cleaning method, the number of the data tables to be cleaned is m, m is an integer greater than 1, and the data cleaning server cluster can implement at least two cleaning methods; Determining a target server for data cleaning of each data table to be cleaned according to current status information of each candidate server in a candidate server set corresponding to each data table to be cleaned; Sending each of the data tables to be cleaned to a corresponding target server, so that the target server performs data cleaning on the data table to be cleaned according to a target cleaning method corresponding to the data table to be cleaned; The cleaned data tables of each target server are obtained, and according to the association rules matching the m data tables to be cleaned, the m cleaned data tables are associated to obtain n associated data tables; wherein n is an integer less than m.

2. The method according to claim 1, characterized in that After obtaining n associated data tables, the method further includes: Obtaining field conversion rules that match the m data tables to be cleaned; wherein the field conversion rules are used to indicate that the original fields in the associated data tables are converted into standard fields; According to the field conversion rule, determining the fields to be converted in the n associated data tables; According to the field conversion rule, the fields to be converted in the n associated data tables are converted into corresponding standard fields, and the data corresponding to the fields to be converted are converted into data that conforms to the data format specified by the standard fields corresponding to the fields to be converted, thereby obtaining n target data tables.

3. The method according to claim 1, characterized in that Before determining a set of candidate servers corresponding to each data table to be cleaned from a data cleaning server cluster according to a target cleaning mode matched with each data table to be cleaned, the method further includes: A target cleaning method matching each of the to-be-cleaned data tables input by a user through a data cleaning configuration interface is received; wherein the target cleaning methods corresponding to different to-be-cleaned data tables are the same or different.

4. The method according to claim 1, characterized in that: Before determining a set of candidate servers corresponding to each data table to be cleaned from a data cleaning server cluster according to a target cleaning mode matched with each data table to be cleaned, the method further includes: Get m data tables to be cleaned; Input each of the to-be-cleaned data tables into a pre-trained data feature analysis model to obtain data features of the to-be-cleaned data table output by the data feature analysis model; According to the data features of each of the data tables to be cleaned, a target cleaning method matching the data table to be cleaned is determined.

5. The method according to claim 4, characterized in that The step of determining a target cleaning method matching the data table to be cleaned according to the data features of each data table to be cleaned includes at least one of the following: If the data feature indicates that there is erroneous data in the data table to be cleaned, determining that the target cleaning method of the data table to be cleaned is data verification; If the data feature indicates that duplicate data exists in the data table to be cleaned, determining that a target cleaning method for the data table to be cleaned is data deduplication; If the data feature indicates that there is incomplete data in the data table to be cleaned, determining that a target cleaning method for the data table to be cleaned is data filling; If the data feature indicates that there is a data format in the data table to be cleaned that does not match the format indicated by the field, determining that the target cleaning method of the data table to be cleaned is data format conversion; If the data feature indicates that clusterable data exists in the data table to be cleaned, determining that a target cleaning method for the data table to be cleaned is data grouping; If the data feature indicates that there are data with inconsistent units in the data table to be cleaned, determining that the target cleaning method of the data table to be cleaned is data conversion; If the data feature indicates that there is data of unmatched fields in the data table to be cleaned, it is determined that the target cleaning method of the data table to be cleaned is data marking.

6. The method according to any one of claims 1 to 5, characterized in that: The current status information of the candidate server includes: network rate parameters, network security parameters and processing performance parameters; The step of determining a target server for cleaning data of the data table to be cleaned according to current status information of each candidate server in a candidate server set corresponding to each data table to be cleaned includes: Determine, among the candidate servers, a candidate server whose corresponding network security parameter is greater than a preset network security threshold as a first intermediate server; Determine, among the first intermediate servers, a first intermediate server whose corresponding processing performance parameter is greater than a preset processing performance threshold as a second intermediate server; The target server is determined according to the network rate parameters of each of the second intermediate servers.

7. The method according to claim 6, characterized in that Before determining the target server for data cleaning of the data table to be cleaned according to the current status information of each candidate server in the candidate server set corresponding to each data table to be cleaned, the method further includes: Obtaining network security parameters of each candidate server from a network management device of the candidate server; Obtaining processing performance parameters of each candidate server according to the processing log of the candidate server; The network rate parameters of each candidate server are determined according to the amount of interaction data and the interaction duration of the candidate server.

8. The method according to claim 6, characterized in that The step of determining the target server according to the network rate parameters of each of the second intermediate servers includes: The second intermediate server with the largest corresponding network rate parameter among the second intermediate servers is determined as the target server.

9. The method according to claim 6, characterized in that If the first candidate server set corresponds to at least two first data tables to be cleaned, determining the target server according to the network rate parameters of each of the second intermediate servers includes: Sorting the identifiers of the at least two first data tables to be cleaned in descending order of data volume to obtain a data table sequence of the data tables to be cleaned; Sort the identifiers of the second intermediate servers corresponding to the first candidate server set in descending order according to the corresponding network rates, to obtain a network rate sequence of the second intermediate servers corresponding to the first candidate server set; If the number r of the first data tables to be cleaned is less than or equal to the number t of the second intermediate servers corresponding to the first candidate server set, then according to the identifiers of the second intermediate servers ranked first r in the network rate sequence and the data table sequence, determine the second intermediate server corresponding to each first data table to be cleaned, and determine the second intermediate server corresponding to the first data table to be cleaned as the target server corresponding to the first data table to be cleaned; wherein the position of the first data table to be cleaned in the data table sequence is the same as the position of the second intermediate server corresponding to the first data table to be cleaned in the network rate sequence; If the number r of the first data tables to be cleaned is greater than the number t of the second intermediate servers corresponding to the first candidate server set, then the second intermediate server corresponding to each first data table to be cleaned is determined according to the data table sequence and the network rate sequence, and the second intermediate server corresponding to the first data table to be cleaned is determined as the target server corresponding to the first data table to be cleaned; wherein, the position of the first data table to be cleaned in the first data table to be cleaned in the data table sequence is the same as the position of the second intermediate server corresponding to the first data table to be cleaned in the network rate sequence, and the positions of other first data tables to be cleaned in the first data table to be cleaned except the first data table to be cleaned in the data table sequence are greater than the positions of the second intermediate servers corresponding to the other first data tables to be cleaned in the network rate sequence, and each of the second intermediate servers corresponds to at least one first data table to be cleaned.

10. The method according to any one of claims 1 to 5, characterized in that: The second data table to be cleaned matches at least two target cleaning methods, and the number of target servers corresponding to the second data table to be cleaned is at least two; The method further comprises: Determining interaction indication information of a target server corresponding to the second data table to be cleaned according to the target cleaning mode matched by the second data table to be cleaned; The step of sending each of the to-be-cleaned data tables to a corresponding target server includes: If the interaction indication information indicates that there is no interaction between the target servers corresponding to the second data table to be cleaned, sending the second data table to be cleaned to the corresponding target server; If the interaction indication information includes the interaction order between the target servers corresponding to the second data table to be cleaned, the second data table to be cleaned and the interaction indication information are sent to the first target server indicated in the interaction indication information, so that after the first target server performs data cleaning on the second data table to be cleaned, the data table after the first cleaning is sent to the second target server indicated in the interaction indication information, and so on, until the last target server indicated in the interaction indication information completes data cleaning on the data after the w-1th cleaning; wherein w is the number of target servers corresponding to the second data table to be cleaned indicated in the interaction indication information, and w is an integer greater than 1.

11. The method according to any one of claims 1 to 5, characterized in that: Before obtaining the cleaned data tables of each target server and associating the m cleaned data tables according to the association rules matching the m data tables to be cleaned to obtain n associated data tables, the method further includes: receiving association rules input by a user that match the m data tables to be cleaned; or, Fields of the m data tables to be cleaned are extracted, and association rules matching the m data tables to be cleaned are determined according to the fields of the m data tables to be cleaned.

12. The method according to any one of claims 1 to 5, characterized in that: The data table to be cleaned is a data table generated during the operation of the financial software.

13. A data cleaning system, characterized in that: include: A data cleaning trigger device and a data cleaning server cluster connected to the data cleaning trigger device; wherein the data cleaning server cluster can implement at least two cleaning modes; The data cleaning trigger device is used to execute the data cleaning method according to any one of claims 1 to 12; The target server in the data cleaning server cluster is used to receive the corresponding data table to be cleaned, and perform data cleaning on the received data table to be cleaned according to the target cleaning method corresponding to the received data table to be cleaned.

14. A data cleaning device, characterized in that: include: A first determination module is used to determine, from a data cleaning server cluster, a set of candidate servers corresponding to each data table to be cleaned according to a target cleaning method matched with the data table to be cleaned; wherein all servers in the candidate server set can implement the target cleaning method, the number of the data tables to be cleaned is m, m is an integer greater than 1, and the data cleaning server cluster can implement at least two cleaning methods; A second determination module is used to determine a target server for data cleaning of each data table to be cleaned according to current status information of each candidate server in a candidate server set corresponding to each data table to be cleaned; A sending module, used for sending each of the data tables to be cleaned to a corresponding target server, so that the target server performs data cleaning on the data table to be cleaned according to a target cleaning method corresponding to the data table to be cleaned; The association acquisition module is used to obtain the cleaned data tables of each target server, and associate the m cleaned data tables according to the association rules matching the m data tables to be cleaned to obtain n associated data tables; wherein n is an integer less than m.

15. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the data cleaning method as claimed in any one of claims 1 to 12 when executing the computer program.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data cleaning method according to any one of claims 1 to 12 is implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the data cleaning method according to any one of claims 1 to 12.

Images