Database-based data cleaning method and device, storage medium and electronic equipment

Abstract

The application discloses a database-based data cleaning method and device, a storage medium and electronic equipment. It relates to the technical field of big data. The method comprises the following steps: in response to a data cleaning request, obtaining N pieces of target business data of a target host from a message queue, wherein the target business data comprises primary key data for identifying the target business data, and N is a positive integer; based on the primary key data of each piece of target business data, matching each piece of target business data with business data in a database to obtain a matching result; and based on the matching result, cleaning the business data stored in the database. The application solves the technical problem that it is difficult to accurately clean the data that needs to be cleaned in a distributed database in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, in particular to a database-based data cleaning method and device, a storage medium and an electronic equipment. BACKGROUND

[0002] At present, most financial institutions mainly use large mainframes and database solutions provided by manufacturers to build systems, and the traditional centralized architecture with large mainframes and databases as the core cannot meet the growing large-scale transactions and data processing. On the one hand, the traditional centralized architecture cannot meet the processing needs of the explosive growth of business, and there is a risk of system overload, and on the other hand, the price is relatively expensive, and the maintenance cost is high. With the development of technology having a greater and greater impact on the financial service mode, the financial industry is gradually transforming to digital and distributed architecture.

[0003] However, in the process of distributed architecture transformation, the full amount of data of the large mainframe needs to be synchronized and updated to the platform distributed database. However, the data scale of the financial institution is huge, and the data volume of the business table is more than ten million, and the data table storage volume will continue to increase, so the cleaning work of the database in the financial institution becomes particularly important.

[0004] The large table in the distributed database in the financial institution usually stores large-scale data such as protocol records, transaction records, and customer information, which are used frequently in business scenarios and will frequently modify table records. However, in the process of distributed architecture transformation, the platform distributed database may contain some data that does not belong to the database. Such data that falls into the wrong database has no actual business use, and if it is not cleaned in time, it will consume the database capacity and reduce the database performance.

[0005] In related technologies, the storage time is used as a judgment basis, and the data with a storage time greater than a time threshold is determined as data to be cleaned. Then, an association operation is performed on the data to be cleaned to determine the amount of data that needs to be associated and cleaned, and if the data amount is less than a threshold, the data is batch deleted for cleaning, otherwise the data is retained. However, this method does not conform to the actual business scenario in the financial industry, and batch deleting data will cause the database to be locked for a long time, affecting online transactions. The cleaning amount of such data in a certain database and the time of falling into the wrong database are unknown, and if this method is used for cleaning, it may not be able to accurately filter the data that needs to be cleaned, and the long-time lock table is easy to block online transactions, causing large-scale transaction timeout failure and affecting the stability of the financial institution system.

[0006] At present, no effective solution has been proposed to solve the above problems. SUMMARY

[0007] Embodiments of the present application provide a database-based data cleaning method and device, a storage medium and an electronic device to at least solve the technical problem of being unable to accurately clean data that needs to be cleaned in a distributed database in the related art.

[0008] According to an aspect of the embodiments of the present application, a database-based data cleaning method is provided, comprising: in response to a data cleaning request, obtaining N pieces of target service data of a target host from a message queue, wherein the target service data comprises primary key data for identifying the target service data, and N is a positive integer; based on the primary key data of each piece of the target service data, matching each piece of the target service data with service data in the database to obtain a matching result; and based on the matching result, cleaning the service data stored in the database.

[0009] Further, the cleaning of the service data stored in the database based on the matching result comprises: if the matching result indicates that there is at least one primary key data in the database that is not matched to N pieces of the target service data, deleting the service data of the primary key data in the database; if the matching result indicates that any primary key data in N pieces of the target service data is successfully matched with the primary key data in the database, and the field data of the target service data associated with the primary key data is consistent with the field data of the primary key data in the database, comparing the latest modification time of the target service data associated with the primary key data with the modification time of the service data associated with the primary key data in the database, and in the case that the latest modification time of the target service data associated with the primary key data is later than the modification time of the service data associated with the primary key data in the database, deleting the service data associated with the primary key data in the database; if the matching result indicates that any primary key data in N pieces of the target service data is successfully matched with the primary key data in the database, and the field data of the target service data associated with the primary key data is inconsistent with the field data of the primary key data in the database, adding the service data associated with the primary key data in the database to a target data table, wherein the target data table is used to store service data that needs to be rechecked.

[0010] Further, after adding the service data associated with the primary key data in the database to the target data table, the method further comprises: checking the service data in the target data table to obtain a checking result, and locking the service data associated with the primary key data in the database; after locking the service data associated with the primary key data in the database, updating the service data associated with the primary key data in the database based on the checking result, and unlocking the service data associated with the primary key data in the database after the service data associated with the primary key data in the database is updated.

[0011] Further, before obtaining the N pieces of target service data of the target host from the message queue, the method further comprises: obtaining N pieces of first service data, and updating a latest modification time of each of the first service data to obtain N pieces of second service data, wherein the first service data comprises original service data in the target host; analyzing each of the second service data to obtain the N pieces of target service data, and adding each of the target service data to the message queue.

[0012] Further, the first service data at least comprises a user number, and obtaining the N pieces of first service data comprises: obtaining M pieces of first service data in the target host, wherein M is a positive integer greater than N; determining region information associated with each of the first service data based on the user number in each of the first service data; and screening the first service data with the region information being specified region information from the M pieces of first service data to obtain the N pieces of first service data.

[0013] Further, analyzing each of the second service data to obtain the N pieces of target service data comprises: analyzing each of the second service data to obtain field data of the second service data and primary key data of the second service data; and determining the N pieces of target service data based on the field data of each of the second service data and the primary key data of the second service data.

[0014] Further, after obtaining the N pieces of target service data of the target host from the message queue, the method further comprises: determining routing data associated with each of the target service data based on field data of a target field in each of the target service data; and storing each of the target service data in a database based on the routing data associated with each of the target service data.

[0015] According to another aspect of the embodiments of the present application, a database-based data cleaning device is further provided, comprising: an obtaining unit configured to obtain N pieces of target service data of a target host from a message queue in response to a data cleaning request, wherein the target service data comprises primary key data used for identifying the target service data, and N is a positive integer; a matching unit configured to match each of the target service data and service data in a database based on the primary key data of each of the target service data to obtain a matching result; and a cleaning unit configured to clean the service data stored in the database based on the matching result.

[0016] Further, the cleaning unit comprises: a first deleting sub-unit configured to delete, if the matching result indicates that at least one of the primary key data in the N pieces of target business data fails to match the primary key data in the database, the business data of the primary key data in the database; a second deleting sub-unit configured to, if the matching result indicates that any of the primary key data in the N pieces of target business data matches the primary key data in the database successfully and the field data of the target business data associated with the primary key data is consistent with the field data of the primary key data in the database, compare the latest modification time of the target business data associated with the primary key data with the modification time of the business data associated with the primary key data in the database, and delete, if the latest modification time of the target business data associated with the primary key data is later than the modification time of the business data associated with the primary key data in the database, the business data associated with the primary key data in the database; and an adding sub-unit configured to, if the matching result indicates that any of the primary key data in the N pieces of target business data matches the primary key data in the database successfully and the field data of the target business data associated with the primary key data is inconsistent with the field data of the primary key data in the database, add the business data associated with the primary key data in the database to a target data table, wherein the target data table is configured to store the business data to be rechecked.

[0017] Further, the cleaning sub-unit further comprises: a checking sub-unit configured to, after adding the business data associated with the primary key data in the database to the target data table, check the business data in the target data table to obtain a checking result, and lock the business data associated with the primary key data in the database; and an updating sub-unit configured to, after locking the business data associated with the primary key data in the database, update the business data associated with the primary key data in the database based on the checking result, and unlock the business data associated with the primary key data in the database after updating the business data associated with the primary key data in the database.

[0018] Further, the data cleaning device further comprises: a processing unit configured to, before obtaining the N pieces of target business data of a target host from a message queue, obtain N pieces of first business data, and update the latest modification time of each of the first business data to obtain N pieces of second business data, wherein the first business data comprises original business data in the target host; and an analyzing unit configured to analyze each of the second business data to obtain the N pieces of target business data, and add each of the target business data to the message queue.

[0019] Further, the first service data at least comprises a user number, the obtaining unit comprises: an obtaining subunit, configured to obtain M pieces of first service data in the target host, wherein M is a positive integer greater than N; a first determining subunit, configured to determine the area information associated with each piece of the first service data based on the user number in each piece of the first service data; and a screening subunit, configured to screen the first service data associated with the specified area information from the M pieces of the first service data, to obtain N pieces of the first service data.

[0020] Further, the parsing unit comprises: a parsing subunit, configured to parse each piece of the second service data in the N pieces of the second service data, to obtain the field data of the second service data and the primary key data of the second service data; and a second determining subunit, configured to determine the N pieces of the target service data based on the field data of each piece of the second service data and the primary key data of the second service data.

[0021] Further, the database-based data cleaning device further comprises: a determining unit, configured to determine the routing data associated with each piece of the target service data based on the field data of the target field in each piece of the target service data after obtaining the N pieces of the target service data of the target host from the message queue; and a storage unit, configured to store each piece of the target service data into the database based on the routing data associated with each piece of the target service data.

[0022] According to another aspect of the embodiment of the present application, an electronic device is further provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the database-based data cleaning method of any one of the above via execution of the executable instructions.

[0023] According to another aspect of the embodiment of the present application, a computer readable storage medium is further provided, which stores a computer program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to execute the database-based data cleaning method of any one of the above when the computer program is running.

[0024] In the present application, in response to a data cleaning request, N pieces of target service data of a target host are obtained from a message queue, wherein the target service data comprises primary key data for identifying the target service data, and N is a positive integer; each piece of target service data and service data in a database are matched based on the primary key data of each piece of target service data, and a matching result is obtained; and the service data stored in the database is cleaned based on the matching result. Thus, the technical problem that it is difficult to accurately clean the data to be cleaned in the distributed database in the related art is solved. In the present application, the target service data in the target host of the message queue is obtained, the data in the database is matched according to the primary key uniqueness principle, and the service data in the database is cleaned according to the matching result, thereby avoiding the situation in the related art that the service data to be cleaned in the database is determined only according to a time threshold, which is low in accuracy and can easily cause a long table locking time, so that the technical effects of improving the data cleaning efficiency and cleaning accuracy are achieved. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and serve to explain the present application. In the drawings:

[0026] Figure 1 is a flowchart of an optional database-based data cleaning method according to an embodiment of the present application;

[0027] Figure 2 is a flowchart of another optional database-based data cleaning method according to an embodiment of the present application;

[0028] Figure 3 is a schematic diagram of an optional database-based data cleaning device according to an embodiment of the present application;

[0029] Figure 4 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the protection scope of the present application.

[0031] It should be noted that the terms "first", "second", and the like in the description and in the claims of the present application and the above-described accompanying drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0032] It should be noted that the database-based data cleaning method, device, storage medium and electronic device in the present disclosure can be used in the case of cleaning data in the database in the field of big data, and can also be used in the case of cleaning data in the database in any field other than the field of big data. The application field of the database-based data cleaning method, device, storage medium and electronic device in the present disclosure is not limited.

[0033] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, business data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0034] The present application can be applied to various software products, control systems, client terminals (including but not limited to mobile client terminals, PC terminals, etc.) of various financial institutions. Taking the software product as an example, through the software product installed on the mobile client terminal, the data of the business content (including but not limited to transfer, financial management, fund, payment, account checking, advertising, recommendation, etc.) of the financial institution can be cleaned in the database.

[0035] Embodiment one

[0036] According to the embodiments of the present application, an optional method embodiment of the database-based data cleaning method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0037] Figure 1is a flowchart of an optional database-based data cleaning method according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0038] In step S101, in response to a data cleaning request, N pieces of target service data of a target host are obtained from a message queue, wherein the target service data comprises primary key data for identifying the target service data, and N is a positive integer.

[0039] The target service data described above can comprise primary key data for identifying the target service data. The target service data can be data in the target host that involves a range to be cleaned. Whether the range to be cleaned is involved can be determined based on a region where a user number associated in the service data is located. In order to accurately clean data in the database and avoid the situation that data is wrongly cleaned, the target service data can further comprise a latest modification time in the host, so as to clean data in the database based on the latest modification time.

[0040] In step S102, each piece of target service data and service data in the database are matched based on the primary key data of each piece of target service data, and a matching result is obtained.

[0041] The type of the database can comprise a database of a distributed platform. In the present embodiment, data sent by the large host to the distributed database platform through the message queue can be parsed and matched with service data in the database according to a unique identifier.

[0042] It should be noted that the primary key data of each piece of target service data can be used as a unique identifier of the target service data. According to the principle of uniqueness of the primary key of the distributed platform data and the large host data, the target service data is matched with the service data in the database. The matching content can include but is not limited to whether the primary key data is the same. In the case where the primary key data is consistent, it can also be compared whether the data of each field of the target service data is consistent with the data of each field of the service data in the database corresponding to the primary key data. In the case where the primary key data is consistent, it can also be compared whether the latest modification time of the target service data is consistent with the latest modification time of the service data in the database corresponding to the primary key data, so as to obtain the matching result.

[0043] In step S103, the service data stored in the database is cleaned based on the matching result.

[0044] In the present embodiment, whether the database data is deleted, updated, re-verified, or processed according to the verification result, can be determined according to whether the matching is successful, the order of the last modification time of the corresponding service data in the case where the matching is successful, and the like.

[0045] Through the above steps, in this embodiment, the target business data in the target host in the message queue is obtained, the data in the database is matched according to the primary key uniqueness principle, and the business data in the database is cleaned according to the matching result, which avoids the related art of determining the business data to be cleaned in the database only according to the time threshold, has low precision, and easily causes the case that the table locking time is too long, thereby realizing the technical effects of improving the data cleaning efficiency and cleaning precision. Further, the technical problem that it is difficult to accurately clean the data to be cleaned in the distributed database in the related art is solved.

[0046] Optionally, the business data stored in the database is cleaned based on the matching result, including: if the matching result indicates that there is at least one primary key data in the database that is not matched to the N target business data, the business data of the primary key data is deleted in the database; if the matching result indicates that any primary key data in the N target business data is successfully matched with the primary key data in the database, and the field data of the target business data associated with the primary key data is consistent with the field data of the primary key data in the database, the latest modification time of the target business data associated with the primary key data is compared with the modification time of the business data associated with the primary key data in the database, and in the case that the latest modification time of the target business data associated with the primary key data is later than the modification time of the business data associated with the primary key data in the database, the business data associated with the primary key data in the database is deleted; if the matching result indicates that any primary key data in the N target business data is successfully matched with the primary key data in the database, and the field data of the target business data associated with the primary key data is inconsistent with the field data of the primary key data in the database, the business data associated with the primary key data in the database is added to the target data table, wherein the target data table is used to store the business data that needs to be rechecked.

[0047] The following is described taking the database as an example of a distributed platform database: according to the business data of the distributed platform database and the primary key uniqueness principle of the mainframe data, the unique index (i.e. the primary key data) is matched, after the data is successfully matched, the index field is immediately released, and the table locking time is reduced.

[0048] Specifically, after the unique identifier (i.e., the primary key data) is matched successfully, the other main fields (corresponding to the field data) are compared. If all the main fields are consistent, the matching is successful. If the main fields are inconsistent and the unique identifier is consistent, the data in the distributed platform database is suspicious, and the platform data is temporarily stored in a temporary table (corresponding to the target data table). After subsequent checking, the data can be cleaned up. In the distributed platform database, the data corresponding to the cleaning area and having a latest modification time earlier than the latest modification time of the current mainframe original data (corresponding to the target business data) is screened and deleted, for example, the data corresponding to the cleaning area and having a last update date earlier than the last modification date of the current mainframe original data is screened and deleted.

[0049] If there is data in the distributed platform database that is not matched to the mainframe original data (corresponding to the target business data), the data is error redundancy data and can be directly deleted, thereby achieving the technical effects of reducing the lock table time of the database and improving the accuracy and efficiency of data cleaning.

[0050] Optionally, after the business data associated with the primary key data in the database is added to the target data table, the business data in the target data table is checked to obtain a checking result, and the business data associated with the primary key data in the database is locked. After the business data associated with the primary key data in the database is locked, the business data associated with the primary key data in the database is updated based on the checking result, and the business data associated with the primary key data in the database is unlocked after the business data associated with the primary key data in the database is updated.

[0051] In this embodiment, the business data in the target data table can be checked. Whether the business data in the target data table is the latest data, whether there is data loss, whether there is data error, etc. can be checked based on a preset rule to obtain a checking result. In the case where the checking result indicates that the business data in the database is incorrect and the business data needs to be updated, the business data associated with the primary key data in the corresponding database can be locked based on the primary key data of the business data, and the business data can be updated. After the business data associated with the primary key data in the database is updated, the business data associated with the primary key data in the database is unlocked, thereby avoiding the case that the data table is locked for too long during the data cleaning process in the distributed database, and achieving the technical effect of reducing the data locking time.

[0052] Optionally, before obtaining the N pieces of target service data of the target host from the message queue, the method further comprises: obtaining N pieces of first service data, and updating a latest modification time of each piece of first service data to obtain N pieces of second service data, wherein the first service data comprises original service data in the target host; and parsing each piece of second service data to obtain the N pieces of target service data, and adding each piece of target service data to the message queue.

[0053] In the embodiment, the N pieces of first service data can be service data in the target host (for example, a large host) related to the region to be cleaned up, and the last modification time of the N pieces of first service data in the region to be cleaned up can be updated to ensure that the timestamp of the data table currently containing the N pieces of first service data is the latest. Then, the target service data obtained by parsing the data related to the region to be cleaned up in the target host by using the data parsing tool is written into the message queue.

[0054] Optionally, the first service data at least comprises a user number, and the obtaining of the N pieces of first service data comprises: obtaining M pieces of first service data in the target host, wherein M is a positive integer greater than N; determining region information associated with each piece of first service data based on the user number in each piece of first service data; and screening the first service data with the region information associated therewith being specified region information from the M pieces of first service data to obtain the N pieces of first service data.

[0055] In the embodiment, the range of each cleaning can be divided according to the region of the customer number, for example, M pieces of first service data in the target host can be obtained, and then the region information associated with each piece of first service data is determined based on the user number in each piece of first service data. If the region information of the first service data relates to the specified region information (i.e. the information of the range to be cleaned up), the first service data can be used as one of the N pieces of first service data, thereby achieving the technical effect of accurately determining the service data related to the data cleaning range.

[0056] Optionally, the parsing of each piece of second service data to obtain the N pieces of target service data comprises: parsing each piece of second service data in the N pieces of second service data to obtain field data of the second service data and primary key data of the second service data; and determining the N pieces of target service data based on the field data of each piece of second service data and the primary key data of the second service data.

[0057] In the embodiment, the thread can store and transmit the whole information of the unique primary key of the service data, and the thread can send the message (i.e., the primary key data) of the unique primary key to the distributed platform for analysis, so as to obtain the field data of each field of the second service data, and to form the target service data corresponding to the second service data by the field data of each second service data and the primary key data of the second service data, so as to accurately determine the target service data, and to facilitate subsequent improvement of the accuracy of data cleaning.

[0058] Optionally, after obtaining N target service data of the target host from the message queue, the method further includes: determining the routing data associated with each target service data based on the field data of the target field in each target service data; and storing each target service data into a database based on the routing data associated with each target service data.

[0059] In the embodiment, the routing value (corresponding to the routing data) of the corresponding target service data can be determined according to the specified field, and the corresponding target service data can be written to a specific database (for example, a distributed platform database) according to the obtained routing value. For example, the data in the specified field (for example, some digits in the number in the specified field) of the target service data can be used to indicate the corresponding distributed platform database in which the target service data is stored.

[0060] The primary key data of each target service data is used as the identifier of the target service data. Based on the routing value of the target service data, all target service data are stored in the distributed platform database, and a new field can be added to record the specific information of the distributed platform database in which the target service data is located. Subsequently, the data containing the identifier (primary key data) can be directly sent to the corresponding distributed platform database after analysis, which reduces the time-consuming of re-computation and avoids the situation that the distributed platform database cannot find the old data, thereby achieving the technical effect of improving the efficiency of business data landing.

[0061] Figure 2 FIG. 3 is a flowchart of another optional database-based data cleaning method according to an embodiment of the present application, as shown in FIG. 3, the method includes: Figure 2

[0062] Host modification time update: updating the last modification time of the service data of the large host related to the cleaning area;

[0063] Tool analysis: analyzing the service data of the large host related to the cleaning area by using the analysis tool, and sending the target service data obtained by the analysis to the message queue;

[0064] Data landing: storing the target service data in the message queue into the distributed platform database according to the routing corresponding to the data.​

[0065] Matching index: according to the primary key unique principle, the primary key data domain in the target business data is matched with the business data in the database; the data in the distributed platform database is matched, and the data with the update time less than the host time is deleted; after the matching result, the field is immediately released, the table locking time is reduced, and the business update is normally performed (the table is locked during the update, and is unlocked after the update); if the distributed platform data is not matched, the data is deleted.

[0066] In the embodiment, the range of each cleaning can be divided by the region of the customer number, the last modification time of the large mainframe to-be-cleaned region data is updated, and the data is written to the distributed platform database by using the message queue. Then, the last update time of the distributed platform database is compared, and it can be judged that the data is in the wrong library. After the corresponding data is judged, the table data is immediately released after the row data is deleted, the table locking time is effectively reduced, and the performance capacity of the distributed database is also improved.

[0067] By checking the large mainframe table and the distributed platform database table, the scene of deleting the actually valid data by only judging the last update time of the distributed platform database data is effectively avoided, the large mainframe table and the distributed platform database single and matching unique index data checking mode are used, the data occupation time is effectively reduced, the table locking time is reduced, and the transaction performance is improved. The large mainframe table and the temporary table data are checked by using the message queue mode, and the incremental data can be checked in real time. Through the embodiment, the optimization mode of redundant data can be provided for various distributed database systems.

[0068] Embodiment two

[0069] The embodiment two of the application provides a kind of optional database-based data cleaning device, each implementation unit in the data cleaning device corresponds to each implementation step in embodiment one.

[0070] Figure 3 It is a schematic diagram of an optional database-based data cleaning device according to the embodiment of the application, as shown in Figure 3 The data cleaning device includes: acquisition unit 31, matching unit 32 and cleaning unit 33.

[0071] Acquisition unit 31 is used to obtain N target business data of target host from message queue in response to data cleaning request, wherein the target business data includes: the primary key data for identifying target business data, and N is a positive integer;

[0072] Matching unit 32 is used to match each target business data and business data in the database based on the primary key data of each target business data, and obtain matching result;

[0073] The cleaning unit 33 is used to clean up the business data stored in the database based on the matching results.

[0074] In the database-based data cleaning apparatus provided in Embodiment 2 of this application, the acquisition unit 31 responds to a data cleaning request and acquires N target business data entries from the message queue. These target business data entries include primary key data used to identify the target business data, where N is a positive integer. Then, the matching unit 32 matches each target business data entry with the business data in the database based on its primary key data to obtain a matching result. Finally, the cleaning unit 33 cleans the business data stored in the database based on the matching result. This solves the technical problem in related technologies where it is difficult to accurately clean the data that needs to be cleaned in a distributed database. In this embodiment, the target business data from the target host is acquired from the message queue, matched with the data in the database according to the principle of primary key uniqueness, and the business data in the database is cleaned based on the matching result. This avoids the situation in related technologies where only a time threshold is used to determine the business data that needs to be cleaned in the database, resulting in low accuracy and potentially excessively long table locking times. This achieves the technical effect of improving data cleaning efficiency and accuracy.

[0075] Optionally, in the database-based data cleaning apparatus provided in Embodiment 2 of this application, the cleaning unit includes: a first deletion subunit, configured to delete the business data of the primary key data in the database if the matching result indicates that there is at least one primary key data in the database that does not match any of the N target business data; and a second deletion subunit, configured to compare the latest modification time of the target business data associated with the primary key data with the primary key data in the database if the matching result indicates that any primary key data in the N target business data matches the primary key data in the database, and the field data of the target business data associated with the primary key data is consistent with the field data of the primary key data in the database. Based on the modification time of the associated business data, and if the latest modification time of the target business data associated with the primary key data is later than the modification time of the business data associated with the primary key data in the database, the business data associated with the primary key data is deleted from the database; a sub-unit is added to add the business data associated with the primary key data in the database to the target data table if the matching result indicates that any primary key data in N target business data matches the primary key data in the database, and the field data of the target business data associated with the primary key data is inconsistent with the field data of the primary key data in the database. The target data table is used to store the business data that needs to be rechecked.

[0076] Optionally, in the database-based data cleaning apparatus provided in Embodiment Two of the present application, the cleaning subunit further comprises: a checking subunit, configured to check the business data in the target data table after adding the business data associated with the primary key data to the target data table, to obtain a checking result, and lock the business data associated with the primary key data in the database; and an updating subunit, configured to update the business data associated with the primary key data in the database based on the checking result after locking the business data associated with the primary key data in the database, and unlock the business data associated with the primary key data in the database after updating the business data associated with the primary key data in the database.

[0077] Optionally, in the database-based data cleaning apparatus provided in Embodiment Two of the present application, the data cleaning apparatus further comprises: a processing unit, configured to obtain N pieces of first business data and update the latest modification time of each piece of first business data to obtain N pieces of second business data before obtaining N pieces of target business data of the target host from the message queue, wherein the first business data comprises original business data in the target host; and an analyzing unit, configured to analyze each piece of second business data to obtain N pieces of target business data, and add each piece of target business data to the message queue.

[0078] Optionally, in the database-based data cleaning apparatus provided in Embodiment Two of the present application, the first business data at least comprises a user number, and the obtaining unit comprises: an obtaining subunit, configured to obtain M pieces of first business data in the target host, wherein M is a positive integer greater than N; a first determining subunit, configured to determine the region information associated with each piece of first business data based on the user number in each piece of first business data; and a screening subunit, configured to screen the first business data associated with the specified region information from the M pieces of first business data to obtain N pieces of first business data.

[0079] Optionally, in the database-based data cleaning apparatus provided in Embodiment Two of the present application, the analyzing unit comprises: an analyzing subunit, configured to analyze each piece of second business data in the N pieces of second business data to obtain the field data of the piece of second business data and the primary key data of the piece of second business data; and a second determining subunit, configured to determine the N pieces of target business data based on the field data of each piece of second business data and the primary key data of the piece of second business data.

[0080] Optionally, in the database-based data cleaning device provided in Embodiment Two of the present application, the database-based data cleaning device further comprises a determination unit configured to determine, after obtaining N pieces of target service data of the target host from the message queue, routing data associated with each piece of target service data based on field data of the target field in each piece of target service data; and a storage unit configured to store each piece of target service data into the database based on the routing data associated with each piece of target service data.

[0081] The database-based data cleaning device described above can further comprise a processor and a memory, and the acquisition unit 31, the matching unit 32, the cleaning unit 33 and the like are stored in the memory as program units, and the corresponding functions are realized by the processor executing the program units stored in the memory.

[0082] The processor described above comprises a core, and the core retrieves the corresponding program units from the memory. The core can be one or more, and the core parameters are adjusted to obtain the target service data of the target host in the message queue, to match the data in the database according to the primary key uniqueness principle, and to clean the service data in the database according to the matching result, thereby avoiding the low precision and long table locking time caused by only determining the service data to be cleaned in the database according to the time threshold in the related art, and achieving the technical effects of improving the data cleaning efficiency and cleaning precision.

[0083] The memory described above can comprise a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory comprises at least one memory chip.

[0084] According to another aspect of the embodiments of the present application, an electronic device is further provided, comprising: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to execute the database-based data cleaning method of any one of the above by executing the executable instructions.

[0085] According to another aspect of the embodiments of the present application, a computer readable storage medium is further provided, and the computer readable storage medium stores a computer program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to execute the database-based data cleaning method of any one of the above when the computer program runs.

[0086] Figure 4 is a schematic diagram of an electronic device according to an embodiment of the present application, as Figure 4As shown, the embodiment of the present application provides an electronic device 40, which comprises a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements the database-based data cleaning method of any one of the above when executing the program.

[0087] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0088] In the above-mentioned embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0089] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0090] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed to multiple units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0091] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0092] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0093] The above is only the preferred embodiment of the present application, it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, can make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A database-based data cleaning method, characterized by, The method comprises the following steps: obtaining N pieces of target business data of a target host from a message queue in response to a data cleaning request, wherein the target business data comprises primary key data used for identifying the target business data, and N is a positive integer; matching each piece of the target business data with business data in a database based on the primary key data of each piece of the target business data to obtain a matching result; cleaning the business data stored in the database based on the matching result; wherein the cleaning of the business data stored in the database based on the matching result comprises: if the matching result indicates that at least one primary key data of the N pieces of target business data does not match the database, deleting the business data of the primary key data in the database; if the matching result indicates that any primary key data of the N pieces of target business data matches the primary key data in the database successfully, and the field data of the target business data associated with the primary key data is consistent with the field data of the primary key data in the database, comparing the latest modification time of the target business data associated with the primary key data with the modification time of the business data associated with the primary key data in the database, and deleting the business data associated with the primary key data in the database if the latest modification time of the target business data associated with the primary key data is later than the modification time of the business data associated with the primary key data in the database; if the matching result indicates that any primary key data of the N pieces of target business data matches the primary key data in the database successfully, and the field data of the target business data associated with the primary key data is inconsistent with the field data of the primary key data in the database, adding the business data associated with the primary key data in the database to a target data table, wherein the target data table is used for storing business data that needs to be rechecked.

2. The data cleaning method of claim 1, wherein, After adding the business data associated with the primary key data in the database to the target data table, the method further comprises: checking the business data in the target data table to obtain a checking result, and locking the business data associated with the primary key data in the database; after locking the business data associated with the primary key data in the database, updating the business data associated with the primary key data in the database based on the checking result, and unlocking the business data associated with the primary key data in the database after the business data associated with the primary key data in the database is updated.

3. The data cleaning method of claim 1, wherein, Before obtaining N pieces of target business data of a target host from a message queue, the method further comprises: obtaining N pieces of first business data, and updating the latest modification time of each piece of the first business data to obtain N pieces of second business data, wherein the first business data comprises original business data in the target host; parsing each piece of the second business data to obtain N pieces of the target business data, and adding each piece of the target business data to the message queue.

4. The data cleaning method of claim 3, wherein, The first business data at least comprises a user number, and the obtaining of the N pieces of first business data comprises: M pieces of first service data in the target host are acquired, where M is a positive integer greater than N; region information associated with each piece of the first service data is determined based on a user number in each piece of the first service data; first service data with the region information as specified region information is screened from the M pieces of the first service data, to obtain N pieces of the first service data.

5. The data cleaning method of claim 3, wherein, Each piece of the second service data is parsed to obtain N pieces of target service data, including: The field data of each piece of the second service data and the primary key data of each piece of the second service data are obtained by parsing each piece of the second service data in the N pieces of the second service data; N pieces of the target service data are determined based on the field data of each piece of the second service data and the primary key data of each piece of the second service data.

6. The data cleaning method of claim 1, wherein, After acquiring N pieces of target service data of the target host from the message queue, further comprising: The routing data associated with each piece of the target service data is determined based on the field data of the target field in each piece of the target service data; Each piece of the target service data is stored in the database based on the routing data associated with each piece of the target service data.

7. A database-based data cleaning apparatus, characterized by comprising: Comprising: An acquisition unit is configured to acquire N pieces of target service data of a target host from a message queue in response to a data cleaning request, wherein the target service data includes primary key data for identifying the target service data, and N is a positive integer; A matching unit is configured to match each piece of the target service data with service data in the database based on the primary key data of each piece of the target service data, to obtain a matching result; A cleaning unit is configured to clean the service data stored in the database based on the matching result; The cleaning unit includes a first deletion sub-unit configured to delete the service data of the primary key data in the database if the matching result indicates that at least one primary key data in the N pieces of target service data is not matched; a second deletion sub-unit configured to compare the latest modification time of the target service data associated with the primary key data with the modification time of the service data associated with the primary key data in the database if the matching result indicates that any primary key data in the N pieces of target service data is successfully matched with the primary key data in the database, and the field data of the target service data associated with the primary key data is consistent with the field data of the primary key data in the database; and an addition sub-unit configured to add the service data associated with the primary key data in the database to a target data table if the matching result indicates that any primary key data in the N pieces of target service data is successfully matched with the primary key data in the database, and the field data of the target service data associated with the primary key data is inconsistent with the field data of the primary key data in the database, where the target data table is used to store service data that needs to be rechecked.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein the computer program controls a device where the computer readable storage medium is located to perform the database-based data cleaning method in any one of claims 1 to 6 when the computer program is running.

9. An electronic device, comprising: An apparatus comprising one or more processors and memory storing one or more programs, wherein the one or more programs that, when executed by the one or more processors, cause the one or more processors to perform the database-based data cleaning method in any one of claims 1 to 6.

Images