A data classification method, device, system, electronic device and storage medium

Abstract

The present application discloses a data clearing method, device, system, electronic device and storage medium. The method is executed by a Network Data Center (IDC) and includes: determining a database combination including a target primary database and a target standby database according to a clearing task and the status of each database; the target primary database is the available part of the primary database, and the target standby database is the backup of the unavailable part of the primary database in the standby database; generating an IDC-level database selection and clearing task according to the determined database combination; separately sending the database selection and clearing task to the IDC of the target primary database and the IDC of the target standby database, so that each IDC executes the received database selection and clearing task and returns IDC-level clearing summary information and IDC-level reconciliation information; generating batch summary information and batch clearing reconciliation information. By determining a database combination including a primary database and a standby database, the present application improves the accuracy of the clearing service when the primary database fails.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a data clearing method, device, system, electronic device, and storage medium. Background Art

[0002] In the business of a payment and settlement platform, clearing is the data preparation stage of settlement, and its main content is to summarize, organize, and classify all network transaction data of the day. Currently, the transaction database of the payment and settlement platform is often deployed in multiple IDCs (Internet Data Centers), each IDC has multiple sets of databases, and each IDC includes a primary database, a local backup database (referred to as the local standby database for short), and a remote backup database (referred to as the remote standby database for short). In the primary and standby synchronization mode of the database, the primary database and the local standby database adopt semi-synchronous synchronization, and the local standby database and the remote standby database adopt asynchronous synchronization.

[0003] When an abnormality occurs in a certain primary database, the clearing system will entirely switch to the standby database for clearing. However, since the data synchronization between the primary and standby databases may be interrupted or delayed due to reasons such as the network, the data accuracy of the standby database is often not as high as that of the primary database, thus causing an accuracy risk in the clearing business. Summary of the Invention

[0004] Embodiments of this application provide a data clearing method, device, system, electronic device, and storage medium to solve the problem of the business accuracy risk generated when the existing clearing system uses the standby database to replace the abnormal primary database for clearing.

[0005] Embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, this application proposes a data clearing method, which is executed by an Internet Data Center (IDC). The database relied on for clearing is a primary and standby architecture, and the primary database and the standby database for storing the same data are set on different IDCs. The method includes:

[0007] According to the clearing task and the status of each database, determine a database combination including a target primary database and a target standby database; the target primary database is the available part of the primary databases, and the target standby database is the backup of the unavailable part of the primary databases in the standby databases;

[0008] Generate an IDC-level database selection and clearing task according to the determined database combination;

[0009] Sending the database selection and clearing tasks to the IDC of the target primary database and the IDC of the target standby database respectively, so that the IDC of the target primary database and the IDC of the target standby database execute the received database selection and clearing tasks, and return the IDC-level clearing summary information and IDC-level reconciliation information;

[0010] Batch summary information and batch clearing and reconciliation information are generated based on the IDC-level clearing summary information and IDC-level reconciliation information.

[0011] Optionally, determining a database combination including a target primary database and a target standby database according to the clearing task and the status of each database includes:

[0012] Display the status of each database through the operation and maintenance interface;

[0013] Receive task configuration information fed back from the operation and maintenance interface, wherein the task configuration information includes database table information and data source address;

[0014] A database combination is determined according to the task configuration information.

[0015] Optionally, determining a database combination including a target primary database and a target standby database according to the clearing task and the status of each database includes:

[0016] The available part and the unavailable part in the primary database are determined, and according to the mapping relationship between the primary database and the standby database, the unavailable part in the primary database is replaced with the corresponding backup in the standby database, thereby forming the database combination.

[0017] Optionally, determining a database combination including a target primary database and a target standby database includes:

[0018] Determine the data source address of each database in the database combination;

[0019] The generating of the IDC-level database selection and clearing tasks according to the determined database combination includes: generating database selection and clearing tasks corresponding to each IDC according to the IDC corresponding to each data source address.

[0020] Optionally, generating an IDC-level database selection and clearing task according to the determined database combination includes:

[0021] The transaction aggregation strategy and transaction business strategy of clearing are queried, and the queried transaction aggregation strategy and transaction business strategy are encapsulated into the library selection and clearing task.

[0022] Optionally, the target standby database includes a local standby database and a remote standby database. When both the local standby database and the remote standby database are available, the local standby database is preferentially specified to form a database combination with the target master database.

[0023] Optionally, the IDC of the target primary database and the IDC of the target standby database execute the received database selection and clearing task, including at least one of the following:

[0024] The IDC of the target primary database receives and parses the database selection and clearing task through the management component of this IDC, and splits the database selection and clearing task into subtasks based on the parsing result of the database selection and clearing task, so that the database selection and clearing task is executed at the granularity of subtasks;

[0025] The IDC of the target standby database receives and parses the database selection and clearing task through the management component of this IDC, and splits the database selection and clearing task into subtasks based on the parsing result of the database selection and clearing task, so that the database selection and clearing task is executed at the granularity of subtasks.

[0026] Optionally, the receiving and parsing the database selection and clearing task includes: parsing database table information from the database selection and clearing task;

[0027] The splitting the database selection and clearing task into subtasks based on the parsing result of the database selection and clearing task includes: determining target database tables according to the database table information of this IDC, and generating subtasks corresponding to each target database table.

[0028] Optionally, the making the database selection and clearing task be executed at the granularity of subtasks includes:

[0029] Sending the subtasks to the execution component of this IDC, and processing the database tables specified by the subtasks through the execution component to obtain clearing data.

[0030] In a second aspect, the present application discloses a data clearing device, which is executed by a network data center IDC. The database relied on for clearing is a primary-standby architecture, and the primary database and the standby database for storing the same data are set on different IDCs. The device includes:

[0031] A combination determination module, configured to determine a database combination including a target primary database and a target standby database according to a clearing task and the status of each database. The target primary database is the available part of the primary database, and the target standby database is the backup of the unavailable part of the primary database in the standby database;

[0032] A task generation module, configured to generate a database selection and clearing task at the IDC level according to the determined database combination;

[0033] A task distribution module, configured to distribute the database selection and clearing tasks to the IDCs of the target primary database and the IDCs of the target standby database respectively, so that the IDCs of the target primary database and the IDCs of the target standby database execute the received database selection and clearing tasks, and return the clearing summary information at the IDC level and the reconciliation information at the IDC level;

[0034] A clearing summary module, configured to generate batch summary information and batch clearing reconciliation information according to the clearing summary information at the IDC level and the reconciliation information at the IDC level.

[0035] In a third aspect, the present application discloses a data clearing system, including multiple Internet Data Centers (IDCs), and at least some of the IDCs are provided with the data clearing device as described above.

[0036] In a fourth aspect, the present application discloses an electronic device, including:

[0037] A processor; and

[0038] A memory arranged to store computer-executable instructions, and when the executable instructions are executed, the processor is caused to execute the method as described in any one of the above.

[0039] In a fifth aspect, the present application discloses a computer-readable storage medium, where the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by an electronic device including multiple application programs, the electronic device is caused to execute the method as described in any one of the above.

[0040] At least one of the technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0041] In the present application, by determining a database combination including a target primary database and a target standby database for clearing according to the clearing task and the status of each database, where the target primary database is the available part of the primary database, and the target standby database is the backup of the unavailable part of the primary database in the standby database. Since the database combination uses the available part of the primary database, the clearing data is more reliable and accurate compared to simply using the standby database for clearing, thereby overcoming the problem of data accuracy risk caused by using only the standby database. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation to the present application. In the drawings:

[0043] Figure 1 It is a schematic diagram of the multi-IDB database topology of the payment and settlement platform;

[0044] Figure 2 A flowchart of a data clearing method according to an embodiment of the present application;

[0045] Figure 3 A flowchart of a data clearing method according to another embodiment of the present application;

[0046] Figure 4 A data flow chart to reflect the data advantages of the data clearing method of this application;

[0047] Figure 5 This is a block diagram of a data sorting device in an embodiment of the present application;

[0048] Figure 6 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0050] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0051] Figure 1 This is a schematic diagram of the multi-IDC database topology of the payment clearing platform, such as Figure 1 As shown in the figure, the transaction library is deployed in multiple IDCs. Each IDC has multiple databases, and each IDC includes a primary database, a local backup database, and a remote backup database. Figure 1 The figure shows cities A and B. Taking IDC-1 in city A as an example, it includes a main database (on the right), which has a backup database in the same city in city A, which is located in IDC-2 in city A (in the middle). In addition, the main database also has a remote backup database, which is located in IDC-2 in city B (on the left). Each backup database is used to provide backup data when the main database fails. In terms of data synchronization, the main database and the backup database in the same city are semi-synchronous, and the backup database in the same city and the remote backup database are asynchronously synchronized.

[0052] The current practice is that if a master database fails during clearing, the corresponding standby database is directly started for clearing. However, this solution is highly dependent on the stability of data synchronization. In reality, data synchronization between the master and standby databases is often interrupted or delayed due to network and other reasons, resulting in the data accuracy of the standby database being lower than that of the master database, thus causing clearing accuracy risks.

[0053] Therefore, the data sorting method in the embodiments of the present application is applied to the sorting scenario where the primary database fails, and a technical solution for determining a database combination for sorting according to the sorting task and the status of each database is proposed. The database combination includes a target primary database and a target standby database. Among them, the target primary database is the available part of the primary database, and the target standby database is the backup of the unavailable part of the primary database in the standby database. Since the database combination uses the available part of the primary database, compared with simply using the standby database for sorting, the sorted data is more reliable and accurate, thus overcoming the problem of data accuracy risk caused by using only the standby database.

[0054] Figure 2 It is a schematic flowchart of the data sorting method for an embodiment of the present application. Refer to Figure 2 As shown, the data sorting method of the present application is executed by the Internet Data Center (IDC). The databases relied on for sorting are in a primary-standby architecture, and the primary database and the standby database for storing the same data are set on different IDCs. The method includes:

[0055] Step S110: Determine a database combination including a target primary database and a target standby database according to the sorting task and the status of each database; where the target primary database is the available part of the primary database, and the target standby database is the backup of the unavailable part of the primary database in the standby database.

[0056] That is, according to the sorting task and the status of each database, the present application retains the available part of the primary database and uses the corresponding part in the standby database to replace the unavailable part in the primary database, thereby forming a database combination. Since this database combination uses the available part of the primary database, compared with using only the standby database, the data is more reliable, avoiding the data risk problem of the standby database caused by network problems, and the sorting accuracy is higher.

[0057] Step S120: Generate an IDC-level database selection and sorting task according to the determined database combination. The IDC-level database selection and sorting task refers to a task instruction for the Internet Data Center (IDC), which is used to indicate the task content of the sorting work to the IDC storing the primary database or the standby database. This database selection and sorting task specifies the sorting task and the database combination to be used, including the target primary database and the target standby database, and can inform the corresponding IDC to implement the sorting task according to the determined database combination information.

[0058] Step S130: Send the database selection and clearing tasks to the IDCs of the target primary database and the target standby database respectively, so that the IDCs of the target primary database and the target standby database execute the received database selection and clearing tasks, and return the clearing summary information at the IDC level and the reconciliation information at the IDC level. Specifically, each IDC executes the received database selection and clearing tasks through its management component (Manager) and execution component (Worker), and returns the clearing summary information at the IDC level and the reconciliation information at the IDC level.

[0059] Step S140: Based on the clearing summary information at the IDC level and the reconciliation information at the IDC level, generate batch summary information and batch clearing reconciliation information through operations such as adding timestamp content, and use this information to provide services externally to implement the clearing service.

[0060] In an embodiment of the present application, determining the database combination including the target primary database and the target standby database according to the clearing task and the status of each database includes:

[0061] Display the status of each database through the operation and maintenance operation interface; receive the task configuration information fed back by the operation and maintenance operation interface, where the task configuration information includes database table information and data source addresses; determine the database combination according to the task configuration information.

[0062] Figure 3 It is a schematic flowchart of the database selection and clearing method for another embodiment of the present application. As Figure 3 shown, the database selection and clearing method of the embodiment of the present application starts from the operation and maintenance operation interface. The operation and maintenance operation interface has multiple options such as IDC selection, database selection, and data source selection. The operation and maintenance personnel can specify the IDC, database, and data source through the operation and maintenance operation interface according to the database status. Among them, the status of each database is displayed through the operation and maintenance operation interface, which is convenient for the operation and maintenance personnel to view and operate, so as to select the relevant IDC, database, and data source according to the clearing task. After the operation and maintenance personnel select information such as IDC, database, and data source, click to trigger the task, and the clearing task will be sent to the control component (Controller) of the IDC, and the next task parsing and encapsulation and further sending and execution will be carried out.

[0063] In addition to the above manual selection of the database combination by the operation and maintenance personnel through the operation and maintenance operation interface, in another embodiment of the present application, a computer software program can also be used to automatically generate the database combination. For example, in step S110, determining the database combination including the target primary database and the target standby database according to the clearing task and the status of each database includes:

[0064] Determine the available and unavailable parts in the primary database, and select corresponding backups in the standby database to replace the unavailable parts in the primary database according to the mapping relationship between the primary database and the standby database, thereby forming a database combination.

[0065] That is to say, by utilizing the mapping relationship between the primary database and the standby database, and based on the status of each database, query and retrieve are performed through a computer program, and the available part of the primary database is combined with the backup part of the standby database to generate a database combination. This can save manpower and improve the speed and accuracy of generating database combinations.

[0066] Since in the IDC topology, the database name, table name, and table field of each standby database are exactly the same as those of the primary database, the only difference is the data source address. Therefore, in one embodiment of the present application, the mapping relationship between the primary database and the standby database is expressed as a data source address mapping relationship. In step S110, determining a database combination including a target primary database and a target standby database includes: determining the data source address of each database in the database combination. Correspondingly, in step S120, generating an IDC-level library selection and clearing task based on the determined database combination includes: generating a library selection and clearing task corresponding to each IDC based on the IDC corresponding to each data source address.

[0067] In one embodiment of the present application, generating an IDC-level database selection and clearing task according to a determined database combination includes: querying transaction aggregation strategies and transaction business strategies for clearing, and encapsulating the queried transaction aggregation strategies and transaction business strategies into the database selection and clearing task.

[0068] Among them, the transaction business strategy is used to determine the clearing target, that is, to solve the problem of what to clear, including the type of clearing tasks, what types of transactions to clear, etc., for example, to analyze, organize and summarize debit and credit transaction data respectively; the transaction aggregation strategy is used to determine the clearing method, that is, to solve the problem of how to clear, including which fields of the transaction are aggregated to generate clearing information when clearing transactions, for example, to generate clearing information by aggregating the "debit" and "credit" field information. Therefore, the encapsulated database selection and clearing task includes both the database combination information as the data source and the transaction aggregation strategy and transaction business strategy as the execution information, so that database selection and clearing can be realized and transaction data can be processed.

[0069] In one embodiment of the present application, the target standby database includes a local standby database and a remote standby database. When both the local standby database and the remote standby database are available, the local standby database is preferentially specified to form a database combination with the target master database.

[0070] Since the distance between the same-city backup database and the main database is closer and the network connection is better, the data reliability is higher than that of the off-site backup database. Therefore, when both the same-city backup database and the off-site backup database are available, this application gives priority to using the same-city backup database for clearing data replacement.

[0071] In one embodiment of the present application, the IDC of the target primary database and the IDC of the target standby database execute the received database selection and clearing tasks, including at least one of the following: the IDC of the target primary database receives and parses the database selection and clearing tasks through the management component Manager of the IDC, splits the database selection and clearing tasks into subtasks based on the parsing results of the database selection and clearing tasks, and executes the database selection and clearing tasks at the granularity of subtasks; the IDC of the target standby database receives and parses the database selection and clearing tasks through the management component Manager of the IDC, splits the database selection and clearing tasks into subtasks based on the parsing results of the database selection and clearing tasks, and executes the database selection and clearing tasks at the granularity of subtasks.

[0072] That is, after the database selection and clearing task is sent to the IDC where the primary database or the standby database is located, the management component of the IDC itself can parse and split the task to extract the clearing task to be completed by this IDC from the overall database selection and clearing task, so as to perform specific subtask operations.

[0073] In one embodiment of the present application, the management component of each IDC receives and analyzes the database selection and clearing task, including: parsing database table information from the database selection and clearing task. Further, based on the parsing result of the database selection and clearing task, splitting the database selection and clearing task into subtasks includes: determining the target database table according to the database table information of the IDC, and generating subtasks corresponding to each target database table.

[0074] By parsing and encapsulating tasks through the corresponding management components of each IDC, the extraction and division of the clearing tasks on each IDC is realized, thereby generating clearing subtasks that are only related to the database on each IDC. These subtasks are further sent to the execution component worker to perform specific execution operations of the clearing tasks.

[0075] In one embodiment of the present application, each IDC executes the library selection and clearing task at the granularity of a subtask, including: sending the subtask to the execution component of the IDC, and processing the database table specified by the subtask through the execution component to obtain clearing data.

[0076] Specific combination Figure 3 After the operation and maintenance personnel select the database combination according to the clearing task, the control component Controller will parse the task, query the clearing transaction aggregation strategy and transaction business strategy from the configuration center, encapsulate them together and route them to each IDC.

[0077] Next, after each IDC receives the encapsulated library selection and clearing task, it can parse the task through the management component Manager, and find the corresponding library table information when the IDC processes the clearing task from the configuration center according to the task type. Based on the library table information, the library selection and clearing task is split into subtasks and asynchronously distributed to the execution component Worker.

[0078] After receiving the subtask, the execution component Worker will know the data source involved in the clearing, that is, which table of which database. At the same time, the execution component Worker also obtains the specific execution information of the clearing task, that is, the transaction business strategy and the transaction aggregation strategy. The transaction data to be cleared is determined according to the transaction business strategy, and how to perform the clearing is determined according to the transaction aggregation strategy. That is, the execution component Worker only needs to parse the subtasks and execute the tasks in full accordance with the subtask list to achieve the corresponding clearing tasks and generate clearing summary information and reconciliation information at the library and table level. Then, the management component Manager processes the library and table level clearing summary information and library and table level reconciliation information generated by the execution component Worker of this IDC into IDC-level clearing summary information and reconciliation information, and reports them to the control component Controller.

[0079] The control component Controller then processes the IDC-level clearing summary information and IDC-level reconciliation information into batch clearing summary information and batch clearing reconciliation information by adding timestamps and other methods, and provides services to the outside world, thus achieving a complete data clearing task. The data accuracy advantage of the above clearing process can be achieved through Figure 4 The data flow diagram shown is significantly reflected.

[0080] like Figure 4 As shown, the local transaction database in IDC1 is the main database, and IDC2 has a local backup database of IDC1's local transaction database. When the 01 database in the main database fails, the original clearing task will fail or go wrong. At this time, in this embodiment, the control component Controller of the IDC generates a database selection clearing task and sends it to the management component Manager of different IDCs for re-clearing (the management component Manager is a cluster structure, Figure 4 It is referred to as a management component Manager).

[0081] This database selection and clearing task abandons the traditional method of using only the backup database for clearing, and instead improves data accuracy by selecting a database combination. The database combination includes a target primary database consisting of the available part of the primary database (i.e., the part other than the local transaction database 01 in IDC1), and a target backup database consisting of a backup that replaces the unavailable part of the primary data in the backup database (i.e., the 01 part of the backup database in the same city in IDC2). After receiving the above database selection and clearing task, the management component Manager of each IDC will parse the database selection and clearing task to obtain the clearing task type, and find the corresponding library table information when its own IDC processes the database selection and clearing task based on the task type. Based on the library table information, the database selection and clearing task will be split into subtasks and asynchronously distributed to the execution component Worker for execution.

[0082] After receiving the subtask, each execution component Worker can clear the selected database content according to the database table information. Figure 4 For the illustrated embodiment, it includes the available part of the local transaction database in IDC1 except 01, and the part 01 in the same-city backup database in IDC2.

[0083] It can be seen that in the data clearing method embodiment of the present application, clearing is performed by selecting a database combination including a primary database and a backup database, and only a partial backup in the backup database is used to replace the unavailable part of the primary database, and other data sources still come from the primary database, thereby avoiding the accuracy business risks caused by data synchronization problems when the backup database is used entirely for clearing, and improving the security of the clearing results.

[0084] Figure 5 The data clearing device block diagram of the embodiment of the present application is executed by the network data center IDC, wherein the database relied on by the clearing is a master-slave architecture, and the master database and the backup database for storing the same data are set on different IDCs, such as Figure 5 As shown, the data clearing device includes:

[0085] The combination determination module 510 is used to determine a database combination including a target primary database and a target standby database according to the clearing task and the status of each database. The target primary database is the available part of the primary database, and the target standby database is a backup of the unavailable part of the primary database in the standby database.

[0086] The task generation module 520 is used to generate an IDC-level database selection and clearing task according to the determined database combination.

[0087] The task distribution module 530 is used to distribute the database selection and clearing tasks to the IDCs of the target primary database and the target standby database respectively, so that the IDCs of the target primary database and the target standby database execute the received database selection and clearing tasks, and return the clearing summary information at the IDC level and the reconciliation information at the IDC level.

[0088] The clearing summary module 540 is used to generate batch summary information and batch clearing reconciliation information according to the clearing summary information at the IDC level and the reconciliation information at the IDC level.

[0089] In an embodiment of the present application, the combination determination module 510 is specifically configured to display the status of each database through the operation and maintenance operation interface; receive the task configuration information fed back by the operation and maintenance operation interface, where the task configuration information includes database table information and data source addresses; and determine the database combination according to the task configuration information.

[0090] In an embodiment of the present application, the combination determination module 510 is specifically configured to determine the available part and the unavailable part in the primary database, and select the corresponding backup in the standby database to replace the unavailable part in the primary database according to the mapping relationship between the primary database and the standby database, so as to form a database combination.

[0091] In an embodiment of the present application, the combination determination module 510 is specifically configured to determine the data source addresses of each database in the database combination. Correspondingly, the task generation module 520 is used to generate database selection and clearing tasks corresponding to each IDC according to the IDCs corresponding to each data source address.

[0092] In an embodiment of the present application, the task generation module 520 is used to query the transaction aggregation policy and transaction service policy for clearing, and encapsulate the queried transaction aggregation policy and transaction service policy into the database selection and clearing tasks.

[0093] In an embodiment of the present application, the target standby database includes a local standby database and a remote standby database. When both the local standby database and the remote standby database are available, the combination determination module 510 preferentially designates the local standby database to form a database combination with the target primary database.

[0094] In one embodiment of the present application, the task distribution module 530 causes the IDC of the target primary database and the IDC of the target standby database to execute the received database selection and clearing task, including at least one of the following: causing the IDC of the target primary database to receive and parse the database selection and clearing task through the management component of this IDC, splitting the database selection and clearing task into subtasks based on the parsing result of the database selection and clearing task, and causing the database selection and clearing task to be executed at the granularity of subtasks; causing the IDC of the target standby database to receive and parse the database selection and clearing task through the management component of this IDC, splitting the database selection and clearing task into subtasks based on the parsing result of the database selection and clearing task, and causing the database selection and clearing task to be executed at the granularity of subtasks.

[0095] In one embodiment of the present application, the task distribution module 530 causing the target primary database and / or the target standby database to receive and parse the database selection and clearing task includes: causing the management component to parse out the database table information from the database selection and clearing task; determining the target database tables according to the database table information of this IDC, and generating subtasks corresponding to each target database table.

[0096] In one embodiment of the present application, the task distribution module 530 causing the database selection and clearing task to be executed at the granularity of subtasks includes: sending the subtasks to the execution component of this IDC, and processing the database tables specified by the subtasks through the execution component to obtain the cleared data.

[0097] The present application also discloses a data clearing system, including a plurality of network data centers IDCs, and at least some of the IDCs are provided with the above data clearing device.

[0098] It can be understood that the above data clearing device can implement each step of the data clearing method provided in the foregoing embodiments. The relevant explanations regarding the data clearing method are applicable to the data clearing device and will not be elaborated here.

[0099] The present application also discloses an electronic device, including: a processor; and a memory arranged to store computer-executable instructions, where the executable instructions, when executed, cause the processor to execute any one of the above data clearing methods.

[0100] The present application also discloses a computer-readable storage medium, where the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device is caused to execute any one of the above data clearing methods.

[0101] Figure 6 It is a schematic structural diagram of an electronic device according to an embodiment of the present application. Please refer to Figure 6, at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include internal memory, such as high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk memory, etc. Of course, the electronic device may also include other hardware required for other services.

[0102] The processor, network interface, and memory can be interconnected through the internal bus, and the internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 6 only a bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0103] The memory is used to store programs. Specifically, the program can include program code, and the program code includes computer operation instructions. The memory can include internal memory and non-volatile memory, and provide instructions and data to the processor.

[0104] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a data sorting device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0105] According to the sorting task and the status of each database, determine the database combination including the target primary database and the target standby database; the target primary database is the available part in the primary database, and the target standby database is the backup of the unavailable part in the primary database in the standby database;

[0106] Generate an IDC-level database selection and sorting task according to the determined database combination;

[0107] Send the database selection and sorting tasks to the IDCs of the target primary database and the IDCs of the target standby database respectively, so that the IDCs of the target primary database and the IDCs of the target standby database execute the received database selection and sorting tasks and return the IDC-level sorting summary information and IDC-level reconciliation information;

[0108] Generate batch summary information and batch sorting reconciliation information according to the IDC-level sorting summary information and IDC-level reconciliation information.

[0109] As described above in the present application Figure 5 The method executed by the data sorting device disclosed in the above - shown embodiments can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above - mentioned method can be completed by the integrated logic circuit of the hardware in the processor or the instructions in the form of software. The above - mentioned processor may be a general - purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application - specific integrated circuit (ASIC), a field - programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general - purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by the combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read - only memory, a programmable read - only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above - mentioned method.

[0110] The electronic device can also execute Figure 2 the method executed by the data sorting device in Figure 2 the above - shown embodiments and implement the functions of the data sorting device in the above - shown embodiments. The embodiments of the present application will not be elaborated herein.

[0111] The embodiments of the present application also propose a computer - readable storage medium. The computer - readable storage medium stores one or more programs. The one or more programs include instructions. When the instructions are executed by an electronic device including a plurality of application programs, the electronic device can execute Figure 5 the method executed by the data sorting device in the above - shown embodiments, and specifically used to execute:

[0112] Determine a database combination including a target primary database and a target standby database according to the sorting task and the status of each database; the target primary database is the available part of the primary database, and the target standby database is the backup of the unavailable part of the primary database in the standby database;

[0113] Generate an IDC-level database selection and clearing task based on the determined database combination;

[0114] Send the database selection and clearing task to the IDCs of the target primary database and the target standby database respectively, so that the IDCs of the target primary database and the target standby database execute the received database selection and clearing task and return the IDC-level clearing summary information and the IDC-level reconciliation information;

[0115] Generate batch summary information and batch clearing reconciliation information based on the IDC-level clearing summary information and the IDC-level reconciliation information.

[0116] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0118] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0119] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable device provide means for implementing the functions in the processFigure 1 one or more processes and / or blocks Figure 1 steps of the functions specified in one or more blocks.

[0120] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0121] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

[0122] Computer-readable media includes both permanent and non-permanent, removable and non-removable media implemented by any method or technology for information storage. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0123] It should also be noted that the term "comprises," "comprising," or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0124] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0125] The above are only the embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A data sorting method, which is executed by a network data center IDC, wherein, The database relied on by the clearing house has a primary-standby architecture, and the primary database and the standby database for storing the same data are set in different IDCs. The method includes: Determine a database combination including a target primary database and a target standby database according to the clearing task and the status of each database; the target primary database is the available part of the primary database, and the target standby database is the backup of the unavailable part in the primary database in the standby database; Generate an IDC-level database selection and clearing task according to the determined database combination; Send the database selection and clearing task to the IDC of the target primary database and the IDC of the target standby database respectively, so that the IDC of the target primary database and the IDC of the target standby database execute the received database selection and clearing task, and return the IDC-level clearing summary information and the IDC-level reconciliation information; Generate batch summary information and batch clearing reconciliation information according to the IDC-level clearing summary information and the IDC-level reconciliation information; The IDC of the target primary database and the IDC of the target standby database execute the received database selection and clearing task, including at least one of the following: The IDC of the target primary database receives and parses the database selection and clearing task through the management component of this IDC, and splits the database selection and clearing task into subtasks based on the parsing result of the database selection and clearing task, so that the database selection and clearing task is executed at the granularity of subtasks; The IDC of the target standby database receives and parses the database selection and clearing task through the management component of this IDC, and splits the database selection and clearing task into subtasks based on the parsing result of the database selection and clearing task, so that the database selection and clearing task is executed at the granularity of subtasks.

2. The method according to claim 1, wherein The determining a database combination including a target primary database and a target standby database according to the clearing task and the status of each database includes: Display the status of each database through the operation and maintenance operation interface; Receive the task configuration information fed back by the operation and maintenance operation interface, and the task configuration information includes database table information and data source address; Determine the database combination according to the task configuration information.

3. The method according to claim 1, wherein The determining a database combination including a target primary database and a target standby database according to the clearing task and the status of each database includes: Determine the available part and the unavailable part in the primary database, and according to the mapping relationship between the primary database and the standby database, select the corresponding backup in the standby database to replace the unavailable part in the primary database, so as to form the database combination.

4. The method according to claim 1, wherein The determining a database combination including a target primary database and a target standby database includes: Determine the data source addresses of each database in the database combination; The generating an IDC-level database selection and clearing task according to the determined database combination includes: generating database selection and clearing tasks corresponding to each IDC respectively according to the IDC corresponding to each data source address.

5. The method according to claim 1, wherein The generating an IDC-level database selection and clearing task according to the determined database combination includes: Query the transaction aggregation strategy and transaction business strategy for clearing, and encapsulate the queried transaction aggregation strategy and transaction business strategy into the database selection and clearing task.

6. The method according to claim 1, wherein The target standby database includes a local standby database and a remote standby database. When both the local standby database and the remote standby database are available, the local standby database is preferentially designated to form a database combination with the target primary database.

7. The method according to claim 1, wherein Receiving and parsing the database selection and clearing task includes: parsing database table information from the database selection and clearing task; Splitting the database selection and clearing task into subtasks based on the parsing result of the database selection and clearing task includes: determining target database tables according to the database table information of this IDC, and generating subtasks corresponding to each target database table.

8. The method according to claim 7, wherein, Making the database selection and clearing task execute at the granularity of subtasks includes: Sending the subtasks to the execution components of this IDC, and processing the database tables specified by the subtasks through the execution components to obtain clearing data.

9. A data classification device, which is executed by a network data center IDC, wherein, The database relied on for clearing is a primary-standby architecture, and the primary database and the standby database for storing the same data are set on different IDCs. The device includes: A combination determination module, configured to determine a database combination including a target primary database and a target standby database according to the clearing task and the status of each database. The target primary database is the available part of the primary database, and the target standby database is the backup of the unavailable part of the primary database in the standby database; A task generation module, configured to generate a database selection and clearing task at the IDC level according to the determined database combination; A task distribution module, configured to distribute the database selection and clearing task to the IDC of the target primary database and the IDC of the target standby database respectively, so that the IDC of the target primary database and the IDC of the target standby database execute the received database selection and clearing task, and return the clearing summary information at the IDC level and the reconciliation information at the IDC level; A clearing summary module, configured to generate batch summary information and batch clearing reconciliation information according to the clearing summary information at the IDC level and the reconciliation information at the IDC level; The IDC of the target primary database and the IDC of the target standby database execute the received database selection and clearing task, including at least one of the following: The IDC of the target primary database receives and parses the database selection and clearing task through the management component of this IDC, splits the database selection and clearing task into subtasks based on the parsing result of the database selection and clearing task, and makes the database selection and clearing task execute at the granularity of subtasks; The IDC of the target standby database receives and parses the database selection and clearing task through the management component of this IDC, splits the database selection and clearing task into subtasks based on the parsing result of the database selection and clearing task, and makes the database selection and clearing task execute at the granularity of subtasks.

10. A data classification system includes multiple Internet Data Centers (IDCs), where, At least some IDCs are provided with the data clearing device according to claim 9.

11. An electronic device, including: A processor; And A memory arranged to store computer-executable instructions, which when executed cause the processor to execute the method according to any one of claims 1 to 8.

12. A computer-readable storage medium storing one or more programs which, when executed by an electronic device including a plurality of application programs, cause the electronic device to perform the method according to any one of claims 1 to 8.

Images