Distributed reporting system and method

Abstract

The present invention provides a distributed reporting system and method, which is applicable to the field of big data technology. The system includes: a host system, multiple sets of computer room systems, and an information service system; each set of computer room systems includes: a source table database is used to: after subscribing to the data of the host system, flush the data to the intermediate table, and mark the reporting status field of the intermediate table as unprocessed; subscribe to the reporting status field of the intermediate table of other computer room systems; a processing server is used to: process the data in the intermediate table, obtain the processed data, and store it in a target database; an MQ server is used to: push the processed data to the information service system; if the push is successful, update the reporting status field to successful, otherwise, update to the intermediate state; the target database is used to: read the reporting status field, if there is an intermediate state and the target database has full data, send the data in the target database to the information service system. The present invention can realize the active-active distributed reporting system.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a distributed reporting system and method. Background Art

[0002] With the development trend of host migration, simply moving query transactions downward can no longer meet the requirements. It is necessary to gradually move reporting transactions such as information service reporting on the host to x86. Reporting transactions generally write the main reporting information into an intermediate table, flush the intermediate table to x86, and then retrieve them one by one on x86, and then supplement other information such as account name and organization, write them into the target table and target system, and then update the intermediate table to indicate that the reporting has been successful. It can be seen that the reporting system is not only for query, but also for update. Therefore, how to achieve dual-activeness in the reporting system becomes a difficult problem. Summary of the invention

[0003] The embodiment of the present invention provides a distributed reporting system for realizing active-active distributed reporting system, the system comprising: a host system, multiple sets of computer room systems, and an information service system;

[0004] Among them, each computer room system includes a source table database, a processing server, an MQ server, and a target database;

[0005] The source table database is used to: after subscribing to the data of the host system, flush the data to the intermediate table of the source table database, and mark the reporting status field of the intermediate table as unprocessed, wherein the reporting status field is used to indicate the reporting status of the data; subscribe to the reporting status field of the intermediate table of the source table database of other computer room systems;

[0006] The processing server is used to: process the data in the intermediate table to obtain processed data; store the processed data into the target database;

[0007] The MQ server is used to: push the processed data to the information service system; if the push is successful, update the reporting status field of the intermediate table to successful; otherwise, update the reporting status field of the intermediate table to intermediate status;

[0008] The target database is used to: read the reporting status field of the intermediate table. If the intermediate status exists and the target database has full data, the data in the target database is sent to the information service system. When the sending is successful, the reporting status field of the intermediate table is updated to successful, otherwise, the reporting status field of the intermediate table is updated to unprocessed; the information service system merges the received data.

[0009] The embodiment of the present invention provides a distributed reporting method for realizing active-active distributed reporting system. The method includes:

[0010] After subscribing to the data of the host system, the data is flushed to the intermediate table, and the reporting status field of the intermediate table is marked as unprocessed, and the reporting status field is used to indicate the reporting status of the data; subscribing to the reporting status field of the intermediate table of the source table database of other computer room systems;

[0011] Process the data in the intermediate table to obtain processed data; store the processed data in the target database;

[0012] Push the processed data to the information service system; if the push is successful, update the reporting status field of the intermediate table to successful; otherwise, update the reporting status field of the intermediate table to intermediate status;

[0013] Read the reporting status field of the intermediate table. If the intermediate status exists and the target database has full data, send the data in the target database to the information service system. When the sending is successful, update the reporting status field of the intermediate table to successful. Otherwise, update the reporting status field of the intermediate table to unprocessed. The information service system merges the received data.

[0014] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned distributed reporting method when executing the computer program.

[0015] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned distributed reporting method is implemented.

[0016] An embodiment of the present invention further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the above-mentioned distributed reporting method is implemented.

[0017] In an embodiment of the present invention, a distributed reporting system includes: a host system, multiple sets of computer room systems, and an information service system; wherein each set of computer room systems includes a source table database, a processing server, an MQ server, and a target database; the source table database is used to: after subscribing to the data of the host system, flush the data to the intermediate table of the source table database, and mark the reporting status field of the intermediate table as unprocessed, wherein the reporting status field is used to indicate the reporting status of the data; subscribe to the reporting status field of the intermediate table of the source table database of other computer room systems; the processing server is used to: process the data in the intermediate table to obtain the processed data; and mark the processed data as unprocessed. The data is stored in the target database; the MQ server is used to push the processed data to the information service system; if the push is successful, the reporting status field of the intermediate table is updated to successful, otherwise, the reporting status field of the intermediate table is updated to the intermediate state; the target database is used to read the reporting status field of the intermediate table, if there is an intermediate state and the target database has full data, the data in the target database is sent to the information service system, and when the sending is successful, the reporting status field of the intermediate table is updated to successful, otherwise, the reporting status field of the intermediate table is updated to unprocessed; the information service system merges the received data. In the above process, the distributed reporting system includes multiple sets of computer room systems, so that various types of servers in each computer room system realize dual-active applications. In order to improve efficiency, distributed transactions with two-stage submission are not used, but data consistency is achieved through the reporting status of the data in the reporting status field; if there is an intermediate state and the target database has full data, the data in the target database is sent to the information service system to ensure that the data is reported without omission. The source table database in each computer room system subscribes to the reporting status column of the intermediate table of the source table database of the other computer room system, so that the data of the computer room system that cannot be used can be obtained through replication and update, thus realizing data dual activity. In addition, the two computer room systems can be deployed separately, thus realizing network dual activity. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1 Schematic diagram of a distributed reporting system in an embodiment of the present invention Figure 1 ;

[0020] Figure 2 A schematic diagram of a distributed reporting system in an embodiment of the present invention;

[0021] Figure 3Schematic diagram of a distributed reporting system in an embodiment of the present invention Figure 2 ;

[0022] Figure 4 Flow chart of the distributed reporting method in an embodiment of the present invention;

[0023] Figure 5 Schematic diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0025] In the description of this specification, the terms "include", "including", "have", "contain", etc. are all open terms, which mean including but not limited to. The descriptions with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", etc. mean that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. The order of steps involved in each embodiment is used to schematically illustrate the implementation of the present application, and the order of steps is not limited and can be appropriately adjusted as needed.

[0026] Figure 1 Schematic diagram of a distributed reporting system in an embodiment of the present invention Figure 1 ,like Figure 1 As shown, the system includes:

[0027] Host system 101, multiple computer room systems 102, information service system 103;

[0028] Each computer room system includes a source table database 1021, a processing server 1022, an MQ server 1023 and a target database 1024;

[0029] The source table database 1021 is used to: after subscribing to the data of the host system, flush the data to the intermediate table of the source table database, and mark the reporting status field of the intermediate table as unprocessed, wherein the reporting status field is used to indicate the reporting status of the data; subscribe to the reporting status field of the intermediate table of the source table database of other computer room systems;

[0030] The processing server is used for 1022: processing the data in the intermediate table to obtain processed data; and storing the processed data in the target database;

[0031] The MQ server 1023 is used to: push the processed data to the information service system; if the push is successful, update the reporting status field of the intermediate table to successful; otherwise, update the reporting status field of the intermediate table to intermediate status;

[0032] The target database 1024 is used to: read the reporting status field of the intermediate table. If the intermediate status exists and the target database has full data, the data in the target database is sent to the information service system. When the sending is successful, the reporting status field of the intermediate table is updated to successful, otherwise, the reporting status field of the intermediate table is updated to unprocessed; the information service system merges the received data.

[0033] In an embodiment of the present invention, the distributed reporting system includes multiple sets of computer room systems, so that various types of servers in each computer room system realize active-active applications. In order to improve efficiency, distributed transactions with two-stage submission are not used, but data consistency is achieved through the reporting status of the data in the reporting status field; if there is an intermediate state and the target database has full data, the data in the target database is sent to the information service system, ensuring that the data is reported without omission. The source table database in each computer room system subscribes to the reporting status field of the intermediate table of the source table database of other computer room systems, so that the data of the computer room system that cannot be used can be obtained by copying, updating and refreshing, thereby realizing active-active data. In addition, the two computer room systems can be deployed separately, thereby realizing active-active networks.

[0034] Figure 2 The source table database may be a cluster including multiple source table databases, and the target database may also be a cluster including multiple target databases. Figure 3 Schematic diagram of a distributed reporting system in an embodiment of the present invention Figure 2 , see Figure 2 and Figure 3 , in one embodiment, the system further includes a data synchronization tool 104;

[0035] The source table database is specifically used to: after subscribing to the data of the host system, flush the data to the intermediate table through a data synchronization tool.

[0036] When the source table database subscribes to the host system, since the data in the host system does not have the field of reporting status STAF, all fields except the STAF field are subscribed at this time.

[0037] See also Figure 2 , in one embodiment, the processing server comprises;

[0038] The scheduling server is used to shard the data in the intermediate table and generate shard data tasks; and dispatch the shard data tasks to the application server;

[0039] The application server is used to process the sharding data tasks and obtain the processed data; the processed data is stored in the target database.

[0040] In one embodiment, the scheduling server is specifically used for:

[0041] Determine the number of shards and threads based on the business volume TPS peak, accumulation scenarios, and non-accumulation scenarios;

[0042] Sharding the data in the intermediate table according to the number of shards;

[0043] The application server is also used to: process shard data tasks according to the number of threads.

[0044] For example, the number of shards can be 10 and the number of threads can be 5.

[0045] In one embodiment, the processing server further includes a distributed coordination server, configured to:

[0046] When there are multiple dispatch servers, coordinate the data processed by each dispatch server;

[0047] When there are multiple application servers, coordinate the shard data tasks processed by each application server;

[0048] When there are multiple MQ servers, coordinate the data pushed by each MQ server.

[0049] Since the number of shards and threads is inconsistent, and there are multiple servers, it is necessary to coordinate the data or tasks processed by each server. At this time, a distributed coordination server can be used. In addition, there can be multiple distributed coordination servers.

[0050] In the embodiment of the present invention, in order to improve efficiency, the system does not use two-phase commit distributed transactions, but achieves consistency by applying idempotence, supporting dynamic multi-data source access. The idea of ​​idempotence is to maintain data consistency by reporting status.

[0051] In order to ensure that all reported transactions are complete, this system implements supplementary reporting, that is, if there is an intermediate state and the target database has full data, the data in the target database will be sent to the information service system. If the sending is successful, the reporting status field of the intermediate table will be updated to successful, otherwise, the reporting status field of the intermediate table will be updated to unprocessed. The information service system merges the received data, that is, it does not receive the existing data, but only receives the non-existent data.

[0052] In one embodiment, the target database is also used to:

[0053] After the data in the target database is sent to the information service system, the data in the target database is cleared.

[0054] Clearing can also be scheduled, for example, clear it every night and apply it the next day.

[0055] In order to achieve data dual activity, the source table database subscribes to the reporting status field of the intermediate table of the source table database of other computer room systems; so that the data of the unavailable computer room system can be obtained through replication and update, thus achieving data dual activity.

[0056] In one embodiment, the system further comprises a control table for:

[0057] After receiving the computer room system switching instruction, the computer room system is switched.

[0058] In this way, network active-active mode can be achieved without interactive calls. For example, if an abnormality occurs in the system of computer room A, the system of computer room B can completely take over. Similarly, if an abnormality occurs in the system of computer room B, the system of computer room A can also completely take over, thus achieving network active-active mode and high availability.

[0059] In summary, in the system proposed in the embodiment of the present invention, the distributed reporting system includes multiple sets of computer room systems, so that various types of servers in each computer room system realize active-active applications. In order to improve efficiency, distributed transactions with two-stage submission are not used, but data consistency is achieved through the reporting status of the data in the reporting status field; if there is an intermediate state and the target database has full data, the data in the target database is sent to the information service system, ensuring that the data is reported without omission. The source table database in each computer room system subscribes to the reporting status field of the intermediate table of the source table database of other computer room systems, so that the data of the computer room system that cannot be used can be obtained by copying, updating and refreshing, thereby realizing active-active data. In addition, the two computer room systems can be deployed separately, thereby realizing active-active network.

[0060] The present invention also provides a distributed reporting method, as described in the following embodiments. Since the principle of solving the problem by this method is similar to that of the distributed reporting system, the implementation of the device can refer to the implementation of the distributed reporting system, and the repeated parts will not be repeated.

[0061] Figure 4 The flowchart of the distributed reporting method in an embodiment of the present invention includes:

[0062] Step 401, after subscribing to the data of the host system, flush the data to the intermediate table, and mark the reporting status field of the intermediate table as unprocessed, wherein the reporting status field is used to indicate the reporting status of the data; subscribe to the reporting status field of the intermediate table of the source table database of other computer room systems;

[0063] Step 402, processing the data in the intermediate table to obtain processed data; storing the processed data in the target database;

[0064] Step 403, push the processed data to the information service system; if the push is successful, update the reporting status field of the intermediate table to successful; otherwise, update the reporting status field of the intermediate table to intermediate status;

[0065] Step 404, read the reporting status field of the intermediate table. If the intermediate status exists and the target database has full data, send the data in the target database to the information service system. When the sending is successful, update the reporting status field of the intermediate table to successful. Otherwise, update the reporting status field of the intermediate table to unprocessed. The information service system merges the received data.

[0066] In one embodiment, after subscribing to the data of the host system, flushing the data to the intermediate table includes:

[0067] After subscribing to the data in the host system, use the data synchronization tool to flush the data to the intermediate table.

[0068] In one embodiment, processing the data in the intermediate table to obtain processed data includes:

[0069] Shard the data in the intermediate table and generate shard data tasks; dispatch the shard data tasks to the application server;

[0070] Process the shard data task and obtain the processed data.

[0071] In one embodiment, the method further comprises:

[0072] Determine the number of shards and threads based on the business volume TPS peak, accumulation scenarios, and non-accumulation scenarios;

[0073] Sharding the data in the intermediate table according to the number of shards;

[0074] Processing the shard data task includes: processing the shard data task according to the number of threads.

[0075] In one embodiment, the method further comprises:

[0076] After the data in the target database is sent to the information service system, the data in the target database is cleared.

[0077] In one embodiment, the method further comprises:

[0078] After receiving the computer room system switching instruction, the computer room system is switched.

[0079] In summary, in the method proposed in the embodiment of the present invention, it can be realized that multiple sets of computer room systems are included, so that various types of servers in each set of computer room systems can achieve dual-active applications. In order to improve efficiency, distributed transactions with two-stage submission are not used, but data consistency is achieved through the reporting status of the data in the reporting status field; if there is an intermediate state and the target database has full data, the data in the target database is sent to the information service system, ensuring that the data is reported without omission. By subscribing to the reporting status field of the intermediate table of the source table database of other computer room systems, the data of the computer room system that cannot be used can be obtained by copying, updating and refreshing, thereby achieving data dual-active. In addition, the two sets of computer room systems can be deployed separately, thereby achieving network dual-active.

[0080] An embodiment of the present invention further provides a computer device, Figure 5 This is a schematic diagram of a computer device in an embodiment of the present invention. The computer device 500 includes a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 530, the above-mentioned distributed reporting method is implemented.

[0081] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned distributed reporting method is implemented.

[0082] An embodiment of the present invention further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the above-mentioned distributed reporting method is implemented.

[0083] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program business systems. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program business system implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0084] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program business systems according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0085] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0086] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0087] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A distributed reporting system, characterized in that: include: Host system, multiple computer room systems, information service system; Among them, each computer room system includes a source table database, a processing server, an MQ server, and a target database; The source table database is used to: after subscribing to the data of the host system, flush the data to the intermediate table of the source table database, and mark the reporting status field of the intermediate table as unprocessed, wherein the reporting status field is used to indicate the reporting status of the data; subscribe to the reporting status field of the intermediate table of the source table database of other computer room systems; The processing server is used to: process the data in the intermediate table to obtain processed data; store the processed data into the target database; The MQ server is used to: push the processed data to the information service system; if the push is successful, update the reporting status field of the intermediate table to successful; otherwise, update the reporting status field of the intermediate table to intermediate status; The target database is used to: read the reporting status field of the intermediate table. If the intermediate status exists and the target database has full data, the data in the target database is sent to the information service system. When the sending is successful, the reporting status field of the intermediate table is updated to successful, otherwise, the reporting status field of the intermediate table is updated to unprocessed; the information service system merges the received data.

2. The system according to claim 1, characterized in that Also includes data synchronization tools; The source table database is specifically used to: after subscribing to the data of the host system, flush the data to the intermediate table through a data synchronization tool.

3. The system according to claim 1, characterized in that The processing server comprises: The scheduling server is used to shard the data in the intermediate table and generate shard data tasks; Dispatching sharding data tasks to application servers; The application server is used to process the sharding data tasks and obtain the processed data; the processed data is stored in the target database.

4. The system according to claim 3, characterized in that The scheduling server is specifically used for: Determine the number of shards and threads based on the business volume TPS peak, accumulation scenarios, and non-accumulation scenarios; Sharding the data in the intermediate table according to the number of shards; The application server is also used to: process shard data tasks according to the number of threads.

5. The system according to claim 3, characterized in that The processing server also includes a distributed coordination server, which is used to: When there are multiple dispatch servers, coordinate the data processed by each dispatch server; When there are multiple application servers, coordinate the shard data tasks processed by each application server; When there are multiple MQ servers, coordinate the data pushed by each MQ server.

6. The system according to claim 1, wherein: The target database is also used for: After the data in the target database is sent to the information service system, the data in the target database is cleared.

7. The system according to claim 1, characterized in that Also included are control tables for: After receiving the computer room system switching instruction, the computer room system is switched.

8. A distributed reporting method, characterized in that: include: After subscribing to the data of the host system, the data is flushed to the intermediate table, and the reporting status field of the intermediate table is marked as unprocessed, and the reporting status field is used to indicate the reporting status of the data; subscribing to the reporting status field of the intermediate table of the source table database of other computer room systems; Process the data in the intermediate table to obtain processed data; store the processed data in the target database; Push the processed data to the information service system; If the push is successful, update the reporting status field of the intermediate table to successful; otherwise, update the reporting status field of the intermediate table to intermediate status; Read the reporting status field of the intermediate table. If the intermediate status exists and the target database has full data, send the data in the target database to the information service system. When the sending is successful, update the reporting status field of the intermediate table to successful. Otherwise, update the reporting status field of the intermediate table to unprocessed. The information service system merges the received data.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method of claim 8 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method of claim 8 is implemented.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method of claim 8 is implemented.

Images