Big data processing method and system based on distributed micro-service architecture

Abstract

The invention discloses a big data processing method and system based on a distributed micro-service architecture, a main transaction is allocated and managed into N sub-transactions, the sub-transactions are responsible for executing specific operations on a specific database, such as writing or reading data, and the distributed micro-service architecture is adopted for management. Each sub-transaction works independently and is asynchronous with other sub-transactions, so that the N sub-transactions can execute different database operations in parallel, and communication is realized through the main transaction. This means that the main transaction will send the data and the operation instructions to the corresponding sub-transactions without waiting for completion of each sub-transaction. The asynchronous communication allows the system to maintain high throughput and responsiveness when processing a large amount of data and requests, high efficiency, high expandability and high reliability of big data processing are realized, and the layered and modularized micro-service processing method enables the system to flexibly process complex distributed transactions.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a big data processing method and system based on a distributed microservice architecture. Background Art

[0002] With the explosive growth of data volume, traditional monolithic application architecture faces problems such as poor scalability and difficult maintenance when processing large-scale data. Distributed microservice architecture provides better scalability and flexibility by splitting applications into independent service units, but it also brings challenges such as low efficiency, data consistency, and system complexity. Summary of the invention

[0003] The purpose of the present invention is to provide a big data processing method and system based on a distributed microservice architecture to solve the above-mentioned problems in the prior art.

[0004] To achieve the above object, the present invention provides a big data processing method based on a distributed microservice architecture, comprising the following steps:

[0005] Step S1: The transaction manager receives real-time data from a data source.

[0006] Step S2: the transaction manager creates a main transaction based on the real-time data, and the main transaction is associated with a cluster database;

[0007] Step S3: the transaction manager divides the main transaction into N sub-transactions, each of which is associated with a sub-database in the cluster database;

[0008] Wherein, the cluster database includes at least N sub-databases, and the sub-transactions are all independently and asynchronously executed;

[0009] Step S4: the sub-transaction checks the status of the corresponding sub-database. If the sub-database is not activated, the sub-transaction will immediately send a negative response to the main transaction; if the sub-database is activated, proceed to step S5;

[0010] Step S5: the sub-transaction executes the required data operation in the corresponding sub-database;

[0011] Step S6: After completing the data operation, the sub-transaction responds the execution result to the main transaction;

[0012] Step S7: The main transaction collects the execution results of all the sub-transactions and sends the final result back to the transaction manager.

[0013] In the big data processing method based on the distributed microservice architecture, preferably, the transaction manager sets a process for the sub-transaction to reflect the state of the sub-transaction.

[0014] In the big data processing method based on the distributed microservice architecture, preferably, the cluster database can be obtained by grouping multiple sub-database clusters in a NoSQL (key / value) database.

[0015] In the big data processing method based on the distributed microservice architecture, preferably, the data operation in step S5 is a write / read operation, which specifically also includes:

[0016] Step S51: setting operation parameters, wherein the operation parameters include a waiting interval;

[0017] Step S52: sending data to perform the write / read operation;

[0018] Step S53: Waiting for a response, where the waiting time is determined by the waiting interval parameter set in step S51;

[0019] Step S54: If the response is received within the waiting interval, continue to execute step S6; if the response is not received within the waiting interval, the sub-transaction will immediately send a negative response to the main transaction.

[0020] In the big data processing method based on the distributed microservice architecture, preferably, it also includes a cluster database coordination mechanism, and the cluster database coordination mechanism can define a consistency protocol; the same data is written into the sub-database, and if the number of successfully written sub-databases meets the consistency protocol, the coordination is deemed to be successful; otherwise, the coordination is deemed to have failed.

[0021] In the big data processing method based on the distributed microservice architecture, preferably, a fault tolerance mechanism is also provided, and the fault tolerance mechanism includes a polling check mechanism, and the polling check mechanism also includes:

[0022] Step T1: Set the polling time interval;

[0023] Step T2: Set the maximum number of polling attempts;

[0024] Step T3: retrying the data operation within the polling time interval;

[0025] Step T4: When the number of retries reaches the maximum number of attempts, the sub-transaction returns a temporary failure response to the main transaction.

[0026] In the big data processing method based on the distributed microservice architecture, preferably, the fault tolerance mechanism includes a waiting and re-execution mechanism, and the waiting and re-execution mechanism also includes:

[0027] Step N1: Set the waiting time;

[0028] Step N2: Set the number of retries;

[0029] Step N3: after waiting for the waiting time, retry the data operation according to the retry number;

[0030] Step N4: Continue to retry until the operation is successful.

[0031] In the big data processing method based on the distributed microservice architecture, preferably, the fault tolerance mechanism includes a clear and re-execute mechanism, and the clear and re-execute mechanism includes:

[0032] Step X1: Undo the data operation that has been performed;

[0033] Step X2: Restore the database to its previous state;

[0034] Step X3: Retry the data operation.

[0035] In the big data processing method based on the distributed microservice architecture, preferably, it also includes:

[0036] Each of the sub-databases is configured with two queues, which are respectively used to store operations related to the waiting re-execution mechanism or the clearing re-execution mechanism.

[0037] The present invention also provides a big data processing system based on a distributed microservice architecture, on which a computer program is stored, and is characterized in that when the computer program is executed by a processor, the steps of the big data processing method based on the distributed microservice architecture are implemented.

[0038] The present invention distributes and manages the main transaction into N sub-transactions, and the sub-transactions are responsible for performing specific operations on a specific database, such as writing or reading data, and are managed using a distributed microservice architecture. Since each sub-transaction works independently and is asynchronous with other sub-transactions, the N sub-transactions can perform different database operations in parallel and communicate through the main transaction. This means that the main transaction will send data and operation instructions to the corresponding sub-transactions without waiting for the completion of each sub-transaction. This asynchronous communication allows the system to maintain high throughput and responsiveness when processing large amounts of data and requests, achieving high efficiency, high scalability and high reliability in big data processing. This layered and modular microservice processing method enables the system to flexibly handle complex distributed transactions. At the same time, the present invention supplements the cluster database coordination mechanism and fault-tolerant mechanism, ensures data consistency, and helps maintain data accuracy and integrity. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a flowchart of a big data processing method based on a distributed microservice architecture provided by an embodiment of the present invention;

[0041] Figure 2 It is a flowchart of a write / read operation of a big data processing method based on a distributed microservice architecture provided by an embodiment of the present invention;

[0042] Figure 3 It is a polling check flow chart of a big data processing method based on a distributed microservice architecture provided by an embodiment of the present invention;

[0043] Figure 4 It is a waiting and re-execution flow chart of a big data processing method based on a distributed microservice architecture provided by an embodiment of the present invention;

[0044] Figure 5 It is a clearing and re-executing flowchart of a big data processing method based on a distributed microservice architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following description provides specific application scenarios and requirements of this specification, with the purpose of enabling those skilled in the art to make and use the contents of this specification. Various local modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but to the widest scope consistent with the claims.

[0046] The terms used herein are only used for the purpose of describing specific example embodiments and are not restrictive. For example, unless the context clearly indicates otherwise, as used herein, the singular forms "a", "an" and "the" may also include plural forms. When used in this specification, the terms "include", "comprise" and / or "contain" mean that the associated integers, steps, operations, elements and / or components exist, but do not exclude the existence of one or more other features, integers, steps, operations, elements, components and / or groups or that other features, integers, steps, operations, elements, components and / or groups may be added in the system / method.

[0047] These and other features of the present specification, as well as the operation and function of the related elements of the structure, and the economy of combination and manufacture of the parts can be significantly improved in view of the following description. Reference is made to the accompanying drawings, all of which form a part of this specification. However, it should be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of this specification.

[0048] The flowcharts used in this specification illustrate the operations implemented by the system according to some embodiments in this specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. On the contrary, the operations may be implemented in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0049] Figure 1 An embodiment of the present invention is presented, providing a flowchart of a big data processing method based on a distributed microservice architecture. The method comprises the following steps:

[0050] Step S1: The transaction manager receives real-time data from a data source.

[0051] Step S2: The transaction manager creates a main transaction based on the received real-time data, where the main transaction is associated with a cluster database.

[0052] Step S3: The transaction manager divides the main transaction into N sub-transactions, each of which is associated with a sub-database in the cluster database; wherein the cluster database includes at least N such sub-databases, and each sub-transaction is executed independently and asynchronously.

[0053] Step S4: Each sub-transaction checks the status of its corresponding sub-database. If the sub-database is not activated, the sub-transaction will immediately send a negative response to the main transaction. If the sub-database is activated, execute step S5.

[0054] Step S5: The sub-transaction executes the required data operations in the corresponding sub-database.

[0055] Step S6: After completing the data operation, the sub-transaction responds the execution result back to the main transaction.

[0056] Step S7: The main transaction collects the execution results of all sub-transactions and sends the final result back to the transaction manager.

[0057] In this way, the present invention effectively distributes and manages the main transaction into N subtransactions. The data contained in the main transaction is decomposed into multiple subtransaction data units, and each subtransaction data unit corresponds to a subtransaction. These data units contain all the information required to perform specific database operations, such as operation type (read / write), data content, target database, etc. Specifically, the subtransaction is responsible for performing specific operations on a specific database, such as writing or reading data, and is managed using a distributed microservice architecture. Since each subtransaction works independently and is asynchronous with other subtransactions, N subtransactions can perform different database operations in parallel and communicate through the main transaction. This means that the main transaction will send data and operation instructions to the corresponding subtransaction without waiting for the completion of each subtransaction. This asynchronous communication allows the system to maintain high throughput and responsiveness when processing large amounts of data and requests, achieving high efficiency, high scalability and high reliability of big data processing. This layered and modular microservice processing method enables the system to flexibly handle complex distributed transactions while improving the scalability of the system.

[0058] In some embodiments, in order to facilitate the sub-transaction to understand the status of the sub-database, the transaction manager can set a process associated with each sub-database. The process understands the status of the sub-database through regular checks (for example, every X seconds), so that the sub-transaction can check the status of the corresponding sub-database.

[0059] In some embodiments, the cluster database may be a NoSQL (key / value) database, obtained by clustering multiple databases of the NoSQL (key / value) database. Key / value databases can effectively handle high throughput and data growth in big data applications, and are suitable for real-time analysis and processing. At the same time, key / value databases do not require data to have a fixed schema, which enables them to store data of various structures and can easily adjust the data structure over time.

[0060] Data operations include write / read operations. Figure 2 An embodiment of the present invention is presented, providing a flowchart of write / read operations of a big data processing method based on a distributed microservice architecture. The process includes the following steps:

[0061] Step S51: Setting operation parameters, wherein the parameters include a waiting interval.

[0062] Step S52: Send data for write / read operation.

[0063] Step S53: Wait for a response, where the waiting time is determined by the waiting interval parameter set in step S51.

[0064] Step S54: If a response is received within the waiting interval, continue to step S6; if no response is received within the waiting interval, the subtransaction will immediately send a negative response to the main transaction.

[0065] The present invention achieves multiple technical advantages by setting a specific waiting interval during the data operation process. First, the method can prevent the system from waiting for a response indefinitely when attempting to perform a database operation, thereby avoiding unlimited occupation of system resources. Second, when the database operation fails to respond in time due to some reasons, a reasonable waiting interval can help the system effectively identify and handle the failure, thereby significantly enhancing the robustness of the system.

[0066] In some embodiments of the present invention, the big data processing method based on the distributed microservice architecture further includes a cluster database coordination mechanism, which can effectively handle high throughput and data growth in big data applications, and is suitable for real-time analysis and processing. This mechanism is used to maintain data consistency. Specifically, the method involves writing the same data to each sub-database in the cluster database, thereby creating multiple copies of the same data, and defining a consistency protocol according to actual needs. In some embodiments, the consistency protocol can be defined as: in the cluster database (assuming that the total number of sub-databases is N), the write operation is deemed successful only when the number of successfully written sub-databases exceeds N / 2. This method ensures data consistency while also ensuring high write efficiency. Preferably, in some embodiments, the consistency protocol can be defined as: the write operation is deemed successful only when the data is successfully written to all sub-databases in the cluster database (that is, the number of successfully written sub-databases is equal to N). In this way, the strongest data consistency can be achieved.

[0067] The present invention can maintain data consistency among multiple database copies through a cluster database coordination mechanism and a consistency protocol. In a distributed environment, this mechanism ensures that the accuracy and integrity of the data are maintained. Even if some database nodes fail, the coordination mechanism can still ensure that the remaining healthy nodes continue to provide services, thereby enhancing the reliability and stability of the entire system.

[0068] Optionally, in some embodiments of the present invention, the big data processing method based on the distributed microservice architecture further includes a fault tolerance mechanism. In these embodiments, the fault tolerance mechanism may include a polling check mechanism. When the transaction manager detects that the subtransaction sends a negative response to the main transaction, the polling check mechanism is triggered.

[0069] Figure 3 A polling check flow chart of a big data processing method based on a distributed microservice architecture is provided for one embodiment of the present invention. The flow chart shows the following steps:

[0070] Step T1: Set the polling time interval.

[0071] Step T2: Set the maximum number of polling attempts.

[0072] Step T3: Retry the data operation within the set polling time interval.

[0073] Step T4: When the number of retries reaches the set maximum number of attempts, the subtransaction returns a temporary failure response to the main transaction.

[0074] Specifically, the polling check mechanism ensures that even if some database nodes are temporarily unavailable, the system can still continuously try to perform operations on these nodes, thereby improving overall availability. By continuously checking the availability of the database, the system can continue operations immediately after the database is restored. This mechanism reduces the risk of the system completely stopping service due to a single point of failure, thereby significantly enhancing the system's fault tolerance.

[0075] Optionally, in some embodiments of the present invention, the fault tolerance mechanism further includes a wait-and-re-execute mechanism. The condition for triggering the mechanism is that the transaction manager detects that the subtransaction has sent a negative response or a temporary failure response to the main transaction, and the data operation has complied with the consistency protocol. Figure 4 A waiting and re-execution flow chart of a big data processing method based on a distributed microservice architecture is provided for one embodiment of the present invention. The flow chart shows the following steps:

[0076] Step N1: Set the waiting time.

[0077] Step N2: Set the number of retries.

[0078] Step N3: After waiting for a set waiting time, retry the data operation according to a set number of retries.

[0079] Step N4: Continue to retry until the operation is successful.

[0080] Specifically, the wait-to-reexecute mechanism is applicable to situations where the consistency condition has been met but some databases are still unavailable. For example, when a write / read operation is executed in a database cluster and the consistency condition has been met, but some databases are still unavailable, the wait-to-reexecute mechanism assumes that the database unavailability is temporary and expects that the database will eventually recover and become available. Therefore, if the database does not respond, the wait-to-reexecute mechanism will initiate a retry, check the availability of the database, and retry the write / read operation when the database is available. In some preferred embodiments, the wait-to-reexecute mechanism does not limit the number of attempts to perform the write / read operation.

[0081] Optionally, in some embodiments of the present invention, the fault tolerance mechanism further includes a clear-and-re-execute mechanism. The mechanism is triggered when the transaction manager detects that the subtransaction sends a negative response or a temporary failure response to the main transaction, and the data operation does not comply with the consistency protocol. Figure 5 A clearing and re-execution flow chart of a big data processing method based on a distributed microservice architecture is provided for one embodiment of the present invention. The flow chart shows the following steps:

[0082] Step X1: Undo the data operation that has been performed.

[0083] Step X2: Restore the database to its previous state.

[0084] Step X3: Retry the data operation.

[0085] Specifically, the purge-replay mechanism is used to handle situations where consistency conditions are not met. For example, when a write operation has been executed on a specific database but has not been able to reach consistency in the database cluster, the operations that have been executed on some databases need to be canceled to maintain data correctness. In this case, the purge-replay process is activated to undo the effects of the operations that have been executed and restore the database to its previous state as a compensation mechanism. If a read operation is being executed, the operation can be ignored.

[0086] The present invention ensures that the system can continue to run even if some database nodes fail by setting a fault-tolerant mechanism, thereby improving the overall reliability. In addition, the fault-tolerant mechanism helps maintain the accuracy and integrity of data by detecting and correcting inconsistent states. Specifically, the fault-tolerant mechanism of the present invention includes two supplementary mechanisms, namely, waiting for re-execution and clearing for re-execution. These mechanisms are combined with the consistency protocol to enable the system to recover or undo partially completed transactions when a failure occurs, thereby reducing the impact of system failures.

[0087] Preferably, in some embodiments, each sub-database is configured with two queues, which are respectively used to store operations related to the waiting re-execution mechanism and the clearing re-execution mechanism.

[0088] On the other hand, the present specification provides a non-transitory storage medium, including a computer program, wherein the computer program stores at least one set of executable instructions for big data processing. When the executable instructions are executed by the computer program, the executable instructions instruct the processor to implement a big data processing method based on a distributed microservice architecture described in the present specification. In some possible implementations, various aspects of the present specification can also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer program to execute a big data processing method based on a distributed microservice architecture described in the present specification. The program product for implementing the above method can use a portable compact disk read-only memory (CD-ROM) to include program code and can be run on a computer device or a server device. However, the program product of the present specification is not limited to this. In the present specification, the non-transitory storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system or used in combination with it. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage chip. The readable storage medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device, or any combination of the above.

[0089] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

Claims

1. A big data processing method based on a distributed microservice architecture, characterized in that: The steps include: Step S1: The transaction manager receives real-time data from a data source. Step S2: the transaction manager creates a main transaction based on the real-time data, and the main transaction is associated with a cluster database; Step S3: the transaction manager divides the main transaction into N sub-transactions, each of which is associated with a sub-database in the cluster database; Wherein, the cluster database includes at least N sub-databases, and the sub-transactions are all independently and asynchronously executed; Step S4: the sub-transaction checks the status of the corresponding sub-database. If the sub-database is not activated, the sub-transaction will immediately send a negative response to the main transaction; if the sub-database is activated, proceed to step S5; Step S5: the sub-transaction executes the required data operation in the corresponding sub-database; Step S6: After completing the data operation, the sub-transaction responds the execution result to the main transaction; Step S7: The main transaction collects the execution results of all the sub-transactions and sends the final result back to the transaction manager.

2. The big data processing method based on distributed microservice architecture according to claim 1 is characterized in that: The transaction manager sets a process for the sub-transaction to reflect the status of the sub-transaction.

3. The big data processing method based on distributed microservice architecture according to claim 1 is characterized in that: The cluster database may be obtained by grouping multiple sub-database clusters in a NoSQL (key / value) database.

4. The big data processing method based on distributed microservice architecture according to claim 1 is characterized in that: The data operation in step S5 is a write / read operation, which specifically includes: Step S51: setting operation parameters, wherein the operation parameters include a waiting interval; Step S52: sending data to perform the write / read operation; Step S53: Waiting for a response, where the waiting time is determined by the waiting interval parameter set in step S51; Step S54: If the response is received within the waiting interval, continue to execute step S6; if the response is not received within the waiting interval, the sub-transaction will immediately send a negative response to the main transaction.

5. The big data processing method based on distributed microservice architecture according to claim 1 is characterized in that: Also includes A cluster database coordination mechanism, wherein the cluster database coordination mechanism may define a consistency protocol; The same data is written into the sub-database. If the number of data successfully written into the sub-database satisfies the consistency protocol, the coordination is deemed to be successful; otherwise, the coordination is deemed to have failed.

6. The big data processing method based on the distributed microservice architecture according to claims 1, 3 and 5 is characterized in that: A fault tolerance mechanism is also provided, the fault tolerance mechanism includes a polling check mechanism, and the polling check mechanism also includes: Step T1: Set the polling time interval; Step T2: Set the maximum number of polling attempts; Step T3: retrying the data operation within the polling time interval; Step T4: When the number of retries reaches the maximum number of attempts, the sub-transaction returns a temporary failure response to the main transaction.

7. The big data processing method based on distributed microservice architecture according to claim 6 is characterized in that: The fault tolerance mechanism includes a waiting and re-execution mechanism, and the waiting and re-execution mechanism also includes: Step N1: Set the waiting time; Step N2: Set the number of retries; Step N3: after waiting for the waiting time, retry the data operation according to the retry number; Step N4: Continue to retry until the operation is successful.

8. The big data processing method based on distributed microservice architecture according to claim 6 is characterized in that: The fault tolerance mechanism includes a clear and re-execute mechanism, and the clear and re-execute mechanism includes: Step X1: Undo the data operation that has been performed; Step X2: Restore the database to its previous state; Step X3: Retry the data operation.

9. The big data processing method based on distributed microservice architecture according to claim 6 or 7, characterized in that: Also includes: Each of the sub-databases is configured with two queues, which are respectively used to store operations related to the waiting re-execution mechanism or the clearing re-execution mechanism.

10. A big data processing system based on a distributed microservice architecture, on which a computer program is stored, characterized in that: When the computer program is executed by a processor, the steps of the big data processing method based on a distributed microservice architecture as described in any one of claims 1 to 9 are implemented.

Images