Data processing system, method and device

Abstract

Embodiments of the present specification provide a data processing system, method and apparatus, wherein the data processing system comprises: a client, a database node and an object storage service node; the database node is configured to receive a data write instruction sent by the client, add the data write instruction to a log synchronization queue, generate a log entry for the data write instruction, concurrently write the log entry to multiple task queues based on a pre-established data write link, generate queue overload information when it is determined that the amount of data in any one of the multiple task queues is greater than a preset threshold, and reversely propagate the task queue overload information to the client based on the data write link; the client is configured to store the data to be written in the object storage service node when it is determined that the task queue overload information is received.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of database technology, and in particular to a data processing system. One or more embodiments of this specification also relate to a data processing method, a data processing apparatus, a computing device, and a computer-readable storage medium. Background Art

[0002] In today's cloud computing era, massive amounts of data are placing increasingly high demands on the storage performance of distributed storage systems. The stability of distributed storage systems is extremely important. Only when the stability is high enough can users be provided with continuous and stable storage services in the long term.

[0003] Due to the complexity of applications, a large number of write operations often occur in the storage system at once. If these large numbers of write operations are not handled in time, they are easily accumulated in the memory of the distributed storage system, causing a surge in the memory usage of the cluster, which not only slows down the performance of the distributed storage, but also causes downtime in serious cases. It can be seen that this situation of a large number of write operations pouring in at once makes the distributed storage system less stable. How to handle a large number of write operations at once and avoid the above-mentioned impact has become an important problem that needs to be solved urgently in this field. Summary of the invention

[0004] In view of this, an embodiment of this specification provides a data processing system. One or more embodiments of this specification also relate to a data processing method, a data processing apparatus, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, there is provided a data processing system, including:

[0006] Client, database node, and object storage service node;

[0007] The database node is configured to receive a data write instruction sent by the client, add the data write instruction to a log synchronization queue, generate a log entry of the data write instruction, concurrently write the log entry into multiple task queues based on a pre-established data write link, generate queue overload information when it is determined that the amount of data in any one of the multiple task queues is greater than a preset threshold, and reversely propagate the task queue overload information to the client based on the data write link;

[0008] The client is configured to store the data to be written in the object storage service node when determining to receive the task queue overload information.

[0009] Optionally, the database node is also configured to monitor the load information of the log synchronization queue and the multiple task queues, and generate queue overload information if it is determined that the load of the log synchronization queue and / or any one of the multiple task queues is greater than a preset load threshold.

[0010] Optionally, the database node is also configured to monitor the load information of the log synchronization queue and the multiple task queues. If it is determined that the data volume of the log synchronization queue and / or any one of the multiple task queues is greater than a preset data volume threshold, queue overload information is generated, and the load information includes the data volume of the log synchronization queue and / or any one of the multiple task queues.

[0011] Optionally, the database node is further configured to add the log entry to a log application queue, apply the log entry in the log application queue to the state machine, and store the execution result of the state machine in a message queue.

[0012] Optionally, the database node is further configured to monitor the load information of the log synchronization queue and the message queue, and generate queue overload information if it is determined that the difference between the load of the log synchronization queue and the load of the message queue is greater than a preset threshold.

[0013] Optionally, the database node includes a master database node and at least two slave database nodes;

[0014] The master database node is configured to receive a data write instruction sent by the client, add the data write instruction to a log synchronization queue, generate a log entry for the data write instruction, and concurrently write the log entry to the at least two slave database nodes based on a pre-established data write link.

[0015] Optionally, the master database node is further configured to concurrently write the log entry into the first log application queues of the at least two slave database nodes based on a pre-established data writing link;

[0016] The at least two slave database nodes are configured to apply the log entries in the first log application queue to the first state machine, store the execution results of the first state machine in the first message queue, and send feedback information of successful log submission to the master database node;

[0017] The master database node is further configured to submit the log entry upon receiving the feedback information.

[0018] Optionally, the main database node is also configured to, upon receiving the feedback information, add the log entry to a second log application queue, apply the log entry in the second log application queue to a second state machine, and store the execution result of the second state machine in a second message queue.

[0019] Optionally, the main database node is further configured to monitor the load information of the log synchronization queue and the second message queue, and generate queue overload information if it is determined that the difference between the load of the log synchronization queue and the load of the second message queue is greater than a preset threshold.

[0020] Optionally, the database node is further configured to lock any one of the multiple task queues whose data volume is greater than a preset threshold, and monitor load update information of any one of the task queues whose data volume is greater than the preset threshold, and if it is determined according to the load update information that the loads of the multiple queues are all less than the preset threshold, then send queue unlocking information to the client;

[0021] The client is further configured to receive the queue unlocking information, obtain the data to be written stored in the object storage service node, and send a data write instruction to the database node based on the data to be written.

[0022] According to a second aspect of an embodiment of this specification, a data processing method is provided, including:

[0023] Sending a data writing instruction to the database node based on the data to be written, wherein the data writing instruction carries data identification information of the data to be written;

[0024] When the task queue overload information returned by the database node is received, the data to be written is stored in the object storage service node.

[0025] Optionally, the data processing method further includes:

[0026] Receive queue unlock information returned by the database node, obtain the data to be written stored in the object storage service node, and send a data write instruction to the database node based on the data to be written.

[0027] According to a third aspect of an embodiment of this specification, there is provided a data processing device, including:

[0028] A sending module, configured to send a data writing instruction to a database node based on the data to be written, wherein the data writing instruction carries data identification information of the data to be written;

[0029] The storage module is configured to store the data to be written in the object storage service node when receiving the task queue overload information returned by the database node.

[0030] According to a fourth aspect of an embodiment of this specification, a computing device is provided, including:

[0031] Memory and processor;

[0032] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the data processing method.

[0033] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the data processing method are implemented.

[0034] An embodiment of the present specification receives a data write instruction sent by a client through a database node, adds the data write instruction to a log synchronization queue, generates a log entry of the data write instruction, and concurrently writes the log entry to multiple task queues based on a pre-established data write link. When it is determined that the amount of data in any one of the multiple task queues is greater than a preset threshold, queue overload information is generated, and the task queue overload information is reversely propagated to the client based on the data write link; when the client determines to receive the task queue overload information, the data to be written is stored in the object storage service node.

[0035] In the embodiments of the present specification, a preset threshold is used to control the data volume of each task queue, thereby controlling the write speed, which is beneficial to improving the stability of the distributed database; when high-concurrency write traffic causes the task queue of the database node to be blocked, a back-pressure mechanism is formed to transmit the generated task queue overload information back to the client, so that the client uses the object storage service node as a data cache area to temporarily store the write traffic that cannot be written to the database node normally, which is beneficial to alleviating the write pressure on the database node due to high-concurrency write traffic, and while ensuring the stability of the distributed database, further improves the ability of the database node to carry burst write traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a structural diagram of a data processing system provided by an embodiment of this specification;

[0037] Figure 2 is a flow chart of a data processing method provided by an embodiment of this specification;

[0038] Figure 3It is an interactive schematic diagram of a data processing method provided by an embodiment of this specification;

[0039] Figure 4 is a schematic diagram of a data processing device provided by an embodiment of this specification;

[0040] Figure 5 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0041] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0042] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0043] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0044] First, the terms involved in one or more embodiments of this specification are explained.

[0045] Back pressure: When a peak load in a short period of time causes the system to receive data at a rate much higher than the rate at which it processes data, back pressure will occur, and the system will exert back pressure on upstream writes.

[0046] OSS: Object storage service, which can be stored and accessed anywhere on the Internet using RESTful API, elastically expands capacity and processing power, and provides a variety of storage types to fully optimize storage costs.

[0047] Queue: A queue is a special linear list. The special thing about a queue is that it only allows deletion operations at the front of the list, and insertion operations at the rear of the list. Like a stack, a queue is a linear list with restricted operations. The end where the insertion operation is performed is called the tail of the queue, and the end where the deletion operation is performed is called the head of the queue.

[0048] Consistency protocol: distributed consistency protocols such as raft and paxos.

[0049] In this specification, a data processing system is provided. This specification also relates to a data processing method, a data processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0050] In view of the problem that a large number of write operations often cannot be processed in time in the current distributed storage system, resulting in poor stability of the distributed storage system, the embodiment of this specification provides a data processing system as a solution. The data processing system provided by the embodiment of the specification is described in detail below in conjunction with the embodiments and drawings.

[0051] Figure 1 A schematic diagram of the structure of a data processing system provided according to an embodiment of the present specification is shown, including:

[0052] Client 102, database node 104 and object storage service node 106;

[0053] The database node 104 is configured to receive a data write instruction sent by the client 102, add the data write instruction to a log synchronization queue, generate a log entry of the data write instruction, concurrently write the log entry into multiple task queues based on a pre-established data write link, and generate queue overload information when it is determined that the amount of data in any one of the multiple task queues is greater than a preset threshold, and reversely propagate the task queue overload information to the client 102 based on the data write link;

[0054] The client 102 is configured to store the data to be written in the object storage service node 106 when determining to receive the task queue overload information.

[0055] With the rapid development of Internet technology, the application scope of distributed databases is becoming wider and wider. Since a distributed database contains multiple distributed nodes and the distributed database can provide data read / write services for clients, the client can access data through any distributed node in the distributed database.

[0056] Based on this, a data processing system is provided in an embodiment of the present specification. First, a client 102 sends a data write instruction to a distributed database (database node 104). After the database node 104 receives the data write instruction, it writes a log through a distributed consistency algorithm (Raft), that is, the data write instruction is added to a log synchronization queue to obtain a log entry of the data write instruction, and the log entry is synchronized to other slave distributed nodes based on a pre-established write link.

[0057] Specifically, the database involved in the embodiment of the present invention may be a database based on an LSM structure, including but not limited to an X-DB database, a LevelDB database, a RocksDB database, and the like.

[0058] In addition, the data write instruction may be a database transaction write instruction, or a write instruction for a database operation, etc. The database node 104 may determine the information of the database write task corresponding to the data write instruction according to the data write instruction, and the information of the database write task includes but is not limited to the information of the write task corresponding to the database transaction, such as write log information, disk flushing information of the log data of the database transaction, write memory information of the database transaction, or commit information of the database transaction, etc. It can be understood that the above description is only exemplary, and the embodiments of this specification do not make any limitation to this.

[0059] In some optional embodiments, when determining the information of the database write task corresponding to the data write instruction, the data write instruction can be parsed to obtain the information of the database write task corresponding to the data write instruction, and the information of the database write task is added to the log synchronization queue. It can be understood that the above description is only exemplary and the embodiments of this specification do not impose any limitation on this.

[0060] In addition, log entries can be synchronized through a data write link. Therefore, before synchronizing log entries, the data write link needs to be established in advance. To improve the efficiency of data writing, the embodiment of the present specification adopts a multi-threaded asynchronous processing method to synchronize the log entries. Therefore, when establishing the data write link, the task types corresponding to the multiple task queues can be determined respectively, and the task queues of the same task type can be set at the same link node level of the data write link. For example, the log synchronization queue is set to the first link node level, and the log application queue is set to the second link node level. The first link node level and the second link node level are in an upstream and downstream relationship, and the second link node level contains at least two task queues.

[0061] In addition, the write flow control and back pressure mechanism of the embodiments of this specification are mainly designed based on multiple task queues. In order to avoid the insertion of concentrated super-large logs causing the queue to be overloaded, in the process of writing log entries to multiple task queues, it is necessary to monitor the queue load of any task queue in the multiple task queues, that is, the data volume. When the data volume is greater than the preset threshold, it is determined that there is a task queue that is overloaded. To ensure the data writing efficiency of the database node, when the embodiments of this specification determine that there is any task queue whose load is greater than the preset threshold, the queue will be locked and the task queue overload information will be propagated back to the client 102 through the data writing link. The locking process of any task queue will form back pressure on the data writing link, which will ultimately cause the client 102 to lock the write to the database node 104 (restricted use).

[0062] It should be noted that in the embodiments of this specification, the data volume includes: the number of data and / or the size of data. Among them, the number of data is the maximum number of pending tasks in the task queue, and the size of data is the total size of the pending tasks in the task queue. For each queue, the number of pending tasks in the queue and the data size corresponding to the pending tasks will be monitored. When the number or size and exceeds the preset threshold, the write flow control mechanism is triggered, and the task queue write operation will be locked. The locking of any task queue will form back pressure on the data write link, which will eventually limit the data write of the client.

[0063] When using the data volume of the task queue to control the processing of the write operation, there are multiple optional specific control methods. For example, if the data size or the total data of any task queue is greater than a preset threshold, no new tasks to be processed are added to all task queues of the data write link to which the task queue belongs; if the data size or the total data of any task queue does not reach the preset threshold, new tasks to be processed can continue to be added to all task queues.

[0064] Finally, upon receiving the task queue overload information returned by the database node 104 , the client 102 may temporarily store the data to be written to the object storage service node (OSS).

[0065] In an embodiment of the present specification, when high-concurrency write traffic causes the task queue of a data storage node to be blocked, the object storage service node is used as a data cache area to temporarily store the write traffic that cannot be written normally to the database node. This helps alleviate the write pressure on the database node caused by high-concurrency write traffic and further improves the ability of the database node to carry burst write traffic.

[0066] During specific implementation, in addition to monitoring the data volume of multiple task queues, the load of the log synchronization queue can also be monitored. Therefore, the database node 104 is also configured to monitor the load information of the log synchronization queue and the multiple task queues. If it is determined that the load of the log synchronization queue and / or any one of the multiple task queues is greater than a preset load threshold, queue overload information is generated.

[0067] Furthermore, the load is the amount of data carried by the queue. The database node 104 is also configured to monitor the load information of the log synchronization queue and the multiple task queues. If it is determined that the data volume of the log synchronization queue and / or any one of the multiple task queues is greater than a preset data volume threshold, queue overload information is generated, and the load information includes the data volume of the log synchronization queue and / or any one of the multiple task queues.

[0068] Specifically, in the process of writing log entries to multiple task queues, in addition to monitoring the data volume of any one of the multiple task queues, the data volume of the log synchronization queue can also be monitored. When the data volume of the log synchronization queue and any one of the multiple task queues is greater than a preset threshold, it is determined that the queue is overloaded. To ensure the data writing efficiency of the database node, the embodiment of this specification will lock the queue if it is determined that the load of any queue is greater than a preset threshold, and through the back pressure mechanism, the task queue overload information is reversely propagated to the client 102 based on the data write link, which ultimately causes the client 102 to lock the write to the database node 104 (restricted use).

[0069] As mentioned above, the data volume also includes the number of data and / or the size of data.

[0070] In the embodiments of the present specification, a preset threshold is used to control the data volume of each task queue, thereby controlling the writing speed, which is beneficial to improving the stability of the distributed database.

[0071] In a specific implementation, the database node 104 is further configured to add the log entry to a log application queue, apply the log entry in the log application queue to the state machine, and store the execution result of the state machine in a message queue.

[0072] Furthermore, the database node 104 is also configured to monitor the load information of the log synchronization queue and the message queue, and generate queue overload information if it is determined that the difference between the load of the log synchronization queue and the load of the message queue is greater than a preset threshold.

[0073] Specifically, after the database node adds the data write instruction to the log synchronization queue and generates a log entry of the data write instruction, the log entry is applied to the state machine of the database node, and the execution result of the state machine is stored in the message queue of the database node.

[0074] In addition, in order to avoid the asynchrony of data processing speed between the log synchronization queue and the message queue, the data write traffic monitoring mechanism will also monitor the number (load) of pending tasks (pending data write instructions) in the log synchronization queue and the message queue, and calculate the difference between the number of pending tasks in the message queue and the number of pending tasks in the log synchronization queue. If the difference between the two is greater than the preset threshold, it means that the task processing speed of the message queue lags far behind the task processing speed of the log synchronization queue. Therefore, the write traffic control of the log synchronization queue can be performed to ensure that the data processing progress of each task queue in the database node remains synchronized.

[0075] In a specific implementation, the database node 104 includes a master database node and at least two slave database nodes;

[0076] The master database node is configured to receive a data write instruction sent by the client, add the data write instruction to a log synchronization queue, generate a log entry for the data write instruction, and concurrently write the log entry to the at least two slave database nodes based on a pre-established data write link.

[0077] Further, the master database node is further configured to concurrently write the log entry into the first log application queues of the at least two slave database nodes based on a pre-established data writing link;

[0078] The at least two slave database nodes are configured to apply the log entries in the first log application queue to the first state machine, store the execution results of the first state machine in the first message queue, and send feedback information of successful log submission to the master database node;

[0079] The master database node is further configured to submit the log entry upon receiving the feedback information.

[0080] Furthermore, the main database node is also configured to, upon receiving the feedback information, add the log entry to a second log application queue, apply the log entry in the second log application queue to a second state machine, and store the execution result of the second state machine in a second message queue.

[0081] Specifically, after the distributed database is created, the distributed database can include multiple nodes, one of which is a master database node, and all nodes other than the master database node are slave database nodes. In the event of a failure of the master database node, the master database node can be re-elected from other slave database nodes. In order to ensure that the slave database node can be elected as the master database node and thus provide services to the client 102, each slave database node needs to store the same data as the master database node.

[0082] Among them, the master database node in the distributed database (database node 104) realizes log synchronization through a distributed consistency algorithm. The master database node is used to receive the data write instruction sent by the client 102, and append the data write instruction to the log synchronization queue, obtain the log entry of the data write instruction, and initiate a synchronization request operation to all other slave database nodes. The slave database node copies the log entry to achieve consistency of data stored in each database node in the distributed database.

[0083] Specifically, the master database node synchronizes log entries to the slave database nodes concurrently, that is, one master database node synchronizes log entries to at least two slave database nodes simultaneously. When the log entries are synchronized to most of the slave database nodes, the master database node may send a prompt message for submitting the log to the slave database node.

[0084] Specifically, the master database node concurrently transmits the log entry to the slave database node, that is, the master database synchronizes the log entry to multiple slave database nodes at the same time, and the slave database node adds the log entry to the log application queue and sends a prompt message of successful log replication to the master database node; after receiving the feedback message of successful log replication returned by the majority (more than half) of the slave database nodes, the master database node sends a prompt message of submitting the log to the slave database node;

[0085] After receiving the prompt information of submitting the log sent by the master database node, the slave database node submits the log, specifically, applying the log entry to the first state machine of the slave database node, storing the execution result of the first state machine to the first message queue of the slave database node, and sending feedback information of successful log submission to the master database node;

[0086] The master database node applies the log entries submitted by the slave database node to the second state machine of the master database node. Specifically, the master database node adds the log entries copied by the slave database node to the second log application queue of the master database node, and after receiving the feedback information of successful log submission returned by the slave database node, applies the log entries submitted by the slave database node to the second state machine of the master database node, and stores the execution results of the second state machine to the second message queue of the master database node.

[0087] Through the distributed consistency protocol algorithm (Raft algorithm), the master database node sends log synchronization to each slave database node in parallel, which is conducive to ensuring that all submitted log entries are persistent and will eventually be executed by all available state machines, and is conducive to improving the processing efficiency of data writing instructions. Once the master database node that creates the log entry copies it to more than half of the slave database nodes (that is, the slave database node safely copies the log entry), the log entry will be submitted, and the master database node will apply the log entry submitted by the slave database node to the state machine, which is conducive to ensuring the consistency of the data stored by the master database node and each slave database node.

[0088] In a specific implementation, the main database node is also configured to monitor the load information of the log synchronization queue and the second message queue. If it is determined that the difference between the load of the log synchronization queue and the load of the second message queue is greater than a preset threshold, queue overload information is generated.

[0089] Specifically, in the embodiments of the present specification, for modular design considerations, the log application layer of the main database node does not actually perform data storage work, but is designed in the form of interfaces. What it does is send the data write instructions that need to be processed to the second message queue of the storage layer, and after the data to be written corresponding to any data write instruction in the message queue is written successfully, the log application value (apply index) is increased by 1. This log application value is used to indicate which log entries have completed data storage.

[0090] The storage layer of the master database node itself will ensure that the data in the table queue will be applied successfully through the checkpoint (checkpoint / check mechanism) and replay (replay / replay) mechanisms. Specifically, after the master database node adds the execution result of the second state machine to the second message queue, it can call the checkpoint node (checkpoint) through the replay mechanism (replay mechanism) to ensure that the data in the second message queue is written successfully.

[0091] In addition, in order to avoid the asynchrony of data processing speeds between the log synchronization queue and the second message queue, the data write traffic monitoring mechanism will also monitor the log application value and the number of pending tasks (pending data write instructions) in the second message queue of the storage layer (last index), and calculate the difference (Gap) between the number of pending tasks in the second message queue and the log application value. If the difference between the two is greater than the preset threshold, it means that the task processing speed of the storage layer lags far behind the task processing speed of the log synchronization queue. Therefore, the log synchronization queue can be written to control the flow rate to ensure that the data processing progress of each task queue in the database node remains synchronized.

[0092] In addition, the database node 104 is further configured to lock any one of the multiple task queues whose data volume is greater than a preset threshold, and monitor the load update information of any one of the task queues whose data volume is greater than the preset threshold, and if it is determined according to the load update information that the loads of the multiple queues are all less than the preset threshold, then send queue unlocking information to the client 102;

[0093] The client 102 is further configured to receive the queue unlocking information, obtain the data to be written stored in the object storage service node 106, and send a data write instruction to the database node 104 based on the data to be written.

[0094] Specifically, when the database node determines based on the load monitoring result that the data volume of any task queue among the multiple task queues is greater than a preset threshold, the database node locks any task queue among the multiple task queues whose data volume is greater than the preset threshold, or locks all the multiple task queues, generates queue locking information, and transmits the queue locking information back to the client 102 based on the data write link through the back pressure mechanism.

[0095] After receiving the queue locking information, the data writing operation of the client 102 to the database node 104 is also locked. Therefore, the client 102 can temporarily store the data to be written to the object storage service node.

[0096] The database node 104 continues to monitor the load update information of each task queue. If it is determined based on the load update information that the loads of the multiple queues are all less than the preset threshold, a queue unlocking information is sent to the client 102 so that the client 102 rewrites the data to be written temporarily stored in the object storage service node 106 to the database node 104.

[0097] In an embodiment of the present specification, a database node receives a data write instruction sent by the client, adds the data write instruction to a log synchronization queue, generates a log entry of the data write instruction, and concurrently writes the log entry to multiple task queues based on a pre-established data write link. When it is determined that the amount of data in any one of the multiple task queues is greater than a preset threshold, queue overload information is generated, and the task queue overload information is reversely propagated to the client based on the data write link; when the client determines that it has received the task queue overload information, it stores the data to be written in the object storage service node.

[0098] In the embodiments of the present specification, a preset threshold is used to control the data volume of each task queue, thereby controlling the write speed, which is beneficial to improving the stability of the distributed database; when high-concurrency write traffic causes the task queue of the database node to be blocked, a back-pressure mechanism is formed to transmit the generated task queue overload information back to the client, so that the client uses the object storage service node as a data cache area to temporarily store the write traffic that cannot be written to the database node normally, which is beneficial to alleviating the write pressure on the database node due to high-concurrency write traffic, and while ensuring the stability of the distributed database, further improves the ability of the database node to carry burst write traffic.

[0099] Figure 2 A processing flow chart of a data processing method provided according to an embodiment of the present specification is shown, which is applied to a client and includes steps 202 to 204.

[0100] Step 202: Send a data write instruction to the database node based on the data to be written, wherein the data write instruction carries data identification information of the data to be written.

[0101] Step 204: upon receiving the task queue overload information returned by the database node, storing the data to be written to the object storage service node.

[0102] Specifically, the database node includes a master database node and at least two slave database nodes; the client sends a data write instruction to the master database node, and after the master database node receives the data write instruction sent by the client, it adds the data write instruction to the log synchronization queue, generates a log entry of the data write instruction, and concurrently writes the log entry into the first log application queues of the at least two slave database nodes based on a pre-established data write link;

[0103] The at least two slave database nodes apply the log entries in the first log application queue to the first state machine, store the execution results of the first state machine to the first message queue, and send feedback information of successful log submission to the master database node;

[0104] Upon receiving the feedback information, the master database node adds the log entry to a second log application queue, applies the log entry in the second log application queue to a second state machine, and stores the execution result of the second state machine in a second message queue.

[0105] In addition, the main database node can monitor the load information of the log synchronization queue and the second message queue. If it is determined that the difference between the load of the log synchronization queue and the load of the second message queue is greater than a preset threshold, queue overload information is generated, and the task queue overload information is reversely propagated to the client based on the data write link.

[0106] After receiving the task queue overload information, the client stores the data to be written to the object storage service node.

[0107] Further, the master database node may continue to monitor the load update information of each task queue, and if it is determined according to the load update information that the loads of the multiple queues are all less than the preset threshold, queue unlocking information is sent to the client;

[0108] After receiving the queue unlocking information returned by the database node, the client obtains the data to be written stored in the object storage service node, and sends a data write instruction to the database node based on the data to be written, so as to rewrite the data to be written into the database.

[0109] In the embodiments of the present specification, a preset threshold is used to control the data volume of each task queue, thereby controlling the write speed, which is beneficial to improving the stability of the distributed database; when high-concurrency write traffic causes the task queue of the database node to be blocked, a back-pressure mechanism is formed to transmit the generated task queue overload information back to the client, so that the client uses the object storage service node as a data cache area to temporarily store the write traffic that cannot be written to the database node normally, which is beneficial to alleviating the write pressure on the database node due to high-concurrency write traffic, and while ensuring the stability of the distributed database, further improves the ability of the database node to carry burst write traffic.

[0110] The above is a schematic scheme of a data processing method applied to a client of this embodiment. It should be noted that the technical scheme of the data processing method and the technical scheme of the above data processing system belong to the same concept, and the details not described in detail in the technical scheme of the data processing method can be referred to the description of the technical scheme of the above data processing system.

[0111] The following combination Figure 3 , the data processing method is further described. Figure 3 An interactive schematic diagram of a data processing method provided by an embodiment of the present specification is shown, and the specific steps include step 302 to step 326.

[0112] Step 302: The client sends a first data write instruction to the primary service node.

[0113] Step 304: The primary database node adds the data write instruction to the log synchronization queue and generates a log entry of the data write instruction.

[0114] Step 306: The master database node concurrently synchronizes the log entry to the first log application queue of the slave database node based on the pre-established data writing link.

[0115] Step 308: Apply the log entries in the first log application queue to the first state machine from the database node, and store the execution result of the first state machine in the first message queue.

[0116] Step 310: The slave database node sends feedback information indicating successful log submission to the master database node.

[0117] Step 312: The master database node adds the log entry to a second log application queue, applies the log entry in the second log application queue to a second state machine, and stores the execution result of the second state machine in a second message queue.

[0118] Step 314: The main database node monitors the load information of the log synchronization queue and the second message queue. If it is determined that the difference between the load of the log synchronization queue and the load of the second message queue is greater than a preset threshold, queue locking information is generated.

[0119] Step 316: The main database node propagates the queue locking information back to the client based on the data writing link.

[0120] Step 318: The client stores the data to be written to the object storage service node.

[0121] Step 320: The primary database node monitors the load update information of the log synchronization queue and the second message queue.

[0122] Step 322: If it is determined according to the load update information that the difference between the load of the log synchronization queue and the load of the second message queue is less than or equal to a preset threshold, the main database node sends queue unlocking information to the client.

[0123] Step 324, the client obtains the data to be written stored in the object storage service node.

[0124] Step 326: The client sends a second data write instruction to the primary database node based on the data to be written.

[0125] In the embodiments of the present specification, a preset threshold is used to control the data volume of each task queue, thereby controlling the write speed, which is beneficial to improving the stability of the distributed database; when high-concurrency write traffic causes the task queue of the database node to be blocked, a back-pressure mechanism is formed to transmit the generated task queue overload information back to the client, so that the client uses the object storage service node as a data cache area to temporarily store the write traffic that cannot be written to the database node normally, which is beneficial to alleviating the write pressure on the database node due to high-concurrency write traffic, and while ensuring the stability of the distributed database, further improves the ability of the database node to carry burst write traffic.

[0126] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 4 FIG. 1 is a schematic diagram of a data processing device provided by an embodiment of the present specification. Figure 4 As shown, the device comprises:

[0127] The sending module 402 is configured to send a data writing instruction to the database node based on the data to be written, wherein the data writing instruction carries data identification information of the data to be written;

[0128] The storage module 404 is configured to store the data to be written in the object storage service node when receiving the task queue overload information returned by the database node.

[0129] Optionally, the data processing device further includes:

[0130] The receiving module is configured to receive the queue unlocking information returned by the database node, obtain the data to be written stored in the object storage service node, and send a data writing instruction to the database node based on the data to be written.

[0131] In the embodiments of the present specification, a preset threshold is used to control the data volume of each task queue, thereby controlling the write speed, which is beneficial to improving the stability of the distributed database; when high-concurrency write traffic causes the task queue of the database node to be blocked, a back-pressure mechanism is formed to transmit the generated task queue overload information back to the client, so that the client uses the object storage service node as a data cache area to temporarily store the write traffic that cannot be written to the database node normally, which is beneficial to alleviating the write pressure on the database node due to high-concurrency write traffic, and while ensuring the stability of the distributed database, further improves the ability of the database node to carry burst write traffic.

[0132] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0133] Figure 5 The structure block diagram of a computing device 500 provided according to an embodiment of the present specification is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.

[0134] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0135] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0136] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 500 may also be a mobile or stationary server.

[0137] The memory 510 is used to store computer executable instructions, and the processor 520 is used to execute the following computer executable instructions to implement the steps of the data processing method.

[0138] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0139] An embodiment of the present specification also provides a computer-readable storage medium storing computer instructions, which are used to implement the steps of the data processing method when executed by a processor.

[0140] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0141] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0142] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0143] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0144] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0145] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data processing system, comprising: A client, a database node, and an object storage service node, wherein the database node is any distributed node in a distributed database; The database node is configured to receive a data write instruction sent by the client, add the data write instruction to a log synchronization queue, generate a log entry of the data write instruction, concurrently write the log entry into multiple task queues based on a pre-established data write link, generate queue overload information when it is determined that the data volume of any one of the multiple task queues is greater than a preset threshold, and reversely propagate the task queue overload information to the client based on the data write link, wherein, when the data volume of any one of the task queues is greater than the preset threshold, no new data write instruction to be processed is added to all queues of the data write link to which the task queue belongs; The client is configured to store the data to be written in the object storage service node when determining to receive the task queue overload information.

2. According to the data processing system according to claim 1, the database node is also configured to monitor the load information of the log synchronization queue and the multiple task queues, and if it is determined that the load of the log synchronization queue and / or any one of the multiple task queues is greater than a preset load threshold, queue overload information is generated.

3. According to the data processing system of claim 1 or 2, the database node is also configured to monitor the load information of the log synchronization queue and the multiple task queues. If it is determined that the data volume of the log synchronization queue and / or any one of the multiple task queues is greater than a preset data volume threshold, queue overload information is generated, and the load information includes the data volume of the log synchronization queue and / or any one of the multiple task queues.

4. According to the data processing system of claim 1 or 2, the database node is also configured to add the log entry to a log application queue, apply the log entry in the log application queue to the state machine, and store the execution result of the state machine in a message queue.

5. According to the data processing system of claim 4, the database node is also configured to monitor the load information of the log synchronization queue and the message queue, and if it is determined that the difference between the load of the log synchronization queue and the load of the message queue is greater than a preset threshold, queue overload information is generated.

6. The data processing system according to claim 1, wherein the database node comprises a master database node and at least two slave database nodes; The master database node is configured to receive a data write instruction sent by the client, add the data write instruction to a log synchronization queue, generate a log entry for the data write instruction, and concurrently write the log entry to the at least two slave database nodes based on a pre-established data write link.

7. The data processing system according to claim 6, wherein the master database node is further configured to concurrently write the log entries into the first log application queues of the at least two slave database nodes based on a pre-established data writing link; The at least two slave database nodes are configured to apply the log entries in the first log application queue to the first state machine, store the execution results of the first state machine in the first message queue, and send feedback information of successful log submission to the master database node; The master database node is further configured to submit the log entry upon receiving the feedback information.

8. According to the data processing system of claim 7, the main database node is also configured to, upon receiving the feedback information, add the log entry to a second log application queue, apply the log entry in the second log application queue to a second state machine, and store the execution result of the second state machine in a second message queue.

9. According to the data processing system of claim 8, the main database node is also configured to monitor the load information of the log synchronization queue and the second message queue, and if it is determined that the difference between the load of the log synchronization queue and the load of the second message queue is greater than a preset threshold, queue overload information is generated.

10. The data processing system according to claim 1, wherein the database node is further configured to lock any one of the multiple task queues whose data volume is greater than a preset threshold, and monitor load update information of any one of the task queues whose data volume is greater than the preset threshold, and if it is determined according to the load update information that the loads of the multiple queues are all less than the preset threshold, then send queue unlocking information to the client; The client is further configured to receive the queue unlocking information, obtain the data to be written stored in the object storage service node, and send a data write instruction to the database node based on the data to be written.

11. A data processing method, applied to a client, comprising: Sending a data write instruction to a database node based on the data to be written, wherein the data write instruction carries data identification information of the data to be written, wherein the database node is any distributed node in a distributed database, and the database node is configured to receive the data write instruction sent by the client, add the data write instruction to a log synchronization queue, generate a log entry of the data write instruction, and concurrently write the log entry to multiple task queues based on a pre-established data write link, and generate queue overload information when it is determined that the data volume of any task queue among the multiple task queues is greater than a preset threshold, and reversely propagate the task queue overload information to the client based on the data write link, wherein, when the data volume of any task queue is greater than the preset threshold, no new data write instruction to be processed is added to all queues of the data write link to which the task queue belongs; When the task queue overload information returned by the database node is received, the data to be written is stored in the object storage service node.

12. The data processing method according to claim 11, further comprising: Receive queue unlock information returned by the database node, obtain the data to be written stored in the object storage service node, and send a data write instruction to the database node based on the data to be written.

13. A data processing device, comprising: A sending module, configured to send a data write instruction to a database node based on the data to be written, wherein the data write instruction carries data identification information of the data to be written, wherein the database node is any distributed node in a distributed database, and the database node is configured to receive the data write instruction sent by the client, add the data write instruction to a log synchronization queue, generate a log entry of the data write instruction, and concurrently write the log entry to multiple task queues based on a pre-established data write link, and generate queue overload information when it is determined that the data volume of any task queue among the multiple task queues is greater than a preset threshold, and reversely propagate the task queue overload information to the client based on the data write link, wherein when the data volume of any task queue is greater than the preset threshold, no new data write instruction to be processed is added to all queues of the data write link to which the task queue belongs; The storage module is configured to store the data to be written in the object storage service node when receiving the task queue overload information returned by the database node.

14. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the data processing method according to any one of claims 11 to 12.

15. A computer-readable storage medium storing computer instructions, wherein the computer instructions, when executed by a processor, implement the steps of the data processing method according to any one of claims 11 to 12.

Images