A management method and device of a distributed database, and a data processing method and device

Abstract

A distributed database management method, data processing method, and apparatus include: acquiring database switching information carried in a database switching instruction received by an application; when it is determined from the acquired database switching information that the application is preparing to switch the database for which it is obtaining services from a local database to a remote database, switching the follower node set in the data replica in the remote database to a leader node; after the switch is completed, returning a switch success message to the application, so that the application can switch the database for which it is obtaining services from the local database to the remote database. This embodiment of the present disclosure can greatly shorten the service request response time of the application when switching the database for which it is obtaining services from a local database to a remote database, thus providing services to the application quickly.

Description

Technical Field

[0001] This article relates to database management technology, and more particularly to a management method, data processing method and apparatus for a distributed database. Background Technology

[0002] A distributed database is defined as a group of logically interconnected databases distributed across a computer network. The storage, access, replication, and scheduling of data in a distributed database are all based on shards. Due to the need for disaster recovery, each shard must have a replica. The shard that is actually responsible for reading and writing is called the leader node, while the shard that is responsible for backup is called the follower node.

[0003] In related technologies, in order to enable distributed databases to provide services quickly, leader nodes are usually located in local databases. However, when an application switches the database from the local database to a remote database to obtain services, it needs to access the leader node in the local database across the network, which results in a long service request response time. Summary of the Invention

[0004] This application provides a distributed database management method, data processing method, and apparatus, which can shorten the service request response time of the application and provide services to the application quickly when the application switches the database for obtaining services from a local database to a remote database.

[0005] This application provides a method for managing a distributed database, including: Obtain the database switching information carried in the database switching command received by the application; When it is determined from the obtained database switching information that the application is preparing to switch the database for obtaining services from the local database to the remote database, the follower node set in the remote database data replica will be switched to the leader node. Once the switch is complete, a switch success message is returned to the application, enabling the application to switch the database from the local database to the remote database for obtaining services.

[0006] On the other hand, this application provides a data processing method, including: Receive the target data information corresponding to the data operation request that the application will send to the remote database; The identifier information of the leader node corresponding to the target data information in the remote database is determined based on the pre-stored data storage location information and returned to the application; wherein, the leader node of the remote database is obtained by switching the follower node set in the data replica in the remote database to the leader node when it is determined that the application is preparing to switch the database to obtain services from the local database to the remote database. Receive a data operation request from the application to the leader node corresponding to the identifier information of the determined leader node; wherein the data operation request carries the target data information; In the corresponding leader node, the target data corresponding to the target data information is operated on according to the data operation request.

[0007] In another aspect, this application also provides a management device for a distributed database, comprising: a memory and a processor, wherein the memory is used to store an executable program; The processor is used to read and execute the executable program to implement the distributed database management method described above.

[0008] In another aspect, this application also provides a distributed database, including: a memory and a processor, wherein the memory is used to store an executable program; The processor is used to read and execute the executable program to implement the data manipulation method described above.

[0009] Compared with related technologies, this application includes, when it is determined that the application is preparing to switch the database for obtaining services from a local database to a remote database, switching the follower node set in the data replica in the remote database to a leader node. This allows the application to directly access the leader node in the remote database to perform corresponding operations when initiating a data operation request, thereby avoiding cross-regional data access. Therefore, when the application switches the database for obtaining services from a local database to a remote database, it can greatly shorten the service request response time of the application and provide services to the application quickly.

[0010] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the application. Other advantages of this application can be realized and obtained by means of the solutions described in the description and the accompanying drawings. Attached Figure Description

[0011] The accompanying drawings are used to provide an understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0012] Figure 1 A flowchart illustrating a distributed database management method provided in an embodiment of this application; Figure 2 This application provides a schematic diagram of the structure of a distributed database under a two-site, three-center architecture. Figure 3 This application provides a schematic diagram of a distributed database data backup process. Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Detailed Implementation

[0013] This application describes several embodiments, but these descriptions are exemplary and not restrictive, and it will be apparent to those skilled in the art that many more embodiments and implementations are possible within the scope of the embodiments described herein. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with, or may replace, any feature or element of any other embodiment.

[0014] This application includes and contemplates combinations of features and elements known to those skilled in the art. The embodiments, features, and elements disclosed in this application may also be combined with any conventional features or elements to form a unique inventive scheme as defined by the claims. Any feature or element of any embodiment may also be combined with features or elements from other inventive schemes to form another unique inventive scheme as defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in this application may be implemented individually or in any suitable combination. Therefore, the embodiments are not limited except by the limitations imposed by the appended claims and their equivalents. Furthermore, various modifications and changes may be made within the scope of the appended claims.

[0015] Furthermore, in describing representative embodiments, the specification may have presented methods and / or processes as a specific sequence of steps. However, the method or process should not be limited to the specific order of steps described herein, to the extent that it does not depend on such a specific order. As will be understood by those skilled in the art, other sequences of steps are also possible. Therefore, the specific order of steps set forth in the specification should not be construed as a limitation of the claims. Moreover, the claims concerning the method and / or process should not be limited to the steps performed in the written order, and those skilled in the art will readily understand that these orders can be varied and still remain within the spirit and scope of the embodiments of this application.

[0016] This application provides a method for managing a distributed database, such as... Figure 1 As shown, it includes: Step 101: Obtain the database switching information carried in the database switching instruction received by the application; Step 102: When it is determined from the obtained database switching information that the application is preparing to switch the database for obtaining services from the local database to the remote database, the follower node set in the data replica in the remote database is switched to the leader node. Step 103: After the switch is complete, return a switch success message to the application so that the application can switch the database from the local database to the remote database.

[0017] For example, after a follower node of a remote database replica is promoted to leader, the leader of the local database is automatically demoted to follower, thus completing the automatic adjustment.

[0018] For example, the specific implementation process of obtaining the database switching information carried in the database switching instruction received by the application may include: pre-setting an exposed interface (specifically an HTTP interface) and writing corresponding execution actions into the application so that when the application receives the database switching instruction, it first calls the set interface and passes the database switching information carried in the received database switching instruction to the interface.

[0019] For example, the database switching information may specifically include identification information of the target database to which the application is preparing to switch the database for obtaining services from the current database.

[0020] For example, after the leader node switch is completed, the application switches to ensure that the application continues to provide services after the switch is completed.

[0021] In related technologies, distributed databases by default distribute the leader node evenly across all nodes according to a load balancing strategy, achieving average pressure on each node and fully leveraging the performance of the distributed database. In a two-site, three-center architecture, such as... Figure 2 As shown, the leader will be fixed in the local city (city 1), meaning the leader node will be set in the local database (i.e., the local database and the database within the same city). In drills or real business scenarios, when the application receives a database switch instruction to switch the database for obtaining services from the local database to the database within the same city, the application calls an HTTP interface, passing the database switch information to the interface. When the database switch information determines that the application needs to switch the database for obtaining services from the local database to the database within the same city, it is confirmed as a same-city switch. No leader adjustment is needed, and feedback can be directly provided (specifically, feedback can be set to 1). The application can then directly access the database for obtaining services. The application can switch from its local database to a database in the same city to continue providing services, and service requests initiated by the application can still be responded to quickly. However, when the application receives a database switch instruction to switch the database from its local database to a database in a different location, the application calls an HTTP interface, passing the database switch information to the interface. When the application determines that it needs to switch the database from its local database to the database in a different location based on the database switch information, cross-database data access will occur because the leader node is set in both the local and the same city databases, resulting in long service acquisition times. Therefore, a switch command needs to be initiated, and a function similar to transform leader is used to transfer all leaders to the different location, thereby ensuring the continuous and fast service capability of the database.

[0022] When an application switches the database for obtaining services from the local database to a remote database, the leader node is set in both the local and local databases, which leads to cross-database data access and results in long service retrieval times.

[0023] The distributed database management method provided in this application, when determining that the application is preparing to switch the database for obtaining services from a local database to a remote database, switches the follower node set in the data replica in the remote database to a leader node. This allows the application to directly access the leader node in the remote database to perform corresponding operations when initiating a data operation request, thus avoiding cross-database data access. Therefore, when the application switches the database for obtaining services from a local database to a remote database, it can greatly shorten the service request response time of the application and provide services to the application quickly.

[0024] In one exemplary instance, after switching the follower node set up as the leader node in the remote database data replica, the method further includes: When writing data, the number of replicas required to return a successful write to the application is adjusted from M to N, where M is a preset number of replicas, N is a number of replicas not greater than P, and P is the sum of the number of leader nodes that wrote the data and the number of follower nodes corresponding to that leader node located in the remote database.

[0025] For example, when an application initiates a data write request to write data to a distributed database, the distributed database needs to successfully save a certain number of replicas of the written data in order to return a write success message to the application (the standard for write success is receiving a write success message from the node to which the data was written). That is, it not only needs to write to the leader node, but also needs to save the data written to the leader node to the follower nodes, so that the total number of replicas written to the leader node and the number of replicas written to the follower nodes meets a certain number of replicas.

[0026] However, when an application initiates a write operation request in a remote database, it writes data to the corresponding leader node. Simultaneously, the leader node also backs up the written data to the follower nodes. But if the number of follower nodes requiring backup is large and the remote database doesn't have enough, backups must be made to the local database's follower nodes. A success message is only returned to the application after the required number of replicas have been backed up. This involves cross-network operations, slowing down service response. Therefore, the number of replicas required to return a successful write to the application is adjusted to be greater than the sum of the number of replicas on the leader node in the remote database and the number of its corresponding follower nodes. Thus, when writing data to the leader node, the leader node only needs to write data to its corresponding follower node in the remote database and receive a write success message from that follower node. The distributed database can then return a write success message to the application, thus speeding up service response. As for backups to follower nodes in the local database across databases, there's no need to wait for feedback; asynchronous returns are sufficient. Data backup typically employs Paxos or Raft protocols, as illustrated in the diagram below. Figure 3 As shown.

[0027] For example, M can be determined according to the majority principle. For instance, if there are K leader nodes and K follower nodes storing the same data, then according to the majority principle, M should be greater than [K / 2]. When K equals 7, then M equals 4, and when K equals 5, then M equals 3.

[0028] In one exemplary instance, after returning the switching success information to the application, the method further includes: First, obtain the database switching information carried in the database switching command received by the application; Secondly, when it is determined from the obtained database switching information that the application is preparing to switch the database for obtaining services from a remote database to a local database, the follower node set in the data replica in the local database is switched to a leader node. Finally, once the switch is complete, a switch success message is returned to the application, enabling the application to switch the database from the remote database back to the local database.

[0029] In one exemplary instance, after switching the follower node set in the local database data replica to the leader node, the process further includes: When writing data, the number of copies to be saved that must be satisfied when the write is successful is adjusted back from N to M.

[0030] For example, when switching back to the local database, the number of replicas should also be adjusted back.

[0031] In one exemplary instance, the distributed database management method provided in this application embodiment further includes: First, monitor the distribution of leader nodes; Secondly, when it is detected that the leader node is set in a remote database, and the database for which the application obtains services is not a remote database, the follower node set in the local database that corresponds to the detected leader node will be switched to the leader node.

[0032] For example, if the leader node is set in a remote database and the database from which the application obtains services is not a remote database, it indicates that an error occurred during the leader node switchover process, which needs to be corrected. An alarm will also be triggered, which can be communicated to the developers and database administrators via SMS and email.

[0033] The distributed database management method provided in this application can intelligently manage the distribution of leaders and intelligently adjust the changes of replicas, thereby improving the service capabilities of the distributed database cluster.

[0034] This application provides a data processing method, such as... Figure 4 As shown, it includes: Step 201: Receive the target data information corresponding to the data operation request that the application will initiate to the remote database; Step 202: Determine the identifier information of the leader node corresponding to the target data information in the remote database based on the pre-stored data storage location information, and return it to the application; wherein, the leader node of the remote database is obtained by switching the follower node set in the data replica in the remote database to the leader node when it is determined that the application is preparing to switch the database for obtaining services from the local database to the remote database. Step 203: Receive a data operation request from the application to the leader node corresponding to the identifier information of the determined leader node; wherein the data operation request carries the target data information; Step 204: In the corresponding leader node, perform operations on the target data corresponding to the target data information according to the data operation request.

[0035] For example, a distributed database has a source data node, which acts as a "guide" and stores data storage location information. Before an application makes a data operation request to the distributed database, it first sends the target data information to the source data node, i.e., the corresponding target data information. The source data node obtains the identification information of the leader node where the target data is located based on the data storage location information, and then returns it to the application. The application can then initiate a data operation request to the leader node in the distributed database corresponding to the identification information based on this identification information.

[0036] The data processing method provided in this application, when determining that the application is preparing to switch the database for obtaining services from the local database to the remote database, switches the follower node set in the data replica in the remote database to the leader node. Therefore, when a data operation request is received from the application to the remote database, the corresponding operation is performed directly on the corresponding leader node in the remote database, thus avoiding cross-database data access. This greatly shortens the service request response time of the application and provides services to the application quickly.

[0037] In one exemplary instance, the data storage location information includes: the correspondence between the data information already stored in the database and the identification information of the leader node, and the data storage rule information when the data is written to the database.

[0038] In one exemplary instance, when the data operation request is a read operation request to read target data, determining the identifier information of the leader node corresponding to the target data information in the remote database based on pre-stored data storage location information includes: Based on the correspondence, determine the identifier information of the leader node where the target data is located in the remote database; In one exemplary instance, when the data operation request is a write operation request to write target data, determining the identifier information of the leader node corresponding to the target data information in the remote database based on pre-stored data storage location information includes: Based on the data storage rule information, determine the identifier information of the leader node to which the target data will be written in the remote database.

[0039] For example, a read operation requires determining the location of the data to be read, while a write operation requires determining the location of the data to be written based on predefined data writing rules. Specifically, the data writing rules could be that data from 0 to 10000 is written to the first leader node, data from 10001 to 20000 is written to the second leader node, and so on.

[0040] In one exemplary instance, when the data operation request is a write operation request, the step of operating on the target data corresponding to the target data information in the corresponding leader node according to the data operation request includes: First, write the target data into the corresponding leader node; Next, the target data written to the leader node is synchronized to N-1 follower nodes in the remote database. After receiving synchronization success information from N-1 follower nodes, the execution success information of the data operation request is returned to the application. Here, N is the number of replicas to be saved when returning a successful write to the application with the data written in advance, which is adjusted from M; M is the preset number of replicas to save, N is the number of replicas to save that is not greater than P, and P is the total number of the leader node to which the data was written and the number of the follower nodes corresponding to the leader node located in the remote database.

[0041] In one exemplary instance, the data processing method provided in this application embodiment further includes: The data storage location information is updated whenever the leader node is switched.

[0042] For example, whenever the leader node switches, the actual leader node responsible for reading and writing changes. Therefore, the data storage location information should be updated. Updating the data storage location information includes updating the correspondence between the data information already stored in the database and the identification information of the leader node, as well as updating the data storage rules information when writing data to the database.

[0043] This application embodiment also provides a distributed database management device, including: a memory and a processor, wherein the memory is used to store executable programs; The processor is used to read and execute the executable program to implement the distributed database management method described in any of the above embodiments.

[0044] This application embodiment also provides a distributed database, including: a memory and a processor, wherein the memory is used to store executable programs; The processor is used to read and execute the executable program to implement the data processing method described in any of the above embodiments.

[0045] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

Claims

1. A method for managing a distributed database, characterized in that, include: Obtain the database switching information carried in the database switching command received by the application; When it is determined from the obtained database switching information that the application is preparing to switch the database for obtaining services from the local database to the remote database, the follower node set in the data replica in the remote database will be switched to the leader node. Once the switch is complete, a switch success message is returned to the application, enabling the application to switch the database from the local database to the remote database for obtaining services.

2. The method according to claim 1, characterized in that, After switching the follower node set up as the data replica in the remote database to the leader node, the method further includes: When writing data, the number of replicas required to return a successful write to the application is adjusted from M to N, where M is a preset number of replicas, N is a number of replicas not greater than P, and P is the sum of the number of leader nodes that wrote the data and the number of follower nodes corresponding to that leader node located in the remote database.

3. The method according to claim 2, characterized in that, After returning a successful switch message to the application, the process also includes: Obtain the database switching information carried in the database switching command received by the application; When it is determined from the obtained database switching information that the application is preparing to switch the database for obtaining services from a remote database to a local database, the follower node set in the data replica in the local database will be switched to the leader node. Once the switch is complete, a successful switch message is returned to the application, enabling the application to switch the database from the remote database back to the local database.

4. The method according to claim 3, characterized in that, After switching the follower node set in the local database data replica to the leader node, the process also includes: When writing data, the number of copies to be saved that must be satisfied when the write is successful is adjusted back from N to M.

5. The method according to claim 1, characterized in that, The method further includes: Monitor the distribution of leader nodes; When it is detected that the leader node is set in a remote database, and the database from which the application obtains services is not a remote database, the follower node set in the local database that corresponds to the detected leader node will be switched to the leader node.

6. A data processing method, characterized in that, include: Receive the target data information corresponding to the data operation request that the application will send to the remote database; The identifier information of the leader node corresponding to the target data information in the remote database is determined based on the pre-stored data storage location information and returned to the application; wherein, the leader node of the remote database is obtained by switching the follower node set in the data replica in the remote database to the leader node when it is determined that the application is preparing to switch the database to obtain services from the local database to the remote database. Receive a data operation request from the application to the leader node corresponding to the identifier information of the determined leader node; wherein the data operation request carries the target data information; In the corresponding leader node, the target data corresponding to the target data information is operated on according to the data operation request.

7. The method according to claim 6, characterized in that, The data storage location information includes: the correspondence between the data information already stored in the database and the identification information of the leader node, and the data storage rule information when the data is written to the database; When the data operation request is a read operation request to read target data, determining the identifier information of the leader node corresponding to the target data information in the remote database based on pre-stored data storage location information includes: Based on the correspondence, determine the identifier information of the leader node where the target data is located in the remote database; When the data operation request is a write operation request to write target data, determining the identifier information of the leader node corresponding to the target data information in the remote database based on pre-stored data storage location information includes: Based on the data storage rule information, determine the identifier information of the leader node to which the target data will be written in the remote database.

8. The method according to claim 6, characterized in that, When the data operation request is a write operation request, the step of operating on the target data corresponding to the target data information in the corresponding leader node according to the data operation request includes: Write the target data into the corresponding leader node; The target data written to the leader node is synchronized to N-1 follower nodes in the remote database. After receiving synchronization success information from N-1 follower nodes, the execution success information of the data operation request is returned to the application. Here, N is the number of replicas required to return a successful write to the application when the data is written in advance, which is adjusted from M; M is the preset number of replicas, N is the number of replicas not greater than P, and P is the total number of the leader node to which the data is written and the number of the corresponding follower nodes located in the remote database.

9. The method according to claim 6, characterized in that, The method further includes: The data storage location information is updated whenever the leader node is switched.

10. A management device for a distributed database, characterized in that, include: A memory and a processor, wherein the memory is used to store an executable program; The processor is used to read and execute the executable program to implement the distributed database management method as described in any one of claims 1-5.

11. A distributed database, characterized in that, include: A memory and a processor, wherein the memory is used to store an executable program; The processor is used to read and execute the executable program to implement the data manipulation method as described in any one of claims 6-9.

Images