A database system updating method and related apparatus

Abstract

The application discloses a database system updating method applied to a database including a first node, a second node and a shared storage device. The method comprises the following steps: the first node sends a first instruction to the second node, the first instruction is used for requesting storage position information of first data in the shared storage device, the first data is data transferred by the second node to the first node for processing; the first node acquires first information sent by the second node and acquires mapping information from the shared storage device according to the first information, the mapping information is used for indicating the storage position of the first data in the shared storage device. Based on the method, when the computing resource of the database system is updated, a large amount of data does not need to be migrated between nodes, but only the mapping information used for recording the data storage position is migrated, the updating efficiency of the database system can be improved, and different nodes can simultaneously process different data in the shared storage device.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a database system update method and related apparatus. Background Technology

[0002] With the rapid development of information technology, the volume of data and the number of accesses to a database are growing rapidly. Generally speaking, as the amount of data in a database increases, the number of database accesses also increases rapidly, and the requirements for access efficiency become higher. As the number of database accesses continues to grow, it is necessary to expand the database's computing resources to improve access efficiency.

[0003] Currently, database systems used to implement database functions are typically distributed database systems. A distributed database system comprises multiple subsystems, each responsible for processing different types of data. Within these subsystems, each subsystem includes compute nodes and storage nodes. Storage nodes store data from the database, while compute nodes process data from the storage nodes within the same subsystem based on data processing requests. Distributed database systems enable different compute nodes to process different data simultaneously, improving database access efficiency and effectively preventing different compute nodes from processing the same data concurrently.

[0004] Expanding the computing resources of a database typically requires adding a subsystem that includes compute nodes and storage nodes. Then, some data from the existing storage nodes is migrated to the new storage nodes, where the new compute nodes handle the migrated data. This reduces the data processing load on the existing compute nodes and achieves load balancing among them.

[0005] Clearly, existing methods for scaling up database computing resources typically require migrating a large amount of data between nodes, which takes a long time and results in low database scaling efficiency. Summary of the Invention

[0006] This application provides a database system update method that can improve the update efficiency of the database system and ensure that different nodes can process different data in a shared storage device simultaneously.

[0007] This application provides a database system update method, applied to a database system including a first node, a second node, and a shared storage device. The shared storage device is used to store data, and the first node and the second node are respectively used to process different data in the shared storage device. The method includes: the first node sending a first instruction to the second node, the first instruction being used to request storage location information of first data in the shared storage device, the first data being data transferred from the second node to the first node for processing; and the first node receiving first information sent by the second node, the first information being used to indicate mapping information stored in the shared storage device, the mapping information indicating the storage location of the first data in the shared storage device.

[0008] The first node obtains the mapping information from the shared storage device based on the first information;

[0009] The first node processes the first data in the shared storage device according to the mapping information.

[0010] In this scheme, multiple data processing nodes in the database system share the same storage device, and each node processes different data within the shared storage device. When the database system's computing resources are updated, the data being processed by a second node needs to be transferred to a first node for processing. The second node then sends an instruction to the first node, enabling the first node to retrieve the mapping information of the data storage location from the shared storage device and process the portion of data transferred to the first node based on this mapping information. By having different nodes share the same storage device and process different data based on corresponding mapping information, updates to the database system's computing resources do not require migrating large amounts of data between nodes; instead, only the mapping information for recording data storage locations is migrated. This improves the update efficiency of the database system and ensures that different nodes can simultaneously process different data within the shared storage device.

[0011] In one possible implementation, during database system expansion, the first node is a newly added node in the database, and the second node is an existing node in the database. To ensure load balancing among the nodes in the database system, the second node transfers a portion of its data responsibility to the first node.

[0012] In this solution, when expanding a database system, it is only necessary to migrate the mapping information of the data storage location to the newly added nodes to achieve the expansion of the database system, thus avoiding the migration of a large amount of data and improving the expansion efficiency of the database system.

[0013] In one possible implementation, when the database is scaled down, both the first and second nodes are existing data processing nodes in the database, and the second node is the node to be deleted from the database. The second node needs to transfer the data it is responsible for processing to other nodes in the database system for further processing.

[0014] In this solution, when the database system is downsizing, it is only necessary to migrate the mapping information that records the data storage location of the node to be deleted to other nodes to achieve the downsizing of the database system, avoiding the migration of a large amount of data and improving the downsizing efficiency of the database system.

[0015] In one possible implementation, the shared storage device includes multiple shards, each used to store different data. The mapping information includes a first mapping table and a second mapping table. The first mapping table indicates the target shard where the first data is stored in the shared storage device, and the second mapping table indicates the storage location of the first data in the target shard.

[0016] In this scheme, based on the multi-shard design of the shared storage device, the mapping information of the data storage location is divided into a first mapping table and a second mapping table. The first mapping table indicates the shard where the data is located, while the second mapping table indicates the specific storage location of the data on the shard. This makes it easier for nodes to quickly find the specific storage location of the data and improves the efficiency of node data processing.

[0017] In one possible implementation, the method further includes: the first node obtaining an incremental log from the shared storage device, the incremental log being generated by the second node during the period when the first node obtains the mapping information, the incremental log being used to record data update information in the shared storage device; and the first node updating the mapping information according to the incremental log.

[0018] In this solution, due to the large size of the mapping information in the shared storage device, the time required for the first node to synchronize the mapping information to the first area is often quite long. By having the second node simultaneously perform database services while the first node is synchronizing the mapping information, and then pausing the database service after the first node completes the synchronization, the first node can obtain the information updated by the second node during the synchronization process. This effectively avoids prolonged pauses in database services during the first node's synchronization, thus shortening the duration of database service disruption.

[0019] In one possible implementation, the method further includes: the first node storing the mapping information in a first area of ​​the shared storage device, the first area being an area in the shared storage device used to store data related to the first node; the first node generating second information based on the first area, the second information indicating the storage location of the mapping information.

[0020] In this solution, by storing the first mapping table in a shared storage device and having the first node record the storage location of the first mapping table in the shared storage device, the storage resources of the first node can be saved.

[0021] In one possible implementation, after the first node obtains the mapping information from the shared storage device based on the first information, the method further includes: the first node sending a second instruction to the second node, the second instruction being used to instruct the second node to delete the mapping information related to the first data in the second node.

[0022] In this scheme, after the first node successfully obtains the mapping information, the second node deletes the mapping information, ensuring that the first data in the shared storage device is subsequently processed only by the first node, thus avoiding multiple nodes processing the same data simultaneously.

[0023] A second aspect of this application provides a database system update method applied to a database including a first node, a second node, and a shared storage device. The shared storage device is used to store data, and the first node and the second node are respectively used to process different data in the shared storage device. The method includes: the second node obtaining a first instruction sent by the first node, the first instruction being used to request storage location information of first data in the shared storage device, the first data being data transferred from the second node to the first node for processing; the second node generating first information based on the first instruction and mapping information in the shared storage device, the first information being used to indicate the mapping information stored in the shared storage device, the mapping information being used to indicate the storage location of the first data in the shared storage device; and the second node sending the first information to the first node so that the second node obtains the mapping information.

[0024] In one possible implementation, the first node is a newly added node in the database system.

[0025] In one possible implementation, the second node is the node to be deleted in the database system.

[0026] In one possible implementation, the shared storage device includes multiple shards, each used to store different data. The mapping information includes a first mapping table and a second mapping table. The first mapping table indicates the target shard where the first data is stored in the shared storage device, and the second mapping table indicates the storage location of the first data in the target shard.

[0027] In one possible implementation, the method further includes: the second node generating an incremental log based on a data processing request and storing the incremental log in the shared storage device, wherein the data processing request is used to request an update of the first data, and the incremental log is used to record update information of the first data.

[0028] In one possible implementation, the shared storage device includes multiple shards, each used to store different data, and the incremental log includes multiple log data, each used to record data update information on different shards.

[0029] In one possible implementation, the method further includes: the second node receiving a second instruction sent by the first node, the second instruction instructing the second node to delete mapping information related to the first data in the second node; the second node deleting the mapping information related to the first data in the second node according to the second instruction.

[0030] A third aspect of this application provides a server, which is a first node in a database system. The database system includes a first node, a second node, and a shared storage device. The shared storage device is used to store data. The first node and the second node are respectively used to process different data in the shared storage device. The server includes: a sending unit, used to send a first instruction to the second node, the first instruction being used to request storage location information of first data in the shared storage device, the first data being data transferred from the second node to the first node for processing; a receiving unit, used to obtain first information sent by the second node, the first information being used to indicate mapping information stored in the shared storage device, the mapping information being used to indicate the storage location of the first data in the shared storage device; a processing unit, used to obtain the mapping information from the shared storage device according to the first information; the processing unit is further used to process the first data in the shared storage device according to the mapping information.

[0031] In one possible implementation, the first node is a newly added node in the database system.

[0032] In one possible implementation, the second node is the node to be deleted in the database system.

[0033] In one possible implementation, the shared storage device includes multiple shards, each used to store different data. The mapping information includes a first mapping table and a second mapping table. The first mapping table indicates the target shard where the first data is stored in the shared storage device, and the second mapping table indicates the storage location of the first data in the target shard.

[0034] In one possible implementation, the processing unit is further configured to: obtain an incremental log from the shared storage device, the incremental log being generated by the second node during the period when the first node obtains the mapping information, the incremental log being used to record data update information in the shared storage device; and update the mapping information according to the incremental log.

[0035] In one possible implementation, the shared storage device includes multiple shards, each used to store different data, and the incremental log includes multiple log data, each used to record data update information on different shards.

[0036] In one possible implementation, the processing unit is further configured to: store the mapping information in a first region of the shared storage device, the first region being a region in the shared storage device used to store data related to the first node; and generate second information based on the first region, the second information being used to indicate the storage location of the mapping information.

[0037] In one possible implementation, the sending unit is further configured to send a second instruction to the second node, the second instruction being configured to instruct the second node to delete the mapping information in the second node related to the first data.

[0038] A fourth aspect of this application provides a server, which is a second node in a database system. The database system includes a first node, a second node, and a shared storage device. The shared storage device is used to store data. The first node and the second node are respectively used to process different data in the shared storage device. The server includes: a receiving unit, used to acquire a first instruction sent by the first node, the first instruction being used to request storage location information of first data in the shared storage device, the first data being data transferred from the second node to the first node for processing; a processing unit, used to generate first information based on the first instruction and mapping information in the shared storage device, the first information being used to indicate the mapping information stored in the shared storage device, the mapping information being used to indicate the storage location of the first data in the shared storage device; and a sending unit, used to send the first information to the first node so that the second node acquires the mapping information.

[0039] In one possible implementation, the first node is a newly added node in the database system.

[0040] In one possible implementation, the second node is the node to be deleted in the database system.

[0041] In one possible implementation, the shared storage device includes multiple shards, each used to store different data. The mapping information includes a first mapping table and a second mapping table. The first mapping table indicates the target shard where the first data is stored in the shared storage device, and the second mapping table indicates the storage location of the first data in the target shard.

[0042] In one possible implementation, the processing unit is further configured to generate an incremental log based on a data processing request and store the incremental log in the shared storage device, wherein the data processing request is used to request an update of the first data and the incremental log is used to record update information of the first data.

[0043] In one possible implementation, the shared storage device includes multiple shards, each used to store different data, and the incremental log includes multiple log data, each used to record data update information on different shards.

[0044] In one possible implementation, the receiving unit is further configured to receive a second instruction sent by the first node, the second instruction being used to instruct the second node to delete the mapping information related to the first data in the second node; the processing unit is further configured to delete the mapping information related to the first data in the second node according to the second instruction.

[0045] The fifth aspect of this application provides a database system including a server as described in the third aspect or any implementation thereof, a server as described in the fourth aspect or any implementation thereof, and a shared storage device.

[0046] A sixth aspect of this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first or second aspect above.

[0047] The seventh aspect of this application provides a computer program product that, when run on a computer, causes the computer to perform the method described in the first or second aspect above. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the architecture of a database in related technologies;

[0049] Figure 2 This is a schematic diagram of the database expansion process in related technologies;

[0050] Figure 3 A schematic diagram of a database architecture provided for an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of the structure of a server 101 provided in an embodiment of this application;

[0052] Figure 5 A schematic flowchart of a database system update method 500 provided in an embodiment of this application;

[0053] Figure 6 A schematic flowchart of a database system update method 600 provided in an embodiment of this application;

[0054] Figure 7 A schematic diagram illustrating a database expansion method provided in an embodiment of this application;

[0055] Figure 8 A schematic diagram of the structure of a DN provided in an embodiment of this application;

[0056] Figure 9 A schematic diagram of a database component architecture provided in an embodiment of this application;

[0057] Figure 10 A schematic diagram illustrating the process of synchronizing a system mapping table, provided for an embodiment of this application;

[0058] Figure 11 A schematic diagram illustrating the process of incremental log synchronization of a system mapping table provided in an embodiment of this application;

[0059] Figure 12 A schematic diagram illustrating the process of synchronizing a fragmented mapping table as provided in an embodiment of this application;

[0060] Figure 13 A schematic diagram illustrating the incremental data synchronization process of a sharded mapping table provided in an embodiment of this application;

[0061] Figure 14 A schematic diagram illustrating the process of migrating baseline data in a mapping table, provided as an embodiment of this application;

[0062] Figure 15 A schematic diagram illustrating an incremental data migration process for a mapping table, provided as an embodiment of this application;

[0063] Figure 16 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0064] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0065] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0066] For ease of understanding, the technical terms involved in the embodiments of this application will be introduced below.

[0067] Metadata: A type of data used to describe data. Metadata mainly describes data attributes and is used to assist in data retrieval.

[0068] Baseline data: A version of data based on a specific point in time.

[0069] Incremental data: A new version of data generated based on the baseline data, that is, a new version of data generated after adding, deleting or modifying the baseline data.

[0070] Checkpoint: A mechanism similar to a bookmark. After a checkpoint is set, all previous logs are deleted from the system and permanently stored on the storage disk.

[0071] Primary replica: A data replica that can be read and written.

[0072] Backup copy: A copy of the data that cannot be written to.

[0073] Structured Query Language (SQL): A database query and programming language used to access, query, update, and manage relational database systems. Simply put, SQL is a standard language for clients to manipulate data in a database.

[0074] Coordinator Node (CN): The client access point, responsible for issuing SQL and other commands to the data processing nodes, and storing which data processing nodes are responsible for calculating the tables of each database.

[0075] Dynamic Load Management Center Control Node (CCN): Includes the functions of CN and is responsible for load balancing of data processing nodes.

[0076] Data Node (DN): This refers to the data processing node involved in this embodiment, which is responsible for processing the SQL commands issued by CN.

[0077] Data Definition Language (DDL): Commands in SQL such as creating / deleting databases and tables.

[0078] Storage Abstraction Layer (SAL): Contains metadata but not actual data, and is responsible for functions such as performing calculations on commands.

[0079] Data Function Virtualization (DFV): This refers to the separation of compute and storage.

[0080] XLog: Records transaction log information for log playback, i.e., for rollback recovery, archiving, and other functions.

[0081] Common Log Processor (CLP): Provides abstract functionalities related to XLog.

[0082] PLog is a distributed storage provided by DFV that only allows append-only writes.

[0083] ULog: Provides an infinitely long log-based storage abstraction, built on top of the PLog interface and PLog abstraction provided by DFV, and is an infinitely large log storage abstraction layer implemented with DFV PLog.

[0084] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the architecture of a database system in related technologies. Figure 1 In this context, a zone is a logical concept; a region typically contains one or more zones, and each zone contains one or more servers. For example... Figure 1 As shown, Zone1 includes server 1 and server 2, and Zone2 includes server 3 and server 4.

[0085] A unit is a unit for resource allocation. Resources such as central processing unit (CPU) and input / output (IO) resources on each server are divided into units, and migration is carried out on a unit-by-unit basis when expanding capacity.

[0086] Sharding is a unit of data partitioning, dividing the data in a database table into multiple shards. Each Unit contains multiple shards. Each shard is distributed across multiple Units, with one primary and two backups.

[0087] Please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the process of expanding computing resources in a database system within related technologies. For example... Figure 2 As shown, during the expansion of computing resources in a database system, the process begins by adding new servers to the existing Zone and creating Units on these new servers to store data. Then, the shards of the Units to be migrated from the original servers are copied to the new servers via network transfer, thus migrating the data. Finally, redundant shards and Units are deleted, meaning the migrated Units and shards from the original servers are removed, completing the expansion of the database system's computing resources.

[0088] Depend on Figure 1 and Figure 2As can be seen, in order for different servers to process different data in the database simultaneously, computing and storage resources must be expanded together during database expansion. Furthermore, a large amount of data needs to be migrated between servers during expansion, resulting in a long expansion time and consuming significant network and I / O resources. In addition, because database services are often interrupted during data migration between servers, the database service is affected for a considerable period, potentially impacting normal service operation.

[0089] In view of this, this embodiment provides a database system update method that can improve the update efficiency of the database system and ensure that different nodes in the database system can process different data in the database simultaneously.

[0090] To facilitate understanding of this solution, the embodiments of this application first combine... Figure 3 The architecture of the database provided in this application is described.

[0091] Please refer to Figure 3 , Figure 3 This is a schematic diagram of a database architecture provided for an embodiment of this application. For example... Figure 3 As shown, the database system includes multiple data processing nodes (e.g. Figure 3 The database system comprises data processing nodes 1 and 2, and a shared storage device. Multiple data processing nodes are connected to the shared storage device via network channels. These nodes receive data processing requests and process the data in the shared storage device according to these requests. For example, if the data processing request is a read request, the nodes read data from a specific storage location in the shared storage device; or, if the data processing request is a write request, the nodes write data to a specific storage location in the shared storage device, thus achieving data writing or updating.

[0092] Optionally, the database system may also include a coordinating node, which connects to multiple data processing nodes via a network channel. This coordinating node serves as the client access point, responsible for issuing SQL and other commands to the data processing nodes, and storing which data processing nodes are responsible for calculating each database table. Simply put, different data processing nodes are responsible for processing different storage areas within the shared storage device, while the coordinating node stores the storage areas each data processing node is responsible for processing. When a client initiates a data processing request, the coordinating node, based on the storage area where the requested data is located, forwards the data processing request to the corresponding data processing node for processing.

[0093] In this embodiment, the data processing node in the database system can be, for example, a server.

[0094] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of a server 101 provided in an embodiment of this application. Figure 4 As shown, server 101 includes processor 103, which is coupled to system bus 105. Processor 103 can be one or more processors, each of which can include one or more processor cores. A video adapter 107 drives a display 109, which is coupled to system bus 105. System bus 105 is coupled to input / output (I / O) bus via bus bridge 111. I / O interface 115 is coupled to the I / O bus. I / O interface 115 communicates with various I / O devices, such as input device 117 (e.g., touchscreen), external storage 121 (e.g., hard disk, floppy disk, optical disk, or USB flash drive), multimedia interface, etc. A transceiver 123 (capable of sending and / or receiving radio communication signals) and an external USB port 125 are also included. Optionally, the interface connected to I / O interface 115 can be a USB interface.

[0095] The processor 103 can be any conventional processor, including a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, or a combination thereof. Optionally, the processor can be a special-purpose device such as an ASIC.

[0096] Hard disk drive interface 131 is coupled to system bus 105. Hardware driver interface is connected to hard disk drive 133. Internal memory 135 is coupled to system bus 105. Data running in internal memory 135 may include server 101's operating system (OS) 137, applications 143, and scheduling tables.

[0097] The processor 103 can communicate with the internal memory 135 via the system bus 105, and retrieve instructions and data from the application program 143 from the internal memory 135 to execute the program.

[0098] An operating system consists of the Shell 139 and the kernel 141. The Shell 139 is an interface between the user and the operating system kernel. The shell is the outermost layer of the operating system. The shell manages the interaction between the user and the operating system: waiting for user input, interpreting user input for the operating system, and processing various operating system outputs.

[0099] Kernel 141 consists of the parts of the operating system used to manage memory, files, peripherals, and system resources. Kernel 141 interacts directly with the hardware. The operating system kernel typically runs processes and provides inter-process communication, CPU time-slice management, interrupts, memory management, I / O management, and so on.

[0100] The above describes the equipment used in the embodiments of this application. The data processing method provided in the embodiments of this application will be described in detail below.

[0101] Please refer to Figure 5 , Figure 5 This is a flowchart illustrating a database system update method 500 provided in an embodiment of this application. The database system update method 500 is applied to a database system including a first node, a second node, and a shared storage device. The shared storage device is used to store data, and both the first node and the second node are connected to the shared storage device and are respectively used to process different data in the shared storage device; that is, both the first node and the second node are data processing nodes.

[0102] like Figure 5 As shown, the database system update method 500 includes the following steps 501-503.

[0103] Step 501: The first node sends a first instruction to the second node. The first instruction is used to request the storage location information of the first data in the shared storage device. The first data is data transferred from the second node to the first node for processing.

[0104] When the database system needs updating, the second node needs to transfer the first data, which it was originally responsible for processing, from the shared storage device to the first node for processing. Since this first data was not originally processed by the first node, the first node does not have information about the storage location of the first data in the shared storage device. Therefore, the first node sends a first instruction to the second node to request the storage location information of the first data in the shared storage device. The first data can be a portion of the data stored in the shared storage device.

[0105] In this embodiment, for nodes in the database system, different nodes are responsible for processing different data in the shared storage device. This allows different nodes to process data in the shared storage device simultaneously, preventing data errors caused by different nodes processing the same data at the same time. Therefore, each node in the database system only has the storage location information of the data it is responsible for in the shared storage device, and does not have the storage location information of the data of other nodes in the shared storage device.

[0106] Optionally, in a database system expansion scenario, the first node is a newly added node in the database system, while the second node is an existing node in the database system. To ensure load balancing among the nodes in the database system, the second node transfers a portion of its data to the first node. In this database system expansion scenario, the aforementioned first data can be a portion of the data handled by the second node.

[0107] Optionally, in the scenario of database system scaling down, both the first node and the second node are existing nodes in the database system, and the second node is the node to be deleted from the database system. Therefore, the second node needs to transfer the data it is responsible for processing to other nodes in the database system for processing. In this database system scaling down scenario, the aforementioned first data can be part or all of the data handled by the second node. That is, the second node can transfer all of the data it is responsible for to the first node for processing; or the second node can transfer part of the data it is responsible for to the first node for processing, while the other part of the data handled by the second node is transferred to other nodes besides the first node for processing.

[0108] Optionally, in the case of a coordinating node in the database system, the first node may send the first instruction to the coordinating node, which then forwards the first instruction to the second node. For example, if it is necessary to transfer the responsibility of one or more shards in a shared storage device to the first node, the coordinating node may forward the first instruction to the second node, based on the node responsible for handling the one or more shards being the aforementioned second node.

[0109] Step 502: The second node generates first information based on the first instruction and the mapping information in the shared storage device. The first information is used to indicate the mapping information stored in the shared storage device, and the mapping information is used to indicate the storage location of the first data in the shared storage device.

[0110] In this embodiment, mapping information may be stored in the shared storage device. This mapping information is used to indicate the storage location of the first data in the shared storage device. For example, when the database is used to store order data, the mapping information is used to indicate the storage location of order data for each region in the shared storage device.

[0111] After the second node receives the first instruction from the first node, the second node determines the mapping information corresponding to the first data indicated in the first instruction in the shared storage device, and generates the first information based on the storage location of the mapping information in the shared storage device.

[0112] Step 503: The second node sends the first information to the first node.

[0113] In this process, the second node may send the first information to the first node through a network channel; alternatively, the second node may first send the first information to the coordinating node, which then forwards the first information to the first node.

[0114] Step 504: The first node obtains mapping information from the shared storage device based on the first information.

[0115] After obtaining the first information, the first node can determine the mapping information located in the shared storage device, and thus be able to retrieve the mapping information from the shared storage device based on the first information. For example, the first node can retrieve the mapping information in the shared storage device based on the first information and cache the mapping information in its local storage space. After obtaining the mapping information, the first node can then perform data processing in the shared storage device according to the mapping information.

[0116] Step 505: The first node processes the first data in the shared storage device according to the mapping information.

[0117] After obtaining the mapping information, the first node can process the data in the shared storage device according to the mapping information. For example, when the first node receives a data processing request from a user, the first node determines the storage location of the target data indicated in the data processing request in the shared storage device according to the mapping information, and processes the target data based on the data processing request.

[0118] Optionally, when the shared storage device in the database stores a large amount of data, the size of the mapping information used to record the data storage location is often also relatively large. Therefore, to save storage resources on the first node, the mapping information can be stored in the shared storage device, and the first node can determine the storage location of the data to be processed based on the mapping information in the shared storage device when executing a data processing request. Alternatively, the first node can cache part or all of the content of the mapping information in its local storage space when executing a data processing request.

[0119] For example, after obtaining first information indicating the storage location of the mapping information, the first node determines the storage location of the mapping information in the shared storage device based on the first information, and stores the mapping information in a first area of ​​the shared storage device. The first area is an area in the shared storage device used to store data related to the first node. The first node generates second information based on the first area, and the second information is used to indicate the storage location of the mapping information. The second information can be stored in the first node. During the execution of a data processing request, the first node can first determine the storage location of the mapping information in the shared storage device (i.e., the aforementioned first area) based on the second information, and then query the mapping information in the first area to determine the storage location of the data corresponding to the data processing request in the shared storage device, thereby realizing data processing.

[0120] In simple terms, after the first node determines the mapping information of the second node stored in the shared storage device based on the first information, the first node can copy the mapping information to the first area mentioned above, so as to perform data processing based on the mapping information in the first area.

[0121] In this embodiment, multiple data processing nodes in the database system share the same storage device, and each node processes different data within the shared storage device. When the database system's computing resources are updated, the data being processed by the second node needs to be transferred to the first node for processing. The second node then sends an instruction to the first node, enabling the first node to retrieve the mapping information of the data storage location from the shared storage device and process the portion of data transferred to the first node based on this mapping information. By having different nodes share the same storage device and processing different data based on corresponding mapping information, updates to the database system's computing resources do not require migrating large amounts of data between nodes; instead, only the mapping information for recording data storage locations is migrated. This improves the database system's update efficiency and ensures that different nodes can simultaneously process different data within the shared storage device.

[0122] In one possible embodiment, due to the large size of the mapping information in the shared storage device, the time required for the first node to synchronize the mapping information to the first region and the time required for the first node to synchronize the mapping information to the local storage space are often quite long. To avoid prolonged interruption of database service during the first node's mapping information synchronization process, in this embodiment, the second node can simultaneously perform database service during the first node's mapping information synchronization. After the first node completes the mapping information synchronization, the database service is then paused, and the first node obtains the information updated by the second node during the mapping information synchronization period.

[0123] For example, while the first node acquires mapping information and stores it in the first region, the second node can continue to provide normal database services based on its cached mapping information; that is, the second node processes data processing requests initiated by clients. During the second node's processing of data processing requests, since the data in the shared storage device changes, the mapping information indicating the data's storage location in the shared storage device also needs to be updated accordingly. Therefore, the second node can generate an incremental log based on the data processing request and store the incremental log in the shared storage device. The data processing request is used to request updates to the first data in the shared storage device, and the incremental log is used to record the update information of the first data.

[0124] Specifically, after the first node successfully stores the mapping information to the first region or its own local storage space based on the first information, the first node retrieves an incremental log from the shared storage device. This incremental log is generated by the second node during the period when the first node retrieves the mapping information, and it records data update information in the shared storage device. Then, the first node updates the mapping information stored in the first region or its own local storage space according to the incremental log.

[0125] For example, suppose that before the second node generates the incremental log, the mapping information indicates that the order data of region A is located in shard 1 of the shared storage device, and the order data of region B is located in shard 2 of the shared storage device; while the first node is synchronizing the mapping information, the second node receives a data processing request, which requests that the order data of region C be stored in shard 2. Then, while the second node performs the corresponding data processing according to the data processing request, the second node generates an incremental log, which is used to record the order data of region C stored in shard 2 of the shared storage device.

[0126] After the first node completes the synchronization of the mapping information, i.e., the first node stores the mapping information in the first area, the first node retrieves the incremental log from the shared storage device. The incremental log is generated by the second node during the period when the first node retrieves the second mapping table. Then, the first node updates the mapping information according to the incremental log.

[0127] For example, if the mapping information stored in the first region by the first node indicates that the order data of region A is located in shard 1 of the shared storage device, the order data of region B is located in shard 2 of the shared storage device, and the incremental log is used to record that the order data of region C is stored in shard 2 of the shared storage device, the first node can update the mapping information so that the mapping information indicates that shard 2 stores the order data of both region B and region C.

[0128] Optionally, after the first node completes the synchronization of mapping information, it can trigger a lock on the database structure and table structure, preventing other nodes from updating the mapping information or generating incremental logs. In this way, after locking the database structure and table structure, the first node can obtain the latest mapping information in the database based on the acquired incremental logs and mapping information.

[0129] For example, after the first node obtains the mapping information from the shared storage device based on the first information, the first node sends a second instruction to the second node. The second instruction instructs the second node to delete the mapping information related to the first data in the second node. Thus, after receiving the second instruction, the second node deletes the mapping information related to the first data in the second node according to the second instruction. Therefore, if the second node deletes the mapping information related to the first data and the first node successfully obtains the mapping information of the first data, it can be guaranteed that the first data in the shared storage device will subsequently be processed only by the first node, avoiding multiple nodes processing the same data simultaneously.

[0130] In this embodiment, by first synchronizing the generated mapping information by the first node, and then obtaining the incremental logs generated during the synchronization of the mapping information by the first node, it is possible to avoid pausing the database service during the synchronization of the mapping information by the first node. Instead, the database service is only paused for a short period of time when the first node obtains the incremental logs, which greatly reduces the pause time of the database service and ensures the normal execution of the database service.

[0131] In one possible embodiment, where the shared storage device comprises multiple shards, and each shard is used to store different data, the mapping information may include two different mapping tables. One mapping table indicates which shard in the shared storage device the data is specifically stored in, while the other mapping table indicates the specific storage address of the data within that shard.

[0132] For example, the mapping information may include a first mapping table and a second mapping table. The first mapping table indicates the target shard where the first data is stored in the shared storage device, and the second mapping table indicates the storage location of the first data in the target shard, that is, the second mapping table indicates the specific storage address of the first data in the shard. For example, the first mapping table may indicate that order data in region A is located in shard 1 of the shared storage device, and order data in region B is located in shard 2 of the shared storage device; the second mapping table may indicate the specific storage address of order data from region A at XX month XX day XX hour XX minute in shard 2.

[0133] For example, the structure of the first mapping table can be as shown in Table 1.

[0134] Table 1

[0135] 0001xxxx~0099xxxx Fragment 1 0100xxxx~0199xxxx Fragment 2 0200xxxx~0299xxxx Part 3

[0136] As shown in Table 1, each piece of data in the shared storage device can be represented by an 8-bit data identifier, and different pieces of data have different data identifiers. The first mapping table can determine the shard in the shared storage device where the data is located based on the first 4 bits of the data identifier. For example, for data whose first 4 bits are 0001-0099, the first mapping table 1 can determine that this data is located in shard 1 of the shared storage device, without needing to know the exact last 4 bits. Similarly, data whose first 4 bits are 0100-0199 is located in shard 2, and data whose first 4 bits are 0200-0299 is located in shard 3.

[0137] For example, the structure of the second mapping table can be as shown in Table 2.

[0138] Table 2

[0139] 00010000 Fragment 1, 0x0001 00010001 Fragment 1, 0x0002 00010002 Fragment 1, 0x0003 … …

[0140] As shown in Table 2, the second mapping table includes the correspondence between specific data identifiers and specific addresses within the fragments. Based on any data identifier, it is possible to determine the specific address of the data corresponding to that data identifier within the fragment.

[0141] When the mapping information includes two different mapping tables, the first node can synchronize the first mapping table first, and then synchronize the second mapping table.

[0142] For example, see Figure 6 , Figure 6 This is a flowchart illustrating a database system update method 600 provided in an embodiment of this application. Figure 6 As shown, the method 600 further includes the following steps 601-612.

[0143] Step 601: The first node sends a third instruction to the second node, the third instruction being used to request the first data to be stored in a fragment in the shared storage device.

[0144] The first data is data transferred from the second node to the first node for processing.

[0145] Step 602: The second node generates second information based on the first mapping table in the shared storage device. The second information is used to indicate the storage location information of the first mapping table in the shared storage device. The first mapping table is used to indicate the target fragment where the first data is stored in the shared storage device.

[0146] Step 603: The second node sends the second information to the first node.

[0147] Step 604: The first node retrieves the first mapping table from the shared storage device based on the second information.

[0148] Furthermore, after the first node obtains the first mapping table, it can further obtain the incremental log corresponding to the first mapping table to update the first mapping table based on the incremental log. The incremental log corresponding to the first mapping table is generated by the second node during the process of the first node obtaining the first mapping table.

[0149] Step 605: The first node sends a fourth instruction to the second node, the fourth instruction being used to request the storage location of the first data in the target fragment.

[0150] In this scenario, the first node may send the fourth instruction to the second node via a network channel; alternatively, the first node may first send the fourth instruction to the coordinating node, which then forwards the fourth instruction to the second node.

[0151] Optionally, when it is necessary to transfer the responsibility of some shards among multiple shards managed by the second node to the first node, the first node may carry the identifiers of the partial shards in the fourth instruction to instruct the second node to provide the mapping table information corresponding to the partial shards. For example, when the second node is responsible for processing data in shards 1, 2, 3, and 4 of the shared storage device, after the first node adds a new database, it needs to transfer the responsibility of shards 1 and 2, originally managed by the second node, to the first node. Therefore, the first node may carry the identifiers of shards 1 and 2 in the fourth instruction sent to the second node to instruct the second node to provide the mapping table information corresponding to shards 1 and 2.

[0152] Step 606: The second node generates third information based on the second mapping table in the shared storage device.

[0153] The third information is used to indicate the second mapping table stored in the shared storage device, and the second mapping table indicates the storage location of the first data in the target fragment of the shared storage device. During data processing, the second node is used to process the data in the shared storage device according to the first mapping table and the second mapping table.

[0154] For example, suppose the second node receives a data processing request to read order data from region A at XX month XX day XX hour XX minute. The second node can determine that the order data from region A is located in shard 2 of the shared storage device by querying the first mapping table. Then, the second node can determine the specific storage address of the order data from region A at XX month XX day XX hour XX minute in shard 2 by querying the second mapping table corresponding to shard 2, and thus read the corresponding data based on that storage address.

[0155] Step 607: The second node sends the third information to the first node so that the second node can obtain the second mapping table.

[0156] In this scenario, the second node may send the third information to the first node via a network channel; alternatively, the second node may first send the third information to the coordinating node, which then forwards the third information to the first node.

[0157] Step 608: The first node obtains the second mapping table from the shared storage device based on the third information.

[0158] Optionally, after the first node obtains the second mapping table from the shared storage device, the first node can save the second mapping table in the first node's local storage space so that the first node can quickly query the specific storage address of the data in the shared storage device based on the second mapping table in the local storage space.

[0159] Step 609: The first node sends a fifth instruction to the second node, the fifth instruction being used to instruct the second node to stop updating the second mapping table.

[0160] After the first node successfully obtains the second mapping table based on the second information, the first node sends a fifth instruction to the second node to instruct the second node to stop updating the second mapping table. Specifically, the fifth instruction may be, for example, an interrupt service command or a block service command, used to instruct the second node to interrupt the processing of data processing requests related to the second mapping table, so as to avoid modifying the second mapping table.

[0161] Understandably, the second node executes the data processing request normally before receiving the fifth instruction. During the execution of the data processing request, the second node may update the data in the shared storage device according to the data processing request, thereby triggering an update to the second mapping table.

[0162] Specifically, the second node may generate an incremental log corresponding to the second mapping table based on the data processing request. This incremental log records the update information of the second mapping table. For example, if the second mapping table indicates the specific storage address of order data for region A in shard 2, and assuming the second node receives a data processing request to request the storage of order data for region B in shard 2, then during the execution of this data processing request, the second node generates an incremental log indicating the specific storage address of order data for region B in shard 2.

[0163] Step 610: The second node stops running the target thread, which is used to perform data processing in the target slice indicated by the second mapping table.

[0164] After receiving the fifth instruction from the first node, the second node stops running the target thread, which was used to perform data processing in the target shards indicated by the second mapping table. For example, if the second mapping table indicates the specific storage addresses of data in shard 1 and shard 2, the target thread could be a thread used to perform data processing in shard 1 and shard 2. The second node can kill the target thread used to process data in shard 1 and shard 2, thereby avoiding further updates to the second mapping table.

[0165] Step 611: The second node sends fourth information to the first node. The third information is used to indicate the storage location of the incremental log corresponding to the second mapping table in the shared storage device. The incremental log corresponding to the second mapping table is generated by the second node during the period when the first node obtains the second mapping table. The incremental log corresponding to the second mapping table is used to record the update information of the second mapping table.

[0166] Step 612: The first node obtains the incremental log corresponding to the second mapping table according to the fourth information, and updates the second mapping table according to the incremental log corresponding to the second mapping table.

[0167] After obtaining the fourth information from the second node, the first node obtains the incremental log corresponding to the second mapping table in the shared storage device based on the fourth information, and updates the second mapping table stored in the local storage space based on the obtained incremental log corresponding to the second mapping table.

[0168] Optionally, the second mapping table may include multiple shard mapping tables, each used to indicate the storage location of data in a corresponding shard. That is, each shard in the shared storage device has a corresponding shard mapping table, and this shard mapping table is used to indicate the storage location of data in a specific shard. For example, the second mapping table includes shard mapping table 1, shard mapping table 2, and shard mapping table 3. If the shared storage device includes shard 1, shard 2, and shard 3, then shard mapping table 1 is used to indicate the storage location of data in shard 1, shard mapping table 2 is used to indicate the storage location of data in shard 2, and shard mapping table 3 is used to indicate the storage location of data in shard 3.

[0169] Furthermore, the incremental logs obtained by the first node include multiple log data sets; these log data sets correspond to the multiple shard mapping tables, and are used to record data update information on different shards. In other words, each log data set corresponds to a shard mapping table, and updates to each shard mapping table can be achieved based on these log data sets. Specifically, during the generation of incremental logs, the second node can independently generate corresponding log data for each shard, thus establishing a one-to-one correspondence between the log data and the shard mapping tables.

[0170] In this scheme, by generating log data corresponding to the shard mapping table independently for each shard, the first node can simultaneously update multiple shard mapping tables based on the log data corresponding to each shard mapping table during the process of updating the second mapping table based on the log data, thereby improving the update efficiency of the second mapping table.

[0171] Optionally, after the first node completes the update of the second mapping table, the first node can send a message to the second node that the mapping table update is complete, so that the second node can resume running the target thread mentioned above.

[0172] The above describes a database system update method provided by the embodiments of this application. For ease of understanding, the database system update method will be described in detail below with specific examples.

[0173] You can refer to Figure 7 , Figure 7 This is a schematic diagram illustrating a database expansion method provided in an embodiment of this application.

[0174] like Figure 7 As shown, before the database expansion, the database included CN1, CN2, DN1, DN2, and a DFV cluster. CN1 and CN2 were client access nodes, responsible for sending SQL commands, such as data processing requests, to DN1 and DN2. DN1 and DN2 were data processing nodes, responsible for processing the data processing requests sent by CN1 and CN2. The DFV cluster was a shared storage device used to store the actual user data. Within the DFV cluster, data was managed in units of shards, with different data stored in different shards.

[0175] exist Figure 7 In the DFV cluster, there are eight shards, designated shard 1 through shard 8, each used to store user data. Additionally, the DFV cluster stores a system mapping table, which indicates the shard where data is stored within the DFV cluster. This system mapping table can be, for example, the first mapping table mentioned above. Specifically, system shard DN1 in the DFV cluster is represented by system mapping table 1 stored within DN1, and system shard DN2 in the DFV cluster is represented by system mapping table 2 stored within DN2. Furthermore, the system shard in DN1 is represented by system mapping table information 1 stored within DN1, indicating the storage location of system mapping table 1 within the DFV cluster; similarly, the system shard in DN2 is represented by system mapping table information 2 stored within DN2, indicating the storage location of system mapping table 2 within the DFV cluster.

[0176] Before database expansion, DN1 handles data in shards 1, 3, 5, and 7, while DN2 handles data in shards 2, 4, 6, and 8. Both DN1 and DN2 store shard-related mapping tables. Specifically, tables 1, 3, 5, and 7 in DN1 are shard mapping tables corresponding to shards 1, 3, 5, and 7, respectively, recording the specific storage addresses of data within those shards. For example, the shard mapping tables could be the second mapping table mentioned above. Similarly, tables 2, 4, 6, and 8 in DN2 are shard mapping tables corresponding to shards 2, 4, 6, and 8, respectively, recording the specific storage addresses of data within those shards.

[0177] After database expansion, DN3 and DN4 were newly added to the database. Therefore, shards were evenly distributed among the three DNs: DN1, DN2, DN3, and DN4. Specifically, shards 3 and 7, previously handled by DN1, were transferred to DN3; shards 4 and 8, previously handled by DN2, were transferred to DN4. For example... Figure 7 As shown, taking DN1 and DN3 as examples, after database expansion, DN3 retrieves the system mapping table from DN1 and stores it in the DFV cluster. The system shard DN3 in the DFV cluster then becomes DN3's system mapping table 3 stored in the DFV cluster. Furthermore, DN3 generates system mapping table information based on the storage location of system mapping table 3 in the DFV cluster and stores it in its local storage space. DN3 also retrieves tables 3 and 7 from DN1 and stores them in its local storage space. DN1 then deletes tables 3 and 7 from its local storage space.

[0178] The system mapping table stored in the DFV cluster primarily records the data's location within the DFV cluster using metadata such as the object identifier (OID). The DFV cluster also stores a mapping table that records the specific storage address of the data within each shard. Figure 7 (not shown in the image), for example Figure 7 The mapping tables, such as Table 1 and Table 2, are stored in each DN.

[0179] You can refer to Figure 8 , Figure 8 This is a schematic diagram of a DN structure provided in an embodiment of this application. Figure 8As shown, in this embodiment, a Storage Abstraction Layer (SAL) module is added to the DN. This SAL module includes metadata but does not contain actual data, and is responsible for calculating commands. The other modules included in the SAL module are described below.

[0180] The Common Log Processor (CLP) module provides abstract interfaces related to xlog, such as log writing, reading, and the currently persisted log sequence number (LSN).

[0181] Page Reader module: Provides an interface for reading data files.

[0182] The Slice Manager module is responsible for managing the mapping rules for slices, implementing the mapping from logs to slices and from physical pages to slices. Generally, a slice in a storage device consists of one or more slices, and a slice consists of one or more pages.

[0183] Meta Service module: Manages the persistence and retrieval of data within the sal module itself.

[0184] Recovery Manager module: Provides recovery capabilities for abnormal power outages.

[0185] The ulog module, a common component, encapsulates the plog write interface in the DFV cluster and provides abstract append semantics.

[0186] The public component slice interface module encapsulates the DFV cluster's slice interface and provides abstract semantics for slices.

[0187] For ease of explanation, the following will use DN1 and DN3 as examples to provide a detailed introduction to the database system update method.

[0188] You can refer to Figure 9 , Figure 9 This is a schematic diagram of a database component architecture provided for an embodiment of this application. For example... Figure 9As shown, the CN includes an SQL module, a balancer module, and a slice manager. The SQL module retrieves client data processing requests (SQL commands), parses them, determines the DN (Data Domain) to handle the request, and distributes the request to the corresponding DN. The balancer distributes partitions from existing DNs evenly across newly added DNs during database expansion, ensuring a more even distribution of shards across DNs. The slice manager determines the DN to handle the data processing request based on the parsed request results.

[0189] The DN includes an SQL module and a SAL module. The SQL module in the DN is used to parse SQL commands issued by the CN to execute corresponding data processing steps. The SAL module includes a slice manager, which executes corresponding data processing steps based on data processing requests and updates the relevant system mapping tables or shard mapping tables.

[0190] Specifically, in a database expansion scenario, the database update process mainly includes four stages: system mapping table synchronization, incremental log synchronization of the system mapping table, shard mapping table synchronization, and incremental data synchronization of the shard mapping table. The following will use DN3 as the new node and DN1 as the source node as an example to detail each stage of the database update process.

[0191] The first stage: system mapping table synchronization.

[0192] Please refer to Figure 10 , Figure 10 This is a schematic diagram illustrating a system mapping table synchronization process provided in an embodiment of this application. For example... Figure 10 As shown, after a new node is added to the database, the new node sends a baseline data backup command to the source node. This baseline data backup command instructs the source node to provide the baseline data of the system mapping table back to the new node. Alternatively, the new node can send the baseline data backup command directly to the source node via a network channel; or it can first send the command to the coordinating node, which then forwards it to the source node.

[0193] After receiving the baseline data backup instruction, the source node sets the system mapping table to a state where mappings cannot be deleted, meaning that objects already created in the system mapping table can no longer be deleted.

[0194] Then, the source node performs a checkpoint to obtain the current version information of the system mapping table, that is, to determine which version the system mapping table has been updated to. Generally, the source node can perform system mapping table updates by generating relevant logs, so the source node can obtain the version information of the system mapping table by obtaining the log number corresponding to the current system mapping table. After obtaining the version information of the current system mapping table, the source node stores a backup of the system mapping table in the shared storage device, and after the system mapping table data is written to disk (i.e., the system mapping table is successfully backed up in the shared storage device), it sends the system mapping table information to the newly added nodes. This system mapping table information can be the version information of the current system mapping table, for example, it can be the log number corresponding to the current system mapping table. Alternatively, the system mapping table information can also indicate the storage location of the system mapping table corresponding to the source node in the shared storage device. For example, the system mapping table information can be an identifier of a specific storage space in the shared storage device, based on which the storage space in the shared storage device used to store the system mapping table can be determined; or, for another example, the system mapping table information can directly indicate the specific storage address of the system mapping table in the shared storage device.

[0195] After obtaining the system mapping table information from the source node, the new node can read the system mapping table stored by the source node in the shared storage device. For example, after obtaining the log number corresponding to the system mapping table, the new node can determine the system mapping table updated based on logs prior to that log number, thereby obtaining the system mapping table stored by the source node in the shared storage device. As another example, the new node can read the system mapping table corresponding to the storage address indicated by the system mapping table information in the shared storage device.

[0196] The newly added node distributes the system mapping table to its own system shard, that is, it stores the system mapping table in the area corresponding to the newly added node in the shared storage device, so that the newly added node can query and manage the system mapping table in the future.

[0197] Finally, the newly added node can record the storage location of the system mapping table on the shared storage device and generate relevant system mapping table information. This system mapping table information is stored in the local storage space of the newly added node so that the newly added node can query the system mapping table in the shared storage device based on the system mapping table information.

[0198] The second stage: incremental log synchronization of the system mapping table.

[0199] Please refer to Figure 11 , Figure 11 This is a schematic diagram illustrating the incremental log synchronization process of a system mapping table, provided as an embodiment of this application. For example... Figure 11As shown, after synchronizing the system mapping table, the newly added node triggers a lock on the Data Definition Language (DDL) on the shared storage device. This means adding a DDL lock to the objects in the database, preventing the creation and modification of various objects. Consequently, the source node can no longer create or modify objects in the database, thus ceasing the generation of new logs related to the system mapping table. Specifically, the newly added node can send a DDL lock command to the CN (Data Center), which then notifies all nodes to trigger the DDL lock, ensuring that none of the nodes create or modify any objects in the database.

[0200] Then, the newly added node reads the incremental log of the system mapping table generated by the source node. This incremental log is generated by the source node during the process of synchronizing the system mapping table with the newly added node and is used to indicate the update information of the system mapping table.

[0201] The newly added node distributes the incremental logs of the acquired system mapping table to its own system shards to update the system mapping table. After updating the system mapping table, the newly added node triggers the unlocking of DDL on the shared storage device to ensure the normal operation of the database service.

[0202] The third stage: sharding mapping table synchronization.

[0203] Please refer to Figure 12 , Figure 12 This is a schematic diagram illustrating a process for synchronizing a sharded mapping table, as provided in an embodiment of this application. Figure 12 As shown, after updating the system mapping table, the newly added node sends a command to the source node to request the sharding mapping table.

[0204] After receiving the mapping table information instruction, the source node requests a checkpoint at the SAL layer, which means backing up the current shard mapping table to the shared storage device. The shard mapping table indicates the specific storage address of data within a shard. Then, the source node sends shard baseline mapping table information to the new node, which indicates the shard mapping table stored in the shared storage device.

[0205] Finally, the newly added node can read the sharding baseline mapping table data from the shared storage device based on the sharding baseline mapping table information, and store the sharding baseline mapping table data in the local storage space of the newly added node to facilitate quick querying of the sharding mapping table.

[0206] The fourth stage: incremental data synchronization of the sharded mapping table.

[0207] Please refer to Figure 13 , Figure 13This is a schematic diagram illustrating the incremental data synchronization process of a sharded mapping table, as provided in an embodiment of this application. Figure 13 As shown, after synchronizing the baseline data of the sharding mapping table, the newly added node sends an interrupt service command to the source node to instruct the source node to interrupt the execution of the corresponding database service, thereby preventing the source node from continuing to update the sharding mapping table.

[0208] After receiving the interrupt service instruction, the source node kills the business threads related to the shard to be migrated, thereby preventing these business threads from continuing to update the corresponding shard mapping table.

[0209] Then, the source node sends an incremental data recovery command to the new node, instructing the new node to read the incremental data from the sharding mapping table. The incremental data in the sharding mapping table is the data updated by the source node during the new node's synchronization of the sharding mapping table.

[0210] The newly added node reads the incremental data of the sharding mapping table from the shared storage device and updates the sharding mapping table according to the incremental data to obtain the latest version of the sharding mapping table.

[0211] Finally, after the newly added node completes the synchronization of incremental data in the sharding mapping table, the newly added node notifies the source node so that the source node can resume its business threads and ensure the normal operation of the database service.

[0212] Specifically, in a database scaling-down scenario, database updates mainly include two processes: baseline data migration of the mapping table and incremental data migration of the mapping table. The following will use the example of the node to be deleted being the node to be removed from the database, and the target node being the node in the database responsible for replacing the node to be deleted, to detail each process in the database scaling-down process.

[0213] The first process: migration of the mapping table baseline data.

[0214] Please refer to Figure 14 , Figure 14 This is a schematic diagram illustrating a process for migrating baseline data in a mapping table, as provided in an embodiment of this application. Figure 14 As shown, when deciding to delete an existing node in the database, the CCN is responsible for reallocating the shards managed by the node to be deleted to other nodes. Specifically, in this embodiment, the CCM is responsible for reallocating the shards managed by the node to be deleted to the target node, which requires migrating the shard mapping table on the node to be deleted to the target node.

[0215] First, the CNN sends a command to the node to be deleted to retrieve the mapping table information. This command instructs the node to be deleted to provide relevant information about the shard mapping table.

[0216] Then, the node to be deleted requests a SAL layer checkpoint, which backs up the current shard mapping table to the shared storage device. The shard mapping table indicates the specific storage address of data within a shard. Next, the node to be deleted sends the shard mapping table information to the CCN, indicating the shard mapping table stored in the shared storage device. Optionally, the node to be deleted can also simultaneously send an incremental mapping table ID to the CCN, enabling subsequent target nodes to quickly retrieve the incremental mapping table data. This incremental mapping table ID uniquely identifies the incremental mapping table corresponding to the node to be deleted, and the node to be deleted will record the updated shard mapping table information in the incremental mapping table during the target node's shard mapping table synchronization process.

[0217] After obtaining the shard mapping table information and incremental mapping table ID sent by the node to be deleted, the CCN sends the shard mapping table information and incremental mapping table ID to the target node.

[0218] Finally, the target node can read the baseline data of the sharding mapping table from the shared storage device based on the sharding mapping table information, and store this baseline data in the storage area associated with the new node within the shared storage device. Alternatively, the new node can also store the baseline data of the mapping table in its local storage space for quick lookup of the sharding mapping table.

[0219] The second process: incremental data migration of the mapping table.

[0220] You can refer to Figure 15 , Figure 15 This is a schematic diagram illustrating an incremental data migration process for a mapping table, provided as an embodiment of this application. Figure 15 As shown, after the target node completes the synchronization of the shard mapping table baseline data, the CCN sends an interrupt service command to the node to be deleted. Upon receiving the interrupt service command, the node to be deleted kills the service threads related to the shard to be migrated.

[0221] Then, the CCN sends an incremental data recovery command to the target node, instructing the target node to read the incremental data of the corresponding shard mapping table. Upon receiving the incremental data recovery command, the target node reads the incremental mapping table data from the shared storage device according to the pre-received incremental mapping table ID, and updates the shard mapping table according to the incremental data of the shard mapping table to obtain the latest version of the shard mapping table.

[0222] Finally, after the target node completes the synchronization of incremental data in the sharding mapping table, the aforementioned nodes to be deleted can be removed from the database, thus completing the database scaling down.

[0223] The specific processes for database expansion and contraction have been described above with reference to the accompanying drawings. To facilitate understanding of the beneficial effects of the database system update method provided in this embodiment, relevant test results for database expansion are provided. Please refer to Table 3, which is a schematic diagram of the database expansion test provided in this embodiment.

[0224] Table 3

[0225]

[0226] As shown in Table 3, when using the database system update method provided in this embodiment, during the database expansion process, in a scenario with 3000 data points, when expanding from 3 nodes to 6 nodes and from 3 nodes to 9 nodes, the longest service interruption time for a single node is 219 milliseconds. This solution can shorten the database expansion time to the second level and the database impact time to the millisecond level.

[0227] exist Figures 1 to 15 Based on the corresponding embodiments, in order to better implement the above-described solutions of the embodiments of this application, related equipment for implementing the above-described solutions is also provided below.

[0228] Please refer to Figure 16 , Figure 16 This is a schematic diagram of a server structure provided in an embodiment of this application. The server is a first node in a database. The database system includes a first node, a second node, and a shared storage device. The shared storage device is used to store data. The first node and the second node are respectively used to process different data in the shared storage device. The server includes: a sending unit 1601, used to send a first instruction to the second node, the first instruction being used to request storage location information of first data in the shared storage device, the first data being data transferred from the second node to the first node for processing; a receiving unit 1603, used to obtain first information sent by the second node, the first information being used to indicate mapping information stored in the shared storage device, the mapping information being used to indicate the storage location of the first data in the shared storage device; a processing unit 1602, used to obtain the mapping information from the shared storage device according to the first information; the processing unit 1602 is also used to process the first data in the shared storage device according to the mapping information.

[0229] In one possible implementation, the first node is a newly added node in the database system.

[0230] In one possible implementation, the second node is the node to be deleted in the database system.

[0231] In one possible implementation, the shared storage device includes multiple shards, each used to store different data. The mapping information includes a first mapping table and a second mapping table. The first mapping table indicates the target shard where the first data is stored in the shared storage device, and the second mapping table indicates the storage location of the first data in the target shard.

[0232] In one possible implementation, the processing unit 1602 is further configured to: obtain an incremental log from the shared storage device, the incremental log being generated by the second node during the period when the first node obtains the mapping information, the incremental log being used to record data update information in the shared storage device; and update the mapping information according to the incremental log.

[0233] In one possible implementation, the shared storage device includes multiple shards, each used to store different data, and the incremental log includes multiple log data, each used to record data update information on different shards.

[0234] In one possible implementation, the processing unit 1602 is further configured to: store the mapping information in a first region of the shared storage device, the first region being a region in the shared storage device used to store data related to the first node; and generate second information based on the first region, the second information being used to indicate the storage location of the mapping information.

[0235] In one possible implementation, the sending unit 1601 is further configured to send a second instruction to the second node, the second instruction being configured to instruct the second node to delete the mapping information related to the first data in the second node.

[0236] In another embodiment, the server is a second node in the database. The database system includes a first node, a second node, and a shared storage device. The shared storage device is used to store data. The first node and the second node are respectively used to process different data in the shared storage device. The server includes: a receiving unit 1603, used to receive a first instruction sent by the first node, the first instruction being used to request storage location information of first data in the shared storage device, the first data being data transferred from the second node to the first node for processing; a processing unit 1602, used to generate first information based on the first instruction and mapping information in the shared storage device, the first information being used to indicate the mapping information stored in the shared storage device, the mapping information being used to indicate the storage location of the first data in the shared storage device; and a sending unit 1601, used to send the first information to the first node so that the second node can obtain the mapping information.

[0237] In one possible implementation, the first node is a newly added node in the database system.

[0238] In one possible implementation, the second node is the node to be deleted in the database system.

[0239] In one possible implementation, the shared storage device includes multiple shards, each used to store different data. The mapping information includes a first mapping table and a second mapping table. The first mapping table indicates the target shard where the first data is stored in the shared storage device, and the second mapping table indicates the storage location of the first data in the target shard.

[0240] In one possible implementation, the processing unit 1602 is further configured to generate an incremental log based on a data processing request and store the incremental log in the shared storage device, wherein the data processing request is used to request an update of the first data and the incremental log is used to record update information of the first data.

[0241] In one possible implementation, the shared storage device includes multiple shards, each used to store different data, and the incremental log includes multiple log data, each used to record data update information on different shards.

[0242] In one possible implementation, the receiving unit 1603 is further configured to receive a second instruction sent by the first node, the second instruction being configured to instruct the second node to delete the mapping information related to the first data in the second node; the processing unit 1602 is further configured to delete the mapping information related to the first data in the second node according to the second instruction.

[0243] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0244] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0245] The execution device or terminal device provided in this application embodiment can specifically be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the compilation method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0246] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0247] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0248] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0249] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0250] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

Claims

1. A database system update method, characterized in that, A method is applied to a database system comprising a first node, a second node, and a shared storage device. Each node in the database system only possesses information about the storage location of the data it is responsible for within the shared storage device. The shared storage device is used to store data and includes multiple shards, each used to store different types of data. The first node and the second node are respectively used to process the different types of data within the shared storage device. The first node sends a first instruction to the second node, the first instruction being used to request the storage location information of first data in the shared storage device, the first data being data transferred from the second node to the first node for processing; The first node receives first information sent by the second node. This first information indicates mapping information stored in the shared storage device. The mapping information indicates the storage location of the first data in the shared storage device. The first information is generated by the second node based on the storage location of the first data in the shared storage device. The mapping information includes a first mapping table and a second mapping table. The first mapping table indicates the target shard where the first data is stored in the shared storage device, and the second mapping table indicates the storage location of the first data in the target shard. The first node retrieves the mapping information from the shared storage device based on the first information. The first node processes the first data in the shared storage device according to the mapping information.

2. The method according to claim 1, characterized in that, The first node is a newly added node in the database system.

3. The method according to claim 1, characterized in that, The second node is the node to be deleted in the database system.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: The first node obtains incremental logs from the shared storage device. The incremental logs are generated by the second node during the period when the first node obtains the mapping information. The incremental logs are used to record data update information in the shared storage device. The first node updates the mapping information based on the incremental log.

5. The method according to claim 4, characterized in that, The shared storage device includes multiple shards, each used to store different data. The incremental log includes multiple log data, each used to record data update information on different shards.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The first node stores the mapping information in a first area of ​​the shared storage device, whereby the first area is a region in the shared storage device used to store data related to the first node. The first node generates second information based on the first region, and the second information is used to indicate the storage location of the mapping information.

7. The method according to any one of claims 1-6, characterized in that, After the first node obtains the mapping information from the shared storage device based on the first information, the method further includes: The first node sends a second instruction to the second node, which instructs the second node to delete the mapping information related to the first data in the second node.

8. A database system update method, characterized in that, A method is applied to a database system comprising a first node, a second node, and a shared storage device. Each node in the database system only possesses information about the storage location of the data it is responsible for within the shared storage device. The shared storage device is used to store data and includes multiple shards, each used to store different types of data. The first node and the second node are respectively used to process the different types of data within the shared storage device. The second node obtains a first instruction sent by the first node. The first instruction is used to request the storage location information of the first data in the shared storage device. The first data is data transferred from the second node to the first node for processing. The second node generates first information based on the first instruction and the mapping information in the shared storage device. The first information is used to indicate the mapping information stored in the shared storage device, and the mapping information is used to indicate the storage location of the first data in the shared storage device. The mapping information includes a first mapping table and a second mapping table. The first mapping table is used to indicate the target shard where the first data is stored in the shared storage device, and the second mapping table is used to indicate the storage location of the first data in the target shard. The second node sends the first information to the first node so that the second node can obtain the mapping information.

9. The method according to claim 8, characterized in that, The first node is a newly added node in the database system.

10. The method according to claim 8, characterized in that, The second node is the node to be deleted in the database system.

11. The method according to any one of claims 8-10, characterized in that, The method further includes: The second node generates an incremental log based on the data processing request and stores the incremental log in the shared storage device. The data processing request is used to request an update to the first data, and the incremental log is used to record the update information of the first data.

12. The method according to claim 11, characterized in that, The shared storage device includes multiple shards, each used to store different data. The incremental log includes multiple log data, each used to record data update information on different shards.

13. The method according to any one of claims 8-12, characterized in that, The method further includes: The second node receives a second instruction sent by the first node, the second instruction being used to instruct the second node to delete the mapping information related to the first data in the second node; The second node deletes the mapping information related to the first data in the second node according to the second instruction.

14. A server, characterized in that, The system includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the server performs the method as described in any one of claims 1 to 7.

15. A server, characterized in that, The system includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the server performs the method as described in any one of claims 8 to 13.

16. A database system, characterized in that, include: The server as described in claim 14, the server as described in claim 15, and the shared storage device.

17. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 13.

18. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 13.

Images