A data fragmentation processing method, apparatus, computer equipment, medium, and program product.

Abstract

This application discloses a data sharding processing method, apparatus, computer device, medium, and program product. The method is applied to a target node in a cluster comprising multiple nodes. Specifically, it retrieves ownership information corresponding to multiple data shards from a preset storage location. If, based on the ownership information of a first data shard, it is determined that none of the nodes in the cluster owns the first data shard, then ownership of the first data shard is acquired. If ownership of the first data shard is successfully acquired, the first data shard is loaded. In this process, the node indicated by the ownership information of the first data shard in the preset storage location is updated to the target node. The ownership information of each data shard is uniformly stored in the preset storage location, eliminating the need for frequent communication between multiple nodes in the cluster. Instead, the information is retrieved directly from the preset storage location, which is convenient, fast, reduces network overhead, shortens communication time, and improves the processing efficiency of data shards.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data fragmentation processing method, apparatus, computer equipment, medium, and program product. Background Technology

[0002] The original data is split into multiple independent and orthogonal sub-data, with different sub-data stored on different nodes in the cluster. Each sub-data can be regarded as a data shard.

[0003] In other words, there is a correspondence between nodes and data shards. Therefore, if a processing operation (such as modification or deletion) is performed on a specific data shard, the operation is executed based on the node corresponding to that data shard. If the node corresponding to the data shard experiences a crash or other issues, making it impossible to process that data shard, related technologies typically establish correspondences between other nodes in the cluster and the data shard, allowing processing to be performed based on those other nodes.

[0004] However, for data sharding with high real-time requirements, the above method may result in untimely processing of data shards, leading to low processing efficiency. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a data fragmentation processing method, apparatus, computer equipment, medium, and program product to improve the processing efficiency of data fragmentation.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] On one hand, embodiments of this application provide a data sharding processing method, which is applied to a target node in a cluster comprising multiple nodes, the method comprising:

[0008] The ownership information corresponding to multiple data shards is obtained from a preset storage location. The ownership information of the data shards is used to indicate the nodes in the cluster that own the data shards. The ownership of the data shards is used to identify the nodes that have the authority to process the data shards.

[0009] If it is determined, based on the ownership information of the first data shard, that none of the nodes in the cluster have ownership of the first data shard, then the ownership of the first data shard is acquired, and the first data shard is one of the multiple data shards.

[0010] If ownership of the first data fragment is successfully acquired, the first data fragment is loaded, wherein the node indicated by the ownership information of the first data fragment in the preset storage location is updated to the target node.

[0011] On the other hand, embodiments of this application provide a data sharding processing apparatus, which is applied to a target node in a cluster including multiple nodes. The apparatus includes: an ownership information acquisition unit, an ownership acquisition unit, and a loading unit.

[0012] The ownership information acquisition unit is used to acquire ownership information corresponding to multiple data shards from a preset storage location. The ownership information of the data shards is used to indicate the nodes in the cluster that own the data shards. The ownership of the data shards is used to identify that the nodes have the authority to process the data shards.

[0013] The ownership acquisition unit is configured to acquire ownership of the first data shard if it is determined, based on the ownership information of the first data shard, that none of the nodes in the cluster have ownership of the first data shard, wherein the first data shard is one of the plurality of data shards.

[0014] The loading unit is configured to load the first data fragment if ownership of the first data fragment is successfully acquired, wherein the node indicated by the ownership information of the first data fragment in the preset storage location is updated to the target node.

[0015] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory:

[0016] The memory is used to store computer programs and to transfer the computer programs to the processor;

[0017] The processor is configured to execute the methods described above according to instructions in the computer program.

[0018] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program for performing the methods described above.

[0019] On the other hand, embodiments of this application provide a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the methods described above.

[0020] As can be seen from the above technical solution, establishing ownership information for data shards clarifies the correspondence between nodes and data shards. For example, the ownership information of a data shard determines which node in the cluster owns that data shard, thus enabling processing of that data shard. Furthermore, compared to distributed storage across individual nodes (where each node only stores its own responsible data shards, equivalent to local information), storing the ownership information of all data shards uniformly in a preset storage location (equivalent to global information) provides higher accuracy. Taking a target node in the cluster as an example, the target node retrieves ownership information for multiple data shards from the preset storage location. If, based on the ownership information of the first data shard, it is determined that no node in the cluster owns the first data shard, the target node acquires ownership of the first data shard. If the acquisition of ownership of the first data shard is successful, the target node loads the first data shard for timely processing. Moreover, in the preset storage location, the node indicated by the ownership information of the first data shard is updated to the target node; that is, the ownership information is dynamically updated, ensuring that nodes can promptly obtain the correct ownership information from the preset storage location.

[0021] Therefore, by updating the ownership information of data shards in a timely manner, data shards without corresponding nodes can be loaded by nodes in the cluster in a timely manner, so that the data shards can be processed promptly. Moreover, the ownership information of each data shard is uniformly stored in a preset storage location, so that multiple nodes in the cluster do not need to communicate frequently with each other, but can directly retrieve it from the preset storage location. This is convenient and fast, reduces network overhead, shortens communication time, and improves the processing efficiency of data shards. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of a centralized cluster.

[0024] Figure 2 This is a schematic diagram of a decentralized cluster;

[0025] Figure 3 This is a schematic diagram illustrating an application scenario of a data fragmentation processing method provided in an embodiment of this application.

[0026] Figure 4A flowchart illustrating the data fragmentation processing method provided in this application embodiment;

[0027] Figure 5 A schematic diagram of a game leaderboard provided for an embodiment of this application;

[0028] Figure 6 A schematic diagram illustrating another game leaderboard provided as an embodiment of this application;

[0029] Figure 7 A schematic diagram of a position-adjusted leaderboard provided for an embodiment of this application;

[0030] Figure 8 This is a schematic diagram illustrating the relationship of a data structure provided in an embodiment of this application;

[0031] Figure 9 A schematic diagram illustrating a node acquiring data fragments, provided as an embodiment of this application;

[0032] Figure 10 This is a schematic diagram illustrating how a target node renews a lease for a data shard, as provided in an embodiment of this application.

[0033] Figure 11 A schematic diagram illustrating the periodic checking of a target node as provided in an embodiment of this application;

[0034] Figure 12 This is a schematic diagram of a load balancing method provided in an embodiment of this application.

[0035] Figure 13 This is a schematic diagram of the structure of a data fragmentation processing device provided in an embodiment of this application;

[0036] Figure 14 This application provides a schematic diagram of the structure of a server according to an embodiment of the present application.

[0037] Figure 15 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0038] The embodiments of this application will now be described with reference to the accompanying drawings.

[0039] Taking a leaderboard as an example, a data sharding system with high real-time requirements, a leaderboard reflects the objective strength of similar related entities and involves comparisons between them. For example, game leaderboards need to be updated in real-time, thus having high real-time requirements. Furthermore, games can have different types of leaderboards, such as those reflecting the number of levels a player has completed, those reflecting a player's rank, and those reflecting a player's score. They can also have leaderboards for different regions, such as district, city, and national leaderboards. In other words, there may be a large number of game leaderboards within the same game. Therefore, each leaderboard can be stored in the cluster as a data shard.

[0040] To ensure rapid processing of data shards with high real-time requirements, it's essential to guarantee that each data shard has a corresponding node. This means that nodes load corresponding data shards, allowing for timely processing of the data shard after receiving processing operations. This avoids the need to first allocate a corresponding node to the data shard before executing the processing operation, thus improving the processing efficiency for data shards with high real-time requirements. The following sections illustrate two cluster methods for ensuring that data shards have corresponding nodes.

[0041] The first type is a centralized cluster, which includes a central scheduling server and multiple regional servers (each regional server corresponds to a node in the cluster).

[0042] The game's coverage area can be divided into multiple fixed regions based on geographical location. One regional server is responsible for processing data fragments for one fixed region, and multiple regional servers can be used to process data fragments across the game's coverage area. A central scheduling server receives load reports from each regional server. When the load of a regional server exceeds its limit, the central scheduling server performs a scaling operation, adding a new regional server. This allows the new regional server to share the data fragments for that fixed region with the server exceeding the load threshold, thus preventing the regional server from crashing due to excessive load and ensuring that all data fragments are processed promptly by the regional servers.

[0043] For example, see the leaderboard. Figure 1 This diagram illustrates a centralized cluster. Figure 1In this centralized cluster, there is a central scheduling server and multiple servers, each corresponding to a specific region. For example, the North China server corresponds to the North China region, and the Northeast server corresponds to the Northeast region. When the central scheduling server detects that the load on the East China server exceeds the load threshold, it adds a new server through a scaling operation. This new server is used to share the load pressure of the East China server and, together with the East China server area, processes the leaderboards in the East China region. This ensures that the East China server does not crash due to excessive load, thereby ensuring that all leaderboards in the East China region have servers that can process them in a timely manner, improving the processing efficiency of various leaderboards in the game.

[0044] While the centralized cluster approach described above allows for centralized management of multiple regional servers and simplifies decision-making, the central scheduling server can become a bottleneck because all regional server load calculations rely on it. This can lead to delayed scaling, resulting in slower leaderboard processing and lower leaderboard efficiency. To address the issues arising from centralized clustering in handling data sharding, a decentralized cluster approach is employed to ensure that each data shard has a corresponding node.

[0045] The second type is a decentralized cluster that includes multiple nodes (each node corresponds to a server).

[0046] Each node processes a data shard from a fixed geographical area to ensure timely processing. This fixed area is determined by geographical location. Each node is responsible for calculating its own load, rather than having everything centralized in a central scheduling server. Multiple nodes periodically exchange load information via a gossip protocol, allowing each node to understand the approximate load of other nodes in the decentralized cluster. The gossip protocol is a decentralized distributed protocol where data spreads like a virus from node to node, with frequent information exchange between nodes. When a node in the decentralized cluster detects that its load exceeds its limit, it transfers a portion of its data shard to a node with a lower load. This transfer of the data shard to the lower-load node prevents the high-load node from crashing while ensuring that each data shard has a corresponding node.

[0047] For example, see the leaderboard. Figure 2The diagram illustrates a decentralized cluster. This cluster comprises seven nodes, each corresponding to a server responsible for processing leaderboards for a fixed region. For example, the leaderboard for the East China region is processed by the East China node, allowing its corresponding server to handle leaderboards for that region. If the East China node experiences high load exceeding its load limit, it will transfer a portion of its leaderboard load to the less loaded Northwest node. This reduces the load on the East China node, prevents it from crashing, and ensures that each leaderboard has a corresponding node, thus guaranteeing efficient leaderboard processing.

[0048] Although decentralized clusters do not rely on a central scheduling server (i.e., a single node), multiple nodes in a decentralized cluster need to frequently communicate with each other via a gossip protocol to obtain the latest load information of each node, thereby deciding whether to perform data shard transfer operations and which node to transfer the data shards to. This not only brings significant network overhead but also results in long communication times, which in turn leads to untimely scaling of nodes with high loads. Consequently, some data shards cannot be transferred to other nodes in a timely manner, resulting in problems with data shards not being processed promptly.

[0049] Based on this, this application provides a data sharding processing method. It establishes ownership information for each data shard and, by updating this ownership information promptly, ensures that data shards without corresponding nodes can be loaded by nodes in the cluster in a timely manner, enabling timely processing. Furthermore, the ownership information of each data shard is uniformly stored in a preset storage location, eliminating the need for frequent communication between multiple nodes in the cluster. Instead, the information can be directly retrieved from the preset storage location, which is convenient, fast, reduces network overhead, shortens communication time, and improves the processing efficiency of data shards.

[0050] The data fragmentation processing method provided in this application can be applied to computer devices with data fragmentation processing capabilities, such as terminal devices and servers.

[0051] Specifically, terminal devices can be desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Smart in-vehicle devices can be in-vehicle navigation terminals and in-vehicle computers, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc., but are not limited to these.

[0052] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal devices and servers can be connected directly or indirectly via wired or wireless communication; this application does not impose any restrictions on this.

[0053] To facilitate understanding of the data fragmentation processing method provided in this application embodiment, the following example uses a server as the execution subject of the data fragmentation processing method to illustrate its application scenarios.

[0054] See Figure 3 This figure is a schematic diagram illustrating an application scenario of a data fragmentation processing method provided in an embodiment of this application. For example... Figure 3 As shown, this application scenario includes a cluster 300 and a database 340, which can communicate with each other via a communication network (not shown in the figure). The communication network uses standard communication technologies and / or protocols, typically the Internet, but can also be any network, including but not limited to Bluetooth, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), mobile, private networks, or any combination of virtual private networks. In some embodiments, customized or dedicated data communication technologies may be used to replace or supplement the aforementioned data communication technologies.

[0055] exist Figure 3 In this example, cluster 300 is responsible for processing multiple data shards. Taking a cluster with three nodes as an example, cluster 300 includes servers 310, 320, and 330. Each server corresponds to one node in the cluster and is used to process a portion of the data shards. The data shard a node is responsible for refers to a data shard that has a corresponding relationship with that node, and the ownership of that data shard belongs to that node. The ownership information clarifies the correspondence between nodes and data shards; for example, it identifies which node in the cluster owns a particular data shard, thus enabling the processing of that data shard.

[0056] Database 340 is a preset storage location used to store the ownership information corresponding to each data shard. Moreover, compared to distributed storage on various nodes, where each node only stores the data shard it is responsible for (equivalent to local information), storing the ownership information of each data shard uniformly in the preset storage location (equivalent to global information) provides higher accuracy.

[0057] The following explanation uses the target node (server 330) in the cluster as an example to illustrate the data sharding process. Server 330 retrieves ownership information for each data shard from database 340, thereby determining which node is responsible for each data shard based on this ownership information. If, based on the ownership information of the first data shard, it is determined that no node in the cluster owns the first data shard (i.e., no node is responsible for the first data shard), then server 330 can acquire ownership of the first data shard. If the acquisition of ownership of the first data shard is successful, server 330 will load the first data shard so that it can be processed promptly in the future.

[0058] Furthermore, after server 330 successfully acquires ownership of the first data analysis, server 330 can proactively modify the ownership information of the first data fragment and send it to database 340. Alternatively, database 340 can modify the ownership information of the first data fragment. As a result, in database 340, the node indicated by the ownership information of the first data fragment will be updated to server 330, that is, all permissions of the first data fragment will be updated. In order to obtain the correct ownership information from the preset storage location in a timely manner by dynamically updating the ownership information of the data fragment.

[0059] Therefore, by updating the ownership information of data shards in a timely manner, data shards without corresponding nodes can be loaded by nodes in the cluster in a timely manner, so that the data shards can be processed promptly. Moreover, the ownership information of each data shard is uniformly stored in a preset storage location, so that multiple nodes in the cluster do not need to communicate frequently with each other, but can directly retrieve it from the preset storage location. This is convenient and fast, reduces network overhead, shortens communication time, and improves the processing efficiency of data shards.

[0060] The data fragmentation processing method provided in this application embodiment can be executed by a server. However, in other embodiments of this application, the terminal device may also have similar functions to the server to execute the data fragmentation processing method provided in this application embodiment, or the terminal device and the server may jointly execute the data fragmentation processing method provided in this application embodiment. This embodiment does not limit this.

[0061] The data sharding processing method provided in this application can be applied to various scenarios, including but not limited to distributed systems, cloud storage and cloud computing, big data analytics, the Internet of Things (IoT), and finance. Financial data and data from devices within the IoT, in particular, have high real-time requirements and can therefore be processed as data shards, through distributed storage, big data analytics, cloud computing, etc. The following explanation uses a distributed storage scenario for game leaderboards as an example.

[0062] As mentioned above, the game contains a large number of leaderboards, and these leaderboards have high real-time requirements. Therefore, distributed storage can be used to store the game leaderboards. In this cluster, different nodes are responsible for processing different game leaderboards. Each game leaderboard has its own ownership information, indicating which node in the cluster is responsible for that leaderboard, and the ownership information for multiple game leaderboards is stored in the same preset storage location.

[0063] For example, if the load of a node exceeds its load limit, multiple nodes in the cluster do not need to frequently communicate through gossip protocols to determine which node to transfer the game leaderboard to. Instead, they directly release ownership of some data shards. As a result, the ownership information of the released game leaderboard will be updated to show that no node is processing the game leaderboard. Then, any of the other nodes can acquire ownership of the game leaderboard so that it can be processed by other nodes.

[0064] In other words, if some game leaderboards have corresponding nodes while others do not, taking the target node as an example, the target node retrieves ownership information for multiple game leaderboards from a preset storage location. If, based on the ownership information of the first game leaderboard (i.e., one of the multiple game leaderboards), it is determined that none of the nodes in the cluster owns the first game leaderboard (i.e., no node is responsible for the first game leaderboard), then the target node acquires ownership of the first game leaderboard. If the acquisition of ownership of the first game leaderboard is successful, the first game leaderboard is loaded, and then the node indicated by the ownership information of the first game leaderboard in the preset storage location is updated to the target node.

[0065] It should be noted that the above application scenarios are merely examples, and the data sharding processing method provided in this embodiment can also be applied to other scenarios, which are not limited here.

[0066] The following describes in detail a data sharding processing method provided in this application through method embodiments. It should be noted that this data sharding processing method is applied in a cluster, which includes multiple nodes. Each node is responsible for processing one or more data shards, thus multiple nodes are responsible for processing multiple data shards. The data shards that a node is responsible for refer to data shards that have a corresponding relationship with the node; that is, the ownership of the data shard belongs to that node. The following explanation uses the example of a target node acquiring ownership of a data shard among multiple nodes in the cluster.

[0067] See Figure 4 This figure is a schematic flowchart of the data fragmentation processing method provided in an embodiment of this application. For ease of description, the following embodiments will still use a server as the execution subject of this data fragmentation processing method. Figure 4 As shown, the data fragmentation processing method includes S401-S403.

[0068] S401: Retrieve ownership information for multiple data fragments from a preset storage location.

[0069] The preset storage location is a pre-set location used to store the ownership information corresponding to each data shard. This application embodiment does not specifically limit the preset storage location, such as the location of the database, the location of the node, etc.

[0070] Ownership information for data shards indicates which node in the cluster owns a data shard; that is, ownership information determines which node owns which data shard. Ownership of a data shard identifies which node has the authority to process the data shard, i.e., which node is responsible for processing which data shard.

[0071] Data sharding (segment, fragment, shard, partition) involves dividing a dataset into independent, orthogonal subsets according to certain rules, and then distributing these subsets across different nodes. In this embodiment, data sharding typically has high real-time requirements, such as leaderboards in games. Therefore, it's necessary to ensure that multiple data shards managed by the cluster can be handled by nodes within the cluster (for ease of explanation, data shards without assigned nodes are referred to as isolated data shards). This allows for direct processing of a data shard after receiving a processing operation request, based on the responsible node, rather than first assigning a node to the isolated data shard and then processing it based on that node. This reduces waiting time and improves the processing efficiency of data sharding.

[0072] To ensure that all data shards under the cluster's responsibility can be managed by nodes within the cluster (i.e., the number of isolated data shards in the cluster is small or even nonexistent), in this embodiment, each node obtains ownership information corresponding to the multiple data shards under the cluster's responsibility from a preset storage location. This allows the node to determine which data shard does not have a corresponding node based on the multiple ownership information, thereby acquiring ownership of that data shard. The specific details are as follows.

[0073] S402: If it is determined from the ownership information of the first data shard that none of the nodes in the cluster have ownership of the first data shard, then acquire ownership of the first data shard.

[0074] After obtaining the ownership information corresponding to multiple data shards at the target node, it determines whether the corresponding data shard is an isolated data shard based on each ownership information. That is, it determines whether the multiple data shards under the responsibility of the cluster are all under the responsibility of a corresponding node through multiple ownership information.

[0075] The following explanation uses one of the data shards in a set of multiple data shards, specifically the first data shard, as an example of an isolated data shard.

[0076] If the target node determines, based on the ownership information of the first data shard, that none of the nodes in the cluster own the first data shard, i.e., the first data shard is an isolated data shard, then the target node acquires ownership of the first data shard so that the first data shard is no longer an isolated data shard, but rather the first data shard is taken over by the target node.

[0077] This application does not specifically limit the method by which the target node determines whether the first data shard is an isolated data shard. For example, the ownership information of the first data shard may indicate that the node owning the first data shard is empty. Or, the ownership information of the first data shard may indicate that the node owning the first data shard has failed. These scenarios will be described in detail later, and will not be repeated here.

[0078] S403: If ownership of the first data fragment is successfully acquired, then load the first data fragment.

[0079] If ownership of the first data shard is successfully acquired, it means that the target node can begin to take responsibility for the first data shard. At this time, the target node loads the first data shard and stores it in the target node so that after obtaining the processing operation for the first data shard, the target node can perform the corresponding processing operation on the first data shard.

[0080] At this point, the first data shard is no longer an isolated data shard, but is handled by the target node. The node indicated by the ownership information of the first data shard can be updated to the target node so that other nodes can determine that the first data shard is handled by the target node based on the ownership information of the first data shard.

[0081] This application does not specifically limit the executing entity for updating the ownership information of the first data fragment. Three methods are described below as examples.

[0082] In Method 1, after loading the first data shard, the target node generates updated ownership information for the first data shard, and this updated ownership information indicates that the node with ownership of the first data shard is the target node. Finally, the target node sends the generated updated ownership information for the first data shard to a preset storage location for storage.

[0083] Method 2: The target node can send an update request to a preset storage location. This update request is used to identify that the ownership of the first data fragment has been updated to the target node. The preset storage location modifies the ownership information of the first data fragment according to the update request, that is, the target node has the ownership of the first data fragment and obtains the updated ownership information of the first data fragment.

[0084] Method 3: The preset storage location and data shards interact through an Application Programming Interface (API). This API can be a native interface based on a programming language or a general standard interface. Providing an API for the preset storage location enables nodes in the cluster to communicate and exchange data with the preset storage location, thereby increasing the flexibility and scalability of the preset storage location.

[0085] Based on this, the target node obtains the ownership information corresponding to each of the multiple data shards from the preset storage location through the data storage interface. After determining that none of the nodes in the cluster have ownership of the first data shard based on the ownership information of the first data shard, the target node obtains ownership of the first data shard through the data storage interface, updates the node indicated by the ownership information of the first data shard to the target node through the data storage interface, and obtains the updated ownership information of the first data shard. If the updated ownership information of the first data shard is successfully stored in the preset storage location through the data storage interface, the target node successfully obtains ownership of the first data shard.

[0086] Therefore, enabling interaction between nodes and preset storage locations through a data storage interface not only simplifies access to these locations but also allows for the management of data shard ownership information, reducing node load. Furthermore, if the preset storage location is a distributed database or other non-centrally managed storage space, managing ownership information becomes even more convenient and efficient.

[0087] Before the target node successfully acquires ownership of the first data shard, all other nodes in the cluster (i.e., nodes other than the target node) can acquire ownership of the first data shard. Other nodes can acquire ownership of the first data shard simultaneously with the target node, or they can acquire ownership sequentially. If other nodes acquire ownership of the first data shard before the target node, the target node may fail to acquire ownership, thus preventing it from loading the first data shard.

[0088] Furthermore, to prevent multiple nodes from simultaneously acquiring ownership of the first data shard, an original version identifier can be added to the ownership information. This original version identifier identifies the version of the ownership information before the update, allowing the system to determine whether other nodes modified the ownership information of the first data shard before the target node. This will be explained in detail below.

[0089] Not only does the ownership information of the first data shard update the node indicated by the ownership information to the target node through the data storage interface, but it also increments the original version identifier of the ownership information of the first data shard by a preset value, such as adding 1 to the original version identifier. That is, every time the ownership information is modified, the original version identifier is incremented by 1, resulting in the updated ownership information of the first data shard. Thus, in the updated ownership information of the first data shard, the first data shard is under the responsibility of the target node, and the original version identifier of the first data shard is incremented.

[0090] Before storing the updated ownership information of the first data shard, it is necessary to retrieve the ownership information of the first data shard from the preset storage location again through the data storage interface. If the version identifier included in the ownership information is the same as the original version identifier, it means that no other node has modified the ownership information of the first data shard before the target node. Then, the updated ownership information of the first data shard is stored in the preset storage location through the data storage interface, so that the target node can successfully acquire the ownership of the first data shard.

[0091] If the version identifier included in the ownership information is different from the original version identifier, it means that other nodes have successfully acquired ownership of the first data shard before the target node, and the target node cannot modify the ownership information of the first data shard.

[0092] It should be noted that the ownership information of the data fragments obtained by the node from the preset storage location is generally the ownership information corresponding to the latest modification made to that data fragment, so that the ownership information of the data fragments obtained by the node from the preset storage location can accurately describe the current state of the data fragments.

[0093] Therefore, ownership information not only indicates which node owns the data shard, but also uses a version identifier to indicate the number of times the ownership information has been modified. When a node wants to modify the ownership information of a data shard, when generating the updated ownership information, it not only modifies the ownership of the node, but also increments the version identifier by a preset value. Before storing the updated ownership information, it retrieves the ownership information of the data shard again. If the version identifier in the retrieved ownership information matches the version identifier in the ownership information retrieved before the modification (i.e., the original version identifier), it means that no other node has modified the ownership information of the data shard, and the node can store the updated ownership information in the preset storage location. Thus, the version identifier not only ensures that the same data shard is not modified by multiple nodes simultaneously, but also ensures that the preset storage location accurately stores the ownership information of the data shard.

[0094] As can be seen from the above technical solution, establishing ownership information for data shards clarifies the correspondence between nodes and data shards. For example, the ownership information of a data shard determines which node in the cluster owns that data shard, thus enabling processing of that data shard. Furthermore, storing the ownership information of each data shard uniformly in a preset storage location provides higher accuracy compared to distributed storage across nodes, where each node only stores its own data shard (i.e., local information). This unified storage method, storing ownership information uniformly in a preset storage location, provides global information. Taking a target node in the cluster as an example, the target node retrieves the ownership information corresponding to multiple data shards from the preset storage location. If, based on the ownership information of the first data shard, it is determined that no node in the cluster owns the first data shard, then the target node acquires ownership of the first data shard. If the acquisition of ownership of the first data shard is successful, the target node loads the first data shard for timely subsequent processing. Furthermore, in the preset storage location, the node indicated by the ownership information of the first data shard will be updated to the target node, that is, the ownership information is dynamically updated so that the node can obtain the correct ownership information from the preset storage location in a timely manner.

[0095] Therefore, by updating the ownership information of data shards in a timely manner, data shards without corresponding nodes can be loaded by nodes in the cluster in a timely manner, so that the data shards can be processed promptly. Moreover, the ownership information of each data shard is uniformly stored in a preset storage location, so that multiple nodes in the cluster do not need to communicate frequently with each other, but can directly retrieve it from the preset storage location. This is convenient and fast, reduces network overhead, shortens communication time, and improves the processing efficiency of data shards.

[0096] As mentioned above, nodes in the cluster can not only acquire ownership of data shards, but also actively release ownership of some data shards when they feel that their load is too high. The released data shards become orphaned data shards, and the ownership of orphaned data shards is empty, meaning that no node is responsible for the released data shards.

[0097] Therefore, to reduce the number of isolated data shards among the multiple data shards handled by the cluster, each node can be instructed to check at fixed intervals to ensure timely loading of isolated data shards and maintain data shard processing efficiency. Continuing with the target node as an example, regarding the aforementioned S401, whereby the target node retrieves ownership information corresponding to multiple data shards from a preset storage location, this can be refined to A1, meaning the target node can retrieve ownership information corresponding to multiple data shards from a preset storage location every second preset time interval.

[0098] It should be noted that the shorter the second preset duration, the lower the probability of isolated data shards among the multiple data shards handled by the cluster, but the more load pressure is added to the nodes. Based on this, those skilled in the art can set the second preset duration according to the load of the nodes, and this application embodiment does not make specific limitations in this regard.

[0099] Therefore, nodes in the cluster retrieve ownership information corresponding to multiple data shards from a preset storage location every second preset time interval. This allows them to identify isolated data shards based on the ownership information and acquire ownership of the isolated data shards, thereby reducing the number of isolated data shards and improving the processing efficiency of data shards.

[0100] As one possible implementation, nodes in the cluster can use the same preset time interval (e.g., all using a second preset time interval) to retrieve ownership information of multiple data shards from a preset storage location. This is equivalent to nodes in the cluster checking for the existence of isolated data shards every second preset time interval. It should be noted that although nodes in the cluster use the same preset time interval, the triggering time can be the same or different; this application does not impose specific limitations on this.

[0101] Multiple nodes in the cluster can also retrieve ownership information of multiple data shards from preset storage locations at different preset intervals. That is, the intervals at which nodes in the cluster detect the existence of isolated data shards differ. For example, the target node retrieves ownership information of multiple data shards from the preset storage location every second preset interval, while other nodes retrieve ownership information of multiple data shards from the preset storage location every third preset interval, which is different from the second preset interval. This refines the way nodes check for the existence of isolated data shards in the cluster by using different preset intervals, making the check more applicable to each node in the cluster, such as determining the preset interval for each node based on its load capacity.

[0102] In addition, the preset duration for detecting the existence of isolated data shards can be dynamic. For example, when the current load of a node is low, the detection of whether there are isolated data shards can be actively triggered, which is equivalent to dynamically triggering the detection based on the node's load, making the second preset duration dynamic. This will be explained later based on C1-C3 or D1-D4, and will not be elaborated here.

[0103] As one possible implementation, although it is possible to check whether a data shard is an isolated data shard by performing a check at fixed intervals, sometimes a node may suddenly crash and fail to release the data shards it is responsible for. As a result, although it is determined through ownership information that these data shards are the responsibility of a certain node, the data shards cannot actually be processed quickly by that node because that node has crashed.

[0104] Based on this, the last lease extension time can be added to the ownership information. This timeframe identifies the last time a node acquired ownership of a data shard. This causes nodes to update the last lease extension time every preset period. If a node fails to update the last lease extension time for a data shard for an extended period, the node is effectively released, and other nodes can acquire ownership of that data shard. The following explanation uses the target node as an example; see B1-B3.

[0105] B1: After the first data slice has been loaded for a first preset duration, acquire ownership of the first data slice.

[0106] After the target node loads the first data fragment, the ownership information of the first data fragment in the preset storage location not only indicates that the target node owns the first data fragment, but also indicates that the last lease time is the first time. The first time is used to describe the time when the ownership information of the first data fragment is modified, such as the time when the target node loads the first data fragment, the time when the ownership information of the first data fragment is modified in the preset storage location, etc.

[0107] After the target node loads the first data shard, the target node will acquire ownership of the first data shard every first preset time interval, which is equivalent to the target node renewing its ownership of the first data shard every first preset time interval. The first preset time interval is the renewal time interval corresponding to the target node. This application embodiment does not specifically limit the size of the first preset time interval. Those skilled in the art can set it according to actual needs, such as determining the size of the first preset time interval by referring to the load limit of the target node.

[0108] As one possible implementation, the decision to renew the lease of the first data shard can be determined based on the load of the target node. Specifically, the load of the target node is obtained. If the load of the target node is less than the upper limit, it means that the load pressure on the target node has not exceeded the unbearable range, and it is capable of renewing the lease of the data shards it is responsible for. Taking the renewal of the first data shard among multiple data shards it is responsible for as an example, the target node acquires ownership of the first data shard. This application does not specifically limit the upper limit of the load; those skilled in the art can set it according to actual needs.

[0109] It should be noted that if the target node is responsible for multiple data shards, the data shards that can be renewed and those that will not be renewed can be determined from the multiple data shards it is responsible for, based on factors such as the target node's load, the load corresponding to each data shard it is responsible for, and the historical load trend of each data shard it is responsible for. This will be explained later based on C21-C23 or D4, and will not be repeated here.

[0110] Therefore, the decision to continue renewing the lease of the data shards under the target node's responsibility is determined by the target node's load. Specifically, when the target node's load is less than its load limit, ownership of the first data shard is acquired so that the lease of the first data shard can be renewed. This allows the renewal operation of the first data shard to be performed without affecting the target node's load pressure. This prevents other nodes from acquiring ownership of the data shards under the target node's responsibility in a timely manner in the event of a failure or other anomaly, thereby improving the processing efficiency of the data shards.

[0111] B2: If ownership of the first data fragment is successfully acquired, then continue loading the first data fragment.

[0112] If the acquisition of ownership of the first data shard is successful, meaning the target node successfully renews its lease of ownership of the first data shard, the target node will continue to load the first data shard. Furthermore, in the preset storage location, the ownership information of the first data shard, including the time, will be updated to a second time, which is later than the first time. Ignoring the time of various execution operations, the first time plus the first preset duration equals the second time.

[0113] B3: If acquiring ownership of the first data fragment fails, then release the first data fragment.

[0114] If acquiring ownership of the first data shard fails, meaning the target node fails to renew its lease of the first data shard, the target node releases the first data shard, meaning it is no longer responsible for it. Furthermore, in a preset storage location, the ownership information of the first data shard can indicate that the node is no longer the target node, or that it will no longer update the latest lease renewal time (e.g., keeping the latest lease renewal time as the first time), to help other nodes acquire ownership of the first data shard based on this ownership information.

[0115] Therefore, by adding the latest lease renewal time to the ownership information and renewing the ownership of data shards at preset intervals, the normality of nodes can be verified. Specifically, if the ownership of the first data shard is successfully acquired, its corresponding latest lease renewal time is updated, which is equivalent to the target node not crashing and can continue to be responsible for the first data shard; if the ownership of the first data shard fails to be acquired, its corresponding latest lease renewal time is not updated, which is equivalent to the target node crashing and cannot continue to be responsible for the first data shard. Thus, other nodes can determine whether there is actually a corresponding node for the first data shard based on the ownership information of the first data shard, preventing the problem of data shards not being processed quickly after a node in the cluster malfunctions, and improving the processing efficiency of data shards.

[0116] As mentioned above, although ownership information can determine which node is responsible for which data shard, if a node crashes, it may not be able to release the data shard it is responsible for in time, or it may be able to release the data shard it is responsible for in time. The corresponding ownership information will also be different. The following is a detailed explanation of S402 based on the two situations, that is, if it is determined from the ownership information of the first data shard that none of the nodes in the cluster have ownership of the first data shard, then the ownership of the first data shard is obtained.

[0117] Scenario 1: If the node indicated by the ownership information of the first data shard is empty, then it is determined that none of the nodes in the cluster have ownership of the first data shard.

[0118] If the node indicated by the ownership information of the first data shard is empty, it means that the ownership of the first data shard has not been acquired by other nodes. This could be due to other nodes actively releasing their ownership of the first data shard, or the first data shard being a newly added data shard. In this case, it is determined that none of the nodes in the cluster have ownership of the first data shard, and the target node can acquire ownership of the first data shard. Here, "other nodes" refers to any node other than the target node among the multiple nodes in the cluster.

[0119] Scenario 2: If the ownership information of the first data shard indicates another node, then the ownership of the first data shard is acquired by the other node.

[0120] If other nodes are functioning normally, the first data shard can be processed normally by other nodes. Other nodes will continuously renew the lease of the first data shard, so that the lease renewal time is constantly updated, and the target node cannot acquire ownership of the first data shard.

[0121] If other nodes malfunction, the ownership of the first data shard belongs to those other nodes in the ownership information. Because of the malfunction, these other nodes cannot process the first data shard in a timely manner, resulting in a single point of failure causing the data shard to be unable to be processed quickly. Since the malfunction of other nodes prevents them from renewing the lease for the data shards they are responsible for at preset intervals, the lease renewal time in the ownership information of the first data shard is no longer updated after the malfunction of other nodes, causing the ownership of the first data shard by the other nodes to expire.

[0122] Based on this, the target node can obtain a third time to identify the current time, which refers to the time when the target node wants to acquire the first data shard. The ownership information of the first data shard includes the time difference between the last lease renewal time and the third time. If the ownership information of the first data shard indicates that the node is another node, and the time difference is greater than the lease renewal interval corresponding to other nodes, it indicates that other nodes may be experiencing anomalies, resulting in the last lease renewal time not being updated in time. Therefore, it is determined that none of the nodes in the cluster owns the first data shard, and the target node can acquire ownership of the first data shard. In this case, the node indicated by the ownership information of the first data shard is updated from another node to the target node.

[0123] The lease renewal time interval describes the time interval between two consecutive lease renewals of a data shard by a node. For example, the lease renewal time interval for the target node on the first data shard can be a first preset duration, which is not specifically limited in this application. In addition, the lease renewal time interval can be determined not only based on the node, but also based on the data shard, such as using different lease renewal time intervals for different data shards.

[0124] Therefore, by adding the latest lease renewal time to the ownership information, it is possible to accurately determine whether a node is normally responsible for the data shard. That is, a node whose latest lease renewal time is continuously updated is normally responsible for the corresponding data shard, while a node whose latest lease renewal time has stopped updating is abnormal. This is equivalent to the corresponding data shard no longer being leased by that node, and the ownership of the data shard is released. Other nodes can then acquire the ownership of the data shard, thereby avoiding the problem of a node being unable to release the ownership of the data shard in a timely manner due to a single point of failure, and improving the processing efficiency of data shards.

[0125] As one possible implementation, the load on the nodes in the cluster may change. In order to ensure load balance in the cluster, the data shards responsible for each node can be adjusted according to the load of each node in the cluster. The following will continue to illustrate with the target node as an example. When the load is too high, data shards need to be released (see C1-C3), and when the load is too low, data shards can be added (see D1-D4).

[0126] C1: Get the load of the target node.

[0127] The load of a target node refers to the number of tasks running on that node and the resource utilization rate. The load level can affect the performance and efficiency of the entire cluster. Therefore, maintaining a balanced node load is crucial.

[0128] C2: If the load of the target node is greater than the load limit, then determine the second data shard from the data shards owned by the target node.

[0129] If the load on the target node exceeds the load limit, it means that the target node is about to exceed its capacity. At this time, it is necessary to reduce the load on the target node. For example, by reducing the number of data shards that the target node is responsible for, the second data shard can be determined from the data shards that the target node owns, thereby releasing the ownership of the second data shard. That is, the target node is no longer responsible for the second data shard, so as to reduce the load on the target node.

[0130] The load limit is the upper limit of the target node's load, which can be set by those skilled in the art according to actual needs; this application embodiment does not specifically limit this. The second data shard is a data shard owned by the target node. It should be noted that if the target node's load is still greater than the load limit after releasing a second data shard, a second data shard can be determined from the data shards owned by the target node, and ownership of the second data shard can be released until the target node's load is less than or equal to the load limit.

[0131] The embodiments of this application do not specifically limit the method of determining the second data fragment. The following description uses one method as an example, see C21-C23.

[0132] C21: Obtain multiple permission data shards.

[0133] Permission-based data sharding refers to data shards that the target node owns.

[0134] C22: Obtain the influencing factors corresponding to each permission data shard.

[0135] Influencing factors include the load of permission data shards and the historical load trend of permission data shards. The load of permission data shards refers to a measure of resource utilization consumed by nodes running permission data shards. The historical load trend of permission data shards refers to the change in resource utilization consumed by nodes running permission data shards over a historical period, which refers to a period preceding the current time.

[0136] This application does not specifically limit the load distribution method of the computational permission data shards. Taking a game leaderboard as an example, the load distribution can be determined based on the number of accounts that affect the game leaderboard. For example, the more accounts that affect the game leaderboard, the more resources are consumed to manage the game leaderboard.

[0137] Furthermore, the load of the target node can be determined based on the number of game leaderboards it manages and the number of accounts on each game leaderboard. Specifically, the load corresponding to each data shard owned by the target node is obtained. The load of each data shard is determined based on the number of accounts participating in the calculation of that data shard. The load of the target node is then obtained by summing the loads of all data shards owned by the target node.

[0138] See Figure 5 The figure is a schematic diagram of a game leaderboard provided in an embodiment of this application. Changes in the numerical value corresponding to each account in the leaderboard will cause changes in the leaderboard itself; for example, playing another game might move a player's ranking up by one place. Therefore, the computational complexity of the leaderboard will vary depending on the number of accounts participating in its calculation.

[0139] See Figure 6 This figure is a schematic diagram of another game leaderboard provided in an embodiment of this application. Compared to Figure 5 The leaderboard shown is calculated based on the number of accounts that have reached a certain level.

[0140] Therefore, if the data shards are used to create leaderboards, such as game leaderboards, each account participating in the leaderboard calculation will affect the leaderboard's update. This means each account participating in the leaderboard calculation will affect the leaderboard's load. Thus, the leaderboard load can be determined based on the number of accounts participating in the leaderboard calculation. This, combined with the number of leaderboards a node is responsible for, determines the node's load. This is equivalent to determining the node load based on business computational load. Compared to determining the node load based on its hardware parameters, this business computational load method is more accurate. Therefore, a more accurate load determination increases the probability of load balancing across nodes in the cluster, thereby improving the processing efficiency of data shards.

[0141] C23: Determine the second data fragment from the permission data fragments based on influencing factors.

[0142] Let's take one of the multiple permission data shards, namely the target permission data shard, as an example. The higher the load on the target permission data shard, the more resources the target node consumes to handle that target permission data shard, and the higher the probability that the target permission data shard will be identified as the second data shard.

[0143] If the historical load trend of the target permission data shard indicates that the future load of the target permission data shard is higher than the load of other permission data shards, where the future load is predicted based on the historical load trend and is used to indicate that the load of the data shard is about to increase and that the increased load is higher than the load of other permission data shards, it means that the resources consumed by the target node to manage the target permission data shard are gradually increasing and will exceed the load of other permission data shards managed by the target node at some point in the future. In this case, the target permission data shard is more likely to be identified as the second data shard. Here, other permission data shards are the permission data shards among multiple permission data shards excluding the target data shard.

[0144] It should be noted that the difference between the future moment or the time that is about to be reached as indicated in this embodiment and the current time is less than the time threshold. In other words, even the future moment will not differ too much from the current time (i.e., the time when the target node load is determined to be greater than the load limit). It is only to reserve a little advance for releasing the data shards with permissions in order to reduce the additional load brought to the target node by the removal operation.

[0145] Therefore, if the target node is responsible for multiple data shards, and the load of the target node exceeds the load limit, it indicates that the target node is under high load. Continuing to be responsible for the multiple data shards it is currently responsible for may cause the target node to malfunction. In this case, based on the load of the permission data shards and their historical load trends, permission data shards with high load or those whose load is about to become high can be identified from the multiple data shards and released. That is, data shards whose resource consumption is about to become high are released first, thereby reducing the number of data shards released and reducing resource consumption.

[0146] C3: Release ownership of the second data fragment.

[0147] Therefore, after determining the second data shard, the target node releases ownership of the second data shard. Consequently, in the preset storage location, the node indicating the ownership information of the second data shard is updated from the target node to empty, meaning that no node in the cluster is responsible for the second data shard.

[0148] For example, if the execution entity for updating ownership information is the target node, the target node updates the node indicated by the ownership information of the second data shard to empty, obtains the updated ownership information of the second data shard, and sends the updated ownership information of the second data shard to a preset storage location so that the ownership information of the second data shard stored in the preset storage location is the latest, that is, the node indicated by the ownership information of the second data shard is updated to empty from the target node.

[0149] Therefore, if the load of a node in the cluster exceeds the load limit, it indicates that the node is under heavy load. One or more data shards can be released from the multiple data shards it is responsible for to reduce the load on the node. This not only ensures the normal operation of the node but also ensures the load balance of each node in the cluster, thereby ensuring the performance and efficiency of the entire cluster.

[0150] After introducing how a node can actively release one or more data shards it is responsible for when its load is high, the following describes how a node can actively acquire one or more isolated data shards when its load is low. In this embodiment, we take the active acquisition of the first data shard when the target node's load is low as an example, and refer to D1-D4 for details.

[0151] D1: Get the load of the target node.

[0152] D2: If the load of the target node is less than the lower load limit, then obtain the ownership information corresponding to multiple data shards from the preset storage location.

[0153] If the load of the target node is less than the lower limit of the load, it means that the target node can still handle more data shards. At this time, the load of the target node can be increased. For example, if the number of data shards handled by the target node is increased, the ownership information corresponding to multiple data shards can be obtained from the preset storage location.

[0154] The lower load limit is the lower limit of the load of the target node. The lower load limit is less than the upper load limit. Those skilled in the art can set it according to actual needs. This application does not make specific limitations on this.

[0155] D3: Determine multiple isolated data fragments based on multiple ownership information.

[0156] Ownership information can indicate the node responsible for the corresponding data shard. Therefore, based on the ownership information corresponding to multiple data shards, it is possible to determine which of the multiple data shards the cluster is responsible for is an isolated data shard, that is, a shard that none of the nodes in the cluster owns.

[0157] D4: Determine the first data shard from multiple isolated data shards based on the load of the target node and the load of each isolated data shard.

[0158] Based on the load of the target node and the data of each isolated data shard, the first data shard is determined from multiple isolated data shards. The sum of the load of the first data shard and the load of the target node is less than the load limit. In other words, the first data shard to be acquired will not cause the load of the target node to exceed the load limit, so as to ensure that the target node will not malfunction.

[0159] It should be noted that if the target node's load is still less than the lower load limit after adding a first data shard, the target node can continue to select a first data shard from multiple isolated data shards and continue to load a new first data shard until the target node's load exceeds the upper load limit when loading a new first data shard.

[0160] Therefore, if the load of a node in the cluster is less than the lower load limit, it means that the load pressure on that node is relatively low. One or more isolated data shards can continue to be loaded. This reduces the number of isolated data shards in the cluster without exceeding the load capacity of that node. This not only ensures the normal operation of that node, but also ensures the load balance of each node in the cluster, thereby ensuring the performance and efficiency of the entire cluster and improving the processing efficiency of data shards.

[0161] As one possible implementation, this application embodiment also provides a specific implementation of D4, namely, obtaining the historical load trend of each isolated data shard, and determining the first data shard from multiple isolated data shards based on the load of the target node, the load of each isolated data shard, and the historical load trend of each isolated data shard.

[0162] In other words, when determining the first data shard, not only the load of the target node and the load of each isolated data shard are considered, but also the historical load trend of the isolated data shards. Therefore, the sum of the future load of the first data shard determined based on historical load and the load of the target node is less than the upper load limit. Here, future load refers to the load of the data shard at a certain future point in time, predicted based on historical load trends.

[0163] Therefore, based on the load of the target node, the load of each isolated data shard, and the historical load trend of each isolated data shard, the first data shard suitable for loading by the target node is determined from multiple isolated data shards. This not only considers the current load of the isolated data shard, but also takes into account the possibility of changes in the isolated data shard. This avoids the situation where, after loading a data shard, the load of the target node exceeds the load limit at some point in the future, requiring the target node to perform a release operation. This reduces the operation frequency of the target node while ensuring load balance among the nodes in the cluster.

[0164] After loading a data shard, a node can also have an access identifier, which identifies the data shard that the target node owns. This allows for the rapid execution of data shard update requests based on the access identifier. See E1-E5 for details.

[0165] E1: Get the data shard update request including the identifier to be updated.

[0166] The target node obtains a data shard update request. The data shard update request includes an identifier to be updated. The identifier to be updated is the identifier of the data shard waiting to be updated. That is, the identifier to be updated is used to indicate the data shard waiting to be updated.

[0167] E2: If the target node includes the same permission identifier as the identifier to be updated, then the data shard corresponding to the identifier to be updated will be updated.

[0168] If the target node includes the same permission identifier as the identifier to be updated, it means that the target node owns the data shard indicated by the identifier to be updated, and thus the target node can directly update the data shard corresponding to the identifier to be updated.

[0169] E3: If the target node does not include the same permission identifier as the identifier to be updated, then obtain the ownership information corresponding to multiple data shards from the preset storage location.

[0170] If the target node does not contain a permission identifier identical to the identifier to be updated, it means that the target node does not own the data shard indicated by the identifier to be updated. In this case, the target node cannot process the data shard corresponding to the identifier to be updated and needs to send the data shard update request to the node that owns the data shard indicated by the identifier to be updated for processing. At this time, the target node obtains the ownership information corresponding to multiple data shards from a preset storage location.

[0171] E4: Determine the permission node based on multiple ownership information and the identifier to be updated.

[0172] The ownership of the data shard indicated by the identifier to be updated is determined based on multiple ownership information. This ownership is determined by the node holding the ownership, i.e., the node with the authority. The node with the authority is the one that owns the data shard corresponding to the identifier to be updated; that is, the node that can process the data shard corresponding to the identifier to be updated.

[0173] As one possible implementation, ownership information can also include identifiers of data shards, thereby quickly determining the permission node by matching the identifier to be updated with the identifiers of each data shard.

[0174] E5: Forward the data shard update request to the authorized node.

[0175] After determining that the ownership of the data fragment indicated by the identifier to be updated is with the authorizing node, the target node forwards the data fragment update request to the authorizing node so that the authorizing node can update the data fragment indicated by the identifier to be updated.

[0176] Therefore, after the target node receives a data shard update request, if it is determined from the update identifier that the target node owns the data shard corresponding to the update request, the target node can directly process that data shard. If it is determined from the update identifier that the target node does not own the data shard, the target node obtains the ownership information of each data shard from a preset storage location. Based on multiple ownership information, it determines the authorized node where the data shard indicated by the update identifier resides, and then forwards the data shard update request to the authorized node for processing. This not only allows for quick determination of whether the target node can directly respond to the data shard update request through the update identifier, but also allows for determination of which node the data shard update request should be forwarded to from multiple ownership information stored in the preset storage location, thereby improving the processing efficiency of data shards.

[0177] As one possible implementation, data shards are related; for example, an update to the third data shard leads to an update to the fourth data shard, even though the third and fourth data shards are different data shards. Let's continue with the example of a leaderboard using data shards.

[0178] See Figure 7 This figure is a schematic diagram of a position-adjusted leaderboard provided in an embodiment of this application. Figure 7 In the game, players can use a location filter to switch between the leaderboards they want to view, such as the leaderboard for province A, the leaderboard for city B within province A, the leaderboard for district C within city B, and so on. In other words, the leaderboards are hierarchically related based on geographical location, and this hierarchy leads to cascading updates between them. For example, an update to the leaderboard in district C will trigger an update to the leaderboard in city B, and may even cause an update to the leaderboard in province A.

[0179] For ease of explanation, if an update to the third data shard leads to an update to the fourth data shard, then the third and fourth data shards are cascaded, with the fourth data shard being the parent data shard and the third data shard being the child data shard. In other words, an update to the child data shard will lead to an update to the parent data shard.

[0180] If data shards with a cascading relationship are assigned to different nodes in the cluster, after a lower-level data shard is updated, the node responsible for the lower-level data shard will send an update request to the node responsible for the upper-level data shard, thus ensuring the accuracy of the data sharding. However, this approach leads to a large amount of communication between nodes, resulting in high communication overhead, and because communication also takes time, the processing efficiency of data sharding is low.

[0181] Based on this, the embodiments of this application assign ownership of data fragments with cascading relationships to the same node. That is, the node that owns the third data fragment and the node that owns the fourth data fragment are the same node. For example, the target node owns the ranking list of province A, the ranking list of city B in province A, and the ranking list of district C in city B. Therefore, after the third data fragment is updated, no communication is required to continue updating the fourth data fragment, which not only reduces communication overhead but also shortens communication time and improves the processing efficiency of data fragments.

[0182] It should be noted that when a node acquires orphaned data shards and releases data shards with permissions, it can also consider the cascading relationships between data shards. For example, a target node may prioritize acquiring orphaned data shards that are cascading with the data shards it owns. Similarly, when releasing data shards with permissions, a target node may prioritize releasing data shards that are not cascading with the updated data shards, as detailed in F1-F2.

[0183] If the third and fourth data shards are cascaded, and the fifth and fourth data shards are also cascaded, but the third and fifth data shards are not cascaded with each other. For example, if the fourth data shard is the ranking list for City B, the third data shard is the ranking list for District C of City B, and the fifth data shard is the ranking list for District D of City B, then District C and District D are at the same level. Updates to either of them will not trigger updates to the other, but updates to their parent data shards will both trigger updates to their parent data shards.

[0184] F1: Get the load of the target node.

[0185] F2: If the load on the third data shard increases to the point that the load on the target node exceeds the load limit, then release ownership of the fifth data shard.

[0186] If the increased load on the third data shard causes the load on the target node to exceed its load limit, such as when there is a sudden surge in traffic to the leaderboard in Zone C (e.g., when the leaderboard is updated), the load on the leaderboard in Zone C will increase, which in turn will increase the load on the target node responsible for the leaderboard in Zone C. If the load on the target node exceeds its load limit, it may cause the target node to malfunction. In this case, the target node needs to release some of the data shards it is responsible for. This can be done not only based on the load and historical load trend of the data shards it is responsible for, but also based on the cascading relationship between data shards. For example, it can prioritize releasing unrelated data shards. Unrelated data shards refer to data shards that have no cascading relationship with the data shard that caused the load increase, i.e., the fifth data shard, such as the leaderboard in Zone D.

[0187] After the target node releases ownership of the fifth data shard, it needs to update the ownership information of the fifth data shard. That is, in the preset storage location, the node indicated by the ownership information of the fifth data shard is updated to be empty, thereby ensuring the correctness of the ownership information stored in the preset storage location.

[0188] Therefore, if a node is responsible for multiple data shards, some of which are cascaded and others are not, and the load on one of the data shards increases, causing the node's load to increase and exceeding its load limit, then to prevent the node from malfunctioning, unrelated data shards can be released first. This ensures the normal operation of the node while reducing communication overhead between nodes and improving the processing efficiency of data shards.

[0189] To facilitate a further understanding of the technical solutions provided in the embodiments of this application, the following description takes the execution subject of the data sharding processing method provided in the embodiments of this application as a server, and the data sharding as a game leaderboard as an example, and provides an overall exemplary introduction to the data sharding processing method.

[0190] The core data structure will be explained below.

[0191] This embodiment provides four core data structures: ownership information, nodes, data shards, and data storage interfaces. The following section will combine... Figure 8 Please provide an explanation.

[0192] See Figure 8 This figure is a schematic diagram of the relationship of a data structure provided in an embodiment of this application.

[0193] (1) Node (LeaderboardNode).

[0194] A node refers to a node in the cluster that is responsible for multiple game leaderboards. It can handle data shard update requests for game leaderboards, manage game leaderboards, and participate in load balancing. Its key fields and methods are as follows, where 1) and 2) represent key fields, and 3) and 4) represent key methods.

[0195] 1) NodeID: A unique identifier for a node, which can be represented by a string.

[0196] 2) OwnedShards: All shards currently owned by the node. This can be represented by a map, indicating the mapping relationship between the game leaderboard name and the data shard. The game leaderboard name can be represented by a string, and LeaderboardShard represents the data shard.

[0197] 3) AcquireShard: Used to indicate the acquisition of ownership of a new data shard.

[0198] 4) ReleaseShard: Used to identify the release of ownership of a data shard that has been previously held by the user.

[0199] (2) Ownership Information (Leaderboard Shard Ownership)

[0200] Ownership information represents the ownership details of a game leaderboard, used to detect the status of each leaderboard shard, including which node owns it, the last lease time, load, etc. Its key fields are as follows:

[0201] 1) AdCode: Administrative region code, uniquely identifies a game leaderboard, and can be represented by a string.

[0202] 2) OwnerNode: The unique identifier of the node that currently owns this data shard, which can be represented by a string.

[0203] 3) LastUpdateTime: The last lease time, used to detect whether a node is alive. It can be expressed as a datetime.

[0204] 4) Version: Original version identifier, used to prevent concurrent conflicts, can be represented by a signed 64-bit integer data type (int64).

[0205] 5) Load: The load of data shards, used for load balancing decisions, can be represented by a signed 64-bit integer data type (int64).

[0206] (3) Data Sharding (LeaderboardShard)

[0207] A data shard represents a specific game leaderboard, used to store and manage specific game leaderboard data, handle ranking updates and queries, etc. Its key fields and key methods are as follows, where 1), 2), and 3) represent key fields, and 4) and 5) represent key methods.

[0208] 1) AdCode: Administrative region code, which corresponds to the AdCode in the ownership information and can be represented by a string.

[0209] 2) Leaderboard: The actual data structure for storing leaderboard data, which can be represented by a skip list.

[0210] 3) Load: The load of the current data shard, which can be represented by a signed 64-bit integer data type (int64).

[0211] 4) UpdateScore: Updates the player's score. The player's unique identifier (playerID) can be represented by a string, or the score can be represented by a signed 64-bit integer data type (int64).

[0212] 5) GetRanking: Retrieves the player's ranking. Players can be represented by a string (playerID) as their unique identifier, and player rankings can be represented by an integer (int).

[0213] (4) Data storage interface (Store)

[0214] The data storage interface is responsible for interacting with the underlying database (such as Redis), i.e., the preset storage location, and provides four operations for data sharding, which is key to achieving distributed coordination. Its key methods are as follows.

[0215] 1) AcquireShardOwnership: Acquire ownership of data shards. This can be represented by a string (AdCode) or a string (NodeID).

[0216] 2) RenewShardOwnership: Renews ownership of data shards. It can be represented by a string (AdCode) or a string (NodeID).

[0217] 3) ReleaseShardOwnership: Releases ownership of the data shard. The game leaderboard (AdCode) can be represented by a string, and the node (NodeID) can be represented by a string.

[0218] 4) GetAllShardOwnerships: Retrieves ownership information for all data shards. Ownership of data shards can be represented by a list (List) (LeaderboardShardOwnership).

[0219] The following is based on Figure 8 The data structure shown, combined with Figures 9-12 The data sharding operation will be explained separately.

[0220] The first step is cluster initialization.

[0221] Create LeaderboardNode instances, each representing a node. Initialize LeaderboardShardOwnership in the underlying database using Store, ensuring that each leaderboard has a corresponding LeaderboardShardOwnership.

[0222] Then, the process of obtaining data shards by the nodes is explained.

[0223] See Figure 9 This figure is a schematic diagram of a node acquiring data fragments according to an embodiment of this application. Figure 9 The following explanation will continue using the target node in the cluster as an example.

[0224] S901: Target node starts.

[0225] Once the target node is successfully created, it will begin processing the data shards.

[0226] S902: The target node obtains ownership information corresponding to multiple data shards from the preset storage location.

[0227] The ownership information of data shards is used to indicate which node in the cluster owns the data shard. In other words, the ownership information can determine which node owns which data shard.

[0228] S903: The target node determines whether there are isolated data shards based on multiple ownership information. If not, it executes S904; if so, it executes S905.

[0229] An isolated data shard is a data shard among multiple data shards that are not managed by any node in the cluster, such as the first data shard mentioned above.

[0230] S904: Wait and retry executing S903.

[0231] For example, S903 is executed again after a second preset time interval, as detailed in A1 above, and will not be repeated here.

[0232] S905: The target node acquires ownership of the isolated data shard.

[0233] If the target node acquires ownership of the first data shard, LeaderboardNode attempts to acquire ownership of the first data shard by calling AcquireShardOwnership through the Store interface.

[0234] S906: Determine whether ownership of the orphaned data fragment has been successfully acquired. If yes, execute S907; otherwise, execute S903.

[0235] S907: Load the orphaned data fragment.

[0236] If ownership of the first data shard is successfully acquired, the first data shard is loaded, such as by creating a corresponding LeaderboardShard instance and adding it to OwnedShards.

[0237] S908: Periodic check of target node.

[0238] That is, the target node renews the lease of the first data shard every first preset time interval, thereby realizing periodic checks. See B1-B3, or... Figure 10 and Figure 11 .

[0239] The following is based on Figure 10 This section explains how the target node renews the lease for data shards.

[0240] See Figure 10 The figure is a schematic diagram of a target node renewing a lease for a data shard, provided in an embodiment of this application.

[0241] S1001: The target node traverses the data shards it is responsible for.

[0242] S1002-S10 are executed for each data shard it is responsible for. The following explanation uses the first data shard as an example.

[0243] S1002: The target node calculates the load of the first data shard.

[0244] S1003: The target node generates the ownership information of the first data shard after it has been updated.

[0245] The ownership information after the first data shard is updated includes AdCode, NodeID, and Load.

[0246] S1004: The target node sends the ownership information of the first data shard updated to the underlying database through the data storage interface.

[0247] For example, obtain the load of the target node. If the load of the target node is less than the load limit, then attempt to acquire ownership of the first data shard.

[0248] S1005: The underlying database returns the lease renewal result through the data storage interface.

[0249] The lease renewal result is used to indicate whether the target node has successfully renewed the lease, that is, whether the target node has successfully acquired ownership of the first data shard.

[0250] S1006: If ownership of the first data fragment is successfully acquired, continue loading the first data fragment.

[0251] If the acquisition of ownership of the first data shard is successful, it means that the target node has successfully renewed the lease for the first data shard. At this time, in the underlying database, the last lease renewal time, which is included in the ownership information of the first data shard, is updated to the second time, i.e., the time used to identify the successful lease renewal.

[0252] S1007: If acquiring ownership of the first data fragment fails, release the first data fragment.

[0253] At this point, in the underlying database, the ownership information of the first data shard, including the last lease renewal time, remains the first time, which is used to identify the time when the target node last successfully renewed the lease.

[0254] The following is based on Figure 11 This section explains the periodic checks performed on the target node.

[0255] See Figure 11 This figure is a schematic diagram of a target node performing periodic checks according to an embodiment of this application. In this embodiment, if node 1 experiences a single point of failure, although the ownership of the data shards managed by node 1 is indicated by node 1, the latest lease time included in the ownership of the data shards managed by node 1 is not updated in a timely manner.

[0256] S1101: Node 2 obtains ownership information corresponding to multiple data shards from the underlying database through the data storage interface.

[0257] If node 2 detects that node 1 has failed using the Store's GetAllShardOwnerships method, node 2 can attempt to acquire ownership of the data shards that node 1 was responsible for. Node 2 can be the target node mentioned above, and node 1 can be any other node.

[0258] S1102: The underlying database returns ownership information corresponding to multiple data shards through the data storage interface.

[0259] S1103: Node 2 determines the ownership information of the first data shard, including the time difference between the last lease time and the current time.

[0260] S1104: If the ownership information of the first data shard indicates that the node is another node, and the time difference is greater than the renewal time interval corresponding to the other node, then node 2 determines that none of the nodes in the cluster have ownership of the first data shard.

[0261] If the ownership information of the first data shard indicates that the node is another node, and the time difference is greater than the renewal time interval of the other node, then the ownership of the first data shard by the other node has expired.

[0262] S1105: Node 2 acquires ownership of the first data shard through the data storage interface.

[0263] S1106: The data storage interface generates updated ownership information for the first data shard.

[0264] S1107: The data storage interface sends the updated ownership information for the first data shard to the underlying database.

[0265] S1108: The underlying database returns the update result to the data storage interface.

[0266] In the underlying database, the node indicated by the ownership information of the first data shard is updated to the target node.

[0267] S1109: The data storage interface successfully acquired ownership of the first data fragment by sending a message to node 2.

[0268] S1110: Node 2 loads the first data shard.

[0269] Furthermore, when a data shard update request is received, if the LeaderboardNode owns the data shard indicated by the update request, it will process that data shard itself. If the LeaderboardNode does not own the data shard, it will find the corresponding LeaderboardShard and call its UpdateScore method to update the leaderboard. It can also update the LeaderboardShard's Load value, as detailed in E1-E5, which will not be elaborated here.

[0270] The following is through Figure 12 This section explains the load balancing of nodes.

[0271] See Figure 12 This figure is a schematic diagram of a load balancing method provided in an embodiment of this application. Figure 12 In this embodiment, the target node will continue to be used as an example for explanation.

[0272] S1201: Obtain the load of the target node.

[0273] For example, LeaderboardNode periodically checks its own load and the load of all LeaderboardShards. If it finds an imbalance in load, it releases some data shards or tries to acquire new data shards and updates LeaderboardShardOwnership through the Store.

[0274] S1202: Determine whether the load of the target node is less than the lower load limit. If yes, execute S1203; otherwise, execute S1206.

[0275] S1203: Determine the first data shard from multiple isolated data shards based on the load of each isolated data shard on the target node.

[0276] For example, obtain ownership information corresponding to multiple data shards from the underlying database, determine multiple isolated data shards based on the multiple ownership information, and determine the first data shard from the multiple isolated data shards based on the load of the target node and the load of each isolated data shard. See D1-D4 for details, which will not be elaborated here.

[0277] S1204: Acquire ownership of the first data fragment.

[0278] In the underlying database, the node indicated by the ownership information of the first data shard is updated to the target node.

[0279] S1205: If ownership of the first data fragment is successfully acquired, then load the first data fragment.

[0280] S1206: Determine whether the load of the target node is greater than the load limit. If yes, execute S1207; otherwise, execute S1209.

[0281] S1207: Determine the second data shard from the data shards owned by the target node.

[0282] For details on how to determine the second data sharding method, please refer to C21-C23, which will not be repeated here.

[0283] S1208: Release ownership of the second data fragment.

[0284] In the underlying database, the node indicating the ownership information of the second data shard is updated to be empty.

[0285] S1209: Maintain the current state.

[0286] That is, the target node neither releases nor loads data shards.

[0287] In addition, the upper or lower load limit can be adjusted based on factors such as the load of the target node, so that the upper or lower load limit better meets the business needs of the target node.

[0288] Therefore, the cluster enables distributed game leaderboard management while ensuring load balancing and fault recovery. Each data structure has its specific responsibility, and they work together to support the updating and other processing of the entire large-scale regional game leaderboard. The above solution can achieve at least the following technical effects:

[0289] 1) Dynamically adapt to load changes: Nodes can dynamically adjust the game leaderboards they are responsible for based on their own load conditions.

[0290] 2) Improve resource utilization: When the load of a node is greater than the upper limit, some game leaderboards can be released; when the load of a node is less than the lower limit, new game leaderboards can be acquired, thereby reducing the number of isolated game leaderboards in the cluster and improving the processing efficiency of game leaderboards.

[0291] 3) Enhanced system availability: Decentralized design eliminates the risk of single points of failure.

[0292] 4) Reduced synchronization overhead: Nodes do not need to frequently exchange ownership information used to describe the global state; it can be quickly obtained from a preset storage location.

[0293] In response to the data fragmentation processing method described above, this application also provides a corresponding data fragmentation processing apparatus so that the above data fragmentation processing method can be applied and implemented in practice.

[0294] See Figure 13 This figure is a schematic diagram of the structure of a data fragmentation processing device provided in an embodiment of this application. Figure 13 As shown, the data sharding processing device 1300 is applied to a target node in a cluster including multiple nodes. The data sharding processing device 1300 includes: an ownership information acquisition unit 1301, an ownership acquisition unit 1302, and a loading unit 1303.

[0295] The ownership information acquisition unit 1301 is used to acquire ownership information corresponding to multiple data shards from a preset storage location. The ownership information of the data shards is used to indicate the node in the cluster that owns the data shards. The ownership of the data shards is used to identify that the node has the right to process the data shards.

[0296] The ownership acquisition unit 1302 is used to acquire ownership of the first data shard if it is determined from the ownership information of the first data shard that none of the nodes in the cluster have ownership of the first data shard. The first data shard is one of the multiple data shards.

[0297] The loading unit 1303 is used to load the first data fragment if the ownership of the first data fragment is successfully acquired, wherein the node indicated by the ownership information of the first data fragment in the preset storage location is updated to the target node.

[0298] As can be seen from the above technical solution, establishing ownership information for data shards clarifies the correspondence between nodes and data shards. For example, the ownership information of a data shard determines which node in the cluster owns that data shard, thus enabling processing of that data shard. Furthermore, compared to distributed storage across individual nodes (where each node only stores its own responsible data shards, equivalent to local information), storing the ownership information of all data shards uniformly in a preset storage location (equivalent to global information) provides higher accuracy. Taking a target node in the cluster as an example, the target node retrieves ownership information for multiple data shards from the preset storage location. If, based on the ownership information of the first data shard, it is determined that no node in the cluster owns the first data shard, the target node acquires ownership of the first data shard. If the acquisition of ownership of the first data shard is successful, the target node loads the first data shard for timely processing. Moreover, in the preset storage location, the node indicated by the ownership information of the first data shard is updated to the target node; that is, the ownership information is dynamically updated, ensuring that nodes can promptly obtain the correct ownership information from the preset storage location.

[0299] Therefore, by updating the ownership information of data shards in a timely manner, data shards without corresponding nodes can be loaded by nodes in the cluster in a timely manner, so that the data shards can be processed promptly. Moreover, the ownership information of each data shard is uniformly stored in a preset storage location, so that multiple nodes in the cluster do not need to communicate frequently with each other, but can directly retrieve it from the preset storage location. This is convenient and fast, reduces network overhead, shortens communication time, and improves the processing efficiency of data shards.

[0300] As one possible implementation, the ownership information also includes the latest lease renewal time, which is used to identify the time when the node last acquired ownership of the data shard. In the preset storage location, the latest lease renewal time in the ownership information of the first data shard is a first time. After the first data shard is loaded for a first preset duration, the device further includes a lease renewal unit for:

[0301] Acquire ownership of the first data shard, where the first preset duration is the lease renewal time interval corresponding to the target node;

[0302] If the ownership of the first data fragment is successfully acquired, the first data fragment is kept loaded. In the preset storage location, the ownership information of the first data fragment, including the latest lease time, is updated to a second time, which is later than the first time.

[0303] If the acquisition of ownership of the first data fragment fails, the first data fragment is released, and in the preset storage location, the ownership information of the first data fragment, including the latest lease time, remains the first time.

[0304] As one possible implementation, the device further includes a lease renewal unit for:

[0305] Obtain the load of the target node;

[0306] If the load of the target node is less than the load limit, then the ownership of the first data shard is acquired.

[0307] As one possible implementation, the device further includes an ownership determination unit for:

[0308] If the node indicated by the ownership information of the first data shard is empty, then it is determined that none of the nodes in the cluster have ownership of the first data shard; or,

[0309] Get the third time used to identify the current time;

[0310] The ownership information of the first data segment includes the time difference between the last lease time and the third time.

[0311] If the ownership information of the first data shard indicates that the node is another node, and the time difference is greater than the lease renewal time interval corresponding to the other node, then it is determined that none of the nodes in the cluster have ownership of the first data shard, and the other nodes are nodes other than the target node among the multiple nodes included in the cluster.

[0312] As one possible implementation, the device further includes a release unit for:

[0313] Obtain the load of the target node;

[0314] If the load of the target node is greater than the load limit, then a second data shard is determined from the data shards owned by the target node;

[0315] Release ownership of the second data fragment;

[0316] In the preset storage location, the node indicated by the ownership information of the second data fragment is updated to be empty.

[0317] As one possible implementation, if the target node owns multiple data shards, the device further includes a release unit for:

[0318] Acquire multiple permission data shards, wherein the permission data shards are data shards owned by the target node;

[0319] Obtain the influencing factors corresponding to each of the permission data shards, including the load of the permission data shard and the historical load trend of the permission data shard;

[0320] The second data fragment is determined from the permission data fragment based on the influencing factors;

[0321] Specifically, if the load of the target permission data shard is higher, the probability that the target permission data shard will be identified as the second data shard is higher; or, if the historical load trend of the target permission data shard indicates that the future load of the target permission data shard is higher than the load of other permission data shards, the probability that the target permission data shard will be identified as the second data shard is higher. The target permission data shard is one of the multiple permission data shards, and the other permission data shards are the permission data shards other than the target data shard among the multiple permission data shards. The future load is the load predicted based on the historical load trend.

[0322] As one possible implementation, the ownership information of the data shard also includes the payload of the data shard.

[0323] As one possible implementation, the ownership information acquisition unit 1301 is specifically used for:

[0324] Obtain the load of the target node;

[0325] If the load of the target node is less than the lower load limit, then the ownership information corresponding to each of the multiple data shards is obtained from the preset storage location.

[0326] The device further includes a data fragmentation determination unit, used for:

[0327] Multiple isolated data shards are determined based on multiple ownership information, wherein the isolated data shards are shards in which no node in the cluster has ownership;

[0328] Based on the load of the target node and the load of each of the isolated data shards, the first data shard is determined from the plurality of isolated data shards, wherein the sum of the load of the first data shard and the load of the target node is less than the upper limit of the load, and the upper limit of the load is greater than the lower limit of the load.

[0329] As one possible implementation, the apparatus further includes a data fragmentation determination unit, used for:

[0330] Obtain the historical load trend of each isolated data shard;

[0331] Based on the load of the target node, the load of each isolated data shard, and the historical load trend of each isolated data shard, the first data shard is determined from a plurality of isolated data shards. The sum of the future load of the first data shard determined based on the historical load of the isolated data shard and the load of the target node is less than the load upper limit value.

[0332] As one possible implementation, if the data shards are a leaderboard, the device further includes a load determination unit, used for:

[0333] Obtain the load corresponding to each data shard owned by the target node, wherein the load of the data shard is determined based on the number of accounts participating in the calculation of the data shard;

[0334] The load of the target node is obtained by summing the loads of all the data shards owned by the target node.

[0335] As one possible implementation, the target node includes at least one permission identifier, which is used to identify the data shard that the target node owns. The apparatus further includes an update unit for:

[0336] Obtain a data shard update request including an identifier to be updated, wherein the identifier to be updated is the identifier of the data shard waiting to be updated;

[0337] If the target node includes a permission identifier that is the same as the identifier to be updated, then the data shard corresponding to the identifier to be updated is updated.

[0338] If the target node does not include a permission identifier that is the same as the identifier to be updated, then the ownership information corresponding to the multiple data shards is obtained from the preset storage location.

[0339] Based on the multiple ownership information and the identifier to be updated, an authorized node is determined, wherein the authorized node is a node that owns the data shard corresponding to the identifier to be updated;

[0340] The data shard update request is forwarded to the authorized node.

[0341] As one possible implementation, if an update to the third data shard leads to an update to the fourth data shard, the apparatus further includes a partitioning unit for:

[0342] The ownership of the third data shard and the ownership of the fourth data shard are assigned to the target node.

[0343] As one possible implementation, if updating the fifth data shard causes updating the fourth data shard, and updating the fifth data shard does not cause updating the third data shard, then the device further includes a partitioning unit and a releasing unit.

[0344] The partitioning unit is used for:

[0345] The ownership of the third data shard, the fourth data shard, and the fifth data shard is allocated to the target node;

[0346] The release unit is used for:

[0347] Obtain the load of the target node;

[0348] If the increased load on the third data shard causes the load on the target node to exceed the load limit, then the ownership of the fifth data shard is released.

[0349] In the preset storage location, the node indicated by the ownership information of the fifth data shard is updated to be empty.

[0350] As one possible implementation, the ownership information acquisition unit 1301 is specifically used for:

[0351] The ownership information corresponding to the multiple data fragments is obtained from the preset storage location through the data storage interface;

[0352] The ownership acquisition unit 1302 is specifically used for:

[0353] Ownership of the first data fragment is obtained through the data storage interface;

[0354] The device also includes a data storage interface for:

[0355] The node indicated by the ownership information of the first data fragment is updated to the target node through the data storage interface to obtain all the permission information of the first data fragment after the update.

[0356] If the updated ownership information of the first data fragment is successfully stored in the preset storage location through the data storage interface, then the target node successfully acquires ownership of the first data fragment.

[0357] As one possible implementation, the ownership information of the data fragment also includes an original version identifier, and the device further includes a data storage interface for:

[0358] The node indicated by the ownership information of the first data segment is updated to the target node through the data storage interface, and the original version identifier of the ownership information of the first data segment is increased by a preset value through the data storage interface to obtain all the permission information of the first data segment after the update.

[0359] The ownership information of the first data fragment is obtained from the preset storage location through the data storage interface;

[0360] If the version identifier in the ownership information of the first data fragment is the same as the original version identifier, then the updated ownership information of the first data fragment is stored in the preset storage location through the data storage interface.

[0361] This application also provides a computer device, which can be a server or a terminal device. The computer device provided in this application will be described below from a hardware implementation perspective. Figure 14 The diagram shown is a structural schematic of the server. Figure 15 The diagram shown is a structural schematic of the terminal device.

[0362] See Figure 14 This figure is a schematic diagram of a server structure provided in an embodiment of this application. The server 1400 can vary considerably due to different configurations or performance. It may include one or more processors 1422, such as a central processing unit (CPU), memory 1432, and one or more application programs 1442 or data storage media 1430 (e.g., one or more mass storage devices). The memory 1432 and storage media 1430 can be temporary or persistent storage. The program stored in the storage media 1430 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the processor 1422 may be configured to communicate with the storage media 1430 and execute the series of instruction operations in the storage media 1430 on the server 1400.

[0363] Server 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input / output interfaces 1458, and / or one or more operating systems 1441, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0364] The steps performed by the server in the above embodiments can be based on this Figure 14 The server structure shown.

[0365] The processor 1422 is used to perform the following steps:

[0366] The ownership information corresponding to multiple data shards is obtained from a preset storage location. The ownership information of the data shards is used to indicate the nodes in the cluster that own the data shards. The ownership of the data shards is used to identify the nodes that have the authority to process the data shards.

[0367] If it is determined, based on the ownership information of the first data shard, that none of the nodes in the cluster have ownership of the first data shard, then the ownership of the first data shard is acquired, and the first data shard is one of the multiple data shards.

[0368] If ownership of the first data fragment is successfully acquired, the first data fragment is loaded, wherein the node indicated by the ownership information of the first data fragment in the preset storage location is updated to the target node.

[0369] Optionally, the processor 1422 may also execute method steps of any specific implementation of the data fragmentation processing method in the embodiments of this application.

[0370] See Figure 15 This figure is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. The description will be based on a smartphone as an example. Figure 15 The diagram shown is a partial structural block diagram of the smartphone, which includes: a radio frequency (RF) circuit 1510, a memory 1520, an input unit 1530, a display unit 1540, a sensor 1550, an audio circuit 1560, a Wi-Fi module 1570, a processor 1580, and a power supply 1590, among other components. Those skilled in the art will understand that... Figure 15 The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0371] The following is combined Figure 15 A detailed introduction to the various components of a smartphone:

[0372] The RF circuit 1510 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 1580; in addition, it transmits uplink data to the base station.

[0373] The memory 1520 can be used to store software programs and modules, and the processor 1580 runs the software programs and modules stored in the memory 1520 to realize various functions and data processing of the smartphone.

[0374] Input unit 1530 can be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the smartphone. Specifically, input unit 1530 may include touch panel 1531 and other input devices 1532. Touch panel 1531, also known as a touch screen, can collect touch operations on or near the user and drive corresponding connected devices according to a pre-set program. In addition to touch panel 1531, input unit 1530 may also include other input devices 1532. Specifically, other input devices 1532 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0375] The display unit 1540 can be used to display information input by the user or information provided to the user, as well as various menus of the smartphone. The display unit 1540 may include a display panel 1541, which may optionally be configured as a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0376] Smartphones may also include at least one sensor 1550, such as a light sensor, a motion sensor, and other sensors. Other sensors that smartphones may also be equipped with, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be detailed here.

[0377] Audio circuit 1560, speaker 1561, and microphone 1562 provide an audio interface between the user and the smartphone. Audio circuit 1560 converts received audio data into electrical signals and transmits them to speaker 1561, where speaker 1561 converts them into sound signals for output. On the other hand, microphone 1562 converts collected sound signals into electrical signals, which are received by audio circuit 1560, converted into audio data, and then processed by processor 1580 before being transmitted via RF circuit 1510 to, for example, another smartphone, or the audio data can be output to memory 1520 for further processing.

[0378] The processor 1580 is the control center of the smartphone, connecting various parts of the smartphone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 1520, and by calling data stored in the memory 1520. Optionally, the processor 1580 may include one or more processing units.

[0379] The smartphone also includes a power supply 1590 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 1580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0380] Although not shown, smartphones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0381] In this embodiment of the application, the memory 1520 included in the smartphone can store computer programs and transmit the computer programs to the processor.

[0382] The processor 1580 included in the smartphone can execute the data fragmentation processing method provided in the above embodiments according to the instructions in the computer program.

[0383] This application also provides a computer-readable storage medium for storing a computer program that executes the data fragmentation processing method provided in the above embodiments.

[0384] On the other hand, embodiments of this application provide a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the data fragmentation processing method provided in various optional implementations of the above aspects.

[0385] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk or optical disk, and other media that can store computer programs.

[0386] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0387] All data collected in this application (such as data fragments, ownership information of data fragments, etc.) are collected with the consent and authorization of the data owner (such as users, institutions or enterprises), and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0388] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0389] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0390] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data fragmentation processing method, characterized in that, The method is applied to a target node in a cluster comprising multiple nodes, and the method includes: The ownership information corresponding to multiple data shards is obtained from a preset storage location. The ownership information of the data shards is used to indicate the nodes in the cluster that own the data shards. The ownership of the data shards is used to identify the nodes that have the authority to process the data shards. If it is determined, based on the ownership information of the first data shard, that none of the nodes in the cluster have ownership of the first data shard, then the ownership of the first data shard is acquired, and the first data shard is one of the multiple data shards. If ownership of the first data fragment is successfully acquired, the first data fragment is loaded, wherein the node indicated by the ownership information of the first data fragment in the preset storage location is updated to the target node.

2. The method according to claim 1, characterized in that, The ownership information also includes the latest lease renewal time, which identifies the time when the node last acquired ownership of the data shard. In the preset storage location, the latest lease renewal time in the ownership information of the first data shard is a first time. After the first data shard is loaded for a first preset duration, the method further includes: Acquire ownership of the first data shard, where the first preset duration is the lease renewal time interval corresponding to the target node; If the ownership of the first data fragment is successfully acquired, the first data fragment is kept loaded. In the preset storage location, the ownership information of the first data fragment, including the latest lease time, is updated to a second time, which is later than the first time. If the acquisition of ownership of the first data fragment fails, the first data fragment is released, and in the preset storage location, the ownership information of the first data fragment, including the latest lease time, remains the first time.

3. The method according to claim 2, characterized in that, The acquisition of ownership of the first data fragment includes: Obtain the load of the target node; If the load of the target node is less than the load limit, then the ownership of the first data shard is acquired.

4. The method according to claim 2 or 3, characterized in that, The method further includes: If the node indicated by the ownership information of the first data shard is empty, then it is determined that none of the nodes in the cluster have ownership of the first data shard; or, Get the third time used to identify the current time; The ownership information of the first data segment includes the time difference between the last lease time and the third time. If the ownership information of the first data shard indicates that the node is another node, and the time difference is greater than the lease renewal time interval corresponding to the other node, then it is determined that none of the nodes in the cluster have ownership of the first data shard, and the other nodes are nodes other than the target node among the multiple nodes included in the cluster.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the load of the target node; If the load of the target node is greater than the load limit, then a second data shard is determined from the data shards owned by the target node; Release ownership of the second data fragment; In the preset storage location, the node indicated by the ownership information of the second data fragment is updated to be empty.

6. The method according to claim 5, characterized in that, If the target node owns multiple data shards, then determining the second data shard from the data shards owned by the target node includes: Acquire multiple permission data shards, wherein the permission data shards are data shards owned by the target node; Obtain the influencing factors corresponding to each of the permission data shards, including the load of the permission data shard and the historical load trend of the permission data shard; The second data fragment is determined from the permission data fragment based on the influencing factors; Specifically, if the load of the target permission data shard is higher, the probability that the target permission data shard will be identified as the second data shard is higher; or, if the historical load trend of the target permission data shard indicates that the future load of the target permission data shard is higher than the load of other permission data shards, the probability that the target permission data shard will be identified as the second data shard is higher. The target permission data shard is one of the multiple permission data shards, and the other permission data shards are the permission data shards other than the target data shard among the multiple permission data shards. The future load is the load predicted based on the historical load trend.

7. The method according to claim 1, characterized in that, The ownership information of the data shard also includes the payload of the data shard. The step of obtaining ownership information corresponding to multiple data shards from a preset storage location includes: Obtain the load of the target node; If the load of the target node is less than the lower load limit, then the ownership information corresponding to each of the multiple data shards is obtained from the preset storage location. The method further includes: Multiple isolated data shards are determined based on multiple ownership information, wherein the isolated data shards are shards in which no node in the cluster has ownership; Based on the load of the target node and the load of each of the isolated data shards, the first data shard is determined from the plurality of isolated data shards, wherein the sum of the load of the first data shard and the load of the target node is less than the upper limit of the load, and the upper limit of the load is greater than the lower limit of the load.

8. The method according to claim 7, characterized in that, The step of determining the first data shard from multiple isolated data shards based on the load of the target node and the load of each isolated data shard includes: Obtain the historical load trend of each isolated data shard; Based on the load of the target node, the load of each isolated data shard, and the historical load trend of each isolated data shard, the first data shard is determined from a plurality of isolated data shards. The sum of the future load of the first data shard determined based on the historical load of the isolated data shard and the load of the target node is less than the load upper limit value.

9. The method according to any one of claims 3, 5-8, characterized in that, If the data shards are leaderboards, the method further includes: Obtain the load corresponding to each data shard owned by the target node, wherein the load of the data shard is determined based on the number of accounts participating in the calculation of the data shard; The load of the target node is obtained by summing the loads of all the data shards owned by the target node.

10. The method according to claim 1, characterized in that, The target node includes at least one permission identifier, the permission identifier being used to identify that the target node owns the data shard, and the method further includes: Obtain a data shard update request including an identifier to be updated, wherein the identifier to be updated is the identifier of the data shard waiting to be updated; If the target node includes a permission identifier that is the same as the identifier to be updated, then the data shard corresponding to the identifier to be updated is updated. If the target node does not include a permission identifier that is the same as the identifier to be updated, then the ownership information corresponding to the multiple data shards is obtained from the preset storage location. Based on the multiple ownership information and the identifier to be updated, an authorized node is determined, wherein the authorized node is a node that owns the data fragment corresponding to the identifier to be updated; The data shard update request is forwarded to the authorized node.

11. The method according to claim 1, characterized in that, If an update to the third data shard leads to an update to the fourth data shard, the method further includes: The ownership of the third data shard and the ownership of the fourth data shard are assigned to the target node.

12. The method according to claim 11, characterized in that, If an update to the fifth data shard leads to an update to the fourth data shard, and an update to the fifth data shard does not lead to an update to the third data shard, then the step of allocating ownership of the third data shard and the fourth data shard to the target node includes: The ownership of the third data shard, the fourth data shard, and the fifth data shard is allocated to the target node; The method further includes: Obtain the load of the target node; If the increased load on the third data shard causes the load on the target node to exceed the load limit, then the ownership of the fifth data shard is released. In the preset storage location, the node indicated by the ownership information of the fifth data shard is updated to be empty.

13. The method according to claim 1, characterized in that, The step of obtaining ownership information corresponding to multiple data fragments from a preset storage location includes: The ownership information corresponding to the multiple data fragments is obtained from the preset storage location through the data storage interface; The acquisition of ownership of the first data fragment includes: Ownership of the first data fragment is obtained through the data storage interface; The method further includes: The node indicated by the ownership information of the first data fragment is updated to the target node through the data storage interface to obtain all the permission information of the first data fragment after the update. If the updated ownership information of the first data fragment is successfully stored in the preset storage location through the data storage interface, then the target node successfully acquires ownership of the first data fragment.

14. The method according to claim 13, characterized in that, The ownership information of the data shard also includes the original version identifier. The step of updating the node indicated by the ownership information of the first data shard to the target node through the data storage interface yields all permission information for the updated first data shard, including: The node indicated by the ownership information of the first data segment is updated to the target node through the data storage interface, and the original version identifier of the ownership information of the first data segment is increased by a preset value through the data storage interface to obtain all the permission information of the first data segment after the update. The method further includes: The ownership information of the first data fragment is obtained from the preset storage location through the data storage interface; If the version identifier in the ownership information of the first data fragment is the same as the original version identifier, then the updated ownership information of the first data fragment is stored in the preset storage location through the data storage interface.

15. The method according to claim 1, characterized in that, The step of obtaining ownership information corresponding to multiple data fragments from a preset storage location includes: Ownership information corresponding to the multiple data fragments is retrieved from the preset storage location every second preset time interval.

16. A data fragmentation processing apparatus, characterized in that, The device is applied to a target node in a cluster comprising multiple nodes, and the device includes: an ownership information acquisition unit, an ownership acquisition unit, and a loading unit; The ownership information acquisition unit is used to acquire ownership information corresponding to multiple data shards from a preset storage location. The ownership information of the data shards is used to indicate the nodes in the cluster that own the data shards. The ownership of the data shards is used to identify that the nodes have the authority to process the data shards. The ownership acquisition unit is configured to acquire ownership of the first data shard if it is determined, based on the ownership information of the first data shard, that none of the nodes in the cluster have ownership of the first data shard, wherein the first data shard is one of the plurality of data shards. The loading unit is configured to load the first data fragment if ownership of the first data fragment is successfully acquired, wherein the node indicated by the ownership information of the first data fragment in the preset storage location is updated to the target node.

17. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is configured to perform the method according to any one of claims 1-15 according to the computer program.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method according to any one of claims 1-15.

19. A computer program product comprising a computer program, characterized in that, When it is run on a computer device, it causes the computer device to perform the method described in any one of claims 1-15.

Images