Partition processing system and methods

Abstract

This specification provides a partitioning system and method. The partitioning system includes a scheduling node, a primary partition, a first partition, and a second partition. The task set of the primary partition is stored in a storage unit. Upon receiving a partition splitting instruction from the scheduling node, the primary partition writes the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into the storage unit. Upon receiving a first partition loading instruction from the scheduling node, the first partition determines its first task set to be processed based on the first partition information in the task set and stores the first partition state in the storage unit, completing the loading of the first partition. Upon receiving a second partition loading instruction from the scheduling node, the second partition determines its second task set to be processed based on the second partition information in the task set and stores the second partition state in the storage unit, completing the loading of the second partition. This achieves partition isolation and ensures data consistency.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a partitioning processing system and method. Background Technology

[0002] Distributed storage services, with their advantages of high reliability, availability, and scalability, are widely used in scenarios such as cloud computing, big data analytics, and data storage. However, distributed storage also presents some challenges, such as high complexity in management and maintenance, and difficulty in ensuring data consistency.

[0003] In existing technologies, distributed storage services typically employ partitioning schemes based on distributed locks to ensure data consistency. However, during the use of distributed locks, the lock holder may still commit dirty writes after the lock is acquired. Partitioning designs based on distributed locks usually require the use of other data protection mechanisms and middleware to achieve the desired data consistency. Therefore, a more effective partitioning system is urgently needed to solve these problems. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a partitioning system. One or more embodiments of this specification also relate to partitioning methods, partitioning apparatuses, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a partitioning system is provided, including: a scheduling node, a main partition, a first partition, and a second partition, wherein the task set of the main partition is stored in a storage unit;

[0006] The primary partition is used to respond to the partition splitting instruction sent by the scheduling node by writing the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into the storage unit.

[0007] The first partition is used to respond to the first partition loading instruction sent by the scheduling node, determine the first task set in the task set based on the first partition information, and store the first partition status in the storage unit;

[0008] The second partition is used to respond to the second partition loading instruction sent by the scheduling node, determine the second task set in the task set based on the second partition information, and store the second partition state in the storage unit.

[0009] According to a second aspect of the embodiments of this specification, another partitioning system is provided, including: a scheduling node, a third partition, a fourth partition, and a merged partition, wherein a third task set of the third partition and a fourth task set of the fourth partition are both stored in a storage unit.

[0010] The third partition is used to write the third partition information into the storage unit in response to the partition merging instruction sent by the scheduling node;

[0011] The fourth partition is used to write the fourth partition information into the storage unit in response to the partition merging instruction sent by the scheduling node;

[0012] The merge partition is configured to, in response to a partition loading instruction sent by the scheduling node, obtain the third partition information and the fourth partition information in the storage unit, determine the third task set in the storage unit based on the third partition information, and determine the fourth task set in the storage unit based on the fourth partition information; merge the third task set and the fourth task set into a merged task set in the storage unit, and store the merged partition status in the storage unit.

[0013] According to a third aspect of the embodiments of this specification, a partitioning method is provided, comprising:

[0014] In response to the partition splitting instruction sent by the scheduling node, the primary partition writes the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into the storage unit, and the task set of the primary partition is stored in the storage unit.

[0015] The first partition responds to the first partition loading instruction sent by the scheduling node, determines the first task set in the task set based on the first partition information, and stores the state of the first partition in the storage unit;

[0016] The second partition responds to the second partition loading instruction sent by the scheduling node, determines the second task set in the task set based on the second partition information, and stores the second partition state in the storage unit.

[0017] According to a fourth aspect of the embodiments of this specification, another partitioning method is provided, including:

[0018] In response to the partition merging instruction sent by the scheduling node, the third partition writes the third partition information into the storage unit, and the third task set of the third partition is stored in the storage unit.

[0019] In response to the partition merge instruction sent by the scheduling node, the fourth partition writes the fourth partition information into the storage unit, and the fourth task set of the fourth partition is stored in the storage unit;

[0020] In response to the partition loading instruction sent by the scheduling node, the merge partition obtains the third partition information and the fourth partition information in the storage unit, determines the third task set in the storage unit based on the third partition information, and determines the fourth task set in the storage unit based on the fourth partition information; merges the third task set and the fourth task set into a merged task set in the storage unit, and stores the merged partition status in the storage unit.

[0021] According to a fifth aspect of the embodiments of this specification, a partitioning processing apparatus is provided, comprising:

[0022] The write module is configured to respond to the partition splitting instruction sent by the scheduling node in response to the primary partition, and write the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into the storage unit, wherein the task set of the primary partition is stored in the storage unit.

[0023] The first storage module is configured to, in response to the first partition loading instruction sent by the scheduling node, determine the first task set in the task set based on the first partition information, and store the state of the first partition in the storage unit.

[0024] The second storage module is configured to, in response to the second partition loading instruction sent by the scheduling node, determine the second task set in the task set based on the second partition information, and store the second partition status in the storage unit.

[0025] According to a sixth aspect of the embodiments of this specification, another partitioning processing apparatus is provided, comprising:

[0026] The third storage module is configured to write the third partition information into the storage unit in response to the partition merge instruction sent by the scheduling node, and the third task set of the third partition is stored in the storage unit.

[0027] The fourth storage module is configured to write the fourth partition information into the storage unit in response to the partition merge instruction sent by the scheduling node, and the fourth task set of the fourth partition is stored in the storage unit.

[0028] The determination module is configured to merge partitions in response to a partition loading instruction sent by the scheduling node, obtain the third partition information and the fourth partition information in the storage unit, determine the third task set in the storage unit based on the third partition information, and determine the fourth task set in the storage unit based on the fourth partition information; merge the third task set and the fourth task set into a merged task set in the storage unit, and store the merged partition status in the storage unit.

[0029] According to a seventh aspect of the embodiments of this specification, a computing device is provided, comprising:

[0030] Memory and processor;

[0031] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the partitioning process described above.

[0032] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the partitioning method described above.

[0033] According to a ninth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the partitioning processing method described above.

[0034] This specification provides a partitioning system according to one embodiment, including a scheduling node, a primary partition, a first partition, and a second partition. The task set of the primary partition is stored in a storage unit. The scheduling node sends a partition splitting instruction to the primary partition to split the task set of the primary partition into two task sets, which are then processed by the first and second partitions respectively. After receiving the partition splitting instruction from the scheduling node, the primary partition writes the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into the storage unit. The scheduling node sends a first partition loading instruction and a second partition loading instruction to the first and second partitions respectively to load the first and second partitions. After receiving the first partition loading instruction from the scheduling node, the first partition determines the first task set to be processed in the task set based on the first partition information and stores the state of the first partition in the storage unit, completing the loading of the first partition. After receiving the second partition loading instruction from the scheduling node, the second partition determines the second task set to be processed in the task set based on the second partition information and stores the state of the second partition in the storage unit, completing the loading of the second partition, thus achieving partition isolation between the first and second partitions. After the first and second partitions determine the first and second task sets respectively, they store the states of the first and second partitions in the storage unit respectively. The primary partition can be split without the use of plugins. Determining the first and second partitions before splitting the primary partition's task set improves partition splitting efficiency, ensures data consistency during the partition splitting process and subsequent task execution, and is suitable for services with high data accuracy requirements. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the structure of a partitioning system provided in one embodiment of this specification;

[0036] Figure 2 This is a schematic diagram of another partitioning system provided in one embodiment of this specification;

[0037] Figure 3 This is a schematic diagram illustrating the partition splitting interaction of a partition processing system provided in one embodiment of this specification;

[0038] Figure 4 This is a schematic diagram of the partition merging interaction of a partition processing system provided in one embodiment of this specification;

[0039] Figure 5 This is a flowchart of a partitioning method provided in one embodiment of this specification;

[0040] Figure 6 This is a schematic diagram of the structure of a partitioning processing device provided in one embodiment of this specification;

[0041] Figure 7 This is a flowchart of another partitioning method provided in one embodiment of this specification;

[0042] Figure 8 This is a schematic diagram of another partitioning processing device provided in one embodiment of this specification;

[0043] Figure 9 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0044] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0045] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0046] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0047] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0048] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0049] Partition: A key concept in distributed services, it typically represents a subset of a service.

[0050] Partition isolation refers to the ability to maintain data consistency and availability when faced with network partitions, node failures, or other unstable factors.

[0051] Partition atomicity: refers to the ability to perform transactions on a single partition, such as atomically writing data to two data entries in the same partition.

[0052] Splitting: refers to the process of splitting a service partition into two partitions.

[0053] Merging: refers to the process of merging two service partitions into one partition.

[0054] Parent / Child Partition: refers to the new (child) and old (parent) partitions during the splitting and merging process.

[0055] In a distributed storage system based on object storage, there exists a data persistence service that is horizontally scalable through partitioning and possesses atomic write capabilities for each partition. This data persistence service enables atomic writes of a single row and multiple rows within a single partition. The distributed storage system also includes a management node responsible for partition scheduling. This node handles basic partition scheduling, including partition loading, migration, splitting, and merging. The partition IDs managed by the management node are globally unique (at least logically). The distributed storage system is a widely used key-value (KV) storage system, where the basic data unit is a row, each row is located by a primary key, uniquely corresponding to a value. This value can be structured or unstructured.

[0056] Each computation partition in a distributed storage system needs to output some computation results for subsequent analysis and summarization. The primary key of the partition computation result in the KV storage system consists of the partition ID and the characteristics of the computation task corresponding to the current computation result (e.g., time interval). The primary key of the partition computation status in the KV storage system consists of the partition ID and a specially customized prefix to distinguish it from the partition computation result. The computation status contains at least two types of data: one is the SessionID, i.e., the partition identifier, used for partition isolation; the other is computation task information, used to initialize the computation task (e.g., initialize the computation data source). The partition computation logic is as follows: After the partition is loaded, it first reads the computation status from the KV storage system; it initializes the local computation task using the task information in the computation status information; after initialization, the partition actively updates the SessionID in the computation status to the KV storage system; it executes the computation task; after the computation task is completed, it submits the computation result to the KV storage system; it updates the computation status information to the KV storage system, mainly updating the task information, while the SessionID remains unchanged. At the same time, this step atomically checks whether the SessionID in the computation status information of the partition in the current KV storage system is still the SessionID stored in the KV storage system after initialization. If the SessionID has changed, the update fails and the latest SessionID is returned; if the SessionID has changed, the partition detects the update failure, determines that it is invalid, stops computation, and waits to exit; if the SessionID has not changed, it continues to execute the subsequent steps.

[0057] The SessionID mechanism is key to partition isolation. In a key-value storage system, the SessionID in the partition computation status information uniquely identifies a valid partition instance. The validity of the computation result is uniquely determined by the computation status information of the corresponding partition. Therefore, even if an invalid partition commits dirty data, its subsequent commits of computation status information will fail. In other words, data committed by an invalid partition is also invalid, thus achieving basic partition isolation (and invalid partitions will exit after recognizing their invalidity).

[0058] This specification provides a partitioning system, and also relates to a partitioning method, a partitioning apparatus, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail in the following embodiments.

[0059] See Figure 1 , Figure 1A schematic diagram of a partition processing system 100 according to an embodiment of this specification is shown. The partition processing system 100 includes: a scheduling node 110, a primary partition 120, a first partition 130, and a second partition 140. The task set of the primary partition 120 is stored in a storage unit. The primary partition 120 is used to write first partition information corresponding to the first partition 130 and second partition information corresponding to the second partition 140 into the storage unit in response to a partition splitting instruction sent by the scheduling node 110. The first partition 130 is used to determine a first task set in the task set based on the first partition information and store the first partition status in the storage unit in response to a first partition loading instruction sent by the scheduling node 110. The second partition 140 is used to determine a second task set in the task set based on the second partition information and store the second partition status in the storage unit in response to a second partition loading instruction sent by the scheduling node 110.

[0060] Specifically, the primary partition can be a subset of a service in a distributed service, and the set of tasks to be executed by the primary partition is stored in the storage unit. The first and second partitions are obtained by splitting the primary partition along the task dimension. The primary partition and the first partition have a parent-child relationship; that is, the primary partition is the parent partition of the first partition, and the first partition is a child partition of the primary partition. Similarly, the primary partition and the second partition also have a parent-child relationship; that is, the primary partition is the parent partition of the second partition, and the second partition is a child partition of the primary partition. The storage unit can be a key-value (KV) storage system, where the basic data unit is a row, each row is located by a primary key, and the primary key corresponds to a value. This value can be structured or unstructured. The storage unit is used to store the task data, computation status information, computation result data of the primary partition, and the split data of the primary partition along the task dimension. For the first and second partitions, the storage unit is also used to store the task data, computation status information, computation result data corresponding to the first and second partitions respectively, as well as the parent partition data corresponding to the first and second partitions respectively. The data stored in the storage unit is used to achieve partition isolation between the primary partition, the first partition, and the second partition.

[0061] The scheduling node interacts with the primary partition, the first partition, and the second partition to schedule the primary partition splitting process and monitor its progress. The scheduling node is responsible for basic partition scheduling, including but not limited to partition loading, migration, splitting, and merging. After the primary partition is split, it is unloaded, and when the first and second partitions are determined, they are loaded. The primary partition's task set stores tasks that need to be executed by the primary partition, including but not limited to computational tasks, storage tasks, and other data- or logical-dimensional tasks. The partition splitting command sent by the scheduling node informs the primary partition that a partition split is about to begin. The first partition information can be the task splitting information corresponding to the first partition, representing the task splitting information for splitting the primary partition's task set. Based on the first partition information, the number and specific tasks handled by the first partition in the primary partition's task set can be determined. The second partition information can be the task splitting information corresponding to the second partition, representing the task splitting information for splitting the primary partition's task set. Based on the second partition information, the number and specific tasks handled by the second partition in the primary partition's task set can be determined.

[0062] The first partition load instruction is used to instruct the first partition to perform a partition load, thereby taking over tasks in the primary partition. The first task set contains the tasks in the primary partition's task set that are taken over by the first partition. The first partition status indicates the takeover status of the tasks taken over by the first partition, and includes the first partition identifier and computation status of the first partition. The computation status can be the computation status of tasks in the first partition, including computation completed, computation in progress, and not computed. The second partition load instruction is used to instruct the second partition to perform a partition load, thereby taking over tasks in the primary partition. The second task set contains the tasks in the primary partition's task set that are taken over by the second partition. The second partition status indicates the takeover status of the tasks taken over by the second partition, and includes the second partition identifier and computation status of the second partition.

[0063] In practical applications, when splitting a primary partition, the number of partitions can be two, i.e., including a first partition and a second partition. Alternatively, the number of partitions can be multiple, including a first partition, a second partition, a third partition, and so on. In other words, when splitting a primary partition as the parent partition, it can be split into at least two child partitions, and the number of partitions can be set according to actual needs.

[0064] Based on this, the partition processing system 100 includes a scheduling node 110, a primary partition 120, a first partition 130, and a second partition 140. The task set of the primary partition is stored in a storage unit. The scheduling node sends a partition splitting instruction to the primary partition to split the task set of the primary partition into two task sets, which are then processed by the first partition and the second partition, respectively. After receiving the partition splitting instruction from the scheduling node, the primary partition writes the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into the storage unit. The scheduling node sends a first partition loading instruction and a second partition loading instruction to the first partition and the second partition, respectively, to load the first partition and the second partition. After receiving the first partition loading instruction from the scheduling node, the first partition determines the first task set it needs to process from the task set based on the first partition information and stores the state of the first partition in the storage unit, thus completing the loading of the first partition. After receiving the second partition loading instruction from the scheduling node, the second partition determines the second task set it needs to process from the task set based on the second partition information and stores the state of the second partition in the storage unit, thus completing the loading of the second partition and achieving partition isolation between the first partition and the second partition. After determining the first task set and the second task set in the first and second partitions respectively, the states of the first and second partitions are stored in the storage units. This allows for the partitioning of the primary partition without the need for plugins. Determining the first and second partitions before splitting the task set of the primary partition improves partition splitting efficiency, ensures data consistency during the partition splitting process and subsequent task execution, and is suitable for services with high data accuracy requirements.

[0065] Furthermore, considering that the task set corresponding to the primary partition 120 may contain no tasks or may contain at least one task, the task set cannot be split if it contains no tasks. Before splitting the primary partition 120, it is necessary to determine whether the task set passes the partition condition check. Specifically, the primary partition 120 is used to perform a partition condition check on the task set in response to the partition splitting instruction sent by the scheduling node 110. If the task set passes the check, the first partition information corresponding to the first partition 130 and the second partition information corresponding to the second partition 140 are written into the storage unit.

[0066] Specifically, a partitioning condition check is performed on the task set to determine whether it meets the partitioning criteria, i.e., whether the task set can be further divided. If the task set contains no tasks or only one task, the task set determining the primary partition fails the partitioning condition check. If the task set contains at least two tasks, the task set determining the primary partition passes the partitioning condition check.

[0067] Based on this, the primary partition responds to the partition splitting command sent by the scheduling node by performing a partition condition check on the task set. If the check passes, it indicates that the task set of the primary partition meets the conditions for partition splitting, that is, it meets the partitioning conditions, and the task set can be further divided to allocate tasks to both the first and second partitions. At this time, the first partition information corresponding to the first partition and the second partition information corresponding to the second partition can be written into the storage unit.

[0068] For example, in a distributed service scenario, after the primary partition receives a partition splitting instruction from the scheduling node, it can first check if it meets the splitting conditions. If it does, the partition splitting can proceed; otherwise, the splitting is terminated. Based on the task set of the primary partition, a partitioning condition check is performed. If the task set contains no tasks, the primary partition has failed the partitioning condition check. If the task set contains 10 tasks, the task set can be further split. The primary partition is then split into a first partition and a second partition. Tasks 1-5 are assigned to the first partition, and tasks 6-10 are assigned to the second partition. The information for the first partition and the information for the second partition are then written to the storage unit.

[0069] When allocating tasks in a task set, you can choose to distribute them equally between the first and second partitions. That is, if the task set contains N tasks from 1 to N, allocate 1 to 2 / N tasks to the first partition and allocate 2 / NN tasks to the second partition.

[0070] In summary, before writing the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into the storage unit, the task set of the main partition is checked for partition conditions to avoid resource waste caused by writing partition information when the task set cannot be further divided.

[0071] Furthermore, as the control node responsible for basic partition scheduling, scheduling node 110 needs to monitor the partition splitting progress. After the first partition information and the second partition information are written to the storage unit in the primary partition 120, and the writing is completed, the writing completion information needs to be sent to scheduling node 110 to inform scheduling node 110 of the partition splitting progress. In specific implementation, the primary partition 120 is also used to generate writing completion information and send the writing completion information to scheduling node 110; the scheduling node 110 is used to generate a partition unload instruction corresponding to the primary partition 120, a first partition load instruction corresponding to the first partition 130, and a second partition load instruction corresponding to the second partition 140 based on the writing completion information; and sends the partition unload instruction to the primary partition 120, the first partition load instruction to the first partition 130, and the second partition load instruction to the second partition 140.

[0072] Specifically, the write completion message is generated after the first and second partition information is written to the storage unit, indicating that the first and second partition information has been written to the storage unit. The partition unload command instructs the primary partition to unload the partition, preventing subsequent tasks from failing to process properly if the primary partition is not unloaded. The first partition load command instructs the first partition to load the first partition, completing partition preparation before task processing. The second partition load command instructs the second partition to load the second partition, completing partition preparation before task processing.

[0073] Based on this, once the primary partition has written the first and second partition information to the storage unit and the writing is complete, it generates a write completion message and sends it to the scheduling node. This message informs the scheduling node that both the first and second partition information have been written to the storage unit. The scheduling node then generates a partition unload instruction for the primary partition, a first partition load instruction for the first partition, and a second partition load instruction for the second partition based on the write completion message. The scheduling node sends the partition unload instruction to the primary partition, the first partition load instruction to the first partition, and the second partition load instruction to the second partition, thus achieving partition unloading of the primary partition and partition loading of the first and second partitions. This ensures the successful execution of the primary partition split and improves the efficiency of the primary partition split.

[0074] Furthermore, after receiving the first partition loading instruction sent by the scheduling node 110, the first partition 130 can enter the data preparation stage and write the first partition status information to the storage unit, indicating that the first partition 130 takes over the processing of tasks in the main partition task set from the main partition 120. After receiving the second partition loading instruction sent by the scheduling node 110, the second partition 140 can enter the data preparation stage and write the second partition status information to the storage unit, indicating that the second partition 140 takes over the processing of tasks in the main partition task set from the main partition 120. Specifically, the first partition 130 is used to determine the first partition identifier and the first task set information, and stores the first partition identifier and the first task set information as the first partition status in the storage unit; the second partition 140 is used to determine the second partition identifier and the second task set information, and stores the second partition identifier and the second task set information as the second partition status in the storage unit.

[0075] Specifically, the first partition identifier represents the ID of the first partition. This identifier is stored in the storage unit when the first partition is loaded, indicating that the first partition has been loaded. The first task set information records the number and information of tasks in the task set allocated to the first partition. The second partition identifier represents the ID of the second partition. This identifier is stored in the storage unit when the second partition is loaded, indicating that the second partition has been loaded. The second task set information records the number and information of tasks in the task set allocated to the second partition.

[0076] Based on this, after receiving the first partition loading instruction from the scheduling node, the first partition determines its own first partition identifier and first task set information, and stores the first partition identifier and the first task set information corresponding to the allocated first task set as the first partition status in the storage unit, indicating that the first partition has been loaded and can subsequently execute tasks. Similarly, the second partition determines its own second partition identifier and the second task set information of the allocated second task set, and stores the second partition identifier and second task set information as the second partition status in the storage unit, indicating that the second partition has been loaded and can subsequently execute tasks.

[0077] Continuing with the previous example, after the first partition receives the first partition load instruction from the scheduling node, it can determine its own session ID and computation status information (task set information), and write the session ID and computation status information into the storage unit, completing the data preparation before task processing. After the second partition receives the second partition load instruction from the scheduling node, it can determine its own session ID and computation status information (task set information), and write the session ID and computation status information into the storage unit, completing the data preparation before task processing.

[0078] In summary, the first partition stores its first partition identifier and first task set information as its first partition state in the storage unit, and the second partition stores its second partition identifier and second task set information as its second partition state in the storage unit. Both the first and second partitions have completed partition loading and can then directly execute their respective assigned tasks. The existence of the first and second partition identifiers ensures data consistency during subsequent task processing.

[0079] Furthermore, during the splitting of the primary partition 120 and the execution of tasks in the first partition 130 and the second partition 140, the execution results of the first task and the second task may become unreliable or unusable due to factors such as task scheduling, computing environment, and task version. This situation manifests as changes in the identifiers of the first and second partitions in the storage unit. Therefore, before storing the execution results of the first and second tasks into the storage unit, it is necessary to check the first partition identifier and the second partition identifier. Specifically, the first partition 130 is also used to execute the first task contained in the first task set to obtain the execution result of the first task; if the first partition identifier contained in the first partition state has not changed, the execution result of the first task is stored into the storage unit, and the first partition state is stored into the storage unit; if the first partition identifier contained in the first partition state has changed, the execution result of the first task is discarded. The second partition 140 is also used to execute the second task contained in the second task set to obtain the execution result of the second task; if the second partition identifier contained in the second partition state has not changed, the execution result of the second task is stored into the storage unit, and the second partition calculation state is stored into the storage unit; if the second partition identifier contained in the second partition state has changed, the execution result of the second task is discarded.

[0080] Based on this, after determining the first task set, the first partition can execute the first task contained in the first task set and obtain the execution result of the first task. Before writing the execution result of the first task to the storage unit, it is necessary to determine whether the first partition identifier contained in the first partition state stored in the storage unit is the same as the target first partition identifier of the current first partition. If the first partition identifier stored in the storage unit is the same as the target first partition identifier, it means that the first partition identifier has not changed, and the execution result of the first task can be stored in the storage unit, as well as the first partition state, indicating that the tasks in the first task set have been processed. If the first partition identifier contained in the first partition state has changed, it is determined that the first partition is non-compliant, and the execution result of the first task obtained from processing the tasks in the first task set is unreliable. There may be other concurrent operations that have taken over the tasks of the first partition, or the computation state of the first partition may have been reset for some reason. At this time, the execution result of the first task needs to be discarded, and the first partition will also wait to exit. When checking the first partition identifier, atomic checks are performed. Atomic checks play an important role in the partition computation process in ensuring data consistency and integrity and avoiding concurrent conflicts.

[0081] Once the second task set is determined, the second partition can execute the second tasks contained within it and obtain the execution results. Before writing the execution results to the storage unit, it is necessary to determine whether the second partition identifier stored in the storage unit's second partition state is the same as the target second partition identifier of the current second partition. If the second partition identifier stored in the storage unit is the same as the target second partition identifier, it means that the second partition identifier has not changed, and the execution results of the second tasks can be stored in the storage unit, along with the computational state of the second partition, indicating that the tasks in the second task set have been completed. If the second partition identifier in the second partition state has changed, the second partition is determined to be non-compliant, and the execution results of the second tasks obtained from processing the tasks in the second task set are unreliable. Other concurrent operations may have taken over the tasks of the second partition, or the computational state of the second partition may have been reset for some reason. In this case, the execution results of the second tasks need to be discarded, and the second partition will wait to exit. An atomic check is performed when checking the second partition identifier.

[0082] In summary, before storing the execution result of the first task to the storage unit, an atomic check is performed on the first partition identifier of the first partition; before storing the execution result of the second task to the storage unit, an atomic check is performed on the second partition identifier of the second partition to ensure data consistency and integrity.

[0083] See Figure 2 , Figure 2A schematic diagram of another partition processing system according to an embodiment of this specification is shown. The partition processing system 200 includes: a scheduling node 210, a third partition 220, a fourth partition 230, and a merge partition 240. The third task set of the third partition 220 and the fourth task set of the fourth partition 230 are both stored in a storage unit. The third partition 220 is used to write third partition information into the storage unit in response to a partition merge command sent by the scheduling node 210. The fourth partition 230 is used to write fourth partition information into the storage unit in response to a partition merge command sent by the scheduling node 210. The merge partition 240 is used to obtain the third partition information and the fourth partition information from the storage unit in response to a partition loading command sent by the scheduling node 210, and to determine the third task set and the fourth task set in the storage unit based on the third partition information. The third task set and the fourth task set are merged into a merge task set in the storage unit, and the merged partition state is stored in the storage unit.

[0084] Specifically, the third and fourth partitions are subsets of a service in the distributed service. The set of third tasks to be executed in the third partition is stored in the storage unit, and the set of fourth tasks to be executed in the fourth partition is also stored in the storage unit. Each set of third and fourth tasks contains at least one task to be executed. The merged partition is obtained by merging the third and fourth partitions along the task dimension. The merged partition and the third partition have a parent-child relationship; that is, the third and fourth partitions are the parent partitions of the merged partition, and the merged partition is the child partition of the third and fourth partitions. The storage unit can be a key-value (KV) storage system, where the basic data unit is a row, each row is located by a primary key, and the primary key corresponds to a value, which can be structured or unstructured. The storage unit is used to store the task data, computation status information, computation result data corresponding to the third and fourth partitions, as well as the split data of the third and fourth partitions along the task dimension. For the merged partition, the storage unit is also used to store the task data, computation status information, computation result data of the merged partition, and the parent partition data corresponding to the merged partition. The data stored in the storage unit is used to achieve partition isolation between the merged partition, the third partition, and the fourth partition.

[0085] The scheduling node interacts with the merge partition, the third partition, and the fourth partition to schedule the merge process of the third and fourth partitions and monitor the partition merge progress. The scheduling node is responsible for basic partition scheduling, including but not limited to partition loading, migration, and split-merging. After the merge partition completes the merge, it loads the merge partition and unloads the third and fourth partitions. The third partition information can be the task merge information corresponding to the third partition, representing the task merge information for merging the third task set of the third partition. Based on the third partition information, the number and specific tasks in the third task set of the third partition that need to be merged and taken over by the merge partition can be determined. The fourth partition information can be the task merge information corresponding to the fourth partition, representing the task merge information for merging the fourth task set of the fourth partition. Based on the fourth partition information, the number and specific tasks in the fourth task set of the fourth partition that need to be merged and taken over by the merge partition can be determined. The partition merge command is used to inform the third and fourth partitions that a task merge is about to occur. The partition load command is used to instruct the merge partition to load the partition to take over the tasks of the third and fourth partitions. The third task set contains all the tasks in the third task set of the third partition that need to be taken over by the merge partition. The fourth task set contains all tasks in the fourth task set of the fourth partition that are taken over by the merged partition; that is, all tasks in the fourth task set. The merged task set is the task set obtained by merging the tasks in the third task set and the tasks in the fourth task set. The merged partition status indicates the takeover status of the tasks taken over by the merged partition, and includes the merged partition identifier and the calculation status. The calculation status refers to the task calculation status of the tasks taken over by the merged partition, including the calculation in progress, not calculated, and calculated completed status.

[0086] Based on this, the partition processing system 200 includes a scheduling node, a third partition 220, a fourth partition 230, and a merging partition 240. The third task set of the third partition and the fourth task set of the fourth partition are both stored in the storage unit. After receiving a partition merging instruction from the scheduling node, the third partition writes its information into the storage unit. Similarly, the fourth partition writes its information into the storage unit after receiving the same instruction. Upon receiving a partition loading instruction from the scheduling node, the merging partition retrieves the third and fourth partition information from the storage unit, determines the third and fourth task sets based on the third and fourth partition information, and merges them into a single merged task set in the storage unit, storing the merged partition state therein. The merging partition then takes over the third task set of the third partition and the fourth task set of the fourth partition, merging the third and fourth partitions. By managing the third and fourth task sets, and executing tasks within the merged task set, logical isolation between partitions can be achieved. A rigorous partition merging process ensures data consistency during the partition merging process and subsequent task execution.

[0087] Furthermore, as the control node responsible for basic partition scheduling, scheduling node 210 needs to monitor the partition merging progress. After the third partition 220 writes its information to the storage unit and the fourth partition 230 writes its information to the storage unit, and both are completed, the third and fourth write completion messages need to be sent to scheduling node 210 to inform it of the partition merging progress. Specifically, the third partition 220, after writing its information to the storage unit, sends the third write completion message to scheduling node 210; the fourth partition 230... After writing the fourth partition information to the storage unit, the scheduling node 210 sends the fourth write completion information to the scheduling node 210; the scheduling node 210 is used to generate a partition load instruction corresponding to the merged partition 240, a third partition unload instruction corresponding to the third partition 220, and a fourth partition unload instruction corresponding to the fourth partition 230 based on the third write completion information and the fourth write completion information; and sends the partition load instruction to the merged partition 240, the third partition unload instruction to the third partition 220, and the fourth partition unload instruction to the fourth partition.

[0088] Specifically, the third write completion message is generated after the third partition information is written to the storage unit, indicating that the third partition information has been written to the storage unit. The fourth write completion message is generated after the fourth partition information is written to the storage unit, indicating that the fourth partition information has been written to the storage unit. The partition load instruction and the merge partition load instruction are used to instruct the merged partitions to load the merged partition, completing the partition preparation before task processing. The third partition unload instruction is used to instruct the third partition to unload the partition. The fourth partition unload instruction is used to instruct the fourth partition to unload the partition.

[0089] Based on this, when the third partition has written its information to the storage unit and the write operation is complete, it generates a third write completion message and sends it to the scheduling node to inform the scheduling node that the third partition information has been written to the storage unit. Similarly, when the fourth partition has written its information to the storage unit and the write operation is complete, it generates a fourth write completion message and sends it to the scheduling node to inform the scheduling node that the fourth partition information has been written to the storage unit. The scheduling node generates partition load instructions for the merged partition, third partition unload instructions for the third partition, and fourth partition unload instructions for the fourth partition based on the third and fourth write completion messages. The scheduling node sends the partition load instructions to the merged partition to load it. The scheduling node also sends the third partition unload instructions to the third partition and the fourth partition unload instructions to the fourth partition to unload them, ensuring the successful execution of the merge of the third and fourth partitions, improving the efficiency of the merge, and ensuring data consistency.

[0090] Furthermore, after receiving the partition loading instruction, the merged partition 240 can enter the data preparation stage and write merged partition status information to the storage unit, indicating that the merged partition 240 takes over from the third partition 220 and the fourth partition 230 to process the tasks in the task set of the third partition and the task set of the fourth partition. In specific implementation, the merged partition 240 is used to determine the merged partition identifier and the merged task set information, and stores the merged partition identifier and the merged task set information as the merged partition status in the storage unit.

[0091] Specifically, the merge partition identifier is an identification information representing the merge partition ID. This identifier is stored in the storage unit when the merge partition is loaded, indicating that the merge partition has been loaded. The merge task set information records the number of tasks and task information in the task set assigned to the merge partition.

[0092] Based on this, after the merged partition receives the partition loading instruction sent by the scheduling node, the merged partition determines its own merged partition identifier and merged task set information, and stores the merged partition identifier and the merged task set information corresponding to the allocated merged task set as the merged partition status in the storage unit, thereby indicating that the merged partition has been loaded and can execute tasks subsequently.

[0093] Following the previous example, after the merged partition receives the partition loading instruction sent by the scheduling node, it can determine its own session ID and computation status information (task set information), write the session ID and computation status information into the storage unit, and complete the data preparation before task processing.

[0094] In summary, merging partitions stores the merge partition identifier and merge task set information as the merge partition status in the storage unit. Subsequently, tasks from the merge task set stored in the storage unit can be executed directly, ensuring data consistency during subsequent task processing.

[0095] Furthermore, during the merging of the third partition 220 and the fourth partition 230 into a merged partition 240, and during the execution of tasks on the merged partition 240, the results of the merged task execution may become unreliable or unusable due to factors such as task scheduling, computing environment, and task version. This manifests as a change in the merged partition identifier in the storage unit. Therefore, before storing the merged task execution results in the storage unit, the merged partition identifier needs to be checked. Specifically, the merged partition 240 is also used to execute the merged tasks contained in the merged task set to obtain the merged task execution results. If the merged partition identifier contained in the merged partition status remains unchanged, the merged task execution results are stored in the storage unit, and the merged partition status is also stored in the storage unit. If the merged partition identifier contained in the merged partition status changes, the merged task execution results are discarded.

[0096] Specifically, after determining the set of merge tasks, the merge partition can execute the merge tasks contained in the set and obtain the merge task execution results. Before writing the merge task execution results to the storage unit, it is necessary to determine whether the merge partition identifier contained in the merge partition status stored in the storage unit is the same as the target merge partition identifier of the current merge partition. If the merge partition identifier stored in the storage unit is the same as the target merge partition identifier, it means that the merge partition identifier has not changed, and the merge task execution results and the merge partition status can be stored in the storage unit, indicating that the tasks in the merge task set have been processed. If the merge partition identifier contained in the merge partition status has changed, it is determined that the merge partition is non-compliant, and the merge task execution results obtained from processing the tasks in the merge task set are unreliable. There may be other concurrent operations that have taken over the merge partition's tasks, or the calculation state of the merge partition may have been reset for some reason. In this case, the merge task execution results need to be discarded, and the merge partition will also wait to exit. When checking the merge partition identifier, atomic checks are performed. Atomic checks play an important role in the partition calculation process in ensuring data consistency and integrity and avoiding concurrent conflicts.

[0097] In summary, before storing the results of the merge task into the storage unit, an atomic check is performed on the merge partition identifier of the merged partition to ensure data consistency and completion.

[0098] Furthermore, after the third partition 220 and the fourth partition 230 are uninstalled, both the third partition 220 and the fourth partition 230 become unusable. The uninstallation status of the third partition 220 and the fourth partition 230 can be marked by changing the third partition identifier and the fourth partition identifier. In specific implementation, the merged partition 240 is also used to perform partition uninstallation update on the third partition identifier contained in the third partition status information of the third partition 220, and to perform partition uninstallation update on the fourth partition identifier contained in the fourth partition status information of the fourth partition 230.

[0099] Specifically, when updating the partition unload of the third partition's identifier (found in the third partition's status information), the third partition identifier can be changed to a special identifier indicating that the third partition is unloaded, thus achieving an unload mark for the third partition. Similarly, when updating the partition unload of the fourth partition's identifier (found in the fourth partition's status information), the fourth partition identifier can be changed to a special identifier indicating that the fourth partition is unloaded, thus achieving an unload mark for the fourth partition.

[0100] The following is in conjunction with the appendix Figure 3 Taking the partitioning method provided in this specification as an example in the application of partition splitting, the partitioning method will be further explained. Figure 3This specification illustrates a partition splitting interaction diagram of a partition processing system provided in one embodiment, which specifically includes the following steps.

[0101] Step 302: The partition scheduling node sends a partition split instruction to the primary partition to split the primary partition.

[0102] Step 304: The primary partition performs a partition split check to see if it meets the conditions for splitting.

[0103] Step 306: If the primary partition is determined to meet the conditions for splitting, write the splitting information for the first and second partitions in the storage unit.

[0104] Storage units can be key-value (KV) systems used to store data during partition splitting, partition merging, task execution, and other processes. The basic data unit of a storage unit is a row, and each row is located by a primary key, uniquely corresponding to a value. This value can be structured or unstructured.

[0105] Step 308: The primary partition sends a write completion message to the partition scheduling node.

[0106] Step 310: The partition scheduling node sends a partition unload command to the primary partition to unload the primary partition.

[0107] Step 312: The partition scheduling node sends a partition load command to the first partition to load the first partition.

[0108] Step 314: The partition scheduling node sends a partition load command to the second partition to load the second partition.

[0109] Step 316: The first partition obtains the split information corresponding to the first partition recorded in the storage unit, identifies the split process, and determines that it is in the split process.

[0110] Step 318: Obtain the first set of tasks stored in the storage unit of the first partition.

[0111] When obtaining the first task set in the first partition, taking the log stream set as an example, the task set of the main partition contains N log streams. The first partition inherits the 1st to N / 2th log stream, and the second partition inherits the (N / 2+1)th to Nth log stream. This splitting is pre-agreed and fixed. As long as the tasks obtained by the first partition and the second partition are equal to the task set, consistency can be guaranteed.

[0112] Step 320: The first partition updates the state of the first partition to the storage unit.

[0113] In practical applications, the status of the first partition includes its own session ID and computation status information (task set information). Simultaneously, the session ID of the primary partition needs to be updated to a completed status, indicating that the primary partition has been unloaded and is no longer used for task execution.

[0114] Step 322: The second partition obtains the split information corresponding to the second partition recorded in the storage unit, identifies the split process, and determines that it is in the split process.

[0115] Step 324: Obtain the second set of tasks stored in the storage unit of the second partition.

[0116] Step 326: The second partition updates the second partition status to the storage unit.

[0117] In practical applications, the status of the second partition includes its own session ID and computation status information (task set information). Simultaneously, the session ID of the primary partition needs to be updated to a completed status, indicating that the primary partition has been unloaded and is no longer used for task execution.

[0118] It should be noted that steps 316-320 for the first partition and steps 322-326 for the second partition can be executed synchronously.

[0119] During partition splitting, partition isolation is achieved solely through the atomic partitioning capabilities of general storage services, without relying on other middleware, thus broadening its applicability. The consistency of calculation results is strictly guaranteed during partition splitting, making it suitable for services with high data accuracy requirements.

[0120] As can be seen from the above primary partition splitting process, the entire splitting process is mainly divided into two stages (this is determined by the characteristics of the partition scheduling node), therefore the overall time consumption is mainly divided into four parts:

[0121] 1. The primary partition is checked to see if it can be split and the status information is prepared for the sub-partitions (first partition and second partition);

[0122] 2. The first and second partitions read the computational state information from the storage unit and initialize it respectively;

[0123] III. Time consumed in request communication between partitions and partition scheduling nodes;

[0124] IV. Data request communication time between partitions and storage units.

[0125] Since partition services and partition scheduling nodes are generally connected via an intranet, the time consumed in the third part is negligible. Similarly, the fourth part is also an intranet request, and the data size is usually very small (mostly status information), so its time consumption is also negligible. The first part mainly occurs in memory, and its time consumption is also almost negligible. The main time consumption in the second part occurs in the initialization of the local task state based on task information, and this part may vary in length depending on the characteristics of each task. Therefore, the overall time consumption in the splitting process is mainly the time spent initializing the local task state of the sub-partitions. This part of the time is shifted to the task calculation step by making it asynchronous. The final effect is that the overall time consumption when processing splitting requests is very short (usually less than 1 second). The partition merging process is similar and will not be described in detail here.

[0126] The following is in conjunction with the appendix Figure 4 Taking the partition processing method provided in this specification in the application of partition merging as an example, the partition processing method will be further explained. Among other things, Figure 4 This specification illustrates a partition merging interaction diagram of a partition processing system according to an embodiment of the present specification, which specifically includes the following steps.

[0127] Step 402: The partition scheduling node sends a partition merge instruction to the third partition to inform the third partition that a partition merge is about to take place.

[0128] Step 404: The partition scheduling node sends a partition merge instruction to the fourth partition to inform the fourth partition that a partition merge is about to take place.

[0129] Step 406: The third partition responds to the partition merge instruction by writing merge information to the storage unit.

[0130] After receiving the partition merge instruction, the third partition writes its own computation status information (task set 1 information) into the storage unit.

[0131] Step 408: The fourth partition responds to the partition merge instruction by writing merge information to the storage unit.

[0132] After receiving the partition merge instruction, the fourth partition writes its own computation status information (task set 2 information) into the storage unit.

[0133] Step 410: The third partition sends the write completion information to the partition scheduling node, informing the partition scheduling node that the first stage of processing is complete.

[0134] Step 412: The fourth partition sends the write completion information to the partition scheduling node, informing the partition scheduling node that the first stage of processing is complete.

[0135] Step 414: The partition scheduling node sends a partition loading instruction to the merged partition to load the merged partition.

[0136] Step 416: The partition scheduling node sends a partition unload command to the third partition to unload the third partition.

[0137] Step 418: The partition scheduling node sends a partition unload command to the fourth partition to unload the fourth partition.

[0138] Step 420: After receiving the partition load instruction, the merged partition obtains the merge information from the storage unit and recognizes that it is in the merge process.

[0139] Step 422: Initialize the merge task set for merging partitions.

[0140] In practical applications, merging partitions can obtain task set 1 and task set 2 from the parent partitions (the third and fourth partitions) according to a pre-defined method. Taking log stream sets as an example, assuming task set 1 contains N log streams and task set 2 contains M log streams, the merged partition inherits these N+M log streams.

[0141] Step 424: Update the merged partition status to the storage unit.

[0142] The merged partition status includes its own session ID and computation status information (task set information). Simultaneously, the session IDs of the third and fourth partitions need to be updated to a completed status (changed to a special ID), indicating that the third and fourth partitions have been unloaded and are no longer used for task execution.

[0143] During the partition merging process, partition isolation is achieved solely through the atomic partitioning capabilities of general storage services, without relying on other middleware, thus broadening its applicability. The consistency of calculation results is strictly guaranteed during the partition merging process, making it suitable for services with high data accuracy requirements.

[0144] Corresponding to the above system embodiments, this specification also provides embodiments of partitioning processing methods. Figure 5 A flowchart illustrating a partitioning method provided in one embodiment of this specification is shown. Figure 5 As shown, the partitioning methods include:

[0145] Step 502: In response to the partition splitting instruction sent by the scheduling node, the primary partition writes the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into the storage unit, and the task set of the primary partition is stored in the storage unit;

[0146] Step 504: In response to the first partition loading instruction sent by the scheduling node, the first partition determines a first task set in the task set based on the first partition information, and stores the first partition state in the storage unit;

[0147] Step 506: In response to the second partition loading instruction sent by the scheduling node, the second partition determines the second task set in the task set based on the second partition information, and stores the second partition state in the storage unit.

[0148] Optionally, the primary partition is configured to respond to a partition splitting instruction sent by the scheduling node, perform partition condition detection on the task set, and if the task set detection passes, write the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into the storage unit.

[0149] Optionally, the primary partition is further configured to generate write completion information and send the write completion information to the scheduling node; the scheduling node is configured to generate a partition unload instruction corresponding to the primary partition, a first partition load instruction corresponding to the first partition, and a second partition load instruction corresponding to the second partition based on the write completion information; and send the partition unload instruction to the primary partition, the first partition load instruction to the first partition, and the second partition load instruction to the second partition.

[0150] Optionally, the first partition is used to determine a first partition identifier and first task set information, and store the first partition identifier and first task set information as the first partition state in the storage unit; the second partition is used to determine a second partition identifier and second task set information, and store the second partition identifier and second task set information as the second partition state in the storage unit.

[0151] Optionally, the first partition is further configured to execute a first task contained in the first task set to obtain a first task execution result; if the first partition identifier contained in the first partition state remains unchanged, the first task execution result is stored in the storage unit, and the first partition state is stored in the storage unit; if the first partition identifier contained in the first partition state changes, the first task execution result is discarded; the second partition is further configured to execute a second task contained in the second task set to obtain a second task execution result; if the second partition identifier contained in the second partition state remains unchanged, the second task execution result is stored in the storage unit, and the second partition calculation state is stored in the storage unit; if the second partition identifier contained in the second partition state changes, the second task execution result is discarded.

[0152] This specification provides a partitioning method in one embodiment. A scheduling node sends a partition splitting instruction to the primary partition to split the task set of the primary partition into two task sets, which are then processed by a first partition and a second partition, respectively. Upon receiving the partition splitting instruction from the scheduling node, the primary partition writes the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into a storage unit. The scheduling node sends a first partition loading instruction and a second partition loading instruction to the first and second partitions, respectively, to load the first and second partitions. Upon receiving the first partition loading instruction from the scheduling node, the first partition determines its first task set to be processed from the task set based on the first partition information and stores the first partition state in the storage unit, completing the loading of the first partition. Upon receiving the second partition loading instruction from the scheduling node, the second partition determines its second task set to be processed from the task set based on the second partition information and stores the second partition state in the storage unit, completing the loading of the second partition, thus achieving partition isolation between the first and second partitions. After the first and second partitions determine their first and second task sets, respectively, they store their first and second partition states in the storage unit. This partition splitting of the primary partition can be completed without the use of plugins. After determining the first and second partitions, the task set of the primary partition is split, which can improve the efficiency of partition splitting, ensure data consistency during the partition splitting process and subsequent task execution, and adapt to services with high data accuracy requirements.

[0153] The above is an illustrative scheme of a partitioning method according to this embodiment. It should be noted that the technical solution of this partitioning method and the technical solution of the partitioning system described above belong to the same concept. For details not described in detail in the technical solution of the partitioning method, please refer to the description of the technical solution of the partitioning system described above.

[0154] Corresponding to the above method embodiments, this specification also provides embodiments of a partitioning processing device. Figure 6 A schematic diagram of a partitioning processing apparatus according to one embodiment of this specification is shown. Figure 6 As shown, the device includes:

[0155] The write module 602 is configured to respond to the partition splitting instruction sent by the scheduling node in the primary partition and write the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into the storage unit. The task set of the primary partition is stored in the storage unit.

[0156] The first storage module 604 is configured to respond to the first partition loading instruction sent by the scheduling node, determine the first task set in the task set based on the first partition information, and store the first partition status in the storage unit.

[0157] The second storage module 606 is configured to, in response to the second partition loading instruction sent by the scheduling node, determine the second task set in the task set based on the second partition information, and store the second partition state in the storage unit.

[0158] In an optional embodiment, the writing module 602 is further configured to:

[0159] In response to the partition splitting instruction sent by the scheduling node, the primary partition performs partition condition detection on the task set. If the detection passes, it writes the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into the storage unit.

[0160] In an optional embodiment, the writing module 602 is further configured to:

[0161] The primary partition generates write completion information and sends the write completion information to the scheduling node; the scheduling node is used to generate a partition unload instruction corresponding to the primary partition, a first partition load instruction corresponding to the first partition, and a second partition load instruction corresponding to the second partition based on the write completion information; and sends the partition unload instruction to the primary partition, the first partition load instruction to the first partition, and the second partition load instruction to the second partition.

[0162] In an optional embodiment, the second storage module 606 is further configured to:

[0163] The first partition determines a first partition identifier and first task set information, and stores the first partition identifier and first task set information as the first partition state in the storage unit; the second partition determines a second partition identifier and second task set information, and stores the second partition identifier and second task set information as the second partition state in the storage unit.

[0164] In an optional embodiment, the second storage module 606 is further configured to:

[0165] The first partition executes the first task contained in the first task set and obtains the first task execution result; if the first partition identifier contained in the first partition state remains unchanged, the first task execution result is stored in the storage unit, and the first partition state is stored in the storage unit; if the first partition identifier contained in the first partition state changes, the first task execution result is discarded. The second partition executes the second task contained in the second task set and obtains the second task execution result; if the second partition identifier contained in the second partition state remains unchanged, the second task execution result is stored in the storage unit, and the second partition calculation state is stored in the storage unit; if the second partition identifier contained in the second partition state changes, the second task execution result is discarded.

[0166] This specification provides a partitioning processing apparatus in one embodiment. A scheduling node sends a partition splitting instruction to the primary partition to split the task set of the primary partition into two task sets, which are then processed by a first partition and a second partition, respectively. Upon receiving the partition splitting instruction from the scheduling node, the primary partition writes the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into a storage unit. The scheduling node sends a first partition loading instruction and a second partition loading instruction to the first and second partitions, respectively, to load the first and second partitions. Upon receiving the first partition loading instruction from the scheduling node, the first partition determines its first task set to be processed from the task set based on the first partition information and stores the first partition state in the storage unit, completing the loading of the first partition. Upon receiving the second partition loading instruction from the scheduling node, the second partition determines its second task set to be processed from the task set based on the second partition information and stores the second partition state in the storage unit, completing the loading of the second partition, thus achieving partition isolation between the first and second partitions. After the first and second partitions determine their first and second task sets, respectively, they store their first and second partition states in the storage unit. Partition splitting of the primary partition can be completed without the use of plugins. After determining the first and second partitions, the task set of the primary partition is split, which can improve the efficiency of partition splitting, ensure data consistency during the partition splitting process and subsequent task execution, and adapt to services with high data accuracy requirements.

[0167] The above is a schematic scheme of a partitioning processing device according to this embodiment. It should be noted that the technical solution of this partitioning processing device and the technical solution of the partitioning processing method described above belong to the same concept. For details not described in detail in the technical solution of the partitioning processing device, please refer to the description of the technical solution of the partitioning processing method described above.

[0168] Corresponding to the above system embodiments, this specification also provides another embodiment of the partitioning processing method. Figure 7 A flowchart illustrating another partitioning method provided in one embodiment of this specification is shown. Figure 7 As shown, the partitioning methods include:

[0169] Step 702: In response to the partition merge instruction sent by the scheduling node, the third partition writes the third partition information into the storage unit, and the third task set of the third partition is stored in the storage unit;

[0170] Step 704: In response to the partition merge instruction sent by the scheduling node, the fourth partition writes the fourth partition information into the storage unit, and the fourth task set of the fourth partition is stored in the storage unit;

[0171] Step 706: In response to the partition loading instruction sent by the scheduling node, the merging partitions obtain the third partition information and the fourth partition information in the storage unit, and determine the third task set and the fourth task set in the storage unit based on the third partition information; merge the third task set and the fourth task set into a merged task set in the storage unit, and store the merged partition status in the storage unit.

[0172] Optionally, after writing the third partition information into the storage unit, the third partition sends a third write completion message to the scheduling node; the fourth partition is configured to send a fourth write completion message to the scheduling node after writing the fourth partition information into the storage unit; the scheduling node is configured to generate a partition load instruction corresponding to the merged partition, a third partition unload instruction corresponding to the third partition, and a fourth partition unload instruction corresponding to the fourth partition based on the third write completion message and the fourth write completion message; and send the partition load instruction to the merged partition, the third partition unload instruction to the third partition, and the fourth partition unload instruction to the fourth partition.

[0173] Optionally, the merge partition is used to determine the merge partition identifier and the merge task set information, and store the merge partition identifier and the merge task set information as the merge partition status in the storage unit.

[0174] Optionally, the merge partition is further configured to execute the merge tasks contained in the merge task set and obtain the merge task execution results; if the merge partition identifier contained in the merge partition status has not changed, the merge task execution results are stored in the storage unit and the merge partition status is stored in the storage unit; if the merge partition identifier contained in the merge partition status has changed, the merge task execution results are discarded.

[0175] Optionally, the merging of partitions is further configured to perform partition unloading and updating on the third partition identifier contained in the third partition status information of the third partition, and to perform partition unloading and updating on the fourth partition identifier contained in the fourth partition status information of the fourth partition.

[0176] This specification provides another partitioning method according to an embodiment, including a scheduling node, a third partition, a fourth partition, and a merged partition. The third task set of the third partition and the fourth task set of the fourth partition are both stored in a storage unit. After receiving a partition merge instruction from the scheduling node, the third partition writes its information to the storage unit. Similarly, the fourth partition writes its information to the storage unit after receiving the same instruction. Upon receiving a partition load instruction from the scheduling node, the merged partition retrieves the third and fourth partition information from the storage unit, determines the third task set based on the third partition information, and determines the fourth task set based on the fourth partition information. The third and fourth task sets are merged into a merged task set in the storage unit, and the merged partition state is stored therein. By storing the merged partition state in the storage unit, the merged partition takes over the third task set of the third partition and the fourth task set of the fourth partition, achieving logical isolation between partitions and ensuring data consistency during the partition merge process and subsequent task execution.

[0177] The above is an illustrative scheme of another partitioning method in this embodiment. It should be noted that the technical solution of this partitioning method and the technical solution of the other partitioning system described above belong to the same concept. For details not described in detail in the technical solution of the other partitioning method, please refer to the description of the technical solution of the other partitioning system described above.

[0178] Corresponding to the above method embodiments, this specification also provides embodiments of a partitioning processing device. Figure 8 A schematic diagram of another partitioning processing apparatus provided in one embodiment of this specification is shown. Figure 8 As shown, the device includes:

[0179] The third storage module 802 is configured to write the third partition information into the storage unit in response to the partition merging instruction sent by the scheduling node, and the third task set of the third partition is stored in the storage unit.

[0180] The fourth storage module 804 is configured to write the fourth partition information into the storage unit in response to the partition merging instruction sent by the scheduling node, and the fourth task set of the fourth partition is stored in the storage unit.

[0181] The determination module 806 is configured to merge partitions in response to a partition loading instruction sent by the scheduling node, obtain the third partition information and the fourth partition information in the storage unit, determine the third task set in the storage unit based on the third partition information, and determine the fourth task set in the storage unit based on the fourth partition information; merge the third task set and the fourth task set into a merged task set in the storage unit, and store the merged partition status in the storage unit.

[0182] In an optional embodiment, the fourth storage module 804 is further configured to:

[0183] After writing the information of the third partition into the storage unit, the third partition sends the third write completion information to the scheduling node.

[0184] After writing the fourth partition information into the storage unit, the fourth partition sends the fourth write completion information to the scheduling node.

[0185] The scheduling node generates a partition loading instruction corresponding to the merged partition, a third partition unloading instruction corresponding to the third partition, and a fourth partition unloading instruction corresponding to the fourth partition based on the third write completion information and the fourth write completion information; it sends the partition loading instruction to the merged partition, the third partition unloading instruction to the third partition, and the fourth partition unloading instruction to the fourth partition.

[0186] In an optional embodiment, the determining module 806 is further configured to:

[0187] The merge partition determines the merge partition identifier and merge task set information, and stores the merge partition identifier and merge task set information as the merge partition status in the storage unit.

[0188] In an optional embodiment, the determining module 806 is further configured to:

[0189] The merged partition is also used to execute the merge tasks contained in the merge task set and obtain the merge task execution results; if the merge partition identifier contained in the merged partition status has not changed, the merge task execution results are stored in the storage unit and the merged partition status is stored in the storage unit; if the merge partition identifier contained in the merged partition status has changed, the merge task execution results are discarded.

[0190] In an optional embodiment, the determining module 806 is further configured to:

[0191] The merged partition is also used to perform partition unload update on the third partition identifier contained in the third partition status information of the third partition, and to perform partition unload update on the fourth partition identifier contained in the fourth partition status information of the fourth partition.

[0192] This specification provides another partitioning processing apparatus according to one embodiment, including a scheduling node, a third partition, a fourth partition, and a merging partition. The third task set of the third partition and the fourth task set of the fourth partition are both stored in a storage unit. After receiving a partition merging instruction from the scheduling node, the third partition writes its information into the storage unit. Similarly, the fourth partition writes its information into the storage unit after receiving the same instruction. Upon receiving a partition loading instruction from the scheduling node, the merging partition retrieves the third and fourth partition information from the storage unit, determines the third task set based on the third partition information, and determines the fourth task set based on the fourth partition information. The third and fourth task sets are merged into a merged task set in the storage unit, and the merged partition state is stored therein. By storing the merged partition state in the storage unit, the merging partition takes over the third task set of the third partition and the fourth task set of the fourth partition, achieving logical isolation between partitions and ensuring data consistency during the partition merging process and subsequent task execution.

[0193] The above is an illustrative scheme of another partitioning processing device in this embodiment. It should be noted that the technical solution of this partitioning processing device and the technical solution of the other partitioning processing method described above belong to the same concept. For details not described in detail in the technical solution of the other partitioning processing device, please refer to the description of the technical solution of the other partitioning processing method described above.

[0194] Figure 9 A structural block diagram of a computing device 900 according to one embodiment of this specification is shown. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0195] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 940 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0196] In one embodiment of this specification, the above-described components of the computing device 900 and Figure 9 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 9 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0197] The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 900 can also be a mobile or stationary server.

[0198] The processor 920 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described partitioning method.

[0199] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the partitioning processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the partitioning processing method described above.

[0200] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the partitioning method described above.

[0201] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the partitioning processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the partitioning processing method described above.

[0202] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the partitioning processing method described above.

[0203] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the partitioning processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the partitioning processing method described above.

[0204] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0205] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0206] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0207] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0208] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A partitioning system, comprising: The system includes a scheduling node, a primary partition, a first partition, and a second partition, with the task set of the primary partition stored in a storage unit. The primary partition is used to respond to the partition splitting instruction sent by the scheduling node by writing the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into the storage unit. The first partition is used to respond to the first partition loading instruction sent by the scheduling node, determine the first task set in the task set based on the first partition information, and store the first partition status in the storage unit; The second partition is used to respond to the second partition loading instruction sent by the scheduling node, determine the second task set in the task set based on the second partition information, and store the second partition state in the storage unit.

2. The partitioning system as described in claim 1, wherein the primary partition is configured to, in response to a partition splitting instruction sent by the scheduling node, perform partition condition detection on the task set, and, if the task set detection passes, write the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into the storage unit.

3. The partition processing system as described in claim 1, wherein the primary partition is further configured to generate write completion information and send the write completion information to the scheduling node; The scheduling node is used to generate, based on the write completion information, a partition unload instruction corresponding to the primary partition, a first partition load instruction corresponding to the first partition, and a second partition load instruction corresponding to the second partition; The partition unload command is sent to the primary partition, the first partition load command is sent to the first partition, and the second partition load command is sent to the second partition.

4. The partitioning system as described in claim 1, wherein the first partition is used to determine a first partition identifier and first task set information, and to store the first partition identifier and the first task set information as the first partition status in the storage unit; The second partition is used to determine the second partition identifier and the second task set information, and to store the second partition identifier and the second task set information as the second partition status in the storage unit.

5. The partition processing system as described in claim 1, wherein the first partition is further configured to execute a first task contained in the first task set to obtain a first task execution result; if the first partition identifier contained in the first partition state remains unchanged, the first task execution result is stored in the storage unit, and the first partition state is stored in the storage unit; if the first partition identifier contained in the first partition state changes, the first task execution result is discarded. The second partition is further configured to execute the second task contained in the second task set and obtain the execution result of the second task; if the second partition identifier contained in the second partition state has not changed, the execution result of the second task is stored in the storage unit and the calculation state of the second partition is stored in the storage unit; if the second partition identifier contained in the second partition state has changed, the execution result of the second task is discarded.

6. A partitioning system, comprising: The scheduling node, the third partition, the fourth partition, and the merged partition, with the third task set of the third partition and the fourth task set of the fourth partition all stored in the storage unit; The third partition is used to write the third partition information into the storage unit in response to the partition merging instruction sent by the scheduling node; The fourth partition is used to write the fourth partition information into the storage unit in response to the partition merging instruction sent by the scheduling node; The merged partition is used to respond to the partition loading instruction sent by the scheduling node, obtain the third partition information and the fourth partition information in the storage unit, determine the third task set in the storage unit based on the third partition information, and determine the fourth task set in the storage unit based on the fourth partition information. The third task set and the fourth task set are merged into a merged task set in the storage unit, and the merged partition status is stored in the storage unit.

7. The partition processing system as described in claim 6, wherein the third partition is configured to send third write completion information to the scheduling node after the third partition information is written to the storage unit; The fourth partition is used to send fourth write completion information to the scheduling node after the fourth partition information is written to the storage unit. The scheduling node is configured to generate a partition loading instruction corresponding to the merged partition, a third partition unloading instruction corresponding to the third partition, and a fourth partition unloading instruction corresponding to the fourth partition based on the third write completion information and the fourth write completion information; and send the partition loading instruction to the merged partition, the third partition unloading instruction to the third partition, and the fourth partition unloading instruction to the fourth partition.

8. The partition processing system as described in claim 6, wherein the merge partition is used to determine the merge partition identifier and the merge task set information, and to store the merge partition identifier and the merge task set information as the merge partition status in the storage unit.

9. The partition processing system as described in claim 6, wherein the merging partition is further configured to execute merging tasks contained in the merging task set to obtain merging task execution results; if the merging partition identifier contained in the merging partition status remains unchanged, the merging task execution results are stored in the storage unit, and the merging partition status is stored in the storage unit; if the merging partition identifier contained in the merging partition status changes, the merging task execution results are discarded.

10. The partition processing system as described in claim 6, wherein merging partitions is further configured to perform partition unloading update on the third partition identifier contained in the third partition status information of the third partition, and to perform partition unloading update on the fourth partition identifier contained in the fourth partition status information of the fourth partition.

11. A partitioning method, comprising: In response to the partition splitting instruction sent by the scheduling node, the primary partition writes the first partition information corresponding to the first partition and the second partition information corresponding to the second partition into the storage unit, and the task set of the primary partition is stored in the storage unit. The first partition responds to the first partition loading instruction sent by the scheduling node, determines the first task set in the task set based on the first partition information, and stores the first partition state in the storage unit; The second partition responds to the second partition loading instruction sent by the scheduling node, determines the second task set in the task set based on the second partition information, and stores the second partition state in the storage unit.

12. A partitioning method, comprising: In response to the partition merging instruction sent by the scheduling node, the third partition writes the third partition information into the storage unit, and the third task set of the third partition is stored in the storage unit. In response to the partition merge instruction sent by the scheduling node, the fourth partition writes the fourth partition information into the storage unit, and the fourth task set of the fourth partition is stored in the storage unit; In response to the partition loading instruction sent by the scheduling node, the merged partition obtains the third partition information and the fourth partition information in the storage unit, and determines the third task set in the storage unit based on the third partition information, and determines the fourth task set in the storage unit based on the fourth partition information. The third task set and the fourth task set are merged into a merged task set in the storage unit, and the merged partition status is stored in the storage unit.

13. A computing device, comprising: Memory and processor; The memory is used to store computer programs or instructions, and the processor is used to execute the computer programs or instructions, which, when executed by the processor, implement the steps of the method of claim 11 or 12.

14. A computer-readable storage medium storing a computer program or instructions that, when executed by a processor, implement the steps of the method of claim 11 or 12.

15. A computer program product comprising a computer program or instructions that, when executed by a processor, implement the steps of the method of claim 11 or 12.

Images