Method for realizing distributed fragmentation calculation based on Java

Abstract

The invention provides a method for realizing distributed fragmentation calculation based on Java, which comprises the following steps of: performing task initialization and scheduling server configuration to set execution conditions of tasks and the number of servers; establishing a long connection and heartbeat mechanism between the task execution server and the task scheduling server so as to ensure real-time communication and state monitoring between the servers; task allocation and task execution server number generation lay a foundation for parallel computing of fragmented tasks by dynamically selecting and numbering servers. According to the method for achieving distributed fragmentation calculation based on Java, the complexity of overall implementation of a system is reduced, execution servers are maintained through long connection, server numbers are dynamically appointed during task allocation and used for achieving fragmentation calculation and the like, and the requirement that some websites conduct fragmentation parallel calculation when some tasks are conducted can be met.

Description

Technical Field

[0001] The present invention belongs to the field of data processing, and in particular relates to a method for implementing distributed sharding computing based on Java. Background Art

[0002] With the rapid development of Internet applications, the backend server architecture of modern websites generally adopts distributed deployment to cope with the needs of massive data processing and high concurrent access. In actual business processing, especially when processing data in large database tables, it is often necessary to shard the data and process it in parallel by multiple servers to improve the overall data processing speed and efficiency. However, the existing distributed computing framework has problems such as high complexity and difficult maintenance when implementing data sharding computing. Summary of the invention

[0003] In view of this, the present invention aims to propose a method for implementing distributed sharding computing based on Java, which realizes data parallel processing in a multi-server environment through reasonable task allocation and resource management, thereby effectively improving data processing performance.

[0004] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0005] In a first aspect, the present solution discloses a method for implementing distributed sharding computing based on Java, comprising the following steps:

[0006] Perform task initialization and scheduling server configuration to set task execution conditions and server quantity;

[0007] Establish a long connection and heartbeat mechanism between the task execution server and the task scheduling server to ensure real-time communication and status monitoring between servers;

[0008] Task allocation and task execution server number generation, by dynamically selecting and numbering servers, lays the foundation for parallel computing of sharded tasks;

[0009] The task execution server performs data sharding and business calculations according to the number, realizes parallel processing through data distribution, and speeds up the overall execution of tasks;

[0010] Summarize the calculation results of each task execution server and complete the task. By centralizing the processing results, the integrity and final output of the task are ensured.

[0011] Carry out system optimization and exception handling, improve system performance and ensure smooth completion of tasks through monitoring and dynamic adjustment.

[0012] Furthermore, the task initialization and scheduling server configuration include:

[0013] Administrators set task execution conditions through the management platform, including data sharding strategies, task priorities, and resource configuration required for sharding calculations;

[0014] The administrator sets the number of task scheduling servers on the management platform to ensure that tasks are processed in parallel on multiple servers;

[0015] After the administrator completes the configuration, the task scheduling server is called through the management platform to send the initial information of the task to the scheduling server, including the conditions for task execution and the sharding computing strategy.

[0016] Furthermore, the establishment of a long connection and heartbeat mechanism between the task execution server and the task scheduling server includes:

[0017] After the task execution server is started, it first establishes a persistent connection with the task scheduling server to ensure that tasks can be scheduled between servers in real time;

[0018] The task execution server periodically sends heartbeat signals to the task scheduling server, which receives and records these signals to maintain a list of currently online task execution servers;

[0019] If the task execution server does not send a heartbeat within the specified time, the task scheduling server will mark the server as faulty or offline, and promptly update the task execution server list in the cache to ensure the stability of task execution.

[0020] Furthermore, the task allocation and task execution server number generation include:

[0021] After receiving the task instruction from the administrator, the task scheduling server selects an appropriate number of task execution servers from the cache according to the preset number of task scheduling servers;

[0022] The task scheduling server assigns a temporary number to each task execution server. The number increases from 0 and is used to identify the task slice number of each server.

[0023] The task scheduling server sends the task conditions, the assigned number, and the number of servers participating in the task execution to the selected task execution server through a persistent connection.

[0024] Furthermore, the task execution server performs data sharding and business calculation according to the serial number, including:

[0025] After receiving the task information from the scheduling server, the task execution server first shards the data according to the assigned number;

[0026] Specifically, the server performs a modulo operation on the data to be executed, and extracts the data fragment that matches its own number;

[0027] After obtaining the corresponding data fragments, the task execution server performs business calculations on the data according to the predetermined task logic;

[0028] Among them, the shard data calculation method of the task execution server is:

[0029] D i ={d j |(j mod N)=i};

[0030] in,

[0031] Di: the set of data shards to be processed by the server numbered i that executes the task;

[0032] dj: the jth data in the original data set to be processed;

[0033] N: the total number of task execution servers, that is, the number of servers that execute tasks in parallel;

[0034] i: the number of the server executing the current task, ranging from 0 to N-1;

[0035] j: The index of the data in the original data set, ranging from 0 to the total amount of data minus one.

[0036] Furthermore, the step of aggregating the calculation results of each task execution server and completing the task includes:

[0037] When all task execution servers complete their sharding tasks, the task scheduling server is responsible for collecting the processing results returned by each server and summarizing the results;

[0038] The task scheduling server returns the final summary results to the management platform, notifying the administrator that the task has been completed. At this time, the scheduling server will also clean up the related temporary task data and release resources.

[0039] Furthermore, the system optimization and exception handling include:

[0040] Monitor the load of the task execution server in real time during the task execution process, dynamically adjust the task allocation strategy, and ensure the optimal use of server resources;

[0041] If an exception occurs on a task execution server during execution, the scheduling server will ensure the successful completion of the task by reallocating the server's tasks to other available servers. At this time, the unfinished shard data will be reallocated and executed to avoid task failure.

[0042] In a second aspect, the present solution discloses an electronic device, comprising a processor and a memory that is communicatively connected to the processor and is used to store executable instructions of the processor, wherein the processor is used to execute a method for implementing distributed sharded computing based on Java as described in the first aspect above.

[0043] In the third aspect, the present solution discloses a server, comprising at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes the method for implementing distributed sharded computing based on Java as described in the first aspect.

[0044] In a fourth aspect, the present solution discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a Java-based distributed sharding computing method as described in the first aspect.

[0045] Compared with the prior art, the method for implementing distributed sharding computing based on Java described in the present invention has the following advantages:

[0046] The method for implementing distributed sharding computing based on Java described in the present invention reduces the complexity of the overall implementation of the system. By maintaining the execution server through long connections and dynamically specifying the server number for sharding computing when allocating tasks, it can meet the needs of some websites for performing sharding parallel computing when performing some tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0048] Figure 1 A schematic diagram of a method for implementing distributed sharding computing based on Java according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0050] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0051] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0052] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0053] In a first aspect, the present solution discloses a method for implementing distributed sharding computing based on Java, comprising the following steps:

[0054] Perform task initialization and scheduling server configuration to set task execution conditions and server quantity;

[0055] Establish a long connection and heartbeat mechanism between the task execution server and the task scheduling server to ensure real-time communication and status monitoring between servers;

[0056] Task allocation and task execution server number generation, by dynamically selecting and numbering servers, lays the foundation for parallel computing of sharded tasks;

[0057] The task execution server performs data sharding and business calculations according to the number, realizes parallel processing through data distribution, and speeds up the overall execution of tasks;

[0058] Summarize the calculation results of each task execution server and complete the task. By centralizing the processing results, the integrity and final output of the task are ensured.

[0059] Carry out system optimization and exception handling, improve system performance and ensure smooth completion of tasks through monitoring and dynamic adjustment.

[0060] Furthermore, the task initialization and scheduling server configuration include:

[0061] Administrators set task execution conditions through the management platform, including data sharding strategies, task priorities, and resource configuration required for sharding calculations;

[0062] The administrator sets the number of task scheduling servers on the management platform to ensure that tasks are processed in parallel on multiple servers;

[0063] After the administrator completes the configuration, the task scheduling server is called through the management platform to send the initial information of the task to the scheduling server, including the conditions for task execution and the sharding computing strategy.

[0064] Furthermore, the establishment of a long connection and heartbeat mechanism between the task execution server and the task scheduling server includes:

[0065] After the task execution server is started, it first establishes a persistent connection with the task scheduling server to ensure that tasks can be scheduled between servers in real time;

[0066] The task execution server periodically sends heartbeat signals to the task scheduling server, which receives and records these signals to maintain a list of currently online task execution servers;

[0067] If the task execution server does not send a heartbeat within the specified time, the task scheduling server will mark the server as faulty or offline, and promptly update the task execution server list in the cache to ensure the stability of task execution.

[0068] Furthermore, the task allocation and task execution server number generation include:

[0069] After receiving the task instruction from the administrator, the task scheduling server selects an appropriate number of task execution servers from the cache according to the preset number of task scheduling servers;

[0070] The task scheduling server assigns a temporary number to each task execution server. The number increases from 0 and is used to identify the task slice number of each server.

[0071] The task scheduling server sends the task conditions, the assigned number, and the number of servers participating in the task execution to the selected task execution server through a persistent connection.

[0072] Furthermore, the task execution server performs data sharding and business calculation according to the serial number, including:

[0073] After receiving the task information from the scheduling server, the task execution server first shards the data according to the assigned number;

[0074] Specifically, the server performs a modulo operation on the data to be executed, and extracts the data fragment that matches its own number;

[0075] After obtaining the corresponding data fragments, the task execution server performs business calculations on the data according to the predetermined task logic;

[0076] Among them, the shard data calculation method of the task execution server is:

[0077] D i ={d j |(j mod N)=i};

[0078] in,

[0079] Di: the set of data shards to be processed by the server numbered i that executes the task;

[0080] dj: the jth data in the original data set to be processed;

[0081] N: the total number of task execution servers, that is, the number of servers that execute tasks in parallel;

[0082] i: the number of the server executing the current task, ranging from 0 to N-1;

[0083] j: The index of the data in the original data set, ranging from 0 to the total amount of data minus one.

[0084] The core idea of ​​the formula involved in the shard data calculation method of the task execution server is to determine which task execution server Di will process the data dj by performing a modulo operation (jmodN) on the index j of the data.

[0085] When the data index j is modulo the total number of servers N, if the result is equal to the number i of the current server, then the data dj is processed by the task execution server numbered i;

[0086] In this way, data can be evenly distributed to different task execution servers, thus achieving parallel processing.

[0087] Examples:

[0088] Assume that there is a data set containing 100 data items numbered from 0 to 99, and there are 5 task execution servers in the system (i.e., N=5), then:

[0089] Server D0 will process data: j satisfies jmod5=0, that is, D0 processes d0, d5, d1 0,….

[0090] Server D1 will process data: j satisfies jmod5=1, that is, D1 processes d1, d6, d 1 1,….

[0091] And so on, until all data has been allocated.

[0092] Furthermore, the step of aggregating the calculation results of each task execution server and completing the task includes:

[0093] When all task execution servers complete their sharding tasks, the task scheduling server is responsible for collecting the processing results returned by each server and summarizing the results;

[0094] The task scheduling server returns the final summary results to the management platform, notifying the administrator that the task has been completed. At this time, the scheduling server will also clean up the related temporary task data and release resources.

[0095] Furthermore, the system optimization and exception handling include:

[0096] Monitor the load of the task execution server in real time during the task execution process, dynamically adjust the task allocation strategy, and ensure the optimal use of server resources;

[0097] If an exception occurs on a task execution server during execution, the scheduling server will ensure the successful completion of the task by reallocating the server's tasks to other available servers. At this time, the unfinished shard data will be reallocated and executed to avoid task failure.

[0098] In a second aspect, the present solution discloses an electronic device, comprising a processor and a memory that is communicatively connected to the processor and is used to store executable instructions of the processor, wherein the processor is used to execute a method for implementing distributed sharded computing based on Java as described in the first aspect above.

[0099] In the third aspect, the present solution discloses a server, comprising at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes the method for implementing distributed sharded computing based on Java as described in the first aspect.

[0100] In a fourth aspect, the present solution discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a Java-based distributed sharding computing method as described in the first aspect.

[0101] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for implementing distributed sharding computing based on Java, characterized in that: The following steps are involved: Perform task initialization and scheduling server configuration to set task execution conditions and server quantity; Establish a long connection and heartbeat mechanism between the task execution server and the task scheduling server to ensure real-time communication and status monitoring between servers; Task allocation and task execution server number generation, by dynamically selecting and numbering servers, lays the foundation for parallel computing of sharded tasks; The task execution server performs data sharding and business calculations according to the number, realizes parallel processing through data distribution, and speeds up the overall execution of tasks; Summarize the calculation results of each task execution server and complete the task. By centralizing the processing results, the integrity and final output of the task are ensured. Carry out system optimization and exception handling, improve system performance and ensure smooth completion of tasks through monitoring and dynamic adjustment.

2. According to a method for implementing distributed sharding computing based on Java in claim 1, it is characterized in that: The task initialization and scheduling server configuration include: Administrators set task execution conditions through the management platform, including data sharding strategies, task priorities, and resource configuration required for sharding calculations; The administrator sets the number of task scheduling servers on the management platform to ensure that tasks are processed in parallel on multiple servers; After the administrator completes the configuration, the task scheduling server is called through the management platform to send the initial information of the task to the scheduling server, including the conditions for task execution and the sharding computing strategy.

3. According to the method for implementing distributed sharding computing based on Java in claim 1, it is characterized in that: The establishment of a long connection and heartbeat mechanism between the task execution server and the task scheduling server includes: After the task execution server is started, it first establishes a persistent connection with the task scheduling server to ensure that tasks can be scheduled between servers in real time; The task execution server periodically sends heartbeat signals to the task scheduling server, which receives and records these signals to maintain a list of currently online task execution servers; If the task execution server does not send a heartbeat within the specified time, the task scheduling server will mark the server as faulty or offline, and promptly update the task execution server list in the cache to ensure the stability of task execution.

4. The method for implementing distributed sharding computing based on Java according to claim 1, characterized in that: The task allocation and task execution server number generation includes: After receiving the task instruction from the administrator, the task scheduling server selects an appropriate number of task execution servers from the cache according to the preset number of task scheduling servers; The task scheduling server assigns a temporary number to each task execution server. The number increases from 0 and is used to identify the task slice number of each server. The task scheduling server sends the task conditions, the assigned number, and the number of servers participating in the task execution to the selected task execution server through a persistent connection.

5. The method for implementing distributed sharding computing based on Java according to claim 1, characterized in that: The task execution server performs data sharding and business calculation according to the number, including: After receiving the task information from the scheduling server, the task execution server first shards the data according to the assigned number; Specifically, the server performs a modulo operation on the data to be executed, and extracts the data fragment that matches its own number; After obtaining the corresponding data fragments, the task execution server performs business calculations on the data according to the predetermined task logic; Among them, the shard data calculation method of the task execution server is: D i ={d j |(j mod N)=i}; in, Di: the set of data shards to be processed by the server numbered i that executes the task; dj: the jth data in the original data set to be processed; N: the total number of task execution servers, that is, the number of servers that execute tasks in parallel; i: the number of the server executing the current task, ranging from 0 to N-1; j: The index of the data in the original data set, ranging from 0 to the total amount of data minus one.

6. The method for implementing distributed sharding computing based on Java according to claim 1, characterized in that: The step of aggregating the calculation results of each task execution server and completing the task includes: When all task execution servers complete their sharding tasks, the task scheduling server is responsible for collecting the processing results returned by each server and summarizing the results; The task scheduling server returns the final summary results to the management platform, notifying the administrator that the task has been completed. At this time, the scheduling server will also clean up the related temporary task data and release resources.

7. The method for implementing distributed sharding computing based on Java according to claim 1, characterized in that: The system optimization and exception handling include: Monitor the load of the task execution server in real time during the task execution process, dynamically adjust the task allocation strategy, and ensure the optimal use of server resources; If an exception occurs on a task execution server during execution, the scheduling server will ensure the successful completion of the task by reallocating the server's tasks to other available servers. At this time, the unfinished shard data will be reallocated and executed to avoid task failure.

8. An electronic device, comprising a processor and a memory connected to the processor for storing instructions executable by the processor, characterized in that: The processor is used to execute a method for implementing distributed sharding computing based on Java as described in any one of claims 1 to 7.

9. A server, characterized in that: It includes at least one processor and a memory that is communicatively connected to the processor, wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes a method for implementing distributed sharded computing based on Java as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for implementing distributed sharding computing based on Java as described in any one of claims 1 to 7 is implemented.

Images