Distributed job processing system and method, and distributed container node collaboration device
By using a distributed job processing system and container node collaboration device, and employing independent clocks and random competition strategies, deadlock and scalability issues in collaborative processing within a distributed system are resolved, achieving efficient and stable job processing.
Patent Information
- Application Number
- CN202510504804.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-21
- Publication Date
- 2025-11-07
AI Technical Summary
Existing distributed systems suffer from deadlocks, performance bottlenecks, increased complexity, and difficulties in maintaining state consistency during collaborative processing, which affect scalability and stability.
A distributed job processing system is adopted, which utilizes multiple nodes, distributed container nodes, and shared memory. Through the collaborative work of independent clocks, job initiation modules, job delivery modules, and job execution modules, combined with random latency and contention strategies, the efficient allocation and execution of job processing requests are achieved.
It improves the system's coordination efficiency and stability, reduces the risk of deadlock, optimizes resource utilization, and enhances the system's scalability and reliability.
Smart Images
Figure CN120909711A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to distributed systems and containerization technology, in particular to a distributed job processing system and method, and a distributed container node coordination device. BACKGROUND
[0002] The cross-border RMB payment system (CIPS) access project uses a cloud native architecture based on container technology to achieve agile deployment and elastic scaling of application services. Container technology is lightweight, highly portable, and fully utilizes resources. However, in a cloud native stateless container environment, especially in a distributed multi-machine collaboration problem that uses a unified solution to address various demand scenarios, it becomes a major challenge for application development.
[0003] The current distributed multi-machine collaboration method and existing problems are as follows:
[0004] (1) Synchronization lock-based collaboration method: distributed locks are used to ensure mutual exclusion of tasks, thereby avoiding concurrent conflicts. However, this method can cause deadlock problems, and due to resource competition for locks, it can also cause system performance bottlenecks. In a high-concurrency environment, the overhead of locks can easily become a performance bottleneck, and the complexity of lock management increases when the system is expanded, limiting scalability.
[0005] (2) Message queue-based collaboration method: by introducing a message queue, asynchronous processing of tasks can be achieved, thereby reducing system coupling. Although this method can improve system response speed in some scenarios, it can also cause processing delays and increase system complexity. Message queues can cause task processing delays, especially when the queue is backed up. System complexity increases, involving message serialization, transmission, and other links. High load increases the demand for storage resources.
[0006] (3) State sharing-based collaboration method: collaboration is achieved by sharing state information between multiple nodes. Although it can improve system collaboration efficiency, state consistency maintenance is difficult in a multi-node environment, requiring complex synchronization mechanisms. State sharing reduces system maintainability, and as the system size grows, state synchronization overhead increases, affecting scalability. SUMMARY
[0007] To solve the above problems in the prior art, the present application aims to provide a distributed job processing system and method that can enhance coordination efficiency and stability, and a distributed container node coordination device.
[0008] The distributed job processing system of one aspect of the present application has a plurality of nodes, a distributed container node coordination device, and a shared memory, characterized in that,
[0009] each of the plurality of nodes has an independent clock,
[0010] The distributed container node coordination device generates a job processing request according to the received task requirement and issues the job processing request to the shared memory,
[0011] One node or multiple nodes of the plurality of nodes obtain the job processing request from the shared memory and execute the job processing request.
[0012] Optionally, each node of the plurality of nodes has a job initiation module, a job communication module, an independent clock processing module, and a job execution module,
[0013] The job initiation module is configured to obtain the job processing request from the shared memory through a specified competition strategy,
[0014] The job communication module is configured to forward the job processing request obtained by the job initiation module to the job execution module or the independent clock processing module,
[0015] The independent clock processing module is configured to manage the job processing request forwarded by the job communication module and forward the job processing request to the job execution module when the job processing request meets a preset timing,
[0016] The job execution module is configured to execute the job processing request forwarded by the job communication module or the independent clock processing module.
[0017] Optionally, each node of the plurality of nodes reports a node state to the distributed container node coordination device in real time.
[0018] Optionally, the independent clock processing module of each node of the plurality of nodes activates a respective independent clock when the distributed container node coordination device is started, and each independent clock is assigned a random delay time value at the time of starting.
[0019] Optionally, the job initiation module broadcasts the job processing request obtained from the shared memory to the job communication module of each node,
[0020] The job communication module of each node determines whether to participate in competition for the job processing request according to a specified competition strategy,
[0021] When the job communication module of the node successfully participates in the competition for the job processing request, the job communication module forwards the job processing request to the job communication module or the independent clock processing module.
[0022] Optionally, the competition strategy comprises one or more of the following:
[0023] a node state related competition condition; and
[0024] a random weight or a random selection mechanism.
[0025] Optionally, the node state related competition condition comprises one or more of the following:
[0026] a current load, energy consumption, historical success rate of the node.
[0027] Optionally, the job conveying module is configured to determine whether the job processing request obtained by the job initiating module needs to be executed immediately or at a timing, and in a case where it is determined that the job processing request needs to be executed immediately, the job processing request is forwarded to the job executing module, and in a case where it is determined that the job processing request needs to be executed at the timing, the job processing request is forwarded to the independent clock processing module.
[0028] Optionally, the independent clock processing module sets a clock task when the job processing request is received, and according to the clock task, the job processing request is forwarded to the executing module when the set clock is reached.
[0029] Optionally, the random delay time value is obtained based on a preset random distribution model.
[0030] Optionally, the random delay time value is in a millisecond level.
[0031] Optionally, the distributed container node coordination apparatus comprises:
[0032] a job receiving module configured to analyze a received task demand and generate a job processing request according to the task demand;
[0033] a job scheduling module configured to assign a priority according to the job processing request; and
[0034] a job dispatching module configured to dispatch the job processing request according to the priority and a preset working mode.
[0035] Optionally, the working mode comprises:
[0036] a concurrent mode in which multiple nodes process job requests in parallel;
[0037] an exclusive mode in which only one node is allowed to process a job request; and
[0038] a designated mode in which a specific node is designated to process a job request.
[0039] The distributed container node coordination apparatus of one aspect of the present application comprises:
[0040] a job receiving module configured to parse a received task requirement and generate a job processing request according to the task requirement;
[0041] a job scheduling module configured to assign a priority according to the job processing request; and
[0042] a job dispatching module configured to dispatch the job processing request according to the priority and a preset working mode.
[0043] Optionally, the working mode comprises:
[0044] a concurrent mode in which multiple nodes process job requests in parallel;
[0045] an exclusive mode in which only one node is allowed to process a job request; and
[0046] a designated mode in which a specific node is designated to process a job request.
[0047] Optionally, the distributed container node coordination apparatus receives state information of each node from the multiple nodes.
[0048] A distributed job processing method according to an aspect of the present application comprises the following steps:
[0049] generating a job processing request according to a received task requirement;
[0050] generating a scheduling strategy according to the job processing request and state of one or more nodes;
[0051] issuing the job processing request to a shared memory according to the scheduling strategy;
[0052] the one or more nodes obtaining the job processing request from the shared memory; and
[0053] the one or more nodes executing the job processing request.
[0054] Optionally, each node of the multiple nodes obtains the job processing request from the shared memory through competition.
[0055] Optionally, each node of the multiple nodes has an independent clock and each independent clock is given a random delay time value at start-up.
[0056] Optionally, the one or more nodes obtaining the job processing request from the shared memory comprises:
[0057] the one or more nodes broadcasting the job processing request obtained from the shared memory;
[0058] The one or more nodes decide whether to participate in the competition for the job processing request according to a specified competition strategy.
[0059] Optionally, the competition strategy comprises one or more of the following:
[0060] a node state related competition condition; and
[0061] a random weight or a random selection mechanism.
[0062] Optionally, the node state related competition condition comprises one or more of the following:
[0063] a current load, energy consumption, historical success rate of the node.
[0064] The computer readable medium of one aspect of the present application has a computer program stored thereon, which, when executed by a processor, implements the distributed job processing method.
[0065] The computer device of one aspect of the present application comprises a storage module, a processor and a computer program stored on the storage module and executable on the processor, and the processor implements the distributed job processing method when executing the computer program.
[0066] The computer program product of one aspect of the present application comprises a computer program, which, when executed by a processor, implements the distributed job processing method. BRIEF DESCRIPTION OF DRAWINGS
[0067] Figure 1 is a structural schematic diagram of the distributed job processing system of the present application.
[0068] Figure 2 is a structural schematic diagram of the distributed container node coordination device 200 of one embodiment of the present application.
[0069] Figure 3 is a working principle schematic diagram of the distributed container node coordination device of one embodiment of the present application.
[0070] Figure 4 is a structural module of each node in the distributed job processing system of one embodiment of the present application and the data involved.
[0071] Figure 5 is a timing diagram of the distributed container node coordination device of the present application of one embodiment of the present application.
[0072] Figure 6 is a working flowchart of the distributed job processing method of one embodiment of the present application.
[0073] The and other objects and advantages of the application will be more fully understood from the following detailed description taken in conjunction with the accompanying drawings, in which like or similar elements refer to like or similar elements throughout. DETAILED DESCRIPTION
[0074] The following description of several embodiments of the application is intended to provide an overview of the application. It is not intended to identify key or critical elements of the application or to delineate the scope of the claimed subject matter.
[0075] For purposes of brevity and clarity, the principles of the application are described herein with reference to exemplary embodiments thereof. However, it will be readily appreciated by those skilled in the art that the same principles can be equally applied to all types of distributed job processing systems and methods, and distributed container node coordination apparatus, and that these same principles can be implemented therein, and that any such variations do not depart from the true spirit and scope of the present patent application.
[0076] Moreover, in the following description, reference is made to the accompanying drawings that form a part hereof, and in which are shown, by way of illustration, specific embodiments. Changes in the electrical, mechanical, logical, and structural details can be made to the embodiments described herein without departing from the spirit and scope of the application. Additionally, although a feature can be disclosed in connection with only one embodiment / implementation, alterations, improvements, or modifications can be suggested to one of ordinary skill in the art that such features can be incorporated into other embodiments / implementations within the scope of the present application as well. Accordingly, the following description is not intended to be in limitation of the scope of the application, as defined in the appended claims, and the entire disclosure is intended to be encompassed by potential claims.
[0077] The terms such as "have", "include", "contain", "have" and "comprise" indicate that the technical scheme of the application does not exclude the presence of other elements (modules) and steps not directly or explicitly expressed in the specification and claims.
[0078] Figure 1 is a schematic diagram of the construction of the distributed job processing system of the application.
[0079] As shown in Figure 1 The distributed job processing system of the application comprises a plurality of nodes 100, a distributed container node coordination apparatus 200 and a shared memory 300, wherein the plurality of nodes 100 comprises a plurality of nodes of node 1, node 2, …, node n.
[0080] The distributed container node coordination apparatus 200 is used to realize the cooperative work of the plurality of nodes. The shared memory 300 adopts a distributed shared memory, which is a technology that provides a shared memory abstraction in a distributed system, allowing multiple processors or multiple nodes to access a shared memory space through a network, just like accessing local memory.
[0081] The following will be described in detail for the distributed container node coordination device.
[0082] Figure 2 Fig. 1 is a structural schematic diagram of a distributed container node coordination device 200 according to an embodiment of the present application.
[0083] As shown in Fig. 1, the distributed container node coordination device 200 according to an embodiment of the present application comprises: Figure 2
[0084] A job receiving module 210 for receiving, parsing and packaging a job processing request;
[0085] A job scheduling module 220 for assigning a priority according to the job processing request;
[0086] A job dispatching module 230 for dispatching the job processing request according to a preset scheduling strategy and the priority, for example, to a current running instance (i.e. a node) through a shared memory 300; and
[0087] A job execution module 240 for receiving a job instruction, judging whether the job is executed, executing the job and reporting the execution situation.
[0088] Figure 3 Fig. 2 is a working principle schematic diagram of the distributed container node coordination device according to an embodiment of the present application.
[0089] Figure 3 In Fig. 2, a1-a3 represent a plurality of nodes, each of which has an independent clock, b1 and b2 represent the distributed container node coordination device and the shared memory, and the independent clock refers to a clock which is not dependent on an external clock source but is maintained and synchronized by the system itself, which can ensure the consistency and synchronization of each node when processing tasks in a distributed system.
[0090] As shown in Fig. 2, the distributed container node coordination device according to an embodiment of the present application operates according to the following four core processing logics: Figure 3
[0091] ① The node realizes real-time state reporting to the distributed container node coordination device through a heartbeat mechanism to maintain system keep-alive;
[0092] ② Triggering of a task demand, for example, triggered by a user instruction through a web interface, triggered through an inter-system call or a timing task, etc., the task demand is transmitted to the distributed container node coordination device after being triggered;
[0093] ③ The distributed container node coordination device generates a job processing request according to the task demand and transmits the job processing request to the shared memory;
[0094] ④To adapt to different operation requirements, the distributed container node coordination device supports multiple operation modes, including concurrent, mutual exclusion and designated node modes.
[0095] Among them, the concurrent mode refers to: suitable for pre-division data tasks, each node receives the job and directly processes in parallel. The mutual exclusion mode refers to: only one node is allowed to process the job, and the working node is selected by means of permission token contention, consensus mechanism confirmation or database unique primary key. The designated node mode refers to: the job has specified a specific node for processing when it is initiated, and no additional competition or selection process is required. These modes allow different machine nodes to take and process jobs according to their own independent clocks without interfering with each other.
[0096] Among them, the heartbeat mechanism is a communication method of sending state information periodically, and the node will send a heartbeat packet to the system at a certain time interval, which contains information such as the current state information of the node.
[0097] Next, the construction of the node is explained.
[0098] Figure 4 The construction module of each node in the distributed job processing system of an embodiment of the present application and the data involved are shown.
[0099] The distributed job processing system includes a plurality of nodes 200, and each node includes a job initiation module 210, a job transmission module 220, an independent clock processing module 230 and a job execution module 240. The actions performed by these modules are as follows:
[0100] The job request is configured and initiated by the user, and the job initiation module 210 of each node competes to receive the job request. On the other hand, the independent clock processing module 230 receives the clock signal from the job initiation module 210;
[0101] Once the job request is obtained, the job transmission module 220 intervenes in the processing, and decides whether the job needs to be executed at a fixed time. If it needs to be executed immediately, it instructs the job execution module 240 to execute immediately. If it needs to be executed at a fixed time, it is handed over to the independent clock processing module 230 for management;
[0102] The independent clock processing module 230 instructs the job execution module 240 to execute when the fixed time is reached.
[0103] The data involved in these modules is as follows:
[0104] (1) The static configuration data involved in the job initiation module 210 includes the following data:
[0105] System identification: job processing subsystem;
[0106] Group name: user trigger group;
[0107] Clock name: specified time sending device;
[0108] Clock description: specified time sending transaction;
[0109] Clock type: crontab expression;
[0110] Clock mode: single machine mutual exclusion;
[0111] Whether to support parallel: non-parallel;
[0112] Machine identification: machine X / task machine;
[0113] Whether to support multiple machines: not support
[0114] Specified time expression: 00133000;
[0115] Time interval: none;
[0116] Timeout time: 30 minutes Log size: 50M;
[0117] Operation status: pending;
[0118] Job processing program name: x process / x class.
[0119] (2) The job task involved in the job initiation module 210 includes the following data:
[0120] Operation identification: task A;
[0121] Machine identification: machine X / task machine;
[0122] Group name: user trigger group;
[0123] Clock name: specified time sending transaction;
[0124] Operation type: start task;
[0125] Operation parameter: transaction parameter;
[0126] Operation status: pending.
[0127] (3) As the clock task involved in the transmission module 220 includes the following data:
[0128] Trigger primary key: 20241010133020000;
[0129] System identification: job processing subsystem;
[0130] Trigger instance identification: 123465789;
[0131] Machine label: Machine X;
[0132] Group name: User triggered group;
[0133] Clock name: Scheduled time sending device;
[0134] Task status: To be processed.
[0135] (4) The clock state involved in the independent clock processing module 230 includes the following data: system identification: job processing subsystem;
[0136] Machine label: Machine X;
[0137] Group name: User triggered group;
[0138] Clock name: Scheduled time sending device;
[0139] Clock status: Normal.
[0140] (5) The clock log involved in the job execution module 240 includes the following data:
[0141] Log identification: 00000000000;
[0142] Trigger instance identification: 123465789;
[0143] System identification: Job processing subsystem;
[0144] Machine label: Machine X;
[0145] Group name: User triggered group;
[0146] Clock name: Scheduled time sending device;
[0147] Clock status: Normal;
[0148] Clock log content: The job has been started.
[0149] Among them, the data structure of the non-difference job in the application mainly involves three core data structures: machine state, job configuration and job instruction. The so-called non-difference job refers to the job that is not affected by the state of other nodes when each node in the distributed job system processes the task, which can ensure the efficient cooperation of the system in the multi-machine environment and is not affected by the state difference between nodes.
[0150] Specifically, the machine status is used to record the machine information of each node and is updated according to the clock cycle of the node, such as central identification, subsystem, node IP, node description, etc. The job configuration is used to store the static information of the job, including the execution mode, time and business logic, etc., such as subsystem, group name, job name, job description, job type, job mode, whether to support parallel, whether to be multi-machine, CRONTAB expression, period, timeout time, job class, etc. The job instruction is dynamic, which is used to guide the machine to pull and execute the job according to the clock cycle, such as operation ID, subsystem, machine number, group name, job name, operation type, operation parameter, operation state, operation.
[0151] The main solution of the present application is to propose a two-fold clock random processing mechanism, the first random processing mechanism is the random start-up deviation design of the clock, and the second random processing mechanism is the randomness involved in processing instructions. The following will specifically explain the two clock random processing mechanisms.
[0152] First, the first random processing mechanism is explained.
[0153] The first random processing mechanism is reflected in the random start-up deviation of the clock. The independent clock processing module is responsible for receiving and forwarding various tasks. In order to ensure the independence of the clock, the independent clock processing module of any node is delayed for a random millisecond when starting up, so that the clock of each node will have a small deviation of unknown degree. This deviation ensures that the actual forwarding time point after receiving the job will be unpredictable. The first random processing mechanism mainly includes the following aspects:
[0154] (1) Start-up of independent clock processing module
[0155] The independent clock processing module of each node is activated when the distributed container node coordination device starts up. At startup, each clock module will introduce a random delay Δ i This delay value is extracted from a pre-set random distribution (such as uniform distribution or normal distribution), and the range is usually a small value of milliseconds.
[0156] (2) Influence of random deviation
[0157] Since each node's independent clock processing module has a different random start-up delay, even if they are initially synchronized, over time, these small random deviations will cause small differences in the clocks between nodes.
[0158] This design makes it impossible for any external observer or the node itself to accurately predict the clock state of other nodes, thereby enhancing the security and attack resistance of the system.
[0159] (3) Maintenance of clock deviation
[0160] Clock reading T of each node i (t) can be represented as T i (t) = t + Δ i where t is the global time, Δ i is the clock skew of node i.
[0161] This skew Δ i remains constant throughout the node's runtime, unless the node is restarted or the clock module is recalibrated.
[0162] Next, the second randomization mechanism is explained.
[0163] The second randomization is reflected in the randomness of processing instructions. After receiving a job request, the independent clock processing module does not immediately process it. Instead, it will compete for processing rights based on the job's configuration requirements after the job is landed.
[0164] The second randomization processing mechanism mainly includes the following aspects:
[0165] (1) Job request competition
[0166] When the job initiation module receives a job request, it broadcasts this request to all nodes' job communication modules.
[0167] Each node's job communication module decides whether to participate in the competition for this job request based on certain competition conditions (for example, based on the node's current load, energy consumption, historical success rate, and other factors).
[0168] (2) The role of randomness in competition
[0169] Even if the node meets the competition conditions for participating in the competition, whether it can successfully obtain the job request is also affected by random factors such as random weights or random selection mechanisms. This randomness ensures that no single node can continuously monopolize job requests, thus promoting fair distribution of resources and balanced load of the system.
[0170] (3) Uncertainty of job processing
[0171] Once the job communication module successfully competes for the job request, it will decide whether to immediately execute the job based on the job's static configuration and dynamic instructions, or schedule it for execution at a future time point. If the job needs to be executed at a fixed time, the independent clock processing module will set a clock task, and the trigger time of this task is also determined based on the current clock reading plus a random deviation, increasing the uncertainty of the job execution time.
[0172] As described above, through the two random processing mechanisms, the distributed container node coordination device of the present application can increase the flexibility and unpredictability of job processing while maintaining the independence of inter-node operations, thereby improving the robustness and security of the entire system. This design is particularly suitable for distributed system environments that require high security and high reliability.
[0173] Figure 5 is a timing diagram of the distributed container node coordination device of the present application according to an embodiment of the present application. In Figure 5 the distributed container node coordination device is referred to as "device" for short.
[0174] As Figure 5 shown, the distributed container node coordination device of the present application according to an embodiment of the present application implements a multiple clock random processing mechanism in the following manner:
[0175] The node reports the status to the system in real time through the heartbeat mechanism to ensure high availability of the system (S1);
[0176] triggered by a task requirement, Figure 5 in the present application, the task requirement is given as an example by an instruction issued by a job issuer, of course, it can also be an inter-system call, or a certain timing task, etc. (S2);
[0177] After receiving the task requirement, the distributed container node coordination device triggers the task to be issued to the shared memory (S3). In the present application, the distributed container node coordination device issues the job instruction through the shared memory, and the shared memory plays a role in coordinating and distributing the job instruction. The job issuer can issue the job instruction in three modes: concurrent mode, the job instruction is sent to node A and node B at the same time, allowing them to process the job in parallel; mutual exclusion mode, the job instruction is first sent to node A, and after node A finishes processing, it is sent to node B; and specified node mode, the job instruction is directly sent to node C, which is processed by node C alone;
[0178] The job instruction in the concurrent mode is issued to node A (S4), and node A processes the concurrent job and feeds back the result to the system (S7);
[0179] Node B processes the job in the mutual exclusion mode after waiting for node A to finish processing (S5), and feeds back the result to the system (S8);
[0180] Node C processes the job independently in the specified node mode (S6), and feeds back the result to the system (S9).
[0181] Figure 6 is a workflow diagram of the distributed job processing method according to an embodiment of the present application.
[0182] As Figure 6 shown, the workflow of the distributed job processing method of an embodiment of the present application includes:
[0183] Node 1, node 2, …, node n report real-time status to the distributed container node coordination device through the heartbeat mechanism (equivalent to "independent keep-alive");
[0184] Job receiving step: the distributed container node coordination device receives and preliminarily analyzes the job request, extracts key parameters, and encapsulates information according to the preset data structure;
[0185] Job scheduling step: the distributed container node coordination device creates a job according to the encapsulated data content, and allocates priority to the job according to the functional characteristics, resource requirements and processing logic, to ensure that the job processing order meets the system efficiency and business requirements;
[0186] Job dispatching step: the distributed container node coordination device allocates the job to the appropriate running instance (i.e. each node) according to the scheduling strategy and system resource status, to ensure the optimal matching of the job and the instance, and improve the efficiency and stability of job execution;
[0187] Node 1, node 2, …, node n execute the job; and
[0188] The execution includes the following steps: the running instance (node 1, node 2, …, node n) generates a report after the job execution is completed.
[0189] In the present application, by designing the random clock delay deviation, the independent clock module of each node is activated and introduced with random delay when the system starts, and the millisecond level deviation is extracted from the preset random distribution. This random start deviation leads to a small difference in the clock between nodes, which can enhance the system security. The node clock reading is composed of global time plus fixed deviation, and can be maintained during the entire running period (unless the node is restarted or the clock is calibrated).
[0190] Further, by designing the random clock processing instruction, the job initiation module receives the job request, broadcasts it to the job communication module of all nodes, triggers the competition strategy based on the node state, introduces random factors such as weight or selection mechanism when competing for the job request, ensures fair allocation of resources, can prevent node monopoly of requests, and the job communication module decides the immediate execution or future scheduling of the job according to the job configuration and dynamic instruction, which can increase the flexibility and uncertainty of processing.
[0191] As described above, according to the distributed job processing system and the distributed job processing method of the present application, the job processing in a multi-machine environment is optimized by setting a double random clock mechanism, and the cooperation efficiency and stability are enhanced. Further, by real-time monitoring and accurate instruction distribution, the stable operation of the system can be ensured, and the reliability is improved. Further, the decoupled architecture simplifies system maintenance, improves maintainability, and makes technical upgrading and business adjustment more flexible and low-cost.
[0192] The above merely describes specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Those skilled in the art can think of other feasible changes or replacements according to the technical scope disclosed in the present application, and such changes or replacements are all covered in the protection scope of the present application. The embodiments and features in the embodiments can also be combined with each other without conflict. The protection scope of the present application is subject to the description of the claims.
Claims
1. A distributed job processing system, comprising: a plurality of nodes, a distributed container node coordination device, and a shared memory, wherein each of the plurality of nodes has an independent clock, the distributed container node coordination device generates a job processing request according to a received task requirement and issues the job processing request to the shared memory, and one or more of the plurality of nodes acquires the job processing request from the shared memory and executes the job processing request. 2.The distributed job processing system of claim 1, wherein each of the plurality of nodes comprises: a job initiation module, a job communication module, an independent clock processing module, and a job execution module, the job initiation module is configured to acquire the job processing request from the shared memory through a specified competition strategy, the job communication module is configured to forward the job processing request acquired by the job initiation module to the job execution module or the independent clock processing module, the independent clock processing module is configured to manage the job processing request forwarded by the job communication module and forward the job processing request to the job execution module when the job processing request meets a preset timing, and the job execution module is configured to execute the job processing request forwarded by the job communication module or the independent clock processing module. 3.The distributed job processing system of claim 1, wherein each of the plurality of nodes reports a node state to the distributed container node coordination device in real time. 4.The distributed job processing system of claim 1, wherein the independent clock processing module of each of the plurality of nodes activates a respective independent clock when the distributed container node coordination device is started and each independent clock is assigned a random delay time value when started. 5.The distributed job processing system of claim 3, wherein the job initiation module broadcasts the job processing request acquired from the shared memory to the job communication module of each node, the job communication module of each node determines whether to participate in competition for the job processing request according to a specified competition strategy, and the job communication module forwards the job processing request to the job communication module or the independent clock processing module when the job communication module of the node successfully participates in the competition for the job processing request. 6.The distributed job processing system of claim 4, wherein the competition strategy comprises one or more of: a node state related competition condition, and a random weight or random selection mechanism. 7.The distributed job processing system of claim 5, wherein the node state related competition condition comprises one or more of: a current load, energy consumption, and historical success rate of the node. 8.The distributed job processing system of claim 4, wherein The job communication module is configured to determine whether the job processing request obtained by the job initiation module needs to be executed immediately or at a scheduled time, and forward the job processing request to the job execution module if it is determined that the job processing request needs to be executed immediately, or forward the job processing request to the independent clock processing module if it is determined that the job processing request needs to be executed at a scheduled time.
9. The distributed job processing system of claim 7, wherein, The independent clock processing module sets a clock task when receiving the job processing request, and forwards the job processing request to the execution module when the set clock is reached according to the clock task.
10. The distributed job processing system of claim 3, wherein, The random delay time value is obtained based on a preset random distribution model.
11. The distributed job processing system of claim 9, wherein, The random delay time value is in the order of milliseconds.
12. The distributed job processing system of claim 2, wherein, The distributed container node coordination device comprises: a job receiving module configured to analyze a received task demand and generate a job processing request according to the task demand; a job scheduling module configured to assign a priority according to the job processing request; and a job dispatching module configured to dispatch the job processing request according to the priority and a preset working mode.
13. The distributed job processing system of claim 11, wherein, The working mode comprises: a concurrent mode in which multiple nodes process job requests in parallel; an exclusive mode in which only one node is allowed to process job requests; and a designated mode in which a specific node is designated to process job requests. a job receiving module configured to analyze a received task demand and generate a job processing request according to the task demand; 14. A distributed container node coordination apparatus, comprising: a job scheduling module configured to assign a priority according to the job processing request; and a job dispatching module configured to dispatch the job processing request according to the priority and a preset working mode.
15. The distributed container node coordination device of claim 14, wherein, The working mode comprises: a concurrent mode in which multiple nodes process job requests in parallel; an exclusive mode in which only one node is allowed to process job requests; and a designated mode in which a specific node is designated to process job requests.
16. The distributed container node coordination device of claim 14, wherein, The distributed container node coordination device receives state information of each of the multiple nodes. The method comprises the following steps: generating a job processing request according to a received task demand; generating a scheduling strategy according to the job processing request and the state of one or more nodes; 17. A method of distributed job processing, characterized by, issuing the job processing request to a shared memory according to the scheduling strategy; the one or more nodes obtaining the job processing request from the shared memory; and the one or more nodes executing the job processing request.
18. The distributed job processing method of claim 17, wherein, Each of the plurality of nodes acquires the job processing request from the shared memory by contention.
19. The distributed job processing method of claim 18, wherein, Each of the plurality of nodes has an independent clock and is assigned a random delay time value at start-up time for each independent clock.
20. The distributed job processing method of claim 19, wherein, The one or more nodes acquiring the job processing request from the shared memory comprises: The one or more nodes broadcasting the job processing request acquired from the shared memory; The one or more nodes deciding whether to participate in contention for the job processing request according to a specified contention strategy.
21. The distributed job processing method of claim 20, wherein, The contention strategy comprises one or more of: a node state related contention condition; and a random weight or random selection mechanism.
22. The distributed job processing method of claim 21, wherein, The node state related contention condition comprises one or more of: a current load, energy consumption, historical success rate of the node.
23. A computer readable medium having stored thereon a computer program, wherein, The computer program, when executed by a processor, implements the distributed job processing method of any one of claims 17-22.
24. A computer device comprising a storage module, a processor, and a computer program stored on the storage module and executable on the processor, wherein, The processor, when executing the computer program, implements the distributed job processing method of any one of claims 17-22.
25. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the distributed job processing method of any one of claims 17-22.