A backend process engine method, system and apparatus with high concurrency processing capability

Abstract

The application discloses a kind of back-end process engine methods, systems and devices with high concurrent processing capacity, it is related to distributed system architecture technical field. Including have: S1: dynamic hierarchical fault-tolerant processing: according to the type of node failure, the type of the node failure and task label are matched, while the priority of task label is acquired, hierarchical node failure processing is carried out;S2: predictive elastic scaling processing: by LSTM traffic prediction model, traffic trend is acquired, the number of pre-scaling nodes is determined, and the task at the fault node is migrated to the healthy node for processing.The application realizes the second-level identification of fault type through three-level fault detection model, i.e., network jitter standard deviation threshold, heartbeat timeout determination, resource scoring formula, and significantly improves the success rate of key task recovery through task priority dynamic adjustment algorithm.

Description

Technical Field

[0001] This invention relates to the field of distributed system architecture technology, specifically to a backend process engine method, system, and apparatus with high concurrency processing capabilities. Background Technology

[0002] With the development of information technology, enterprises and organizations have an increasing demand for business process automation. Workflow engines, as a core component for achieving this demand, have been widely used in various industries. However, traditional relational database-based workflow engines often fall short when facing high concurrency and large-scale data processing.

[0003] Traditional workflow engines such as Activiti and Flowable, while capable of supporting complex business logic and task allocation, do not adequately consider high-concurrency scenarios in the internet environment. For example, these systems typically rely on a single database instance for process definition and instance data storage, leading to significant performance bottlenecks under high concurrency. Furthermore, the tight coupling between business logic and process control often necessitates rewriting the software code when business rules change, which is unacceptable for companies seeking rapid market responses.

[0004] Chinese invention patent CN117573187A discloses a method for automatically generating different SDKs and sample code based on variable parameters, including the following steps: S1. Define specifications and create SDK templates; S2. Inject variable parameters into the SDK and provide downloads. Step S1 includes the following sub-steps: S11. Define SDK interface specifications and SDK code structure specifications; S12. Write SDK templates for different programming languages ​​according to the definitions, and set variable parameters in the SDK templates; S13. Upload the templates to a template file. This invention improves the standardization, maintainability, and scalability of application SDK development, reduces the technical requirements, development complexity, development workload, and debugging time for SDK users; it can better standardize the SDK development process, improve the standardization and consistency of SDK development in various programming languages, and enhance the code quality of each SDK.

[0005] Most of the aforementioned and similar process engines suffer from rigid fault tolerance mechanisms (such as a fixed 5-second retry interval), resulting in excessively long node failure recovery times (>10 seconds), leading to decreased service availability (SLA<99.9%). Furthermore, due to the characteristic that configuration updates depend on downtime maintenance (such as missing BPMN file hot reloading), business interruptions (request loss rate >1%) and iteration lags (hourly update cycles) are further caused, making it difficult to meet the core requirements of elastic scaling, millisecond-level fault recovery, and real-time hot updates of business logic in high-concurrency scenarios. Summary of the Invention

[0006] The purpose of this invention is to provide a backend process engine method, system, and apparatus with high concurrency processing capabilities to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a backend process engine method with high concurrency processing capabilities, comprising:

[0008] S1: Dynamic hierarchical fault tolerance processing: Based on the type of node fault, the type of node fault is matched with the task label, and the priority of the task label is obtained to process the node fault in a hierarchical manner.

[0009] S2: Predictive Elastic Scaling Processing: Utilizing an LSTM traffic prediction model, traffic trends are obtained to determine the number of nodes to be pre-scaled. Tasks from failed nodes are then migrated to healthy nodes for processing. This includes:

[0010] Step S2.1: Pre-allocation of expansion nodes: Using time series data and external features, a traffic prediction model is constructed to determine the number of nodes to be pre-expanded, specifically as follows:

[0011]

[0012] in: This refers to the number of nodes to be pre-expanded. For at a certain point in time The predicted request rate for the next 15 seconds. To achieve the target utilization rate, This represents the current number of active nodes.

[0013] S2.2: Delayed scaling down strategy: Determine the scaling down delay time based on the peak and valley traffic at the node, and compare the scaling down delay time with a preset delay time. When the scaling down delay time is greater than the preset delay time, the task corresponding to the scaling down delay time is dynamically truncated; otherwise, the task corresponding to the scaling down delay time runs normally.

[0014] S2.3: Task Distribution: Based on the number of pre-expanded nodes and the expected request rate, expand the nodes and allocate the expanded nodes to the constructed hash ring. At the same time, based on the shrinkage delay time and utilization rate of each node, perform node task migration processing.

[0015] Furthermore, node failures are handled in a tiered manner, including:

[0016] S1.1: Determine the fault type: Based on the real-time data between each node, determine the fault type of each node, which includes network jitter, node downtime, and resource exhaustion;

[0017] S1.2: Task Priority Classification: Each task at the node is labeled with an integer tag. Simultaneously, the tag level for each task is adjusted based on the tag level corresponding to the task attributes and the system load rate. Specifically:

[0018]

[0019] in: The adjusted tag level corresponding to the task attributes. The original tag level corresponding to the task attribute. The current system load rate, The load rate segmentation threshold;

[0020] S1.3: Strategy Selection: Based on the fault type of the node, the tasks of the node are processed using a retry strategy or a migration strategy, specifically:

[0021] When the fault type is network jitter, the retry strategy is executed to process the task of the node; otherwise, the migration strategy is executed to process the task of the node.

[0022] Furthermore, the fault type of each node is determined, including:

[0023] S1.1.1: Network Jitter Detection: By setting a preset number of ICMP Ping packets or application layer heartbeat packets, a sliding window is set to obtain the latency standard deviation, latency fluctuation coefficient, and packet loss rate corresponding to the sliding window at each node. The latency standard deviation, latency fluctuation coefficient, and packet loss rate are compared with preset latency standard deviation, latency fluctuation coefficient, and packet loss rate. Based on the comparison result, it is determined whether the fault type at the node is network jitter. Specifically:

[0024] When the delay standard deviation, delay fluctuation coefficient, and packet loss rate are all greater than the preset delay standard deviation, delay fluctuation coefficient, and packet loss rate, the fault type at the node is network jitter; otherwise, the fault type at the node is not network jitter.

[0025] S1.1.2: Node Downtime Detection: The cluster nodes are divided into sending nodes and receiving nodes. The sending nodes broadcast heartbeat signals to the receiving nodes. Simultaneously, based on the time when the receiving node receives the last heartbeat signal, a timeout threshold for each node downtime is obtained. This timeout threshold is compared with a preset timeout threshold, and based on the comparison result, it is determined whether the fault type at the node is node downtime. Specifically:

[0026] When the timeout threshold for node downtime is greater than the preset timeout threshold, the fault type at the node is node downtime; otherwise, the fault type at the node is not node downtime.

[0027] S1.1.3: Resource Exhaustion Detection: A comprehensive score is obtained by measuring CPU utilization, memory occupancy, and I / O latency. This comprehensive score is compared with a preset score, and based on the comparison result, it is determined whether the fault type at the node is resource exhaustion. Specifically:

[0028] When the overall score is greater than the preset score, the fault type at the node is resource depletion; otherwise, the fault type at the node is not resource depletion.

[0029] Furthermore, the tasks of the nodes are processed through retry or migration strategies, including:

[0030] S1.3.1: Retry Strategy: Based on the adjusted label level, the tasks are divided into critical tasks and non-critical tasks, and the retry interval is determined according to the baseline parameters corresponding to the divided tasks, specifically as follows:

[0031]

[0032] in: This refers to the retry interval for critical tasks. This is the initial retry interval for critical tasks. This represents the number of retries for critical tasks. For the fixed incremental step size corresponding to the critical task, This refers to the retry interval for non-critical tasks. This refers to the number of backoff attempts for non-critical tasks. The initial backoff interval for non-critical tasks;

[0033] S1.3.2: Migration Strategy: Based on the health score corresponding to each node, the nodes are divided into healthy nodes and faulty nodes, and the health score is compared with a preset health score to determine whether the task at the node needs to be migrated, specifically:

[0034] When the health score is not less than the preset health score, the node corresponding to the health score is a healthy node; otherwise, the node corresponding to the health score is a faulty node. At the same time, the tasks executed under the faulty node are migrated to the healthy node for task processing.

[0035] Furthermore, a target node is determined from the healthy nodes, and the tasks executed under the faulty node are migrated to the target node for processing. The process of determining the target node includes:

[0036] W1: Obtain Matching Degree: Based on the CPU and memory requirements of the faulty node, obtain the matching degree between each healthy node and the faulty node, specifically:

[0037]

[0038] in: The degree of matching between healthy and faulty nodes. This refers to the amount of CPU resources required for the task. This represents the remaining CPU resources on the target node. This is the CPU weighting coefficient. The amount of memory resources required for the task. This represents the amount of memory resources remaining on the target node. This refers to the memory weighting coefficient.

[0039] W2: Determine the target node: Compare the matching degree between all the healthy nodes and faulty nodes, and determine the maximum matching degree. The healthy node corresponding to the maximum matching degree is the final target node.

[0040] Furthermore, the migration of node tasks is performed, including:

[0041] S2.3.1: Constructing a hash ring: Assign multiple virtual nodes to each physical node, obtain the hash value corresponding to each virtual node through the SHA-1 algorithm, and map the hash value to the hash ring;

[0042] S2.3.2: Impact of Node Position: Based on the migration and transformation of the physical nodes, the virtual nodes on the hash ring are adjusted. The specific migration ratio of the virtual nodes is as follows:

[0043]

[0044] in: This represents the migration ratio of virtual nodes. This represents the original number of physical nodes. The number of changes in physical nodes. The number of virtual nodes;

[0045] S2.3.3: Task Route Allocation: Based on the task hash value and the corresponding virtual node, the target physical node is determined on the hash ring. Simultaneously, route adjustment is performed using temporary virtual nodes, and the weights of the virtual nodes are adjusted accordingly. Specifically:

[0046] When the physical node is a high-load node, the number of virtual nodes corresponding to the physical node is reduced; when the physical node is a low-load node, the number of virtual nodes corresponding to the physical node is increased; and vice versa, the number of virtual nodes remains unchanged.

[0047] Furthermore, routing adjustments are made through temporary virtual nodes, including:

[0048] S2.3.3.1: Determine the task hash value: The task hash value is determined using the task identifier and the CRC32 algorithm, specifically as follows:

[0049]

[0050] in: For task hash value, It is a cyclic redundancy check algorithm. For session identifier, For modulo operation;

[0051] S2.3.3.2: Determine the target physical node: Compare the task hash value with the hash values ​​corresponding to all virtual nodes, and sort the virtual nodes in ascending order to determine the first hash value that is not less than the task hash value. The virtual node corresponding to the first hash value that is not less than the task hash value is the task virtual node, and the physical node corresponding to the task virtual node is the target physical node.

[0052] S2.3.3.3: Obtain Node Utilization: Obtain the node utilization rate for each node by measuring its CPU and memory utilization rates, compare the node utilization rate with a preset utilization rate, and perform task processing based on the comparison result. Specifically:

[0053] When the node utilization rate is greater than the preset utilization rate, a temporary virtual node is inserted into the hash ring for routing adjustment; otherwise, task processing is performed directly at the node.

[0054] Furthermore, the node utilization rate is compared with a preset utilization rate. When the node utilization rate is greater than the preset utilization rate, the number of virtual nodes is reduced; conversely, the number of virtual nodes is increased. The formula for adjusting the number of virtual nodes is as follows:

[0055]

[0056] in: This is the number after reducing the number of virtual nodes. This is the number after the number of virtual nodes is increased. This is the maximum preset utilization rate. This is the minimum value of the preset utilization rate. For node utilization, This represents the number of virtual nodes.

[0057] A backend process engine system with high concurrency processing capability is characterized by using a backend process engine method with high concurrency processing capability as described in any one of the above claims.

[0058] A backend process engine apparatus with high concurrency processing capability uses a backend process engine method with high concurrency processing capability as described above.

[0059] Compared with the prior art, the beneficial effects of the present invention are:

[0060] Firstly, this invention achieves second-level fault type identification through a three-level fault detection model, namely network jitter standard deviation threshold, heartbeat timeout judgment, and resource scoring formula. It also significantly improves the success rate of critical task recovery through a task priority dynamic adjustment algorithm. At the same time, it significantly compresses the average recovery time of non-critical tasks through exponential backoff retry and resource-type fault trigger migration strategies.

[0061] Secondly, the dynamic truncation strategy based on traffic peak and valley analysis in this invention maintains node redundancy for 15-30 seconds during the traffic decline phase, thereby preventing false scaling down due to short-term fluctuations and significantly reducing the false scaling down rate.

[0062] Thirdly, this invention achieves a request loss rate of less than 0.01% and service interruption time within 200ms by using a dynamic migration mechanism for hash ring virtual nodes and a temporary virtual node insertion mechanism. Attached Figure Description

[0063] Figure 1 This is a flowchart of the backend process engine method of the present invention;

[0064] Figure 2 This is a comparison chart of the fault recovery time of the present invention (50 tests).

[0065] Figure 3 This is a comparison chart of request loss rates under different concurrency levels according to the present invention;

[0066] Figure 4 This is a comparison chart of the stability of resource utilization rate in this invention (4-hour monitoring).

[0067] Figure 5 This is a comparison chart of the expansion response delay of the present invention (5 tests);

[0068] Figure 6 This is a radar chart comparing the overall performance of the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] Most existing process engines suffer from rigid fault tolerance mechanisms (such as a fixed 5-second retry interval), resulting in excessively long node failure recovery times (>10 seconds), leading to decreased service availability (SLA < 99.9%). Furthermore, the requirement for downtime maintenance for configuration updates (such as missing BPMN file hot reloading) further causes business interruptions (request loss rate > 1%) and iteration lag (hourly update cycles), making it difficult to meet the core requirements of elastic scaling, millisecond-level fault recovery, and real-time hot updates of business logic in high-concurrency scenarios. The technical solution in this application, however, achieves intelligent migration or tiered retries of faulty tasks by real-time detection of node fault types, combined with dynamic adjustment strategies for task priorities and healthy node matching. Simultaneously, it predicts traffic trends over the next 15 seconds using an LSTM model, dynamically adjusts the number of nodes by determining the pre-expansion node quantity, and combines this with a delayed scaling-down strategy and a virtual node hash ring to achieve synergistic optimization of load balancing and resource utilization, ultimately achieving millisecond-level fault recovery, predictive resource scaling, and highly available task distribution.

[0071] Example 1

[0072] refer to Figures 1-6 This embodiment provides a backend process engine method with high concurrency processing capabilities, which specifically includes the following steps:

[0073] Step S1: Dynamic hierarchical fault tolerance processing. This involves real-time fault identification and analysis to determine the type of each fault and matching each fault type with a set task label. Simultaneously, task labels are prioritized, and a hybrid backoff algorithm and health score-based task migration are combined to trigger cross-node task migration. Details are as follows:

[0074] Step S1.1: Determine the fault type. This involves acquiring and analyzing real-time data between nodes to determine the fault type, which includes network jitter, node downtime, and resource exhaustion. Details are as follows:

[0075] Step S1.1.1: Network jitter detection. This involves deploying a monitoring agent on each node of the cluster and sending ICMP Ping packets or application-layer heartbeat packets to other nodes at preset intervals (100ms). It is worth noting that each ICMP Ping packet or application-layer heartbeat packet includes round-trip time and packet loss status.

[0076] Furthermore, based on a preset number of ICMP Ping packets or application layer heartbeat packets, a sliding window is set to obtain the moving average latency and latency standard deviation corresponding to the sliding window, specifically;

[0077]

[0078] in: For moving average latency, The size of the sliding window. Let be the round-trip time for the i-th measurement. The standard deviation of the delay. For the index of the measurement.

[0079] Furthermore, based on the obtained moving average latency and latency standard deviation, the latency fluctuation coefficient and packet loss rate corresponding to the sliding window at each node are determined, specifically:

[0080]

[0081] in: This is the delay fluctuation coefficient. For packet loss rate, For moving average latency, The size of the sliding window. The standard deviation of the delay. This represents the total number of packet losses within the sliding window.

[0082] Specifically, within a preset period (the specific settings are made according to actual needs, so they are not described in detail in this embodiment), the latency standard deviation, latency fluctuation coefficient, and packet loss rate corresponding to the sliding window at each node are obtained. Simultaneously, the obtained latency standard deviation, latency fluctuation coefficient, and packet loss rate are compared with preset latency standard deviation, preset latency fluctuation coefficient, and preset packet loss rate, respectively. Based on the comparison results, it is determined whether the fault type at that node is network jitter.

[0083] When the obtained latency standard deviation, latency fluctuation coefficient, and packet loss rate are all greater than the preset latency standard deviation, preset latency fluctuation coefficient, and preset packet loss rate, respectively, the fault type at this node is network jitter; otherwise, the fault type at this node is not network jitter.

[0084] In the specific implementation process, ICMP Ping packets or application layer heartbeat packets are acquired over three consecutive periods. The round-trip times for the third period are 250ms, 400ms, and 500ms, respectively, with a corresponding sliding window size of 10 and 2 packet losses. That is, the latency standard deviation, latency fluctuation coefficient, and packet loss rate for this third period are 104.2ms, 0.27%, and 20%, respectively. Furthermore, in this embodiment, the preset latency standard deviation, preset latency fluctuation coefficient, and preset packet loss rate are 50ms, 0.3%, and 5%, respectively.

[0085] In other words, although the obtained latency standard deviation and packet loss rate are both greater than the preset latency standard deviation and preset packet loss rate, the obtained latency fluctuation coefficient is less than the preset latency fluctuation coefficient. Therefore, the fault type at this node is not network jitter.

[0086] Step S1.1.2: Node Failure Detection. The cluster nodes are divided into sending nodes and receiving nodes. Each sending node broadcasts a heartbeat signal to the other nodes (receiving nodes) in the cluster at preset intervals. The heartbeat signal includes the node ID, timestamp, and load status. At the same time, the receiving nodes record the time of the last heartbeat signal received.

[0087] Furthermore, based on the time of the last received heartbeat signal obtained at the receiving node, the timeout threshold for each node's downtime is obtained, specifically as follows:

[0088]

[0089] in: This is the timeout threshold for node failure. The interval between heartbeat signal transmissions. This is for network latency tolerance.

[0090] Specifically, the obtained timeout threshold for node downtime is compared with a preset timeout threshold, and based on the comparison result, it is determined whether the fault type at that node is node downtime.

[0091] If the obtained timeout threshold for node downtime is greater than the preset timeout threshold, then the fault type at that node is node downtime; otherwise, the fault type at that node is not node downtime.

[0092] In the specific implementation process, the heartbeat signal transmission interval is 500ms, and the network latency tolerance is 100ms. Therefore, the timeout threshold for node downtime in this embodiment is 1.1s. Meanwhile, the preset timeout threshold in this embodiment is set to 2s. That is to say, if the timeout threshold for node downtime in this embodiment is less than the preset timeout threshold, then the fault type at that node is not node downtime.

[0093] Step S1.1.3: Resource Exhaustion Detection. This involves real-time monitoring of CPU, memory, and disk usage to obtain CPU utilization, memory occupancy, and I / O latency. Based on actual usage requirements, appropriate weights are assigned to determine the overall score.

[0094]

[0095] in: For comprehensive scoring, The weight corresponding to CPU utilization. The weights corresponding to available memory. The weights corresponding to I / O latency. For CPU utilization, Available memory, This refers to the I / O wait time.

[0096] Specifically, the obtained comprehensive score is compared with the preset score, and based on the comparison result, it is determined whether the fault type at that node is resource exhaustion. Specifically:

[0097] When the obtained comprehensive score is greater than the preset score, the fault type at that node is resource depletion; otherwise, the fault type at that node is not resource depletion.

[0098] Furthermore, by reading the ` / proc / stat` file and extracting the CPU data, the CPU idle time and total CPU time per unit time are determined, and the corresponding CPU utilization is calculated. Similarly, by reading the ` / proc / meminfo` file, the total physical memory and available memory of the system are determined, and the corresponding memory occupancy is calculated. Likewise, by reading the ` / proc / diskstats` file, the average I / O request processing time and the number of I / O operations per second are determined, and the I / O wait time is calculated. Specifically:

[0099]

[0100] in: For CPU utilization, This represents the increment of CPU idle time per unit of time. This represents the increment of total CPU time per unit of time. For memory occupancy, Available memory, Total system physical memory For I / O wait time, This represents the average I / O request processing time. I / O operations per second This represents the maximum I / O latency.

[0101] In the specific implementation process, the increment of CPU idle time per unit time is 100, and the increment of total CPU time per unit time is 1000, so the CPU utilization rate is 90%. Similarly, the available memory is 2GB, and the total physical memory of the system is 16GB, so the memory occupancy rate is 87.5%. Similarly, the average I / O request processing time is 5000ms, the number of I / O operations per second is 700, and the maximum I / O wait time is 200ms, so the I / O wait time is 0.04.

[0102] Specifically, in this embodiment, the weights of CPU utilization (0.9), available memory (0.875), and I / O latency (0.04) are 0.6, 0.2, and 0.2, respectively. Therefore, the overall score is: 0.6*0.9+0.2*0.875+0.2*0.04=0.723.

[0103] Furthermore, in this embodiment, the preset score is set to 0.7, which means that the obtained comprehensive score is greater than the preset score, so the fault type at this node is resource depletion.

[0104] Step S1.2: Task Priority Classification. This involves assigning an integer label to each task, with a value range of 1-5, specifically:

[0105] Level 1 tags: highest priority, such as real-time payment and core inventory deduction.

[0106] Level 2 tags: High priority, such as order status updates and transaction compensation.

[0107] Level 3 label: Medium priority, such as asynchronous message notification.

[0108] Level 4 tags: low priority, such as log archiving.

[0109] Level 5 tag: Lowest priority, such as non-critical cache warm-up.

[0110] Furthermore, based on the set rule matrix, the task attributes of each task are mapped to corresponding tags. It's worth noting that when there is no match between the current task attribute and its corresponding tag, the tag for that current task attribute is set to a level 5 tag.

[0111] Specifically, the tag levels for task attributes are adjusted based on the tag level corresponding to the task attribute and the system load rate to obtain the final tag level for the task attribute. It is worth noting that regardless of the obtained tag level, the highest tag level is always level 5. Further, the adjustment formula for the tag level corresponding to the task attribute is as follows:

[0112]

[0113] in: The adjusted tag level corresponding to the task attributes. The original tag level corresponding to the task attribute. The current system load rate, The threshold for load rate segmentation.

[0114] In the specific implementation process, the original label level corresponding to the task attribute is 3, the current system load rate is 70%, and the load rate segment threshold is 20%. Therefore, the adjusted label level corresponding to the task attribute is 5.

[0115] Step S1.3: Strategy Selection. This involves implementing corresponding strategies based on the node failure type determined in Step S1.1. Specifically, when the node failure type is network jitter, the node's task type is divided into critical tasks and non-critical tasks. The set retry strategy is then used to handle these tasks accordingly, ensuring rapid recovery of critical tasks while preventing non-critical tasks from increasing network load.

[0116] To elaborate further, when a node failure is classified as node downtime or resource exhaustion, a migration strategy is triggered to migrate the node without performing local retries. In other words, local retries cannot resolve node-level failures, so the node is migrated to a healthy node for further processing.

[0117] Furthermore, when the node failure type identified in step S1.1 includes network jitter, node downtime, and / or resource exhaustion, then the node failure requires both retrying and migration. Specifically, the priority of retrying and node migration can be determined according to the following formula:

[0118]

[0119] in: Rate the priority of actions. As task priority weight, For the retry interval, Rate your health.

[0120] Specifically, the action priority score corresponding to the retry operation is compared with the action priority score corresponding to the migration operation, and the action priority score with the highest score is determined. The operation corresponding to the highest action priority score is the operation to be executed first.

[0121] In this embodiment, based on the node fault type determined in step S1.1 and the label level determined in step S1.2, the tasks at each node are classified accordingly, and specific processing is performed based on the classification results. Specifically, as follows:

[0122] Step S1.3.1: Retry strategy. Based on the task label size determined in step S1.2, the tasks are divided into critical tasks (label levels 1 and 2) and non-critical tasks (label levels 3, 4 and 5), and the retry interval for each task is obtained.

[0123] Furthermore, based on the baseline parameters (initial retry interval, fixed increment step size, and number of retries) obtained from the critical tasks, the retry interval corresponding to the critical tasks is determined, specifically as follows:

[0124]

[0125] in: This refers to the retry interval for critical tasks. This is the initial retry interval for critical tasks. This represents the number of retries for critical tasks. This is a fixed, incremental step size corresponding to the critical task.

[0126] In the specific implementation, the initial retry interval is set to 100ms, the fixed increment step is set to 50ms, and the maximum number of retries is set to 5. That is, after the task fails on its first run, it will retry every 100ms, and then retry again at 50ms intervals until the maximum number of retries is reached. If the task still fails after the last retry, it will be marked as permanently failed, triggering an alarm and logging the event.

[0127] Furthermore, the baseline parameters (initial backoff interval, randomization range, and number of backoffs) obtained from non-critical tasks determine the retry interval for the corresponding non-critical tasks, specifically:

[0128]

[0129] in: This refers to the retry interval for non-critical tasks. This refers to the number of backoff attempts for non-critical tasks. This is the initial backoff interval for non-critical tasks.

[0130] In the specific implementation process, the initial backoff interval is set to 200ms, and the maximum number of backoffs is set to 8. That is to say, after the task fails on the first run, it will be retried at any time between 0 and 200ms, and each time it will be retried with an exponentially increasing random range (maximum interval of 51s). If it still fails after 8 retries, the task will be canceled and the exception will be recorded.

[0131] Step S1.3.2: Migration Strategy. This involves obtaining a health score for each node based on its failure rate, and then classifying nodes into healthy and faulty nodes based on these scores. Specifically, the obtained health scores are compared with preset health scores to classify nodes and determine whether the tasks at that node need to be migrated.

[0132] When the obtained health score is lower than the preset health score, the node corresponding to that health score is a faulty node; otherwise, the node corresponding to the health score is a healthy node. Simultaneously, the tasks executed under the faulty node are migrated to the healthy node for processing.

[0133] To elaborate further, the formula for obtaining the health score is as follows:

[0134]

[0135] in: Rate your health. , , These are the weighting coefficients. For node failure rate, The remaining resource rate, This represents the average response time.

[0136] Step S2: Predictive Elastic Scaling. This involves using an LSTM traffic prediction model to obtain traffic trends and determining the number of nodes to be scaled up based on these trends. In other words, based on the fault type of a node, appropriate node scaling is performed to migrate tasks from failed nodes to healthy nodes for processing. Details are as follows:

[0137] Step S2.1: Pre-allocation of expansion nodes. This involves constructing a traffic prediction model using an LSTM network structure based on time-series data (e.g., historical traffic data from the past hour, requests per second) and external characteristics (e.g., business activities, promotions, time periods, peak / off-peak periods, holiday markers). Specifically:

[0138]

[0139] in: For at a certain point in time The predicted request rate for the next 15 seconds. For the prediction function of the LSTM model, For time points The historical request volume sequence from -3600 seconds to the current time t. For time points The historical request sequence from -3599 seconds to the current time t. This is the historical request sequence at the current time t.

[0140] Furthermore, based on the expected request rate obtained from the traffic prediction model, the number of nodes to be pre-scaled is determined, specifically as follows:

[0141]

[0142] in: This refers to the number of nodes to be pre-expanded. For at a certain point in time The predicted request rate for the next 15 seconds. To achieve the target utilization rate, This represents the number of currently active nodes.

[0143] During the specific implementation process, at the time point The predicted request rate in 15 seconds is 9200 RPS, and the target utilization rate is set to 70%. The current number of active nodes is 10, so the number of nodes to be pre-expanded is 4. In other words, 4 more nodes need to be pre-expanded.

[0144] Step S2.2: Delayed scaling-down strategy. This involves determining the corresponding scaling-down delay time based on the obtained peak and trough traffic values ​​at each node. Specifically:

[0145]

[0146] in: For the reduction delay time, Peak traffic This represents the lowest possible traffic volume.

[0147] Furthermore, the obtained scaling-down delay time is compared with the preset delay time. If the obtained scaling-down delay time is greater than the preset delay time, the task corresponding to the scaling-down delay time needs to be dynamically truncated; otherwise, it can run normally.

[0148] Step S2.3: Task Distribution. Based on the expansion nodes obtained in Step S2.1, a hash ring is constructed. Simultaneously, based on the constructed hash ring and the expected request rate obtained from the traffic prediction model, and according to the obtained number of pre-expansion nodes, nodes are expanded and allocated to the hash ring. Further, based on the reduction latency and utilization rate obtained for each node, node task migration processing is performed. Specifically as follows:

[0149] Step S2.3.1: Construct a hash ring. This involves allocating multiple virtual nodes to each physical node to ensure even load distribution. Each physical node is uniquely identified by a combination of IP address, port number, and sequence number. Simultaneously, the hash value corresponding to each virtual node is obtained using the SHA-1 algorithm, and this hash value is mapped onto the hash ring. This allows all virtual nodes to be arranged in ascending order based on their hash values, forming a ring structure.

[0150] Step S2.3.2: Impact of Node Position. This involves adjusting the virtual nodes on the hash ring based on the migration and transformation of the physical nodes to maintain correspondence between the virtual and physical nodes. The specific migration ratio of the virtual nodes is as follows:

[0151]

[0152] in: This represents the migration ratio of virtual nodes. This represents the original number of physical nodes. The number of changes in physical nodes. This represents the number of virtual nodes.

[0153] Step S2.3.3: Task Route Allocation. This involves determining the target physical node on the hash ring based on the task hash value and the corresponding virtual node, and then adjusting the route using temporary virtual nodes to adjust the weights of the virtual nodes. Specifically:

[0154] High-load nodes: Reduce the number of their virtual nodes to decrease the probability of new task assignments.

[0155] Low-load nodes: Increase the number of virtual nodes to increase task processing capacity.

[0156] To elaborate further, the specific methods for adjusting the routing are as follows:

[0157] Step S2.3.3.1: Determine the task hash value. This involves using unique fields such as Session ID, Request ID, or User ID as the task identifier, and obtaining a 32-bit hash value using the CRC32 algorithm to determine the task hash value. Specifically:

[0158]

[0159] in: For task hash value, It is a cyclic redundancy check algorithm. For session identifier, This is a modulo operation.

[0160] Step S2.3.3.2: Determine the target physical node. That is, based on the task hash value determined in step S2.3.3.1, compare the task hash value with the hash values ​​of all virtual nodes, and determine the first hash value that is not less than the task hash value. The virtual node corresponding to this first hash value is the task virtual node. Further, based on the mapping relationship between physical nodes and virtual nodes, the physical node corresponding to this task virtual node is the target physical node.

[0161] Step S2.3.3.3: Obtain node utilization. This involves obtaining the node utilization for each node by measuring its CPU and memory utilization. Specifically:

[0162]

[0163] in: For node utilization, CPU utilization at the node This represents the memory utilization rate at the node.

[0164] Furthermore, the obtained node utilization rate is compared with the preset utilization rate. When the obtained node utilization rate is greater than the preset utilization rate, a temporary virtual node needs to be inserted into the hash ring for route adjustment. Otherwise, task processing can be performed directly at the node.

[0165] Specifically, the obtained node utilization rate is compared with the preset utilization rate. When the obtained node utilization rate is greater than the preset utilization rate, the number of virtual nodes is reduced; conversely, the number of virtual nodes is increased. The specific formula for adjusting the number of virtual nodes is as follows:

[0166]

[0167] in: This is the number after reducing the number of virtual nodes. This is the number after the number of virtual nodes is increased. This is the maximum preset utilization rate. This is the minimum value of the preset utilization rate. For node utilization, This represents the number of virtual nodes.

[0168] In the actual implementation, the preset utilization rate is set to 65%-75%, where 65% is the minimum and 75% is the maximum. That is, when the node utilization rate is below 65%, resources are wasted, and the load needs to be increased. When the node utilization rate is between 65% and 75%, it is the ideal state and no adjustment is needed. When the node utilization rate is above 75%, it is overloaded, and the load needs to be reduced.

[0169] To elaborate further, if the current number of virtual nodes is 1000 and the node utilization rate is 60%, then the number of virtual nodes needs to be increased, resulting in 1077 virtual nodes. In other words, adding 77 virtual nodes increases the task distribution probability by 7.7%, and the utilization rate gradually recovers to over 65%.

[0170] refer to Figure 2 , Figure 2 The fault recovery time comparison chart (50 tests) shows that the median of the traditional solution is approximately 12.3 seconds, while the median of the proposed solution is approximately 0.79 seconds. This represents an absolute time reduction of 11.51 seconds, a 15-fold performance improvement. Furthermore, the standard deviation of the traditional solution is 1.2 seconds, while the standard deviation of the proposed solution is 0.15 seconds, indicating an 87.5% reduction in time fluctuation, thus significantly improving system stability.

[0171] refer to Figure 3 , Figure 3 The chart comparing request loss rates under different concurrency levels shows that in a 5000-concurrency scenario, the traditional solution has a loss rate of 2.5%, while the proposed solution has a loss rate of 0.12%, representing an absolute difference of 2.38 percentage points and an improvement of 95.2%. At the critical inflection point of 2000 concurrency, the loss rate of the traditional solution increases rapidly, from 1.2% to 1.8%, and finally to 2.5%, with the growth rate decreasing from 50% to 38%. In contrast, the loss rate of the proposed solution maintains a linear growth, increasing from 0.05% to 0.07%, and then to 0.12%, with the growth rate increasing from 40% to 71%.

[0172] refer to Figure 4 , Figure 4 The chart shows a comparison of resource utilization stability (4-hour monitoring). It can be seen that the traditional solution's resource utilization fluctuates wildly between 30% and 95%, with a peak-to-trough difference of 65 percentage points, repeatedly hitting the 30% idle threshold and the 95% overload threshold. In contrast, the resource utilization of the proposed solution remains stable within the target range of 65%-75%, with a maximum fluctuation of only 10 percentage points, a reduction of 84.6% compared to the traditional solution.

[0173] refer to Figure 5 , Figure 5 The comparison chart of response latency during capacity expansion (5 tests) shows that the average latency of the traditional solution is 44.2 seconds, while the average latency of the proposed solution is 14.8 seconds, a reduction of 29.4 seconds in absolute time, representing a 66.5% improvement. Furthermore, the traditional solution exceeded its nominal threshold of 45 seconds in all 5 tests, while the latency of the proposed solution was less than 17 seconds in all tests, 2.6 times faster than the traditional threshold.

[0174] refer to Figure 6 , Figure 6 The comprehensive performance comparison radar chart shows that the fault recovery time has decreased from 12.4 seconds to 0.8 seconds, a 15.5-fold speedup, achieving sub-second recovery. The request loss rate is 1.8, a 36-fold reduction. Meanwhile, the resource stability fluctuation range is 30, and the standard deviation has decreased from 18.7 to 2.1, a reduction of 88.8%. The scaling response latency has decreased from 45 seconds to 15 seconds, a 3-fold improvement in response speed. The pre-scaling mechanism has reduced peak accumulation by 71%.

[0175] Furthermore, this embodiment also provides a backend process engine system with high concurrency processing capability, which uses the backend process engine method with high concurrency processing capability described above.

[0176] Furthermore, this embodiment also provides a backend process engine system device with high concurrency processing capability, which uses the backend process engine method with high concurrency processing capability described above.

[0177] Example 2

[0178] This embodiment provides a backend process engine method with high concurrency processing capability. Its specific implementation method is the same as that of Embodiment 1. The difference is that the execution tasks under the faulty node are migrated to the healthy node for task processing. The invention will be illustrated below with specific implementation methods of this embodiment.

[0179] In this embodiment, during the process of migrating tasks executed under a faulty node to a healthy node for processing, a target node needs to be determined from among many healthy nodes. In other words, the tasks executed under the faulty node are migrated to a target node that matches the faulty node for processing. Specifically:

[0180] Step W1: Obtain the matching degree. This involves obtaining the matching degree between each healthy node and the faulty node based on the CPU and memory requirements of the tasks in the faulty node. Specifically:

[0181]

[0182] in: The degree of matching between healthy and faulty nodes. This refers to the amount of CPU resources required for the task. This represents the remaining CPU resources on the target node. This is the CPU weighting coefficient. The amount of memory resources required for the task. This represents the amount of memory resources remaining on the target node. This represents the memory weighting coefficient.

[0183] During the specific implementation process, the task requires 2 CPU cores, the target node has 4GB of remaining CPU resources, the task requires 4 memory cores, and the target node has 8GB of remaining memory resources. Therefore, the matching degree between healthy nodes and faulty nodes is 2.0.

[0184] Step W2: Determine the target node. This involves comparing the matching degree between all healthy nodes and faulty nodes, and determining the healthy node with the maximum matching degree. The healthy node corresponding to the maximum matching degree is the final target node.

[0185] Example 3

[0186] This embodiment provides a backend process engine method with high concurrency processing capability. Its specific implementation method is the same as that of Embodiment 1. The difference is that the hash value is mapped to the hash ring so that all virtual nodes can be arranged in ascending order according to the size of the hash value to form a ring structure. The invention will be illustrated below with specific implementation of this embodiment.

[0187] In this embodiment, the uniformity of the hash ring is ensured in the following way:

[0188] Step M1: Multiple hashing. This involves performing multiple hash calculations on each virtual node to generate multiple hash values. These hash values ​​are then distributed and mapped to different locations on the hash ring to reduce distribution bias that might result from a single hash function. Furthermore, even with a small number of physical nodes, multiple hashing can disperse the virtual node locations and reduce hotspot areas.

[0189] Step M2: Virtual Node Redundancy. This involves determining the number of virtual nodes corresponding to each physical node based on the number of physical nodes, ensuring that each physical node has the same number of virtual nodes. This further guarantees that even if the number of physical nodes is small, the total number of virtual nodes is still sufficient, thus avoiding sparse distribution on the hash ring.

[0190] Furthermore, the formula for determining the number of virtual nodes is as follows:

[0191]

[0192] in: This represents the number of virtual nodes. This represents the number of physical nodes.

[0193] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended embodiments and their equivalents.

Claims

1. A backend process engine method with high concurrency processing capability, characterized in that, Comprise: S1: dynamic hierarchical fault-tolerant processing: according to the type of node failure, the type of node failure and task label are matched, and the priority of task label is obtained, and the processing of node failure is carried out hierarchically; S2: predictive elastic scaling processing: through LSTM traffic prediction model, the traffic trend is obtained, the number of pre-scaling nodes is determined, and the tasks at the fault node are migrated to the healthy node for processing, comprising: Step S2.1: pre-allocation of expansion node: through time series data and external features, a traffic prediction model is constructed, and the number of pre-scaling nodes is determined, specifically: ； wherein: is the number of nodes to pre-scale, is the time point, is the predicted expected request rate 15 seconds in the future, is the target utilization, is the current number of active nodes; S2.2: lagging shrinkage strategy: according to the peak and valley of the traffic at the node, the shrinkage delay time is determined, and the shrinkage delay time is compared with the preset delay time, when the shrinkage delay time is greater than the preset delay time, the task corresponding to the shrinkage delay time is dynamically truncated, otherwise, the task corresponding to the shrinkage delay time is normally operated; S2.3: task distribution: according to the number of pre-scaling nodes and expected request rate, node expansion is carried out, and the expanded nodes are allocated to the constructed hash ring, and according to the shrinkage delay time and utilization rate of each node, the migration processing of node task is carried out.

2. The backend process engine method with high concurrency processing capability according to claim 1, wherein, The processing of node failure is carried out hierarchically, comprising: S1.1: determine the fault type: according to the real-time data between each node, the fault type of each node is determined, the fault type includes network jitter, node downtime and resource exhaustion; S1.2: task priority classification: the tasks at the node are marked by integer type label, and the label level corresponding to the task attribute is adjusted according to the label level and system load rate, specifically: ； wherein: is an adjusted label rank corresponding to the task attribute, is an original label rank corresponding to the task attribute, is a current load rate of the system, is a load rate segment threshold value; S1.3: strategy selection: according to the fault type of the node, the task of the node is processed by retry strategy or migration strategy, specifically: When the fault type is network jitter, the retry strategy is executed to process the task of the node, otherwise, the migration strategy is executed to process the task of the node.

3. The method of claim 2, wherein, Determine the fault type of each node, comprising: S1.1.1: network jitter detection: through the preset acquisition times of ICMP Ping packet or application layer heartbeat packet, the sliding window is set, the delay standard deviation, delay fluctuation coefficient and packet loss rate corresponding to the sliding window at each node are obtained, and the delay standard deviation, delay fluctuation coefficient and packet loss rate are compared with the preset delay standard deviation, delay fluctuation coefficient and packet loss rate, according to the comparison result, whether the fault type at the node is network jitter is determined, specifically: When the delay standard deviation, delay fluctuation coefficient and packet loss rate are greater than the preset delay standard deviation, delay fluctuation coefficient and packet loss rate respectively, the fault type at the node is network jitter, otherwise, the fault type at the node is not network jitter; S1.1.2: Node downtime detection: divide the cluster nodes into sending nodes and receiving nodes, and the sending nodes broadcast heartbeat signals to the receiving nodes, while obtaining the time of the last heartbeat signal from the receiving nodes, obtaining the timeout threshold of each node downtime, comparing the timeout threshold with the preset timeout threshold, and determining whether the fault type at the node is node downtime according to the comparison result, specifically: When the timeout threshold of the node downtime is greater than the preset timeout threshold, the fault type at the node is node downtime, otherwise, the fault type at the node is not node downtime; S1.1.3: Resource exhaustion detection: obtain a comprehensive score through CPU utilization, memory occupancy, and I / O latency, and compare the comprehensive score with the preset score, and determine whether the fault type at the node is resource exhaustion according to the comparison result, specifically: When the comprehensive score is greater than the preset score, the fault type at the node is resource exhaustion, otherwise, the fault type at the node is not resource exhaustion.

4. The method of claim 2, wherein the backend process engine has a high concurrency processing capability. Handle the tasks of the node through retry strategy or migration strategy, including: S1.3.1: Retry strategy: divide the tasks into critical tasks and non-critical tasks according to the adjusted label level, and determine the retry interval according to the baseline parameters corresponding to the divided tasks, specifically: ； wherein: is a retry interval corresponding to a critical task, is an initial retry interval corresponding to a critical task, is a number of retries corresponding to a critical task, is a fixed increment step corresponding to a critical task, is a retry interval corresponding to a non-critical task, is a number of backoffs corresponding to a non-critical task, is an initial backoff interval corresponding to a non-critical task; S1.3.2: Migration strategy: divide the nodes into healthy nodes and fault nodes according to the health score corresponding to each node, and compare the health score with the preset health score to determine whether the task at the node needs to be migrated, specifically: When the health score is not less than the preset health score, the node corresponding to the health score is a healthy node, otherwise, the node corresponding to the health score is a fault node, and the executing task under the fault node is migrated to the healthy node for task processing.

5. The method of claim 4, wherein, Determine the target node from the healthy nodes, and migrate the executing task under the fault node to the target node for processing, the determination process of the target node includes: W1: Obtain matching degree: obtain the matching degree between each healthy node and fault node according to the task demand CPU and task demand memory in the fault node, specifically: ； wherein: is a matching degree between the healthy node and the faulty node, is a CPU resource amount of a task requirement, is a remaining CPU resource amount of the target node, is a CPU weight coefficient, is a memory resource amount of a task requirement, is a remaining memory resource amount of the target node, is a memory weight coefficient; W2: Determine target node: compare the matching degrees between all healthy nodes and fault nodes to determine the maximum matching degree, and the healthy node corresponding to the maximum matching degree is the final target node.

6. The method of claim 1, wherein the backend process engine has a high concurrency processing capability. Migrate the tasks of the node, including: S2.3.1: Constructing a hash ring: allocate multiple virtual nodes to each physical node, and obtain the hash value corresponding to each virtual node through SHA-1 algorithm, and map the hash value to the hash ring; S2.3.2: Influence of node location: adjust the virtual nodes on the hash ring according to the migration transformation of the physical nodes, and the migration ratio of the virtual nodes is specifically: ； wherein: is a migration proportion of virtual nodes, is an original number of physical nodes, is a changed number of physical nodes, is a number of virtual nodes; S2.3.3: Task routing distribution: according to the task hash value and the corresponding virtual node, the target physical node is determined on the hash ring, and the routing adjustment is adjusted through the temporary virtual node, and the weight of the virtual node is adjusted, specifically: When the physical node is a high-load node, the number of virtual nodes corresponding to the physical node is reduced, when the physical node is a low-load node, the number of virtual nodes corresponding to the physical node is increased, otherwise, the number of virtual nodes is unchanged.

7. The method of claim 6, wherein the backend process engine has a high concurrency processing capability. The routing adjustment through the temporary virtual node includes: S2.3.3.1: Determine the task hash value: determine the task hash value by task identification and CRC32 algorithm, specifically: ； wherein: is a task hash value, is a cyclic redundancy check algorithm, is a session identifier, is a modulo operation; S2.3.3.2: Determine the target physical node: compare the task hash value with the hash value corresponding to all virtual nodes, and determine the first hash value not less than the task hash value according to the ascending order of the virtual nodes, the virtual node corresponding to the first hash value not less than the task hash value is the task virtual node, and the physical node corresponding to the task virtual node is the target physical node; S2.3.3.3: Get node utilization: get the node utilization corresponding to each node by the CPU utilization and the memory utilization of the node, compare the node utilization with the preset utilization, and according to the comparison result, process the task, specifically: When the node utilization is greater than the preset utilization, a temporary virtual node is inserted in the hash ring for routing adjustment, otherwise, the task is directly processed at the node.

8. The method of claim 7, wherein the backend process engine has a high concurrency processing capability. Compare the node utilization with the preset utilization, when the node utilization is greater than the preset utilization, the virtual node is reduced, otherwise, the virtual node is increased, the number of virtual nodes adjustment formula, specifically: ； wherein: is a number of virtual nodes after reduction, is a number of virtual nodes after increase, is a maximum value of the preset utilization rate, is a minimum value of the preset utilization rate, is a node utilization rate, is a number of virtual nodes.

9. A backend process engine system with high concurrency processing capability, characterized in that, A backend process engine method with high concurrent processing capability is used.

10. A backend process engine apparatus with high concurrency processing capability, characterized by, A backend process engine method with high concurrent processing capability is used.

Images