Load drainage method and device based on high-capacity workflow

By defining requirements and goals in the load drainage method of large-capacity workflow, designing system architecture, formulating drainage strategies, and optimizing data processing, the problem of insufficient data processing fluency and security is solved, and the system is efficient, stable and safe. reduces processor pressure.

CN119052176APending Publication Date: 2024-11-29正天技术有限公司
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411169590.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-24
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing load drainage methods and devices based on large-capacity workflows have shortcomings in data processing fluency and safety, and the processor pressure is high, resulting in system instability.

Method used

By defining requirements and goals, designing system architecture, formulating drainage strategies, optimizing data processing, improving stability and security, including redundant design, fault tolerance mechanism, data backup, security protection, processor load scheduling, data compression and resource management, Ensure the system is efficient, safe and stable.

Benefits of technology

Improve data processing speed and system stability, ensure data security, reduce processor working pressure, and meet the growing workflow needs.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention relates to the technical field of load drainage, in particular to a load drainage method based on high-capacity workflow, and the method comprises the following steps: S1, defining a demand and a target; s2, carrying out architecture design; s3, a drainage strategy; s4, measures for improving load drainage stability; s5, performing safety protection on the drainage data; s6, reducing the working pressure of the processor; s7, optimizing and compressing the drainage data; s8, fault tolerance and fault recovery; s9, resource management and scheduling; and S10, performing performance optimization and expansion. According to the method, the load drainage data is estimated by defining requirements, targets, architecture design and drainage strategies, the data processing speed is improved, and the load drainage stability and reliability are improved by means of the measure of improving the load drainage stability; by reducing the working pressure of the processor, optimizing and compressing the drainage data, managing and scheduling the resources, optimizing and expanding the performance and continuously improving and monitoring, the performance and the stability of the system are improved, and the working pressure of the processor is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of load diversion, and in particular to a load diversion method and device based on a large-capacity workflow. Background Art

[0002] Load diversion usually refers to the application scenario of load balancing equipment, in which link load balancing is generally deployed at the network exit of an enterprise or data center to achieve load sharing of export traffic, reduce traffic pressure, optimize traffic paths, and reduce bandwidth latency. By sensing the server status and using load balancing algorithms, access traffic is reasonably distributed to background servers. Load diversion of large-capacity workflows mainly refers to the use of a series of technologies and methods to disperse, balance and optimize workloads during workflow processing to ensure efficient and stable operation of the workflow.

[0003] However, the load diversion methods and devices based on large-capacity workflows currently on the market directly divert the collected data load, do not understand the processed data, are not conducive to the smoothness of data processing, and are relatively weak in data security protection processing, which is not conducive to the security of working data. The processing pressure of the processor is relatively large during large-capacity workflows, which is not conducive to the stability of data load diversion. Therefore, those skilled in the art provide a load diversion method and device based on large-capacity workflows to solve the problems raised in the above background technology. Summary of the invention

[0004] The purpose of the present invention is to provide a load diversion method and device based on large-capacity workflow to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a load diversion method based on a large-capacity workflow, the steps are as follows:

[0006] S1. Define requirements and goals: Understand the load diversion system through the acquisition module, considering the expected workload, load type, performance indicators, and data security requirements;

[0007] S2. Architecture design: Design the system architecture through simulation modules. The system architecture includes load diverter, queue system, and processor node components. Determine the relationship and interaction between components to achieve efficient workflow processing;

[0008] S3. Traffic diversion strategy: Create a traffic diversion strategy and set weights based on workflow type and load characteristics;

[0009] S4. Measures to improve the stability of load diversion: Improve the stability of load diversion by optimizing the redundant design and fault-tolerant mechanism of the module, node backup and failover, data backup and recovery strategy, load prediction algorithm and application, load prediction of historical data, application and adjustment of prediction results, high availability and redundancy measures;

[0010] S5. Security protection of diversion data: Protect working data through the security protection module and take the following measures:

[0011] S051. Data encryption: Encrypt the transmitted data and use a secure protocol for transmission;

[0012] S052, Access Control: Set access control rules to allow only legal requests to access and restrict illegal requests;

[0013] S053. Firewalls and intrusion detection systems: Deploy firewalls and intrusion detection systems to monitor and block malicious requests and attacks;

[0014] S054. Data backup and recovery: Back up work data regularly to cope with accidental data loss and establish a recovery mechanism;

[0015] S055. Formulation and implementation of disaster recovery plans: Establish a comprehensive monitoring system to monitor the operation status of the workflow system and the load of the processing nodes in real time, set up an alarm mechanism, and notify the administrator in time or automatically trigger emergency measures when abnormal situations or potential risks are detected;

[0016] S6. Reduce the work pressure of the processor: schedule and distribute the work pressure of the processor through the scheduling module, adjust the task allocation according to the processor load, and optimize the use of processor resources;

[0017] S7, optimization and compression of drainage data: optimize and compress drainage data through the optimization module to reduce the working pressure of the processor;

[0018] S8, Fault tolerance and fault recovery: Improve the stability and reliability of the system through fault tolerance processing mechanism;

[0019] S9. Resource management and scheduling: Manage and schedule system resources to ensure reasonable use of resources and achieve optimal performance;

[0020] S10, Performance optimization and expansion: Perform performance testing on the system, optimize it according to the performance evaluation results, formulate reasonable expansion strategies and expand the system according to business needs and data growth trends;

[0021] S11. Continuous improvement and monitoring: Through the monitoring module, data performance monitoring and log analysis can be carried out to identify bottlenecks and potential unstable factors in the system, establish a monitoring system, grasp the system operation status in real time, discover potential problems through log analysis and issue early warnings, simulate various load scenarios, conduct stress tests, and verify the stability and performance of the system. At the same time, emergency drills are carried out regularly to improve the team's ability to respond to emergencies, collect user feedback and suggestions, and continuously improve the system to adapt to the ever-changing needs and environment.

[0022] As a further solution of the present invention: the content of the traffic diversion strategy in S3 also includes:

[0023] Dynamic load balancing: monitor the load of each processing node in real time, dynamically adjust the distribution strategy of workflow tasks according to the load of the node, give priority to assigning tasks to nodes with lower loads, and use load balancing algorithms to optimize task distribution;

[0024] Task queue management: Introduce a task queue mechanism to put the arriving workflow tasks into the queue first, and then take out the tasks from the queue for processing based on the queue priority and task type factors. Set up multiple queues to process workflow tasks of different types or priorities respectively.

[0025] Distributed processing: Split large-volume workflows into multiple subtasks or subprocesses and distribute them to multiple processing nodes for parallel execution, taking advantage of distributed computing to improve overall processing capabilities and efficiency;

[0026] Elastic scaling: Dynamically increase or decrease the number of processing nodes according to the load of the workflow. When the load increases, new nodes are automatically added to enhance the processing capacity. When the load decreases, some nodes are released to save resources. Elastic scaling is achieved through cloud computing platforms or containerization technology.

[0027] Traffic shaping: Shape the traffic entering the workflow system, limit its rate and mode, avoid system overload caused by sudden traffic increase, and use token bucket algorithm and leaky bucket algorithm to implement traffic shaping;

[0028] Caching strategy: For computationally intensive or repetitive tasks, use a caching mechanism to store intermediate or final results. When the same task arrives again, the result can be obtained directly from the cache to reduce repeated calculations.

[0029] As a further solution of the present invention: the content of the high availability and redundancy measures in S4 includes:

[0030] Multiple processor nodes: deploy multiple processor nodes to achieve load balancing and failover;

[0031] Server cluster: Establish server cluster to achieve high availability and load balancing;

[0032] Data backup and synchronization: Regularly back up work data and synchronize data.

[0033] As a further solution of the present invention, the optimization and compression of the drainage data in S7 adopts the following method:

[0034] S071. Data compression: compress data to reduce data transmission bandwidth and storage costs;

[0035] S072. Data preprocessing: preprocess, clean and filter the data to remove invalid data and noise data and reduce the workload of the processor;

[0036] S073. Data sharding and diversion: Split large-volume workflow data into small pieces and distribute them to multiple processor nodes for parallel processing to improve processing efficiency.

[0037] As a further solution of the present invention: the steps of the fault-tolerant processing mechanism in S8 are as follows:

[0038] S081. Fault detection and recovery: Regularly detect the status of processor nodes, promptly detect faulty nodes, and perform fault recovery;

[0039] S082. Data redundancy and backup: Redundant storage and backup of working data to prevent data loss and failure;

[0040] S083. Monitoring and alarm system: Establish a monitoring and alarm system to monitor node status, work flow and performance indicators in real time, and issue an alarm in time and take corresponding measures when an abnormality occurs.

[0041] As a further solution of the present invention: the steps of managing and scheduling system resources in S9 are as follows:

[0042] S091. Resource monitoring and adjustment: Regularly monitor system resources and adjust resources as needed;

[0043] S092. Scheduling algorithm: The scheduling algorithm is used to dynamically adjust task allocation and scheduling according to load conditions and processing node performance to achieve load balancing and efficient utilization.

[0044] A device for a load diversion method based on a large-capacity workflow includes:

[0045] The acquisition module is used to collect data and understand the load diversion system;

[0046] The simulation module is used to design the system architecture and simulate the operation of data;

[0047] The optimization module is used to optimize data processing methods and improve the stability of load diversion;

[0048] The security protection module is used to protect working data;

[0049] The scheduling module is used to divert data and distribute the processor's workload;

[0050] The processor is used for processing data and receiving and sending instructions during the load diversion work of large-capacity workflows;

[0051] The monitoring module is used to monitor data performance and analyze logs to identify bottlenecks and potential unstable factors in the system.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. The load diversion method based on large-capacity workflow of the present invention estimates the data of load diversion by defining requirements and goals, architecture design and diversion strategy, improves the data processing speed, realizes efficient workflow processing, and ensures data synchronization and communication smoothness between distributed systems;

[0054] 2. Improve the stability and reliability of load diversion by taking measures to improve the stability of load diversion, and protect the security of working data by protecting the security of diversion data;

[0055] 3. By reducing the workload of the processor, optimizing and compressing the drainage data, fault tolerance and fault recovery, resource management and scheduling performance optimization and expansion, continuous improvement and monitoring, and improving the performance and stability of the system, the workload of the processor can be reduced to meet the growing workflow needs. DETAILED DESCRIPTION

[0056] Example

[0057] The load diversion method based on large-capacity workflow has the following steps:

[0058] S1. Define requirements and goals: Understand the load diversion system through the acquisition module, considering the expected workload, load type, performance indicators, and data security requirements;

[0059] S2. Architecture design: Design the system architecture through simulation modules. The system architecture includes load diverter, queue system, and processor node components. Determine the relationship and interaction between components to achieve efficient workflow processing;

[0060] S3. Traffic diversion strategy: Create a traffic diversion strategy and set weights based on the workflow type and load characteristics. The content of the traffic diversion strategy also includes:

[0061] Dynamic load balancing: monitor the load of each processing node in real time, dynamically adjust the distribution strategy of workflow tasks according to the load of the node, give priority to assigning tasks to nodes with lower loads, and use load balancing algorithms to optimize task distribution;

[0062] Task queue management: Introduce a task queue mechanism to put the arriving workflow tasks into the queue first, and then take out the tasks from the queue for processing based on the queue priority and task type factors. Set up multiple queues to process workflow tasks of different types or priorities respectively.

[0063] Distributed processing: Split large-volume workflows into multiple subtasks or subprocesses and distribute them to multiple processing nodes for parallel execution, taking advantage of distributed computing to improve overall processing capabilities and efficiency;

[0064] Elastic scaling: Dynamically increase or decrease the number of processing nodes according to the load of the workflow. When the load increases, new nodes are automatically added to enhance the processing capacity. When the load decreases, some nodes are released to save resources. Elastic scaling is achieved through cloud computing platforms or containerization technology.

[0065] Traffic shaping: Shape the traffic entering the workflow system, limit its rate and mode, avoid system overload caused by sudden traffic increase, and use token bucket algorithm and leaky bucket algorithm to implement traffic shaping;

[0066] Caching strategy: For computationally intensive or repetitive tasks, use a caching mechanism to store intermediate or final results. When the same task arrives again, the result can be directly obtained from the cache to reduce repeated calculations.

[0067] S4. Measures to improve load diversion stability: Improve load diversion stability by optimizing module redundancy design and fault tolerance mechanism, node backup and failover, data backup and recovery strategy, load prediction algorithm and application, load prediction of historical data, application and adjustment of prediction results, high availability and redundancy measures. The content of high availability and redundancy measures includes:

[0068] Multiple processor nodes: deploy multiple processor nodes to achieve load balancing and failover;

[0069] Server cluster: Establish server cluster to achieve high availability and load balancing;

[0070] Data backup and synchronization: regularly back up work data and synchronize data;

[0071] S5. Security protection of diversion data: Protect working data through the security protection module and take the following measures:

[0072] S051. Data encryption: Encrypt the transmitted data and use a secure protocol for transmission;

[0073] S052, Access Control: Set access control rules to allow only legal requests to access and restrict illegal requests;

[0074] S053. Firewalls and intrusion detection systems: Deploy firewalls and intrusion detection systems to monitor and block malicious requests and attacks;

[0075] S054. Data backup and recovery: Back up work data regularly to cope with accidental data loss and establish a recovery mechanism;

[0076] S055. Formulation and implementation of disaster recovery plans: Establish a comprehensive monitoring system to monitor the operation status of the workflow system and the load of the processing nodes in real time, set up an alarm mechanism, and notify the administrator in time or automatically trigger emergency measures when abnormal situations or potential risks are detected;

[0077] S6. Reduce the work pressure of the processor: schedule and distribute the work pressure of the processor through the scheduling module, adjust the task allocation according to the processor load, and optimize the use of processor resources;

[0078] S7. Optimization and compression of drainage data: The optimization module is used to optimize and compress drainage data to reduce the workload of the processor. The following methods are used for optimization and compression of drainage data:

[0079] S071. Data compression: compress data to reduce data transmission bandwidth and storage costs;

[0080] S072. Data preprocessing: preprocess, clean and filter the data to remove invalid data and noise data and reduce the workload of the processor;

[0081] S073. Data sharding and diversion: Split large-volume workflow data into small pieces and distribute them to multiple processor nodes for parallel processing to improve processing efficiency;

[0082] S8, Fault tolerance and fault recovery: Improve the stability and reliability of the system through the fault tolerance processing mechanism. The steps of the fault tolerance processing mechanism are as follows:

[0083] S081. Fault detection and recovery: Regularly detect the status of processor nodes, promptly detect faulty nodes, and perform fault recovery;

[0084] S082. Data redundancy and backup: Redundant storage and backup of working data to prevent data loss and failure;

[0085] S083. Monitoring and alarm system: Establish a monitoring and alarm system to monitor node status, work flow and performance indicators in real time, and promptly alarm and take corresponding measures when abnormalities occur;

[0086] S9. Resource management and scheduling: Manage and schedule system resources to ensure reasonable use of resources and achieve optimal performance. The steps for managing and scheduling system resources are as follows:

[0087] S091. Resource monitoring and adjustment: Regularly monitor system resources and adjust resources as needed;

[0088] S092, Scheduling algorithm: The scheduling algorithm is used to dynamically adjust task allocation and scheduling according to load conditions and processing node performance to achieve load balancing and efficient utilization;

[0089] S10, Performance optimization and expansion: Perform performance testing on the system, optimize it according to the performance evaluation results, formulate reasonable expansion strategies and expand the system according to business needs and data growth trends;

[0090] S11. Continuous improvement and monitoring: Through the monitoring module, data performance monitoring and log analysis can be carried out to identify bottlenecks and potential unstable factors in the system, establish a monitoring system, grasp the system operation status in real time, discover potential problems through log analysis and issue early warnings, simulate various load scenarios, conduct stress tests, and verify the stability and performance of the system. At the same time, emergency drills are carried out regularly to improve the team's ability to respond to emergencies, collect user feedback and suggestions, and continuously improve the system to adapt to the ever-changing needs and environment.

[0091] A device for a load diversion method based on a large-capacity workflow, comprising:

[0092] The acquisition module is used to collect data and understand the load diversion system;

[0093] The simulation module is used to design the system architecture and simulate the operation of data;

[0094] The optimization module is used to optimize data processing methods and improve the stability of load diversion;

[0095] The security protection module is used to protect working data;

[0096] The scheduling module is used to divert data and distribute the processor's workload;

[0097] The processor is used for processing data and receiving and sending instructions during the load diversion work of large-capacity workflows;

[0098] The monitoring module is used to monitor data performance and analyze logs to identify bottlenecks and potential unstable factors in the system.

[0099] What is described above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A load diversion method based on a large-capacity workflow, characterized in that: Here are the steps: S1. Define requirements and goals: Understand the load diversion system through the acquisition module, considering the expected workload, load type, performance indicators, and data security requirements; S2. Architecture design: Design the system architecture through simulation modules. The system architecture includes load diverter, queue system, and processor node components. Determine the relationship and interaction between components to achieve efficient workflow processing; S3. Traffic diversion strategy: Create a traffic diversion strategy and set weights based on workflow type and load characteristics; S4. Measures to improve the stability of load diversion: Improve the stability of load diversion by optimizing the redundant design and fault-tolerant mechanism of the module, node backup and failover, data backup and recovery strategy, load prediction algorithm and application, load prediction of historical data, application and adjustment of prediction results, high availability and redundancy measures; S5. Security protection of diversion data: Protect working data through the security protection module and take the following measures: S051. Data encryption: Encrypt the transmitted data and use a secure protocol for transmission; S052, Access Control: Set access control rules to allow only legal requests to access and restrict illegal requests; S053. Firewalls and intrusion detection systems: Deploy firewalls and intrusion detection systems to monitor and block malicious requests and attacks; S054. Data backup and recovery: Back up work data regularly to cope with accidental data loss and establish a recovery mechanism; S055. Formulation and implementation of disaster recovery plans: Establish a comprehensive monitoring system to monitor the operation status of the workflow system and the load of the processing nodes in real time, set up an alarm mechanism, and notify the administrator in time or automatically trigger emergency measures when abnormal situations or potential risks are detected; S6. Reduce the work pressure of the processor: schedule and distribute the work pressure of the processor through the scheduling module, adjust the task allocation according to the processor load, and optimize the use of processor resources; S7, optimization and compression of drainage data: optimize and compress drainage data through the optimization module to reduce the working pressure of the processor; S8, Fault tolerance and fault recovery: Improve the stability and reliability of the system through fault tolerance processing mechanism; S9. Resource management and scheduling: Manage and schedule system resources to ensure reasonable use of resources and achieve optimal performance; S10, Performance optimization and expansion: Perform performance testing on the system, optimize it according to the performance evaluation results, formulate reasonable expansion strategies and expand the system according to business needs and data growth trends; S11. Continuous improvement and monitoring: Through the monitoring module, data performance monitoring and log analysis can be carried out to identify bottlenecks and potential unstable factors in the system. A monitoring system can be established to grasp the system operation status in real time. Potential problems can be found through log analysis and early warning can be issued. Various load scenarios can be simulated to conduct stress tests to test the stability and performance of the system. At the same time, emergency drills can be conducted regularly to improve the team's ability to respond to emergencies, collect user feedback and suggestions, and continuously improve the system to adapt to the ever-changing needs and environment.

2. The method for load diversion based on large-capacity workflow according to claim 1 is characterized in that: The content of the traffic diversion strategy in S3 also includes: Dynamic load balancing: monitor the load of each processing node in real time, dynamically adjust the distribution strategy of workflow tasks according to the load of the node, give priority to assigning tasks to nodes with lower loads, and use load balancing algorithms to optimize task distribution; Task queue management: Introduce a task queue mechanism to put the arriving workflow tasks into the queue first, and then take out the tasks from the queue for processing based on the queue priority and task type factors. Set up multiple queues to process workflow tasks of different types or priorities respectively. Distributed processing: Split large-volume workflows into multiple subtasks or subprocesses and distribute them to multiple processing nodes for parallel execution, taking advantage of distributed computing to improve overall processing capabilities and efficiency; Elastic scaling: Dynamically increase or decrease the number of processing nodes according to the load of the workflow. When the load increases, new nodes are automatically added to enhance the processing capacity. When the load decreases, some nodes are released to save resources. Elastic scaling is achieved through cloud computing platforms or containerization technology. Traffic shaping: Shape the traffic entering the workflow system, limit its rate and mode, avoid system overload caused by sudden traffic increase, and use token bucket algorithm and leaky bucket algorithm to implement traffic shaping; Caching strategy: For computationally intensive or repetitive tasks, use a caching mechanism to store intermediate or final results. When the same task arrives again, the result can be obtained directly from the cache to reduce repeated calculations.

3. The method for load diversion based on large-capacity workflow according to claim 1, characterized in that: The high availability and redundancy measures in S4 include: Multiple processor nodes: deploy multiple processor nodes to achieve load balancing and failover; Server cluster: Establish server cluster to achieve high availability and load balancing; Data backup and synchronization: Regularly back up work data and synchronize data.

4. The method for load diversion based on large-capacity workflow according to claim 1, characterized in that: The optimization and compression of the drainage data in S7 adopts the following method: S071. Data compression: compress data to reduce data transmission bandwidth and storage costs; S072. Data preprocessing: preprocess, clean and filter the data to remove invalid data and noise data and reduce the workload of the processor; S073. Data sharding and diversion: Split large-volume workflow data into small blocks and distribute them to multiple processor nodes for parallel processing to improve processing efficiency.

5. The method for load diversion based on large-capacity workflow according to claim 1, characterized in that: The steps of the fault-tolerance processing mechanism in S8 are as follows: S081. Fault detection and recovery: Regularly detect the status of processor nodes, find faulty nodes in time, and perform fault recovery; S082. Data redundancy and backup: Redundant storage and backup of working data to prevent data loss and failure; S083. Monitoring and alarm system: Establish a monitoring and alarm system to monitor node status, work flow and performance indicators in real time, and issue an alarm in time and take corresponding measures when an abnormality occurs.

6. The method for load diversion based on large-capacity workflow according to claim 1, characterized in that: The steps of managing and scheduling system resources in S9 are as follows: S091. Resource monitoring and adjustment: Regularly monitor system resources and adjust resources as needed; S092. Scheduling algorithm: The scheduling algorithm is used to dynamically adjust task allocation and scheduling according to load conditions and processing node performance to achieve load balancing and efficient utilization.

7. A device for implementing the method for load diversion based on large-capacity workflow as described in any one of claims 1 to 6, characterized in that: include: The acquisition module is used to collect data and understand the load diversion system; The simulation module is used to design the system architecture and simulate the operation of data; The optimization module is used to optimize data processing methods and improve the stability of load diversion; The security protection module is used to protect working data; The scheduling module is used to divert data and distribute the processor's workload; The processor is used for processing data and receiving and sending instructions during the load diversion work of large-capacity workflows; The monitoring module is used to monitor data performance and analyze logs to identify bottlenecks and potential unstable factors in the system.

Citation Information

Cited By

  • Building engineering construction progress information management system and method based on big data

    CN120182044A