Virtualization-based high-concurrency multi-component flow parallel execution method

Abstract

The invention discloses a virtualization-based high-concurrency multi-component flow parallel execution method, which is applied to the technical field of computers and aims at solving the problem that the architecture design of the traditional component flow is generally relatively fixed and is difficult to quickly adapt to the change of service requirements. According to the invention, an efficient virtual parallel computing framework is designed, a component flow parallel computing framework is built through a container and virtual machine double-engine structure, and component flows are distributed to different cluster computing nodes for parallel processing. According to the method disclosed by the invention, not only can the resources be efficiently utilized in a high-concurrency scene, but also a cluster computing node overload phenomenon is avoided in scenes such as a complex computing service and the like; and stable and efficient operation of the cluster can be ensured without manual intervention.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a multi-component stream parallel execution technology. Background Art

[0002] Component-based simulation technology has gradually become a mainstream solution in the simulation field and has been widely used in various projects. As the complexity of simulation systems continues to increase, developers have gradually realized the advantages of decomposing the system into multiple independent components for development and combination. This approach not only improves development efficiency, but also enhances the maintainability and scalability of the system. Therefore, the concept of componentization has become an indispensable part of the simulation system development process.

[0003] Currently, many popular programming languages ​​and frameworks support component-based development. For example, languages ​​and frameworks such as Java, C#, Python, and AngularJS provide a variety of mechanisms to implement componentization. These tools and libraries allow developers to easily create, manage, and reuse software components, thereby promoting the construction of high-quality software. Through componentization, developers can focus on the functional implementation of a single component, reduce interactive interference in complex systems, and improve the efficiency of development and testing.

[0004] Flexible resource management and dynamic expansion capabilities enable components to quickly adapt when requirements change. Different teams develop components in parallel. The architectural design method of layering and component-based decomposition of the system simplifies the development of complex simulation systems. Each algorithm, protocol, and model can be independently developed, tested, deployed, and maintained, greatly improving development efficiency, reducing maintenance costs, and also improving the reusability of functions and codes.

[0005] However, despite the many advantages that software componentization brings, it also faces some challenges. For example, in some systems, there are higher computing requirements. When processing a large number of concurrent requests, traditional component flows can easily become a bottleneck for system performance. Since all requests must pass through the same component, the load on the component is too high, which affects the overall response time. At the same time, the architectural design of traditional component flows is usually relatively fixed and difficult to quickly adapt to changes in business needs. When faced with new functional requirements or traffic peaks, the ability to adjust and expand is limited. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a high-concurrency multi-component stream parallel execution method based on virtualization.

[0007] The technical solution adopted by the present invention is: a high-concurrency multi-component flow parallel execution method based on virtualization, the application scenario includes multiple cluster computing nodes, each cluster computing node is connected through a network; each cluster computing node is provided with a virtual machine, and a number of component flow containers are deployed in the virtual machine; each cluster computing node is provided with a cluster computing node agent module;

[0008] S1, the cluster computing node agent module uses the resource tags in the local configuration file to specify the task objectives, data types, and processing requirements;

[0009] S2, the cluster computing node agent module downloads the resource library from the resource management module, collects local resource information through the resource library and reports it to the resource management module;

[0010] S3, the cluster computing node agent module configures component flow resources according to task objectives, data types and processing requirements;

[0011] S4, the component flow driving module parses the components and their parameter information contained in the component flow;

[0012] S5. The component flow driving module applies to the resource management module for a cluster computing node to execute the task according to the parsing result of step S4, and sends the task to the corresponding cluster computing node.

[0013] S6. The deep reinforcement learning model corresponding to each cluster computing node predicts load changes through historical data based on the current resource usage and task requirements of the cluster computing node, and provides the optimal resource allocation strategy for the current cluster computing node;

[0014] S7, the cluster computing node agent module starts the container loading signal processing component to run according to the optimal resource allocation strategy;

[0015] S8. If the task is completed, stop the task, close the component flow container, and recycle various resources; otherwise, return to step S6.

[0016] Beneficial effects of the present invention: The method of the present invention designs an efficient virtual parallel computing framework based on the characteristics of the computing resource allocation and scheduling mechanism, wherein a component flow parallel computing framework is built through a dual-engine structure of containers and virtual machines. When an abnormality occurs in a cluster computing node or the load is too heavy, the component flow is allocated to different cluster computing nodes for parallel processing. The current resource usage and task requirements are analyzed through a deep reinforcement learning (DQN) model to provide an optimal resource allocation strategy. The method of the present invention can not only efficiently utilize resources in high-concurrency scenarios, but also avoid the overload of cluster computing nodes in scenarios such as complex computing services, and can ensure the stable and efficient operation of the cluster without human intervention. The method of the present invention has the following advantages:

[0017] (1) The present invention can efficiently utilize resources in high-concurrency scenarios;

[0018] Traditional component flow methods often face the problem of low resource utilization, especially in high-concurrency scenarios, where some components may not be able to fully utilize their performance due to excessive load. The intelligent parallel strategy of the present invention improves the computing speed of the system by allocating tasks to different cluster computing nodes. This method ensures that each component can obtain sufficient resources when needed, avoids idleness and waste of resources, and thus improves the performance of the overall system. In addition, the resource allocation strategy realizes parallel processing of data by dividing the data into blocks and processing each block on different cluster computing nodes in the cluster.

[0019] (2) The present invention avoids cluster computing node overload in scenarios such as complex computing services;

[0020] In the general component flow architecture, since all requests must pass through the same component, it is easy to cause performance bottlenecks and delay response time. The present invention can efficiently handle a large number of concurrent requests through the collaborative work of cluster computing node management and component scheduling modules. Through deep reinforcement learning, the system can intelligently determine the optimal resource allocation strategy based on historical data and real-time monitoring information, reduce manual intervention, and improve efficiency. Adaptive load balancing can respond to load changes in real time, ensure the reasonable allocation of resources, and thus improve the overall stability and performance of the system;

[0021] (3) The present invention ensures stable and efficient operation of the cluster.

[0022] The present invention proposes zero copy and cluster fault tolerance based on virtual environment. Zero copy ensures efficient data transmission between components by reducing communication. Cluster fault tolerance monitors the status of cluster computing nodes in real time and performs failover in time after discovering faulty cluster computing nodes. This solution can easily adapt to different application scenarios and business needs, has good flexibility and scalability, and is suitable for a variety of industries and environmental applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the technical process of the present invention;

[0024] Figure 2 A schematic diagram of a high-concurrency multi-component stream parallel framework based on virtualization of the present invention;

[0025] Figure 3 This is a schematic diagram of the deep reinforcement learning structure design of the present invention;

[0026] Figure 4 This is a flowchart of the intelligent allocation decision of the deep reinforcement learning network of the present invention. DETAILED DESCRIPTION

[0027] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0028] A high-concurrency multi-component stream parallel method based on virtualization of the present invention is applied to a componentized signal analysis system, and its overall process is as follows: Figure 1 As shown, including:

[0029] Step 1: Use the cluster computing node agent module to manage users, component libraries, and component flow tasks in the cluster computing nodes through the deep reinforcement learning (DQN) model, clarify the target signal type, data type, and processing requirements through the resource tags in the local configuration file, download the resource library from the resource manager, read the resource library file content, and send it to the cluster computing node agent module;

[0030] Step 2: The cluster computing node agent module loads the resource library, collects local resource information through the resource library and reports it to the resource manager, which stores and manages the resource information to form a computing resource pool. Combined with the dual-engine resource scheduling of containers and virtual machines, the Prometheus monitoring tool is used to regularly collect resource information such as CPU, memory, and network bandwidth, and report it to the resource manager.

[0031] Step 3: Configure component flow resources according to resource tags. After the component flow resources are configured, the component flow driver module parses the driver, including the parameters for component execution: filter frequency range, FFT window size, etc. Use the Kubernetes controller mode to initialize and manage components, and schedule and monitor container and virtual machine resources through K8s and OpenStack. The driver process is divided into initialization, running, and stopping stages;

[0032] Step 4: The component flow driver module parses the components and their parameter information contained in the component flow. The component is the smallest universal encapsulation structure of the algorithm or functional module, including: signal filtering, feature transformation, spectrum analysis, etc. The component flow driver module applies to the resource management module for the cluster computing node to execute the task and sends the task to each cluster computing node. It manages the life cycle of the task through the Deployment object to ensure that the task can be executed on both the container and the virtual machine, and automatically pulls the required Docker image.

[0033] Step 5. The DQN model receives the resource information stored by the resource manager. This information is used as input data to analyze the current resource usage and task requirements. It selects the optimal resource allocation strategy through output actions, including allocating cluster computing node resources, initiating dynamic migration, and reclaiming cluster computing node resources. Define state s as the resource usage and task characteristics of the cluster computing node, including CPU, memory, current load, and network bandwidth. Define action a as the resource allocation strategy, which is to select which cluster computing node to perform a specific subtask. Set the reward function r = -αT + βB + γU to encourage the system to find the best balance between task completion time T, load balancing B, and resource utilization U.

[0034] Step 6: Use Q-learning to update the strategy Q(s,a)←Q(s,a)+α[r+γa′maxQ(s′,a′)-Q(s,a)]

[0035] Among them, s′ is the new state transferred to after executing action a, a′ represents all possible actions that can be taken in the new state s′, α is the learning rate, and γ is the discount factor. The cluster computing node agent module of each cluster computing node starts the container to load the signal processing component to run and perform predefined signal analysis tasks including spectrum analysis, time-frequency transformation, etc.

[0036] Step 7: Control the data distribution and transmission process according to the network connection relationship to ensure the normal operation of the task.

[0037] Step 8: The DQN model monitors the load of each cluster computing node in real time and dynamically adjusts the task allocation strategy based on real-time feedback. If a cluster computing node is abnormal or overloaded, the resource management module will initiate dynamic migration and automatically expand or reduce the number of Pods based on the load conditions in combination with Kubernetes to achieve dynamic load balancing, ensuring the continuous operation of tasks and efficient use of resources.

[0038] Step 9: After the task is completed, stop the task, close the container, and recycle various resources.

[0039] It should be noted that the present invention is based on a virtualized high-concurrency multi-component stream parallel framework such as Figure 2As shown, the resource management module manages the computing resources of the cluster, including cluster formation, resource allocation, deployment, load balancing, etc. Each cluster computing node in the system will deploy a cluster computing node agent module. The cluster computing node agent module is developed based on the dual-engine architecture of containers and virtual machines. The cluster computing node agent module collects hardware resource information on the deployed cluster computing nodes and reports it to the resource management module. At the same time, it receives application deployment tasks during application deployment and manages the component container module process on the cluster computing node. The component container module is deployed on the virtual machine. The container provides lightweight application deployment and fast startup capabilities, while the virtual machine provides stronger isolation and security. The component container module is responsible for assuming the role of a bridge for interaction between components and the system framework, realizing the reception of data from other cluster computing nodes and other components, starting threads to schedule user tasks, or sending data to other components. The component flow driver module realizes resource application and automatic component deployment at runtime based on the design, resource configuration, and analysis of component flows, and drives the component flow for distributed execution.

[0040] The deep reinforcement learning structure designed by the present invention is as follows Figure 3 As shown, after the cluster computing node agent module is started, a thread loop is started to obtain the local resource usage, and the obtained device usage rate is reported to the resource manager module (CFRM, Centralized Federation Resource Manager).

[0041] After receiving the cluster computing node resource information reported by the cluster computing node agent module, CFRM caches the cluster computing node resource information locally and sorts the cluster computing node resources locally according to the usage of the cluster computing node resources. The basis for sorting the cluster computing node resources is the sum of the CPU, GPU, and memory usage.

[0042] In the process of cluster computing node resource load balancing, the DQN model will be used to dynamically predict and optimize resource allocation strategies, select actions with a greedy strategy, and then execute the actions and observe the rewards and the state of the next time step. Then the experience samples are put into the experience pool, samples are randomly selected from it, the predicted value of Q is calculated using the current network, the target value of Q is calculated using the Q-Target network, and then the loss function between the two is calculated:

[0043]

[0044] in, It means uniformly sampling the experience samples in the experience replay buffer D, taking the expected value, and using gradient descent to update the current network parameters.

[0045] When a component flow driver requests resources, CFRM not only returns the cluster computing node with the smallest value of the corresponding hardware resource type based on the current load of the cluster computing node, but also refers to the output of the DQN model. The DQN model outputs the Q value corresponding to each action in the current state, and selects the optimal action based on the Q value, that is, selects the optimal cluster computing node to assign tasks, so as to predict future load changes and thus select the optimal cluster computing node to balance long-term and short-term loads.

[0046] The operation of this process also includes component data balancing, load balancing between different component containers in the same cluster computing node, and task allocation based on computing speed. Here, the DQN model will analyze these status information to dynamically adjust the load balancing to ensure optimal resource allocation, such as Figure 4 shown.

[0047] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A high-concurrency multi-component stream parallel execution method based on virtualization, characterized in that: The application scenario includes multiple cluster computing nodes, each of which is connected through a network; each cluster computing node is provided with a virtual machine, in which several component flow containers are deployed; each cluster computing node is provided with a cluster computing node agent module; S1, the cluster computing node agent module uses the resource tags in the local configuration file to specify the task objectives, data types, and processing requirements; S2, the cluster computing node agent module downloads the resource library from the resource management module, collects local resource information through the resource library and reports it to the resource management module; S3, the cluster computing node agent module configures component flow resources according to task objectives, data types and processing requirements; S4, the component flow driving module parses the components and their parameter information contained in the component flow; S5. The component flow driving module applies to the resource management module for a cluster computing node to execute the task according to the parsing result of step S4, and sends the task to the corresponding cluster computing node. S6. The deep reinforcement learning model corresponding to each cluster computing node predicts load changes through historical data based on the current resource usage and task requirements of the cluster computing node, and provides the optimal resource allocation strategy for the current cluster computing node; S7, the cluster computing node agent module starts the container loading signal processing component to run according to the optimal resource allocation strategy; S8. If the task is completed, stop the task, close the component flow container, and recycle various resources; otherwise, return to step S6.

2. According to the virtualization-based high-concurrency multi-component stream parallel execution method of claim 1, it is characterized in that: The reward function corresponding to the deep reinforcement learning model is: r=-αT+βB+γU Among them, r is the reward, T is the task completion time, B is the load balancing, U is the resource utilization, α is the learning rate, γ is the discount factor, and β is the performance indicator weight coefficient.

3. According to the virtualization-based high-concurrency multi-component stream parallel execution method of claim 2, it is characterized in that: Step S4 is specifically as follows: using the Kubernetes controller mode to initialize and manage components, and scheduling and monitoring component flow containers and virtual machine resources through K8s and OpenStack.

4. According to the virtualization-based high-concurrency multi-component stream parallel execution method of claim 3, it is characterized in that: The optimal resource allocation strategy is specifically one of allocating cluster computing node resources, initiating dynamic migration, and reclaiming cluster computing node resources.

5. The high-concurrency multi-component stream parallel execution method based on virtualization according to claim 4 is characterized in that: The dynamic migration includes performing at least one of the following operations: Expand the component stream container, reduce the component stream container, and distribute the tasks of the current cluster computing node to other cluster computing nodes for processing.

Images