Development and operation optimization system and method for large-scale multi-component systems based on reinforcement learning

Abstract

The present invention discloses a development and operation optimization system and method for a large-scale multi-component system based on reinforcement learning, real-time clustering DevOps, promoting analysis and evaluation of DevOps performance at different abstraction levels, RTCD automated task dependency extraction, sequence generation, task scheduling and task mapping, and dynamically optimizing the system through an optimization engine using algorithm reinforcement learning to improve DevOps key performance indicators KPIs, and learning from historical data to improve task execution time estimation, scheduling, mapping and resource allocation decisions. The present invention integrates reinforcement learning with key performance indicators to continuously analyze DevOps performance indicators, and adjusts process parameters to maximize efficiency and effectiveness, thereby achieving efficient development and deployment of automotive systems.

Description

Technical Field

[0001] The present invention relates to a development and operation optimization method, and in particular to a development and operation optimization system and method for a large-scale multi-component system based on reinforcement learning. Background Art

[0002] In the automotive industry, DevOps is a methodology that combines software development (Dev) and IT operations (Ops) to enhance collaboration, automation, and efficiency in the software development lifecycle. However, optimizing DevOps for large-scale systems in an industry like automotive presents unique challenges due to the complexity and criticality of the systems involved.

[0003] The automotive industry is increasingly dependent on complex software systems to enhance vehicle functionality, safety, and user experience. As software becomes a critical component, efficient DevOps practices are essential for fast and reliable software delivery. However, optimizing DevOps for large-scale multi-component systems faces significant challenges. Summary of the invention

[0004] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a development and operation optimization system and method for a large-scale multi-component system based on reinforcement learning.

[0005] To achieve the purpose of the present invention, the technical solution adopted by the present invention is:

[0006] A development and operation optimization system for large-scale multi-component systems based on reinforcement learning, including a task dependency extractor, a sequence generator, a task scheduler, a task mapper, and an optimization engine;

[0007] Task dependency extractor, which analyzes and extracts the dependencies between various tasks in the waiting queue to form a dependency matrix in the form of a directed acyclic graph (DAG);

[0008] The sequence generator performs static planning based on the dependency matrix generated by the task dependency extractor to generate a series of task plan sequences that can be executed concurrently;

[0009] Task scheduler, the sequence generator completes the allocation of execution plan sequences, and the task scheduler schedules the tasks in each sequence according to a specific scheduling algorithm;

[0010] The task mapper maps tasks in the ready queue to specific CI work processes in the cluster. Each work process represents a certain computing resource in a labeled form.

[0011] The optimization engine uses algorithmic reinforcement learning to dynamically optimize the system, improve DevOps key performance indicators (KPIs), and learn from historical data to improve task execution time estimation, scheduling, mapping, and resource allocation decisions.

[0012] Furthermore, each task in the dependency matrix contains an estimated execution time, the time source of which is the reinforcement learning result based on the task history data or manual estimation; within the task planning sequence, the tasks are sorted by execution time to avoid deadlock.

[0013] Furthermore, the task scheduler defines the order in which tasks are executed by assigning a priority to each task and transfers the highest priority task to the ready queue.

[0014] Furthermore, it also includes a real-time operation monitor and resource manager;

[0015] Run monitors in real time to continuously monitor the performance of the system, providing real-time feedback on resource usage, potential bottlenecks, or any failures;

[0016] The Resource Manager oversees resource allocation and management across the cluster, working with the Task Scheduler and Task Mapper to manage resource availability.

[0017] A method for optimizing development and operation of a large-scale multi-component system based on reinforcement learning, comprising the steps of:

[0018] (1) Generate CI / CD pipeline tasks by submitting code or specific events, use the task dependency extractor to extract any dependencies and generate a task dependency matrix;

[0019] (2) Perform static planning based on the dependency matrix generated by the task dependency extractor, and generate multiple concurrent execution plan sequences based on the task execution time and dependency relationships generated by the previous order;

[0020] (3) The sequence generator completes the allocation of execution plan sequences and uses the scheduling algorithm to allocate tasks according to priority;

[0021] (4) Map tasks in the ready queue to specific CI work processes within the cluster. Each work process represents a certain computing resource in a labeled form.

[0022] (5) Optimize DevOps based on reinforcement learning, optimize key performance indicators, and learn from historical data to improve task execution time estimation, scheduling, mapping, and resource allocation decisions.

[0023] Furthermore, step (5) specifically includes:

[0024] (5.1) Clarify the optimization goals and KPIs of reinforcement learning and select the appropriate algorithm based on these goals;

[0025] (5.2) Modeling tasks and resources through state space, action space, and reward functions;

[0026] (5.3) Select appropriate reinforcement learning algorithms based on different objectives;

[0027] (5.4) Conduct data collection and DevOps environment modeling;

[0028] (5.5) Conduct reinforcement learning training, use a simulated training environment, create a simulated DevOps environment for multiple rounds of training, and repeatedly optimize task scheduling and resource allocation strategies;

[0029] (5.6) After training and optimization are completed, the reinforcement learning model is deployed to the production environment and continues to be optimized and adjusted through real-time feedback;

[0030] (5.7) Regularly check the effectiveness of the model and make adjustments, regularly evaluate the KPIs in DevOps, ensure the optimization effect of the model in the actual production environment, and iterate and optimize the reinforcement learning model based on the actual results.

[0031] Furthermore, in step (5.1), the DevOps optimization KPIs include: build time, task scheduling efficiency, system resource utilization, fault recovery time, and build success rate.

[0032] Furthermore, in step (5.3), task scheduling and resource allocation use deep Q learning, deep deterministic policy gradient, and proximal policy optimization algorithms; fault recovery and load balancing use multi-agent reinforcement learning and model-based reinforcement learning algorithms; and multi-stage task optimization uses a hierarchical reinforcement learning algorithm.

[0033] Furthermore, in step (5.4), the steps of DevOps environment modeling are: collecting log data and indicator collection of CI / CD pipelines; collecting the usage of resources such as CPU, memory, network, and bottleneck data during task execution; using graph models to describe the dependencies between tasks to help optimize task scheduling.

[0034] The beneficial effects of the present invention are that, compared with the prior art, the present invention includes the following advantages:

[0035] Dynamic adaptability: The system dynamically adjusts based on workload conditions and system constraints;

[0036] Real-time optimization: Reinforcement learning algorithms enable real-time decision making to maximize system efficiency and reliability;

[0037] Scalability: The invention is applicable to large-scale multi-component systems with different CI workflows and resource requirements;

[0038] Comprehensive performance analytics: Integrating reinforcement learning with key performance indicators provides insights into system behavior and facilitates continuous improvement.

[0039] The present invention solves the challenge of efficiently managing large-scale multi-component systems in the automotive industry by utilizing reinforcement learning algorithms, continuously learning and adapting, improving key performance indicators, and enhancing the overall development and deployment process, thereby accelerating the innovation cycle, improving product quality, and enhancing customer satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the development and operation optimization method of a large-scale multi-component system based on reinforcement learning described in the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of this application.

[0042] The development and operation optimization system for large-scale multi-component systems based on reinforcement learning described in the present invention uses reinforcement learning algorithms to dynamically adjust DevOps processes to improve key performance indicators such as cycle time, delivery time, uptime, error rate, infrastructure utilization, integration frequency, build success rate, and pipeline stability. The system identifies bottleneck resources in the continuous integration (CI) pipeline, provides actionable improvement suggestions, and dynamically applies them to the system to achieve efficient development and deployment of automotive systems.

[0043] The system includes Real-Time Cluster DevOps (RTCD) and Reinforcement Learning modules. Real-Time Cluster DevOps (RTCD): This is a comprehensive tool that facilitates the analysis and evaluation of DevOps performance at different levels of abstraction. RTCD automates task dependency extraction, sequence generation, task scheduling, and task mapping, enabling designers to optimize CPU utilization for multi-core systems. Reinforcement Learning Module: Reinforcement learning algorithms are incorporated into the system to continuously analyze DevOps performance metrics and adjust process parameters to maximize efficiency and effectiveness.

[0044] Specifically, the development and operation optimization system of a large-scale multi-component system based on reinforcement learning described in the present invention includes a task dependency extractor, a sequence generator, a task scheduler, a task mapper, and an optimization engine.

[0045] The task dependency extractor analyzes and extracts the dependencies between various tasks in the waiting queue to form a dependency matrix in the form of a directed acyclic graph (DAG) to determine which tasks need to be executed in sequence and which tasks can run simultaneously. Each task in the dependency matrix contains an estimated execution time, which is derived from reinforcement learning results based on task history data or manual estimation.

[0046] The sequence generator performs static planning based on the dependency matrix generated by the task dependency extractor to generate a series of task schedule sequences that can be executed concurrently. It considers system resources, task priorities, and other constraints (such as specific resource availability) to create execution schedule sequences. Within each task schedule sequence, tasks are arranged in the most efficient order by execution time while avoiding deadlock.

[0047] Task Scheduler, After the sequence generator completes the allocation of execution plan sequences, the task scheduler starts to schedule the tasks in each sequence according to a specific scheduling algorithm (such as earliest deadline first EDF, rate monotonic RM, etc.). The task scheduler defines the order of task execution by assigning a priority to each task and transfers the highest priority task to the ready queue.

[0048] The task mapper maps tasks in the ready queue to specific CI worker processes within the cluster. Each worker process represents a certain computing resource in a labeled form, ensuring efficient use of CPU cores, memory, and other resources. Dynamically adjust scheduling based on real-time system performance, resource availability, and task priority. Balance the load between available resources to prevent specific nodes from being overloaded while ensuring that all tasks are completed in a timely manner.

[0049] The optimization engine uses algorithmic reinforcement learning to dynamically optimize the system, constantly seeking to improve DevOps key performance indicators KPIs such as build time, error rate, integration frequency, etc. Learn from historical data to improve task execution time estimation, scheduling, mapping, and resource allocation decisions.

[0050] The development and operation optimization system for a large-scale multi-component system based on reinforcement learning described in the present invention also includes a real-time operation monitor and a resource manager.

[0051] Runs a real-time monitor that continuously monitors the system's performance (CPU utilization, memory usage, task progress, etc.). Provides real-time feedback on resource usage, potential bottlenecks, or any failures. Can trigger the rescheduling or remapping of tasks to adapt to system changes or dynamically optimize performance.

[0052] Resource Manager, which oversees resource allocation and management for the entire cluster. Ensures that resources are efficiently allocated between tasks to avoid underutilization or resource contention. Collaborates with the Task Scheduler and Task Mapper to manage resource availability.

[0053] like Figure 1 As shown, the development and operation optimization method of a large-scale multi-component system based on reinforcement learning according to the present invention specifically includes the following steps:

[0054] (1) Dependency generation: Generate CI / CD pipeline tasks by submitting code or specific events, use the task dependency extractor to extract any dependencies and generate a task dependency matrix;

[0055] A directed acyclic graph (DAG) is obtained using a conventional iterative algorithm and then converted into a task dependency matrix. The execution time of each task is estimated. Reinforcement learning is used to estimate the historical records, and new tasks can be estimated manually.

[0056] (2) Sequence generation: static planning based on the dependency matrix generated by the task dependency extractor;

[0057] Generate multiple concurrent execution plan sequences based on the task execution time and dependency relationships generated by the previous order. Sort the tasks in each execution plan task sequence by their execution time.

[0058] (3) Task scheduling: Use scheduling algorithms (such as EDF and RM) to assign tasks according to priority;

[0059] After the sequence generator completes the allocation of execution plan sequences, the task scheduler selects the earliest deadline first (EDF) or rate monotonic RM to schedule the tasks in each sequence. The scheduling strategy can be optimized through reinforcement learning.

[0060] The task scheduler defines the order in which tasks are executed by assigning a priority to each task and moves the highest priority task to the ready queue.

[0061] (4) Task orchestration: Mapping tasks in the ready queue to specific CI work processes within the cluster;

[0062] Each worker process represents certain computing resources in a labeled form. For example, if a task requires a specific compiler or GPU resource, the task mapper needs to schedule the task to a CI worker process with the above label.

[0063] The resource manager needs to allocate sufficient CPU, memory, storage space and other resources to the computing node where the CI is located to ensure the smooth completion of the task.

[0064] (5) Optimizing DevOps based on reinforcement learning (RL). The overall goal is to intelligently improve the task scheduling, resource allocation, load balancing, and fault recovery in the DevOps process through reinforcement learning, and ultimately optimize key performance indicators (KPIs), such as build time, error rate, and integration frequency. This can be achieved through the following steps:

[0065] (5.1) Clarify optimization goals and KPIs;

[0066] In the process of implementing reinforcement learning to optimize DevOps, we must first define clear optimization goals and select corresponding algorithms based on these goals. Common DevOps optimization KPIs include:

[0067] Build time: Reduce build, test, and deployment time in CI / CD pipelines;

[0068] Task scheduling efficiency: improve the concurrency and resource utilization of task scheduling;

[0069] System resource utilization: optimize the use of CPU, memory, disk and other resources;

[0070] Failure recovery time: quickly restore services when a failure occurs;

[0071] Build success rate: Reduce the probability of build failure and increase integration frequency.

[0072] (5.2) Task and resource modeling;

[0073] In the DevOps process, tasks are usually composed of multiple subtasks, each of which may be dependent on other tasks, and resources include computing, storage, network, etc. When using reinforcement learning to optimize task scheduling, it is necessary to model through state space, action space and reward function.

[0074] State space: represents the current state of the system, including resource utilization, task dependencies, execution progress, etc.

[0075] Action space: The actions that the system can take, such as task scheduling, resource allocation, task mapping, etc.

[0076] Reward function: Evaluate the benefits of each action. For example, completing a task, reducing resource contention, and improving system throughput can be used as positive rewards, while task failure, resource overload, etc. can be used as negative rewards.

[0077] (5.3) Select an appropriate reinforcement learning algorithm;

[0078] Choose the appropriate reinforcement learning algorithm based on the characteristics and requirements of DevOps. For different scenarios, use the following methods:

[0079] 1. Task scheduling and resource allocation;

[0080] Deep Q-learning (DQN) or Double DQN (DDQN): In more complex task scheduling, the agent learns the optimal task execution order and resource allocation plan, which can be used to optimize the task execution order of the CI / CD pipeline and reduce the build time.

[0081] Deep Deterministic Policy Gradient (DDPG): For resource allocation problems, DDPG is suitable for processing continuous action spaces, for example, dynamically adjusting computing resources, memory allocation, etc.

[0082] Proximal Policy Optimization (PPO): When multiple tasks and multiple resource pools are involved, PPO can optimize the task scheduling strategy, reduce the risk of over-estimation, and provide a more stable optimization process.

[0083] 2. Fault recovery and load balancing;

[0084] Multi-agent reinforcement learning (MARL): In large-scale distributed systems, multiple agents can represent different nodes or services and collaborate to complete task scheduling, load balancing, and fault recovery.

[0085] Model-Based Reinforcement Learning (Model-Based RL): By simulating system behavior, potential bottlenecks or failure points in the task execution process can be predicted in advance, and adjustments and optimizations can be made in advance.

[0086] 3. Multi-stage task optimization (such as CI / CD pipeline);

[0087] Hierarchical reinforcement learning (HRL): The entire CI / CD pipeline task is broken down into multiple subtasks (such as building, testing, and deployment), and each subtask is optimized using a separate sub-agent. HRL can simplify the decision-making process and improve the efficiency of multi-stage processes.

[0088] (5.4) Data collection and environmental modeling;

[0089] Reinforcement learning relies on interaction with the environment, so a large amount of historical data is required to build a training environment. The following are the key steps for DevOps environment modeling:

[0090] Log and indicator collection: Collect log data of CI / CD pipelines, including build time, build failure rate, task scheduling delay, etc.

[0091] System resource monitoring: collects the usage of CPU, memory, network and other resources, as well as bottleneck data during task execution;

[0092] Task dependency modeling: Use graph models (such as directed acyclic graphs (DAGs)) to describe the dependencies between tasks and help optimize task scheduling.

[0093] (5.5) Reinforcement learning training and exploration;

[0094] To ensure that the RL agent can optimize task scheduling and resource allocation, effective training is required. During the training process, it is necessary to balance exploration and exploitation to ensure that the RL agent can not only use the known optimal strategy, but also explore new possibilities.

[0095] Use the epsilon-greedy strategy to conduct a lot of exploration in the early stages, gradually reduce randomness, and increase experience-based utilization. Use a simulated training environment to create a simulated DevOps environment in which the agent can conduct multiple rounds of training and repeatedly optimize task scheduling and resource allocation strategies without having to execute directly in the production environment every time.

[0096] The reward function is continuously adjusted based on the agent's feedback during training to ensure that the reward mechanism can incentivize the most beneficial behavior (such as optimizing build time or reducing error rate).

[0097] (5.6) Deployment and continuous learning;

[0098] After training and optimization are completed, the reinforcement learning model can be deployed to a production environment and continue to be optimized and adjusted through real-time feedback.

[0099] Through real-time monitoring systems (such as real-time operation monitors), data on environmental changes, such as task progress, resource consumption, and system load, are continuously collected to adjust optimization strategies in a timely manner.

[0100] By adopting online reinforcement learning strategies, as the environment changes (such as resource changes, new tasks, etc.), the agent can continuously update its own strategies to maintain the system's optimization capabilities in a constantly changing environment.

[0101] (5.7) Continuous optimization and feedback;

[0102] The optimization of reinforcement learning models is an ongoing process that requires regular checks on the model's effectiveness and adjustments. Regular evaluation of KPIs in DevOps, such as build time, error rate, task completion rate, etc., ensures the optimization effect of the model in the actual production environment. Iterate and optimize the reinforcement learning model based on the actual results. As the tasks and environment change, the learning strategy of the agent needs to be continuously updated.

[0103] The beneficial effect of the present invention is that, compared with the prior art, the present invention solves the challenge of efficiently managing large-scale multi-component systems in the automotive industry by utilizing reinforcement learning algorithms, continuous learning and adaptation, improving key performance indicators, and enhancing the overall development and deployment process, thereby accelerating the innovation cycle, improving product quality, and enhancing customer satisfaction.

[0104] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. A development and operation optimization system for large-scale multi-component systems based on reinforcement learning, characterized in that: Includes task dependency extractor, sequence generator, task scheduler, task mapper, and optimization engine; The task dependency extractor analyzes and extracts the dependencies between various tasks in the waiting queue to form a dependency matrix in the form of a directed acyclic graph (DAG). Each task in the dependency matrix contains an estimated execution time, which is derived from reinforcement learning results or manual estimates based on task history data. Within the task plan sequence, the execution time is used to sort the tasks to avoid deadlock. The sequence generator performs static planning based on the dependency matrix generated by the task dependency extractor to generate a series of concurrently executed task plan sequences; Task scheduler, the sequence generator completes the allocation of execution plan sequences, and the task scheduler schedules the tasks in each sequence according to a specific scheduling algorithm; The task mapper maps tasks in the ready queue to specific CI work processes in the cluster. Each work process represents a certain computing resource in a labeled form. The optimization engine uses algorithmic reinforcement learning to dynamically optimize the system, improve DevOps key performance indicators (KPIs), and learn from historical data to improve task execution time estimation, scheduling, mapping, and resource allocation decisions.

2. The development and operation optimization system for large-scale multi-component systems based on reinforcement learning according to claim 1, characterized in that: The task scheduler defines the order in which tasks are executed by assigning a priority to each task and moves the highest priority task to the ready queue.

3. The development and operation optimization system for large-scale multi-component systems based on reinforcement learning according to claim 1, characterized in that: It also includes a real-time operation monitor and resource manager; Run monitors in real time to continuously monitor the performance of the system, providing real-time feedback on resource usage, potential bottlenecks, or any failures; The Resource Manager oversees resource allocation and management across the cluster, working with the Task Scheduler and Task Mapper to manage resource availability.

4. A method for optimizing the development and operation of a large-scale multi-component system based on reinforcement learning, based on the system for optimizing the development and operation of a large-scale multi-component system based on reinforcement learning according to any one of claims 1 to 3, characterized in that: Includes steps: (1) Generate CI / CD pipeline tasks by submitting code or specific events, use the task dependency extractor to extract any dependencies and generate a task dependency matrix; (2) Perform static planning based on the dependency matrix generated by the task dependency extractor, and generate multiple concurrent execution plan sequences based on the task execution time and dependency relationships generated by the previous order; (3) The sequence generator completes the allocation of execution plan sequences and uses the scheduling algorithm to allocate tasks according to priority; (4) Map tasks in the ready queue to specific CI work processes within the cluster. Each work process represents a certain computing resource in a labeled form. (5) Optimize DevOps based on reinforcement learning, optimize key performance indicators, and learn from historical data to improve task execution time estimation, scheduling, mapping, and resource allocation decisions.

5. The development and operation optimization method of a large-scale multi-component system based on reinforcement learning according to claim 4 is characterized in that: Step (5) specifically includes: (5.1) Clarify the optimization goals and KPIs of reinforcement learning and select the appropriate algorithm based on these goals; (5.2) Modeling tasks and resources through state space, action space, and reward functions; (5.3) Select appropriate reinforcement learning algorithms based on different objectives; (5.4) Conduct data collection and DevOps environment modeling; (5.5) Conduct reinforcement learning training, use a simulated training environment, create a simulated DevOps environment for multiple rounds of training, and repeatedly optimize task scheduling and resource allocation strategies; (5.6) After training and optimization are completed, the reinforcement learning model is deployed to the production environment and continues to be optimized and adjusted through real-time feedback; (5.7) Regularly check the effectiveness of the model and make adjustments, regularly evaluate the KPIs in DevOps, ensure the optimization effect of the model in the actual production environment, and iterate and optimize the reinforcement learning model based on the actual results.

6. The development and operation optimization method for a large-scale multi-component system based on reinforcement learning according to claim 5 is characterized in that: In step (5.1), DevOps optimization KPIs include: build time, task scheduling efficiency, system resource utilization, fault recovery time, and build success rate.

7. The development and operation optimization method of a large-scale multi-component system based on reinforcement learning according to claim 5, characterized in that: In step (5.3), task scheduling and resource allocation use deep Q learning, deep deterministic policy gradient, and proximal policy optimization algorithms; fault recovery and load balancing use multi-agent reinforcement learning and model-based reinforcement learning algorithms; multi-stage task optimization uses a hierarchical reinforcement learning algorithm.

8. The development and operation optimization method for a large-scale multi-component system based on reinforcement learning according to claim 5, characterized in that: In step (5.4), the steps of DevOps environment modeling are: collecting log data and indicator collection of CI / CD pipelines; collecting the usage of CPU, memory, and network resources, as well as bottleneck data during task execution; using graph models to describe the dependencies between tasks to help optimize task scheduling.

Images