Method for scheduling large-scale agile software projects based on deep reinforcement learning

Abstract

The application discloses a large-scale agile software project scheduling method based on deep reinforcement learning, and comprises the following steps: initializing state information of agile software projects and human resources in a large-scale agile software project scheduling environment; establishing a large-scale agile software project scheduling decision model based on deep reinforcement learning to generate a scheduling scheme for scheduling; collecting trajectory information generated in the scheduling process, storing the trajectory information into an experience replay pool and updating priorities of all trajectories; when the number of trajectories in the experience replay pool reaches a requirement, training parameters of the scheduling decision model by batch sampling trajectories according to the priorities; generating a better scheduling scheme by using the trained scheduling decision model for scheduling; and outputting an optimal scheduling scheme if the total value of developed user stories is stable. The application can generate an optimal scheduling scheme in time according to changes in the environment state in response to random dynamic events or uncertain factors in the agile software development process.

Description

Technical Field

[0001] This invention belongs to the fields of agile software project development scheduling optimization and deep reinforcement learning technology, and particularly relates to a large-scale agile software project scheduling method based on deep reinforcement learning. Background Technology

[0002] As the world slowly enters the post-pandemic era, global economic growth is slowing, and the software industry faces significant challenges. The global software industry is transitioning from traditional software models to the SaaS and cloud era, primarily in the following aspects: First, delivery methods are shifting from traditional standalone software to web-based delivery, requiring the construction of robust and stable hardware and software delivery and maintenance systems. Second, payment models are shifting from one-time payments to continuous payments, quickly addressing core pain points in customers' production and operation processes and cultivating users' willingness to pay continuously. Third, application scenarios are shifting from single-domain to multi-domain, covering the needs of the entire industry chain and achieving collaborative innovation and ecosystem cooperation. Unlike the traditional "waterfall" development methodology, agile development focuses on the evolution of user needs, dividing the development process into several sprints (iterations). At the end of each sprint, a software version is delivered for customer experience, and then continuously improved based on customer satisfaction until the final software version is delivered. Due to its iterative development and continuous delivery characteristics, agile development perfectly aligns with the software industry's transformation challenges regarding delivery methods and payment models. In terms of application scenarios, large-scale agile development projects span multiple functional departments, providing a breakthrough for software companies to transform from single-domain to multi-domain approaches. Therefore, research on large-scale agile software development is essential.

[0003] Scheduling is a crucial aspect of agile software development, playing a vital role in improving the overall efficiency and reducing costs of agile software projects. Optimization algorithms are tools for finding optimal scheduling solutions. Traditional exact algorithms rely on rigorous mathematical methods to establish a mathematical model of the problem and obtain a globally optimal solution using mathematical approaches. However, the introduction of rigorous mathematical methods makes it difficult for exact algorithms to guarantee finding the optimal solution within a finite time when solving large-scale problems. Currently, most researchers use metaheuristic algorithms for solving these problems. Metaheuristics is an iterative generative process that explores and develops the search space through intelligent combinations of different concepts. In this process, learning strategies are used to acquire and master information to effectively discover near-optimal solutions. Metaheuristic algorithms typically do not rely on the specific conditions of a particular problem, thus enabling their application to a wider range of areas. However, metaheuristic algorithms cannot guarantee a globally optimal solution; their performance depends on the specific problem or the designer's experience. Reinforcement learning models the scheduling problem as a Markov decision process, where a learning agent continuously interacts with the scheduling environment to generate experience, learning the optimal strategy (i.e., the scheduling scheme) from this experience to obtain the maximum reward. Reinforcement learning does not rely on specific data; it evaluates policy value based on experience gained from interactions and improves the policy by executing actions that maximize that value. Deep reinforcement learning algorithms further combine the high-dimensional feature extraction capabilities of deep learning, saving significant computational resources and storage space, and demonstrating strong adaptability to solving various problems.

[0004] Currently, existing agile software project scheduling methods cannot cope with dynamically changing user needs, resulting in failed software delivery, low market stickiness, and poor returns. Therefore, a large-scale agile software project scheduling method based on deep reinforcement learning is proposed. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a large-scale agile software project scheduling method based on deep reinforcement learning, which solves the aforementioned problems.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a large-scale agile software project scheduling method based on deep reinforcement learning, comprising the following steps:

[0007] S1: Initialize the status information of agile software projects and human resources in the large-scale agile software project scheduling environment;

[0008] S2: Establish a large-scale agile software project scheduling decision model based on deep reinforcement learning, generate scheduling schemes, and carry out scheduling.

[0009] S3: Collect trajectory information generated during the scheduling process, store it in the experience replay pool, and update the priority of all trajectories;

[0010] S4: When the number of trajectories in the experience replay pool reaches the requirement, sample trajectories in batches according to priority to train the parameters of the scheduling decision model;

[0011] S5: Use the trained scheduling decision model to generate a better scheduling scheme for scheduling;

[0012] S6: If the total value of the developed user stories has not yet stabilized, continue to collect higher priority trajectories during the scheduling process of a better scheduling scheme to replace the lower priority trajectories in the original experience pool, thereby enhancing the parameter training effect of the scheduling decision model; if the total value of the developed user stories stabilizes, then output the optimal scheduling scheme.

[0013] Based on the above technical solutions, the present invention also provides the following optional technical solutions:

[0014] Further technical solution: The large-scale agile software project P is required to be completed within D sprints (l = 1, 2, ..., D), and the agile software project and human resources are scheduled in each sprint;

[0015] Based on user needs, agile software projects are described as N user stories. i (i = 1, 2, ..., N) constitute the set of user stories to be developed, and the attribute values ​​of the user stories include story points, value points and workload;

[0016] Among them, the user story us i Story Points i This is a unitless estimate of the size and complexity of a user story. i Value points VP i This is a unitless estimate of the revenue from the user story market. i workload i Indicates the time required to complete a user story, in hours;

[0017] One of the aforementioned story points is equivalent to approximately 8 hours of work.

[0018] Among them, user stories i Can be decomposed into tn i A specific task t ij (j = 1, 2, ..., tn) i A total of S skills are needed to complete these tasks;

[0019] User Story i Medium task t ij The included attributes are the skills (TSK) required to complete the mission. ij and required duration th ij ;

[0020] The agile development company has a total of R teams. r (r = 1, 2, ..., R), each team has M members. r employee e rk (k = 1, 2, ..., M) r ), employee e rk The skills mastered constitute a skill set (esk) rk Because each sprint dynamic event causes changes in employee working hours, the employee's working hours in each sprint... Also set as a variable value.

[0021] Further technical solution: The scheduling of the large-scale agile software project includes three coupled sub-problems:

[0022] Story selection: Choose a story from the list of stories to be developed and enter the final sprint;

[0023] Story allocation involves assigning user stories entering the sprint phase to various teams; and

[0024] Task allocation breaks down the story that each team needs to complete into tasks and assigns them to employees in the team according to the required skills and time.

[0025] A further technical solution: The scheduling of the sub-problem is represented by three decision variables:

[0026] Story Selection

[0027] Story Allocation

[0028] Task allocation

[0029] Further technical solution: The scheduling scheme generated in S2 satisfies the following conditions:

[0030] Select User Story in the first sprint. i Story Points SP i The sum does not exceed the sprint speed SV l ;

[0031] in

[0032] Sprint speed That is, the team speed of all teams. sum;

[0033] The team speed for the l-th sprint is the average of the total number of story points completed by each team in the first two sprints, denoted as:

[0034] Since no user stories were completed in the first sprint, the sum of all team members' working hours was used to estimate the initial team velocity, with 1 story point ≈ 8 hours. The calculation method is as follows:

[0035] The sprint speed of the second sprint is the sum of the complete user story points completed in the first sprint, calculated as follows: In the formula, OUT 1 This represents the complete set of user stories indicating the completion of the first sprint;

[0036] In the first sprint, the selected story is assigned to the team. r The sum of story points does not exceed the team's team speed.

[0037] The first sprint completes the user story. i Medium task t ij Required skills TSK ij Must be included in the assigned team staff e rk The set of skills mastered (esk) rk Among them;

[0038] The first sprint user story i Task t ij Only one team member can participate. rk To complete;

[0039] The team members were assigned e during the l-th sprint. rk Story mission duration th ij The sum does not exceed their working hours

[0040] Further technical solution: The process of establishing the large-scale agile software project scheduling decision model based on deep reinforcement learning is as follows:

[0041] Deep reinforcement learning is modeled as a quadruple<S,A,R,γ> The Markov decision process is represented;

[0042] in,

[0043] S represents a finite set of states.

[0044] In the formula, This represents the completion rate of the assigned full user story after the l-th sprint. This represents the completion rate of the assigned user story workload after the l-th sprint. This represents the average complete user completion rate for all teams after the l-th sprint. This represents the standard deviation of the complete user story completion rate for all teams after the l-th sprint. This represents the average workload completion rate of all teams after the l-th sprint. This represents the standard deviation of the workload completion rate of all teams after the l-th sprint. This represents the average time utilization rate of all teams after the l-th sprint. This represents the standard deviation of time utilization for all teams after the l-th sprint. This represents the average time utilization rate of all team members after the l-th sprint. This represents the standard deviation of time utilization rate for all team members after the l-th sprint.

[0045] A represents a finite set of actions.

[0046] In the formula, seven general scheduling rules are designed for the three decision variables of large-scale agile software project scheduling. The seven general scheduling rules of the three decision variables are combined to obtain 12 composite scheduling rules that constitute an action set.

[0047] R represents the reward function after the action is performed. It is the immediate reward after the agent executes the action rules in the current sprint state. It is an important guiding signal for the agent to learn and improve its strategies. The reward function is represented as the total value of the complete user story completed after the l-th sprint ends.

[0048] γ is a decay factor, with a value between [0,1], which represents the importance of subsequent decisions to the current state.

[0049] The state S of the agent after the l-th sprint ends l Next, select action A. l ;

[0050] User stories and tasks are assigned in the scheduling environment, employees' remaining working hours decrease, and the number of completed user stories increases;

[0051] After the (l+1)th sprint, the environment rewards the agent with R. l+1 The environment status will be updated to S. l+1 , represented as

[0052] A further technical solution: The trajectory information in S3 is a quintuple. l A l ,R l ,S l+1 ​,done>; where done is a flag to determine whether the scheduling has terminated. When the set of user stories to be developed is empty, the scheduling ends and done = True; otherwise, done = False.

[0053] Further technical solution: The scheduling decision model in S4 is trained using a dual deep Q-network algorithm based on priority experience replay of composite scheduling rules, including the current Q-network, the target Q-network, and the experience replay pool.

[0054] Further technical solution: The specific steps of S4 include:

[0055] S401. Put the trajectory information collected in step S3 into the experience replay pool, calculate the trajectory temporal difference error, and update the priority of all trajectories in the experience replay pool.

[0056] S402. When the number of trajectories in the pool reaches the limit, the trajectories in the pool are sampled in the smallest sampling batch and supplied to the target Q network for Q value prediction.

[0057] S403. The target Q network will pass the predicted Q value to the current Q network for gradient descent training. The current Q network with trained neural network parameters will periodically copy the parameters to the target Q network to improve the prediction ability of the target Q network.

[0058] S404. The current Q network will select the action rule with the largest Q value to execute based on the status characteristics of the agile software project and human resources in each sprint.

[0059] Beneficial effects

[0060] This invention provides a large-scale agile software project scheduling method based on deep reinforcement learning, which has the following advantages compared with existing technologies:

[0061] 1. The invention can be centered on evolving user needs, dividing the development process into several sprints (iterations). At the end of each sprint, a software version is delivered for customer experience. Then, based on customer satisfaction, the software is continuously improved until the final software version is delivered. At the same time, the invention can select the optimal scheduling and allocation scheme based on the project development value obtained by each sprint, which can improve the work efficiency of team members and ensure the rationality and reliability of resource scheduling, thereby improving the operational effect of large-scale agile development in agile software project scheduling.

[0062] 2. This invention adopts a deep reinforcement learning-based experience replay dual deep Q-network algorithm based on composite scheduling rules and a Markov decision process to generate a scheduling scheme, which can effectively adapt to the dynamic changes in user needs and further improve the reliability and rationality of agile software project scheduling.

[0063] 3. Finally, by selecting the optimal scheduling scheme through the total value of the complete user story, this invention can guarantee the overall benefits of the optimal scheduling scheme and the work efficiency of team members, thereby further improving the operational effect of large-scale agile development in agile software project scheduling. Attached Figure Description

[0064] Figure 1 This is a flowchart of the large-scale agile software project scheduling method based on deep reinforcement learning of the present invention.

[0065] Figure 2 This invention provides a framework for solving deep reinforcement learning algorithms.

[0066] Figure 3 This is a schematic diagram illustrating the convergence of the average cumulative reward (total value of the developed user stories) for deep reinforcement learning in this invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0068] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0069] Please see Figures 1-3 A method for scheduling large-scale agile software projects based on deep reinforcement learning is proposed. This method schedules large-scale agile software projects using a deep reinforcement learning algorithm based on the status information of each sprint agile software project and human resources. The specific steps include:

[0070] S1: Initialize the status information of agile software projects and human resources in the large-scale agile software project scheduling environment;

[0071] S2: Establish a large-scale agile software project scheduling decision model based on deep reinforcement learning, generate scheduling schemes, and carry out scheduling.

[0072] S3: Collect trajectory information generated during the scheduling process, store it in the experience replay pool, and update the priority of all trajectories;

[0073] S4: When the number of trajectories in the experience replay pool reaches the requirement, sample trajectories in batches according to priority to train the parameters of the scheduling decision model;

[0074] S5: Use the trained scheduling decision model to generate a better scheduling scheme for scheduling.

[0075] S6: If the total value of the developed user stories has not yet stabilized, continue to collect higher priority trajectories during the scheduling process of a better scheduling scheme to replace the lower priority trajectories in the original experience pool, thereby enhancing the parameter training effect of the scheduling decision model; if the total value of the developed user stories stabilizes, then output the optimal scheduling scheme.

[0076] Specifically, based on actual large-scale agile software development processes, user stories are concise descriptions of features from the customer's perspective. Depending on the type of feature, user stories are categorized into three types: essential stories (basic features), linearly incremental stories (features with continuous benefits), and excitement stories (features with explosive benefits). The software development process is divided into several sprints (iterations, typically around two weeks). At the end of each sprint, a software version consisting of several user stories is delivered for customer testing, and then continuously improved based on customer satisfaction. User stories are broken down into more specific and detailed tasks based on essential software development skills (such as requirements analysis, high-level design, graphic design, programming, and automated testing), with team members corresponding to the relevant skills committing to the time required to complete the task.

[0077] Specifically, the large-scale agile software project P is required to be completed within D sprints (l = 1, 2, ..., D), with each sprint involving the scheduling of agile software projects and human resources. Based on user needs, the agile software project is described as N user stories (us). i (i = 1, 2, ..., N), forming a set of user stories to be developed. Each user story has three attribute values: story point, value point, and workload. (User story us) i Story Points i This is a unitless estimate of the size and complexity of a user story; user story (us) i Value points VP i This is a unitless estimate of the revenue generated by the user story market; user story (us) i workload i This indicates the time required to complete a user story, in hours. One story point is approximately equivalent to 8 hours of work. (User Story - us) i It can be decomposed into tn i A specific task t ij (j = 1, 2, ..., tn) i A total of S skills are required to complete these tasks. (User Story) i Medium task t ij There are two attributes: the skill tsk required to complete the task. ij and required duration th ij The agile development company has a total of R teams. r (r = 1, 2, ..., R), each team has M members.r employee e rk (k = 1, 2, ..., M) r ), employee e rk The skills mastered constitute a skill set (esk) rk .

[0078] Specifically, considering the changes in employee working hours caused by each sprint dynamic change event, the employee's working hours in each sprint... It is also set as a variable value, and the dynamic change events include absence / overtime / vacation, etc.

[0079] Specifically, considering the changing customer needs for each sprint, the projected number of sprints to be completed is estimated based on the sprint speed and the number of completed sprints. If the projected number of sprints to be completed does not exceed D, user stories are randomly selected from the new user story set for priority development; otherwise, user stories with the lowest value per story point are deleted. During the process of adding and deleting user stories, the projected number of sprints to be completed must never exceed the target number of sprints D. That is, user stories will be added or deleted to respond to changes in customer needs for each sprint.

[0080] Specifically, the scheduling of large-scale agile software projects includes three coupled sub-problems:

[0081] Story selection: Choose a story from the list of stories to be developed and enter the final sprint;

[0082] Story allocation involves assigning user stories entering the sprint phase to various teams; and

[0083] Task allocation breaks down the story that each team needs to complete into tasks and assigns them to employees in the team according to the required skills and time.

[0084] Specifically, the scheduling of the subproblem is represented by three decision variables:

[0085] Story Selection

[0086] Story Allocation

[0087] Task allocation

[0088] Specifically, the scheduling scheme generated in S2 satisfies the following conditions:

[0089] (1) Select user story in the lth sprint. i Story Points SP i The sum does not exceed the sprint speed SV l Among them, sprint speed That is, the team speed of all teams. The sum. The team speed of the l-th sprint is the average of the total number of story points completed by each team in the first two sprints, denoted as: Since no user stories were completed in the first sprint, the sum of all team members' working hours was used to estimate the initial team velocity, with 1 story point ≈ 8 hours. The calculation method is as follows: The sprint speed of the second sprint is the sum of the complete user story points completed in the first sprint, calculated as follows: In the formula, OUT 1 This represents the complete set of user stories indicating the completion of the first sprint;

[0090] (2) In the lth sprint, the selected story is assigned to the team. r The sum of story points does not exceed the team's team speed.

[0091] (3) The lth sprint completes the user story. i Medium task t ij Required skills TSK ij Must be included in the assigned team staff e rk The set of skills mastered (esk) rk Among them;

[0092] (4) The first sprint user story i Task t ij Only one team member can participate. rk To complete;

[0093] (5) During the lth sprint, e is assigned to team members. rk Story mission duration th ij The sum does not exceed their working hours

[0094] Specifically, the process of establishing the large-scale agile software project scheduling decision model based on deep reinforcement learning is as follows: the scheduling process of the large-scale agile software project is represented by a Markov decision process, which consists of a set of four tuples.<S,A,R,γ> Indicated;

[0095] Specifically, S represents a finite set of states. In the formula, This represents the completion rate of the assigned full user story after the l-th sprint ends; This represents the completion rate of the user story workload assigned after the l-th sprint ends; This represents the average complete user completion rate for all teams after the l-th sprint. This represents the standard deviation of the complete user story completion rate for all teams after the l-th sprint. This represents the average workload completion rate of all teams after the l-th sprint. This represents the standard deviation of the workload completion rate of all teams after the l-th sprint; This represents the average time utilization rate of all teams after the l-th sprint. This represents the standard deviation of time utilization for all teams after the l-th sprint. This represents the average time utilization rate of all team members after the l-th sprint. This represents the standard deviation of time utilization rate for all team members after the l-th sprint.

[0096] Specifically, A represents a finite set of actions. In the formula, three general scheduling rules were designed for the three decision variables of large-scale agile software project scheduling, as shown in Table 1; the seven general scheduling rules of the three decision variables were combined to obtain 12 composite scheduling rules that constitute an action set. This indicates the composite scheduling rule 1: SSL+STL+TEL; This indicates composite scheduling rule 2: SSL+STL+TES; This indicates composite scheduling rule 3: SSL+STL+TER; This indicates composite scheduling rule 4: SSL+STS+TEL; This indicates composite scheduling rule 5: SSL+STS+TES; This indicates composite scheduling rule 6: SSL+STS+TER; This indicates composite scheduling rule 7: SSA+STL+TEL; This represents composite scheduling rule 8: SSA+STL+TES; This represents composite scheduling rule 9: SSA+STL+TER; This indicates composite scheduling rule 10: SSA+STS+TEL; Composite scheduling rule 11: SSA+STS+TES; This indicates composite scheduling rule 12: SSA+STS+TER;

[0097] Table 1 General Rules for Decision Variables

[0098]

[0099]

[0100] Specifically, the reward function It is the immediate reward for the agent after executing the action rules in the current sprint state, and serves as an important guiding signal for the agent to learn and improve its strategies. The reward function is represented as the total value of the complete user story completed after the l-th sprint.

[0101] Specifically, γ is a decay factor, which takes values ​​between [0,1] and represents the importance of subsequent decisions to the current state.

[0102] Specifically, after the l-th sprint, in state Sl, the agent selects to perform action A1 (based on the current Q-network in the algorithm). User stories and tasks are assigned in the scheduling environment, the remaining working time of employees decreases, and the number of completed user stories increases. After the (l+1)-th sprint, the environment rewards the agent with a reward R. l+1 (The value of completing the full user story in the (l+1)th sprint), the environment state will be updated to S. l+1 ; indicates

[0103] Specifically, the trajectory information is a quintuple. l A l ,R l ,S l+1 `<dispatch>,done>; where `done` is a flag indicating whether the scheduling has terminated. Scheduling ends when the set of user stories to be developed is empty, and `done = True`; otherwise, `done = False`.

[0104] Specifically, in step S4, the scheduling decision model adopts a dual-depth Q-network algorithm based on priority experience replay using composite scheduling rules, including the current Q-network, the target Q-network, and the experience replay pool, and its specific working steps are as follows:

[0105] First, the trajectory information collected in step S3 is put into the experience replay pool, the trajectory temporal difference error is calculated, and the priority of all trajectories in the experience replay pool is updated.

[0106] Secondly, when the number of trajectories in the pool reaches the limit, the trajectories in the pool are sampled in the smallest sampling batch and supplied to the target Q network for Q-value prediction.

[0107] Furthermore, the target Q-network will pass the predicted Q-value to the current Q-network for gradient descent training. The current Q-network, whose neural network parameters have been trained, will periodically copy the parameters to the target Q-network to improve the prediction ability of the target Q-network.

[0108] Finally, the current Q network will select the action rule with the highest Q value to execute based on the status characteristics of the agile software project and human resources in each sprint.

[0109] Specifically, the deep reinforcement learning algorithm framework used in this invention is as follows:​ Figure 2 As shown, before the first sprint begins, the state information of the large-scale agile software project scheduling and human resources is initialized, and the current state characteristics are calculated and passed to the agent (project manager). The agent randomly selects an action (composite scheduling rule) with probability ε, and selects the action with probability 1-ε using the current Q-network based on the maximum Q value. After the action is executed, the environmental state changes, story tasks are selected and assigned, and the remaining working time of team members decreases. At the end of the sprint, the new environmental state characteristics and reward value are calculated to determine whether the scheduling has ended. During each sprint, the current / next state characteristics, composite scheduling rules, reward (value of completing the story), and the determination of whether the scheduling has ended are combined into a trajectory and stored in the replay pool.

[0110] Specifically, the trained Q-network will select the action rule with the largest Q value to execute based on the state characteristics of the agile software project and human resources in each sprint.

[0111] Verification Example

[0112] The initial scheduling environment is set as follows: The total number of sprints is 16, each sprint lasts 2 weeks, with 5 working days per week and 8 hours per day. There are 200 user stories and 5 teams. The number of employees in Teams 1, 2, 3, 4, and 5 are 6, 8, 7, 5, and 5 respectively. The story points and value points of user stories are in the range {2, 3, 5, 8, 13}. The task duration of each user story is between [4, 8]. The skills required to complete the task and the skills possessed by the employees are represented by one of {1, 2, 3, 4, 5}. New user stories are added only considering linearly increasing stories and exciting stories. The selection of story points is the same as the original story range, and the value points are selected from {5, 8, 13}. Considering that employee working hours in each sprint may change due to dynamic events (absence / overtime / vacation, etc.) and internal and external factors, the employee working hours in each sprint will vary between [80, 100]. Considering the changing needs of each sprint customer, estimate the projected number of sprints to be completed based on the sprint speed and the number of completed sprints. If the projected number of sprints to be completed does not exceed 16, randomly select user stories from the new user story set for priority development; otherwise, delete the user stories with the lowest value per story point. Throughout the process of adding and deleting user stories, the projected number of sprints to be completed must never exceed the target number of sprints, which is 16.

[0113] Figure 3 This is a schematic diagram illustrating the convergence of the average cumulative reward (total value of the developed user stories) of the deep reinforcement learning algorithm in this embodiment of the invention.

[0114] This invention proposes a large-scale agile software project scheduling method based on deep reinforcement learning, which can effectively adapt to the dynamic changes in user needs, ensuring that the developed software perfectly meets user expectations and enhances the close relationship between the developer and users. Furthermore, the generated optimal scheduling scheme not only effectively improves employee work efficiency but also further enhances the operational effectiveness of large-scale agile development in agile software project scheduling.

[0115] Existing methods fail to address dynamically changing user needs, leading to software delivery failures, low market stickiness, and poor returns. This invention addresses this by establishing a large-scale agile software project scheduling environment based on actual agile development processes. The invention employs a Markov decision process to design the state characteristics, action rules, and reward function of the large-scale agile software project scheduling environment. Furthermore, the invention optimizes the scheduling decision model's parameters using a dual deep Q-network algorithm with priority experience replay based on composite scheduling rules. Simultaneously, by using the total value of completing a full user story as the reward function, the optimal scheduling scheme is selected, ensuring that the software adapts to dynamic changes in customer needs within the software delivery deadline and enhances the market value of the developed software.

[0116] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for scheduling large-scale agile software projects based on deep reinforcement learning, characterized in that, The method comprises the following steps: S1: initializing the state information of agile software projects and human resources in a large-scale agile software project scheduling environment; S2: establishing a large-scale agile software project scheduling decision model based on deep reinforcement learning to generate a scheduling scheme for scheduling; S3: collecting trajectory information generated in the scheduling process and storing it in an experience replay pool and updating the priority of all trajectories; S4: when the number of trajectories in the experience replay pool reaches the requirement, the trajectories are batch-sampled according to the priority to train the scheduling decision model parameters; S5: using the trained scheduling decision model to generate a better scheduling scheme for scheduling; S6: if the total value of the developed user stories is not stable, continue to collect trajectories with higher priority in the scheduling process of the better scheduling scheme to replace the trajectories with lower priority in the original experience pool, and enhance the parameter training effect of the scheduling decision model; if the total value of the developed user stories is stable, output the optimal scheduling scheme; According to the requirement of the user, the agile software project is described as a user story , , which constitutes a set of user stories to be developed, and the attribute values included in the user story are story points, value points and workloads; wherein the user story has a story point which is a unitless estimate of the size and complexity of the user story has a value point which is a unitless estimate of the market benefit of the user story has a work amount which is the length of time in hours that it takes to complete the user story; The establishment process of the large-scale agile software project scheduling decision model based on deep reinforcement learning is as follows: Deep reinforcement learning is modeled as a Markov decision process represented by a quadruple ​ wherein, representing a limited set of states wherein, represents the completion rate of the assigned complete user stories after the end of the th sprint, represents the completion rate of the assigned user story effort after the end of the th sprint, represents the average of the complete user story completion rates of all teams after the end of the th sprint, represents the standard deviation of the complete user story completion rates of all teams after the end of the th sprint, represents the average of the effort completion rates of all teams after the end of the th sprint, represents the standard deviation of the effort completion rates of all teams after the end of the th sprint, represents the average of the time utilization rates of all teams after the end of the th sprint, represents the standard deviation of the time utilization rates of all teams after the end of the th sprint, represents the average of the employee time utilization rates of all teams after the end of the th sprint, represents the standard deviation of the employee time utilization rates of all teams after the end of the th sprint. representing a limited set of actions 7 general scheduling rules are designed for three decision variables of large-scale agile software project scheduling, and 12 composite scheduling rules are obtained by combining the 7 general scheduling rules of the three decision variables to constitute an action set; represents the reward function after performing the action , is the immediate reward of the agent after performing the action rule in the current sprint state, which is an important guiding signal for the agent to learn and improve the strategy, and the reward function is represented as the total value of the complete user story after the first sprint. is a decay factor, taking values in between 0 and 1, indicating the importance of the subsequent decision to perform an action on the current state; The agent is in a state after the end of the first sprint The agent is in a state after the end of the first sprint The agent is in a state after the end of the first sprint The agent is in a state after the end of the first sprint The user stories and tasks in the scheduling environment are assigned, the remaining working hours of the employees are reduced, and the completed complete user stories are increased; After the end of the episode, the environment gives the agent a reward , and the environment state is updated to , denoted as . ​ 2. The deep reinforcement learning based large-scale agile software project scheduling method of claim 1, wherein, The large-scale agile software project is required to be completed within sprints, Each sprint carries out the scheduling of the agile software project and human resources; One story point is approximately equal to 8 hours of work; User stories can be broken down into specific tasks , which collectively require skills; User stories Mid task Included attributes are the skills required to complete the task and the length of time required ; Agile development companies have teams , with employees , in each team whose skill sets form a skill set collection , the length of work of an employee in each sprint is also set to a varying value due to the dynamic events that occur in each sprint.

3. The deep reinforcement learning based large scale agile software project scheduling method of claim 2, wherein, The large-scale agile software project scheduling comprises three coupled sub-problems: story selection, selecting stories from the list of stories to be developed into sprints; story assignment, assigning user stories entering the sprint to each team; and task assignment, decomposing the stories that each team needs to complete into tasks and assigning them to employees in the team according to the required skills and time.

4. The method of claim 3, wherein, The scheduling of the sub-problems is represented by three decision variables: Story selection ; Story assignment ; Task allocation .

5. The method of claim 2, wherein, The scheduling scheme generated in S2 satisfies the following conditions: The sum of the story points of the user stories selected in a sprint does not exceed the sprint velocity ;​​ wherein sprint speed i.e. the sum of all team's team speed ​ No. The team speed for each sprint is the average of the total number of story points completed by each team in the first two sprints, denoted as: ; Since the first sprint did not complete any user stories, the sum of the work hours of all employees of the team is calculated as ; The sprint velocity of the second sprint is the sum of the complete user story points of the first sprint, calculated as where represents the set of complete user stories of the first sprint; The sum of the story points assigned to a team from the selected stories in a sprint does not exceed the team velocity of the team ;​​ No. A sprint to complete the user story Medium task Required skills Must be included in the assigned team staff Skill set mastered Among them; The first sprint user story task can only be completed by one team member . No. Each sprint is assigned to team members Story mission duration The sum does not exceed their working hours .

6. The method of claim 2, wherein the deep reinforcement learning-based scheduling of large-scale agile software projects is characterized by, The trajectory information in the S3 is a five-tuple ; wherein, is a flag to determine whether the schedule is terminated, and the schedule ends when the set of user stories to be developed is empty, ; otherwise, .

7. The deep reinforcement learning based large scale agile software project scheduling method of claim 1, wherein, The scheduling decision model in S4 is trained using a priority experience replay double deep Q network algorithm based on composite scheduling rules, including a current Q network, a target Q network and an experience replay pool.

8. The deep reinforcement learning based large-scale agile software project scheduling method of claim 7, wherein, The specific steps of S4 include: S401: placing the trajectory information collected in step S3 into the experience replay pool, calculating the trajectory time difference error, and updating the priority of all trajectories in the experience replay pool; S402: when the number of trajectories in the pool reaches the limit, the trajectories in the pool are sampled in batches according to the minimum sampling batch to provide the target Q network for Q value prediction; S403: the target Q network will pass the predicted Q value to the current Q network for gradient descent training, and the neural network parameter trained current Q network will periodically copy the parameters to the target Q network to improve the prediction ability of the target Q network; S404: the current Q network will select the action rule with the maximum Q value according to the state characteristics of the agile software project and human resources in each sprint to execute.

Images