Multi-stage task time delay control method based on semi-Markov decision
By constructing an action value function and potential function based on a semi-Markov decision-making method, and combining a time discount term and a shaping reward, the problem of poor latency optimization in multi-stage tasks is solved, and global latency optimization and resource utilization are improved.
Patent Information
- Application Number
- CN202610047185.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-14
- Publication Date
- 2026-05-19
AI Technical Summary
Traditional latency optimization methods neglect the connections between stages and the constraints of overall latency in multi-stage tasks, resulting in poor global latency optimization, low resource utilization, and an inability to meet the requirements of task completion efficiency in real-world scenarios.
A semi-Markov decision-making approach is adopted to construct an action value function and a potential function. By combining a time discount term and a shaping reward, actions are selected through an exploration-utilization balancing strategy, and the action value function is iteratively optimized to determine the optimal strategy, thereby achieving global latency optimization for multi-stage tasks.
It accelerates the convergence speed of the model, improves the latency control accuracy and resource utilization of multi-stage tasks, and meets the actual needs of complex scenarios.
Smart Images

Figure CN122063864A_ABST
Abstract
Description
Technical Field
[0001] The embodiments in this specification belong to the field of reinforcement learning, and in particular relate to a multi-stage task delay control method based on semi-Markov decision-making. Background Technology
[0002] As industrial production, intelligent manufacturing, and network scheduling become increasingly complex, multi-stage task execution is becoming more common. These tasks have clear dependencies between their different stages, and each stage has significantly different time and resource requirements. Traditional latency optimization methods typically focus only on independent decisions at a single stage, neglecting the connections between stages and the constraints of overall latency. This results in poor global latency optimization, low resource utilization, and an inability to meet the efficiency requirements of real-world scenarios. Therefore, a latency control method that can simultaneously consider global latency and resource efficiency in multi-stage tasks is needed. Summary of the Invention
[0003] Embodiments of this disclosure provide a multi-stage task delay control method based on semi-Markov decision-making, which aims to solve one or more of the above-mentioned problems and other potential problems.
[0004] According to a first aspect of this disclosure, a multi-stage task delay control method based on semi-Markov decision-making is provided. The method includes: constructing an action value function corresponding to the initial action of the target multi-stage task in its initialization state; determining a global reference duration based on historical running data of the target multi-stage task; and constructing a potential function corresponding to each stage state. In a single training round targeting the action value function, a first action corresponding to the current stage state is selected based on an exploration-utilization balancing strategy, and after executing the first action, the transition duration from the current stage state to the next stage state is determined. A time discount term is calculated based on the power of the transition duration using a preset discount factor. A shaping reward is calculated based on the time discount term and the potential function, and an update formula for the action value function is set based on the shaping reward. The action value function is iterated based on the update formula until the action value function meets the convergence condition. Finally, an optimal strategy is determined based on the converged action value function to control the target multi-stage task according to the optimal strategy.
[0005] According to a second aspect of this disclosure, a multi-stage task delay control device based on semi-Markov decision-making is provided. The device includes a function construction module configured to construct an action value function corresponding to the initial action of the target multi-stage task in the initialization state, determine a global reference duration based on historical running data of the target multi-stage task, and construct a potential function corresponding to each stage state; an action selection module configured to select the first action corresponding to the current stage state based on an exploration-utilization balance strategy in a single training round for the action value function, and determine the transition duration from the current stage state to the next stage state after executing the first action; a shaping reward calculation module configured to calculate a time discount term based on the power of the transition duration of a preset discount factor, calculate a shaping reward based on the time discount term and the potential function, and set an update formula for the action value function based on the shaping reward; and an optimization control module configured to iterate the action value function based on the update formula until the action value function meets the convergence condition, and determine the optimal strategy based on the converged action value function to control the target multi-stage task according to the optimal strategy.
[0006] According to a third aspect of this disclosure, an electronic device is provided, including one or more processors and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform a method provided according to a first scheme.
[0007] According to a fourth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided according to the first aspect.
[0008] The scheme provided in the embodiments of this specification can combine the non-fixed-length transition duration characteristics of semi-Markov decision with the shaping reward of unbiased potential function. By using a time discount term to match the actual time difference of each stage, the unbiased shaping reward can solve the reward sparsity problem of traditional reinforcement learning, accelerate convergence without changing the optimal policy, and achieve global latency optimization for multi-stage tasks. Attached Figure Description
[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:
[0010] Figure 1 A flowchart illustrating a multi-stage task delay control method based on semi-Markov decision-making, according to some embodiments of this disclosure, is shown.
[0011] Figure 2A schematic diagram of the structure of a multi-stage task delay control device based on semi-Markov decision-making according to some embodiments of the present disclosure is shown.
[0012] Figure 3 A schematic block diagram of an electronic device according to some embodiments of the present disclosure is shown. Detailed Implementation
[0013] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0014] The terms “comprising” and “having”, and any variations thereof, in this specification, claims, and the foregoing drawings are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. Depending on the context, the word “if” as it applies herein may be interpreted as “when”, “in response to determination”, or “in response to detection”.
[0015] As mentioned earlier, reinforcement learning is widely used for multi-stage decision-making in delay optimization control problems. However, traditional reinforcement learning methods, based on Markov decision processes, are only suitable for cases with fixed transition step sizes (fixed durations), and cannot cope with the variable time consumption of each stage in multi-stage tasks. Furthermore, due to the sparse reward signal problem in reinforcement learning, model training is often unstable and converges slowly in multi-stage, multi-time tasks, making it difficult to quickly learn suitable delay optimization strategies. In addition, while semi-Markov decision-making is suitable for handling tasks with variable delays and can compensate for the shortcomings of traditional reinforcement learning methods, becoming an ideal model for multi-stage delay optimization problems, current reinforcement learning methods based on semi-Markov decision-making lack targeted reward design. They neither guide the model to focus on the core objective of delay optimization through dense rewards nor balance convergence speed and delay control accuracy, resulting in models that still suffer from slow convergence and insufficient delay control capabilities, making it difficult to meet the practical needs of complex scenarios.
[0016] Figure 1The diagram illustrates a flowchart of a multi-stage task delay control method 100 based on semi-Markov decision-making, representing some embodiments of this disclosure. Method 100 can be executed by a terminal, which may include, but is not limited to, mobile phones, tablets, desktop computers, servers, etc. Method 100 can also be applied, for example, to interdisciplinary research in mathematics and formal sciences such as supply chain operations research. Figure 1 As shown, in method 100, step 102 can construct the action value function corresponding to the initial action of the target multi-stage task in the initialization state, determine the global reference duration based on the historical running data of the target multi-stage task, and construct the potential function corresponding to each stage state.
[0017] In this embodiment, the stage state sequence and action set of the target multi-stage task will be obtained. Each stage state represents a different stage of task execution, and the action set contains different strategies or decisions that can be executed at each stage. The action set may include actions such as "execute immediately," "wait n minutes," and "wait n+1 minutes." Next, the Q-value function for each state-action pair will be initialized. That is, the action value function, where This is a stage state. For action, The initial value can be set to zero. Furthermore, the training parameters of the action value function, including the learning rate, can be set according to actual needs. Exploration rate Discount Factor Next, the historical execution data of the multi-stage task is statistically analyzed. Based on the historical execution data, the historical characteristics of the stage duration are determined to calculate the median or mean time of each stage. The global reference duration is then calculated based on the sum of the median or mean times of all stages. This serves as a reference for the expected time to complete the task, providing foundational data for subsequent shaping rewards. Furthermore, the potential function for each stage state will be constructed. The potential function represents the "residual cost" from a stage state to the endpoint. Its design can be customized based on the specific task requirements. Common calculation methods include mapping the expected delay of a stage state or other business-related metrics to numerical values as the basis for shaping rewards, effectively guiding the evaluation of each stage state during the learning process. As an example, the potential function can be calculated as follows:
[0018]
[0019] in, From state The sum of the typical times for all adjacent stages from start to finish. This is the global reference duration, which is the median of the total duration from the start state to the end state of the order.
[0020] In method 100, step 104 may select the first action corresponding to the current stage state based on an exploration-utilization balancing strategy in a single training round for the action value function, and determine the transition duration from the current stage state to the next stage state after executing the first action.
[0021] In this embodiment, Q-learning of a semi-Markov decision process will be used to train the action value function. In a single training round, an exploration-exploitation balance strategy (e.g., Using strategies such as Softmax, the first action is selected based on the current stage state. This ensures that the action value function can be sufficiently explored in the early stages of training, avoiding getting trapped in local optima. By executing the selected first action, the value of the action from the current stage state can be obtained. To the next stage Continuous transfer time The continuous transfer time is the delay caused by the first action, which can be the sum of the base time and the delay caused by the selected first action. The base time is usually the predetermined reference time for each stage, that is, the reference time calculated by the aforementioned process based on historical operating data to estimate the standard time of each stage. The delay caused by the first action is the additional time spent executing the first action.
[0022] In method 100, step 106 can calculate the time discount term based on the power of the transfer duration of the preset discount factor, and then calculate the shaping reward based on the time discount term and the potential function, and set the update formula of the action value function based on the shaping reward.
[0023] In this embodiment, based on the discount factor and transfer duration It can calculate time discount items. The time discount term can influence the weight of future rewards, thereby accelerating latency control optimization during the learning process. Based on the time discount term and the potential function constructed in the preceding steps, the shaping reward can be calculated. The specific calculation method for the shaping reward can be adjusted according to different needs. As an example, the shaping reward... The formula for calculation is:
[0024]
[0025] in, This indicates a penalty for the duration of the task. For time-based discounts, and These are the potential function values for the next stage state and the current stage state, respectively.
[0026] Based on the shaping reward, the update formula for the action value function during training optimization can be set. This update formula updates the function value of the current stage state based on the shaping reward and the maximum function value of the next stage state. As an example, the update formula could be:
[0027]
[0028] in, The learning rate is typically set to 0.1. This is the next stage of the status quo. This is for the next stage of action.
[0029] In method 100, step 108 can iterate the action value function based on the update formula until the action value function meets the convergence condition. Then, the optimal strategy is determined based on the converged action value function, so as to control the target multi-stage task according to the optimal strategy.
[0030] In this embodiment, the action value function can be iteratively optimized according to the pre-set update formula until it meets the preset convergence condition. Based on the converged action value function, the optimal action value for each stage state can be determined, thereby setting the optimal strategy. The execution process of the target multi-stage task is then controlled according to the optimal strategy, that is, the execution control of corresponding actions is performed according to the optimal strategy at different time points and different stage states, thereby achieving global latency optimization of the multi-stage task.
[0031] In one possible implementation, the first action corresponding to the current stage state is selected based on an exploration-utilization balancing strategy, including:
[0032] Random actions are selected based on a preset exploration rate, and the action that maximizes the function value is selected based on the utilization rate. The first action corresponding to the current stage state is obtained, and the utilization rate is the difference between 1 and the exploration rate.
[0033] In this embodiment, the exploration rate will be determined based on a predetermined rate. , with exploration rate Make random selections (i.e., explore) with probability. Choose to The biggest action (i.e., to take advantage of) is to balance between trying unknown options that may be better (i.e., exploring to gain new knowledge and optimize long-term benefits) and choosing the best option that is currently known (i.e., using existing knowledge to gain the maximum short-term benefits).
[0034] In one possible implementation, the action value function is iterated based on the update formula until the action value function satisfies the convergence condition, including:
[0035] After updating the action value function based on the update formula, the current stage state is advanced to the next stage state, and the update formula and the action value function are redefined and updated based on the update formula until the action value function meets the convergence condition. The convergence condition includes completing a preset number of rounds or the average difference of the function values of adjacent rounds being less than a preset threshold.
[0036] In this embodiment, a time discount term is calculated at each stage to set an update formula. After updating the action value function using the update formula, the current stage state is advanced to the next stage state, and the process of selecting the first action, calculating the time discount term based on the first action, and resetting the stage state using a new update formula is repeated. Reaching the final termination stage in the current stage can be considered as completing one round of training. By repeating the above process until a preset number of rounds is completed or the average difference in function values between adjacent rounds is less than a preset threshold, the training of the action value function is complete.
[0037] In one possible implementation, the optimal strategy is determined based on the converged action value function, including:
[0038] Based on the converged action value function, determine the second action that maximizes the function value in each stage state, and generate the optimal strategy based on each second action.
[0039] In this embodiment, the optimal action value function for each stage state can be obtained by iterating through multiple training rounds. Based on the action value function, the second action that maximizes the function value can be selected to generate the optimal policy output, which can then be applied to actual task execution. The optimal policy can be expressed as:
[0040]
[0041] Figure 2 Schematic diagrams of a multi-stage task delay control device 200 based on semi-Markov decision-making, according to some embodiments of this disclosure, are shown. The various embodiments in this specification are described in a progressive manner; similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments. Figure 2As shown, the device 200 includes a function construction module 201, configured to construct the action value function corresponding to the initial action of the target multi-stage task in the initialization state, determine the global reference duration based on the historical running data of the target multi-stage task, and construct the potential function corresponding to each stage state; an action selection module 202, configured to select the first action corresponding to the current stage state based on the exploration-utilization balance strategy in a single training round for the action value function, and determine the transition duration from the current stage state to the next stage state after executing the first action; a shaping reward calculation module 203, configured to calculate the time discount term based on the power of the transition duration of the preset discount factor, calculate the shaping reward based on the time discount term and the potential function, and set the update formula of the action value function based on the shaping reward; and an optimization control module 204, configured to iterate the action value function based on the update formula until the action value function meets the convergence condition, determine the optimal strategy based on the converged action value function, and control the target multi-stage task according to the optimal strategy.
[0042] In one possible implementation, the function construction module 201 is further configured to determine the median duration of each stage based on historical runtime data of the target multi-stage task, so as to determine the global reference duration based on the sum of the medians of each stage duration.
[0043] In one possible implementation, the action selection module 202 is further configured to randomly select actions based on a preset exploration rate, and select the action that maximizes the function value based on the utilization rate, thereby obtaining the first action corresponding to the current stage state, where the utilization rate is the difference between 1 and the exploration rate.
[0044] In one possible implementation, the shaping reward calculation module 203 is further configured to set a penalty term based on a negative value of the transition duration, and set a reward term based on the product of the time discount term and the potential function value of the next stage state; the shaping reward is obtained by subtracting the potential function value of the current stage state from the sum of the penalty term and the reward term.
[0045] In one possible implementation, the update formula is used to update the function value of the current stage state based on the shaping reward and the maximum function value of the next stage state.
[0046] In one possible implementation, the optimization control module 204 is further configured to, after updating the action value function based on the update formula, advance the current stage state to the next stage state, and redetermine the update formula and update the action value function based on the update formula until the action value function meets the convergence condition, which includes completing a preset number of rounds or the average difference of the function values of adjacent rounds being less than a preset threshold.
[0047] In one possible implementation, the optimization control module 204 is further configured to determine the second action that maximizes the function value in each stage state based on the converged action value function, so as to generate an optimal strategy based on each second action.
[0048] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this specification is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).
[0049] Figure 3 A block diagram of an electronic device 300 that can implement various embodiments of the present disclosure is shown. For example... Figure 3 As shown, the electronic device 300 includes a processor 310, a disk drive 320, an input / output interface 330, a network interface 340, and a memory 350. The processor 310, disk drive 320, input / output interface 330, network interface 340, and memory 350 can communicate with each other via a communication bus 360.
[0050] The processor 310 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs in order to implement the technical solution provided in this application.
[0051] The memory 350 can be implemented in the form of ROM (Read Only Memory), RAM (Read Access Memory), static memory, dynamic storage devices, etc. The memory 350 can store the operating system 351 used to control the operation of the electronic device 300, and the basic input / output system (BIOS) 352 used to control the low-level operations of the electronic device 300. Additionally, it can store a web browser 353, a data storage management system 354, etc. In summary, when the technical solution provided in this application is implemented through software or firmware, the relevant program code is stored in the memory 350 and is called and executed by the processor 310.
[0052] Input / output interface 330 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, vibrators, indicator lights, etc.
[0053] Network interface 340 is used to connect a communication module (not shown in the figure) to enable communication and interaction between the device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0054] Bus 360 includes a pathway for transmitting information between various components of the device, such as processor 310, disk drive 320, input / output interface 330, network interface 340, and memory 350.
[0055] It should be noted that although the above-described device only shows the processor 310, disk drive 320, input / output interface 330, network interface 340, memory 350, bus 360, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the method of this application, and does not necessarily include all the components shown in the figures.
[0056] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0057] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Furthermore, although operations are depicted in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.
[0058] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.
Claims
1. A multi-stage task delay control method based on semi-Markov decision-making, characterized in that, The method includes: Construct the action value function corresponding to the initial action of the target multi-stage task in the initialization state, determine the global reference duration based on the historical running data of the target multi-stage task, and construct the potential function corresponding to each stage state. In a single training round targeting the action value function, the first action corresponding to the current stage state is selected based on an exploration-utilization balancing strategy, and after executing the first action, the transition duration from the current stage state to the next stage state is determined. The time discount term is calculated by exponentiation of the transition duration based on the preset discount factor. After calculating the shaping reward based on the time discount term and the potential function, the update formula of the action value function is set based on the shaping reward. The action value function is iterated based on the update formula until the action value function meets the convergence condition. Then, the optimal strategy is determined based on the converged action value function, so as to control the target multi-stage task according to the optimal strategy.
2. The multi-stage task delay control method based on semi-Markov decision-making according to claim 1, characterized in that, The determination of the global reference duration based on the historical operational data of the target multi-stage task includes: Based on the historical operation data of the target multi-stage task, the median time of each stage is determined, and the global reference duration is determined according to the sum of the medians of each stage.
3. The multi-stage task delay control method based on semi-Markov decision-making according to claim 1, characterized in that, The method of selecting the first action corresponding to the current stage state based on the exploration-utilization balance strategy includes: Random actions are selected based on a preset exploration rate, and the action that maximizes the function value is selected based on the utilization rate, thus obtaining the first action corresponding to the current stage state. The utilization rate is the difference between 1 and the exploration rate.
4. The multi-stage task delay control method based on semi-Markov decision-making according to claim 1, characterized in that, The calculation of the shaping reward based on the time discount term and the potential function includes: A penalty term is set based on the negative value of the transition duration, and a reward term is set based on the product of the time discount term and the potential function value of the next stage state. The shaping reward is obtained by subtracting the potential function value of the current stage state from the sum of the penalty and reward terms.
5. The multi-stage task delay control method based on semi-Markov decision-making according to claim 1, characterized in that, The update formula is used to update the function value of the current stage state based on the shaping reward and the maximum function value of the next stage state.
6. The multi-stage task delay control method based on semi-Markov decision-making according to claim 1, characterized in that, The step of iterating the action value function based on the update formula until the action value function satisfies the convergence condition includes: After updating the action value function based on the update formula, the current stage state is advanced to the next stage state, and the update formula is redefined and the action value function is updated based on the update formula until the action value function meets the convergence condition. The convergence condition includes completing a preset number of rounds or the average difference of function values between adjacent rounds being less than a preset threshold.
7. The multi-stage task delay control method based on semi-Markov decision-making according to claim 1, characterized in that, The step of determining the optimal strategy based on the converged action value function includes: Based on the converged action value function, determine the second action that maximizes the function value in each stage state, and generate the optimal strategy based on each second action.
8. A multi-stage task delay control device based on semi-Markov decision-making, characterized in that, The device includes: The function construction module is configured to construct the action value function corresponding to the initial action of the target multi-stage task in the initialization state, determine the global reference duration based on the historical running data of the target multi-stage task, and construct the potential function corresponding to each stage state. The action selection module is configured to select the first action corresponding to the current stage state based on an exploration-utilization balancing strategy in a single training round for the action value function, and determine the transition duration from the current stage state to the next stage state after executing the first action. The shaping reward calculation module is configured to calculate a time discount term based on a preset discount factor raised to the power of the transition duration, and then calculate the shaping reward based on the time discount term and the potential function, and set the update formula of the action value function based on the shaping reward. The optimization control module is configured to iterate the action value function based on the update formula until the action value function meets the convergence condition. Then, it determines the optimal strategy based on the converged action value function to control the target multi-stage task according to the optimal strategy.
9. An electronic device, characterized in that, include: One or more processors, and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of a multi-stage task delay control method based on semi-Markov decision-making as described in any one of claims 1-7.
10. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements a multi-stage task delay control method based on semi-Markov decision-making according to any one of claims 1-7.
