A method and system for intelligent scheduling of monthly tasks based on reinforcement learning

Abstract

The application belongs to the technical field of artificial intelligence, and provides a month-end task intelligent scheduling method and system based on reinforcement learning, which is a technical scheme for constructing a time sequence state vector based on obtained month-end task multi-dimensional state feature data; encoding the time sequence state vector to obtain time sequence features, constructing a task dependency graph, extracting global features of the task dependency graph, fusing the time sequence features and the global features of the task dependency graph to obtain fused features, generating an action probability distribution based on the fused features; introducing hard constraints and soft constraints based on the action probability distribution, constructing a multi-objective reward function, optimizing a reinforcement learning agent based on a proximal policy optimization algorithm, outputting an action policy based on the optimized reinforcement learning agent; and converting the action policy output by the optimized reinforcement learning agent into specific month-end task scheduling instructions. The application realizes dynamic, adaptive and globally optimized scheduling of complex financial month-end processes.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and in particular relates to an intelligent scheduling method and system for monthly settlement tasks based on reinforcement learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] As enterprises increasingly adopt refined operational management, the monthly financial closing process for large group companies has become a core task involving massive amounts of data, multi-system collaboration, and complex dependencies. Taking telecommunications infrastructure service companies as an example, their monthly financial closing process involves cost accounting for expenses such as rental fees, electricity fees, and maintenance fees. This process is characterized by extremely large amounts of data, numerous participating entities, and complex business logic.

[0004] Currently, this process typically relies on traditional workflow engines and static scheduling strategies based on fixed rules, but it has the following drawbacks:

[0005] Static scheduling has poor adaptability: Existing systems mostly use predefined workflow templates, where the task execution order and dependencies are fixed before the process starts. However, the actual business environment is highly dynamic, and task dependencies often change temporarily due to local policy adjustments, system interface upgrades, or data anomalies. Static scheduling models cannot respond to these changes in real time, and are prone to delays or interruptions in the entire process due to a blockage in one place.

[0006] Inefficient resource utilization: The month-end closing process is strictly constrained by computing resources, such as server and database performance, and human resources (number and skills of finance personnel). Especially during peak month-end closing periods, traditional rule-based scheduling, such as first-come-first-served, cannot dynamically allocate resources according to the global resource load, easily creating performance bottlenecks. This results in both idle and congested resources, making it difficult to optimize utilization.

[0007] Unable to achieve global optimization: Monthly settlement optimization objectives are multi-dimensional, requiring simultaneous minimization of total completion time, maximization of resource utilization, and ensuring no delays in critical tasks. Rule-based scheduling strategies based on human experience, such as simply setting priorities, can only achieve local optimization. They lack the ability to learn from historical scheduling experience and cannot make scientific decisions under multiple constraints to achieve global goals.

[0008] Weak ability to respond to emergencies: Processes must be completed within strict time limits (such as T+5 days). When tasks are unexpectedly delayed, traditional systems lack effective dynamic restructuring and emergency scheduling mechanisms and often rely on manual intervention, which is not only inefficient but also difficult to guarantee reliability.

[0009] While reinforcement learning has demonstrated its potential to handle dynamic and complex problems in task scheduling theory research, its direct application in the specific industrial application scenario of financial month-end closing still faces significant challenges. Existing research lacks a comprehensive consideration of ultra-large-scale data, multi-level distributed organizational architecture, heterogeneous system integration, and strong business constraints, and has not yet formed a mature and feasible systematic solution. Summary of the Invention

[0010] To address at least one of the technical problems mentioned above, this invention provides a method and system for intelligent scheduling of month-end tasks based on reinforcement learning. This method enables dynamic, adaptive, and global optimization scheduling of complex financial month-end processes, significantly shortening the month-end cycle, improving resource utilization efficiency, and ensuring process stability and scalability.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] The first aspect of this invention provides an intelligent scheduling method for month-end tasks based on reinforcement learning, comprising the following steps:

[0013] Obtain multi-dimensional status feature data for monthly closing tasks;

[0014] A time-series state vector is constructed based on the acquired multi-dimensional state feature data of the monthly settlement task;

[0015] Construct a reinforcement learning agent, using the temporal state vector as the state space, encode the temporal state vector to obtain temporal features, construct a task dependency graph, extract global features from the task dependency graph, fuse the temporal features and the global features of the task dependency graph to obtain fused features, and generate action probability distributions based on the fused features;

[0016] Based on the action probability distribution, hard constraints and soft constraints are introduced to construct a multi-objective reward function. The reinforcement learning agent is optimized based on the proximal policy optimization algorithm, and the action policy is output based on the optimized reinforcement learning agent.

[0017] The optimized reinforcement learning agent outputs action policies, which are then transformed into specific monthly task scheduling instructions and executed.

[0018] Furthermore, the extraction of global features from the task dependency graph includes:

[0019] For each node i Sample a fixed number of neighboring nodes;

[0020] Aggregate the features of neighboring nodes to generate neighbor messages;

[0021] The current node's features are concatenated with the aggregated neighbor messages, and then updated using a learnable weight matrix and a non-linear activation function to obtain the node's information. i In the k New features of the layer;

[0022] After K layers of convolution, the embedding vectors of all nodes are obtained. Global average pooling is then performed on the embeddings of all nodes to obtain the global features of the graph.

[0023] Furthermore, the hard constraint is to introduce an action masking mechanism, which analyzes the current task dependency graph in real time and dynamically generates an action mask. This mask forces the probability of all actions that violate the current dependency constraint to zero. Subsequently, the probability distribution of the remaining valid actions is renormalized to ensure that the agent only samples from the valid actions.

[0024] Furthermore, the multi-objective reward function is:

[0025] in, , , This represents the adjustment coefficient. Indicates time constraints. Indicates resource constraints. This indicates a delayed punishment. Indicates a stability reward. , , and Indicates the weight.

[0026] Furthermore, the loss function for optimizing the reinforcement learning agent based on the proximal policy optimization algorithm is:

[0027] ,

[0028] ,

[0029] ,

[0030] ,

[0031] ,

[0032] ,

[0033] in, Indicates the total loss. This represents the network loss for updating the Actor policy. This represents the loss for updating the value function. This represents the entropy reward loss. and Indicates weight, It is the probability ratio of the new strategy to the old strategy. It is the dominant function. λ is a hyperparameter, γ is the discount factor, and λ is the GAE parameter. Indicates the importance sampling ratio, and This represents the time series difference error between the future and the present. and The value function representing the state and the next state. Represents the initial state value. Indicates the actual cumulative return. Represents the policy function. Represents the entropy of the strategy.

[0034] Furthermore, when encoding the temporal state vector to obtain temporal features, the temporal state vector is encoded through a fully connected network or a small Transformer encoder to obtain the temporal features.

[0035] Furthermore, the multi-dimensional status feature data of the month-end task includes task status, resource status, dependency graph, and environment status; wherein, task status includes task queue, execution duration, number of remaining tasks, priority, and task type; resource status includes available computing nodes, CPU / memory / disk I / O load, network bandwidth, personnel to be assigned, and skill matrix; dependency graph includes dynamically parsed task DAG dependencies; and environment status includes the time remaining until the month-end deadline, historical latency, and system alarms.

[0036] Furthermore, when constructing the time-series state vector based on the acquired multi-dimensional state feature data of the monthly settlement task, the process includes normalizing the multi-dimensional state feature data, embedding and encoding the normalized data, and using a graph neural network to encode the complex graph structure dependencies into a fixed-length graph embedding vector.

[0037] A second aspect of the present invention provides a reinforcement learning-based intelligent scheduling system for month-end tasks, comprising:

[0038] The data acquisition module is used to acquire multi-dimensional status feature data of the monthly settlement task.

[0039] A time-series state vector is constructed based on the acquired multi-dimensional state feature data of the monthly settlement task;

[0040] The action generation module is used to construct a reinforcement learning agent. It uses the temporal state vector as the state space, encodes the temporal state vector to obtain temporal features, constructs a task dependency graph, extracts global features from the task dependency graph, fuses the temporal features and global features of the task dependency graph to obtain fused features, and generates an action probability distribution based on the fused features.

[0041] The action optimization module is used to construct a multi-objective reward function based on the action probability distribution, introduce hard and soft constraints, optimize the reinforcement learning agent based on the proximal policy optimization algorithm, and output the action policy based on the optimized reinforcement learning agent.

[0042] The task execution module is used to transform the optimized reinforcement learning agent's output action policy into specific monthly task scheduling instructions and execute the monthly task scheduling instructions.

[0043] A third aspect of the present invention provides a computer device.

[0044] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the reinforcement learning-based intelligent scheduling method for month-end tasks as described above.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] This invention innovatively integrates reinforcement learning and graph neural networks into the field of financial month-end scheduling, constructing a closed-loop intelligent scheduling system that integrates real-time multi-source data perception, GNN dependency modeling, proximal policy optimization (PPO) decision optimization, and dynamic execution control. It can perform multimodal fusion of environmental state (time series data) and task topology (graph data), continuously learn and dynamically adjust scheduling strategies based on reward signals, and realize a paradigm shift from "static rules" to "dynamic perception and optimization".

[0047] This invention utilizes the message passing mechanism of GraphSage (GNN) to allow each task node to aggregate information from its upstream and downstream tasks. This enables the system to deeply understand the propagation effect of dependencies, accurately identify critical paths affecting the entire month-end cycle, and intelligently prioritize the allocation of scarce resources to the tasks that can shorten the overall time, achieving true global optimization. It overcomes the bottleneck of traditional schedulers that cannot understand the task chain from a global perspective.

[0048] The reward function designed in this invention considers four dimensions simultaneously: critical path progress, resource utilization, delay penalty, and process stability. The weights can be dynamically configured. Combined with the advantages of the PPO algorithm in training stability and sample efficiency, the system can learn to make optimal trade-offs among multiple competing objectives and adapt to different business strategies. This solves the problem of having a single objective and fixed rules, and can adapt to different stages, such as pursuing speed at the beginning of the month and pursuing stability at the end of the month.

[0049] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0050] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0051] Figure 1 This is a flowchart of an intelligent scheduling method for month-end tasks based on reinforcement learning provided in an embodiment of the present invention;

[0052] Figure 2 This is a block diagram of a monthly task intelligent scheduling system based on reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0053] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0054] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0055] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0056] Example 1

[0057] like Figure 1 As shown, this embodiment provides an intelligent scheduling method for month-end tasks based on reinforcement learning, including the following steps:

[0058] Step 1: Obtain multi-dimensional status feature data for the monthly closing task;

[0059] In this embodiment, when obtaining multi-dimensional status feature data of monthly settlement tasks, various data systems are accessed, and ERP, financial accounting system, asset management system, contract management system, IT monitoring system (server / database performance), human resources system, etc. are connected through RESTful API or message queue (such as Kafka).

[0060] Specifically, the multi-dimensional status feature data of the monthly settlement task obtained includes task status, resource status, dependency graph and environment status;

[0061] The task status includes the task queue, execution time (historical mean / variance), number of remaining tasks, priority, and task type (computation / IO / manual).

[0062] Resource status includes available compute nodes, CPU / memory / disk I / O load, network bandwidth, personnel to be assigned, and skill matrix;

[0063] Dependency graphs include dynamically resolving task DAG dependencies and identifying critical paths;

[0064] The environmental status includes the time remaining until the end of the month, historical delays, and system alarms.

[0065] Step 2: Construct a time-series state vector based on the acquired multi-dimensional state feature data of the monthly settlement task;

[0066] The system states in the process are highly complex due to their multi-dimensional, multi-scale, and dynamically changing nature. In this embodiment, a unified 128-dimensional state vector is constructed. At this stage, it's not a simple splicing process, but rather involves specialized processing techniques, specifically including:

[0067] Normalize the multi-dimensional state feature data;

[0068] For example, normalize resource status metrics such as CPU%, memory (GB), and I / O (MBps) from different dimensions to the same scale;

[0069] Normalized data is embedded and encoded, for example, discrete data such as task type and personnel skills are transformed into low-dimensional dense vectors.

[0070] Graph neural networks are used to encode complex graph structure dependencies into fixed-length graph embedding vectors;

[0071] Finally, the processed multi-dimensional features are used to obtain an original high-dimensional vector, which is then passed through a fully connected layer for dimensionality reduction and feature fusion. In this implementation, the final output is a 128-dimensional temporal state vector, represented as:

[0072] ,

[0073] in, Indicates task status characteristics. Indicates resource status characteristics, This indicates a dependence on spectral features. It indicates the characteristics of the environmental state.

[0074] Step 3: Construct a reinforcement learning agent, using the temporal state vector as the state space, encode the temporal state vector to obtain temporal features, construct a task dependency graph, extract global features from the task dependency graph, fuse the temporal features and the global features of the task dependency graph to obtain fused features, and generate action probability distributions based on the fused features;

[0075] System states change rapidly (e.g., a task suddenly freezes, a server unexpectedly crashes), and intelligent agents cannot obtain a 100% perfect global state. The design aims to provide an effective observation of the current system. The agent needs to learn to make decisions based on incomplete information. By using recurrent networks or taking the state at consecutive time steps as input, the agent can acquire a certain level of historical information perception to cope with dynamic changes in the environment.

[0076] A reinforcement learning agent is constructed, which adopts the Actor-Critic architecture. The Actor and Critic share the state encoding layer. The output dimension of the Actor output layer is equal to the size of the action space. The Softmax function is used to output the probability distribution π(a|s) of selecting each action. The output dimension of the Critic output layer is 1, and the output state value function V(s) represents the long-term expected reward of the current state.

[0077] Its state space is The action space includes: adjusting task priority weights; dynamically allocating CPU cores and memory for computation tasks; assigning financial personnel with corresponding skills to manual tasks; and triggering emergency modes (such as activating backup clusters).

[0078] Specifically, the steps include the following:

[0079] Step 301: Process the time-series state vector Encoding is performed to obtain temporal features ;

[0080] In this embodiment, the 128-dimensional temporal state vector Temporal features are obtained by encoding using a fully connected network or a small Transformer encoder. ;

[0081] Step 302: Construct a task dependency graph and extract global features from the task dependency graph;

[0082] The dependencies between tasks are naturally a graph structure (DAG), and traditional fully connected networks cannot effectively utilize its topological information. This invention employs a graph neural network (GNN), specifically the GraphSAGE (Graph Sample and Aggregate) algorithm, to encode the task dependency graph in order to capture the global context and dependency constraints of the tasks.

[0083] In this embodiment, when constructing the task dependency graph, each node represents a monthly closing task (such as "electricity bill calculation" or "rent accrual"). The initial feature vector of each node is represented as follows: This includes the task's status information, such as: task type (encoded as one-hot), historical average execution time, priority, and remaining data volume. Directed edges represent dependencies between tasks (e.g., "task B can only start after task A is completed"), and the direction of the edge indicates the flow of the dependency.

[0084] In this embodiment, a graph neural network (GNN) is used to extract global features of the task-dependent graph, specifically including:

[0085] GraphSAGE updates the representation of the current node by iteratively aggregating information from neighboring nodes:

[0086] In the k layer:

[0087] Sampling neighbors: for each node i Sample a fixed number of neighboring nodes .

[0088] Aggregating Neighbor Information: Using aggregation functions (such as mean, pool, LSTM) to aggregate the features of neighbor nodes and generate neighbor messages. , is represented as:

[0089]

[0090] in, This represents the new feature of node j at the (k-1)th layer. This represents the function for aggregating neighbor node information, which is used to integrate the feature vectors of all neighbor nodes of the target node into a fixed-length aggregate message;

[0091] Update node state: Concatenate the current node's features with the aggregated neighbor messages, and then apply a learnable weight matrix. and nonlinear activation functions (e.g., ReLU) is used to update and obtain the node. i In the k New features of layers , is represented as:

[0092] ,

[0093] in, Indicates the first i The node at the th k New features of the layer Indicates a splicing operation;

[0094] After K layers of convolution, each node obtains a final embedding vector containing K-hop neighborhood information. To obtain a global representation of the entire graph (for agent decision-making), global mean pooling is performed on all node embeddings to obtain the global graph features, represented as follows:

[0095] ,

[0096] Where N is the total number of nodes.

[0097] Step 303: Extract time series features The environmental state representation is obtained by fusing features with global features from the task dependency graph, and then finally encoded through a fusion network (fully connected layer);

[0098] The environmental state is represented as follows:

[0099] ,

[0100] in, Indicates fusion characteristics, Represents the matrix weights. Represents high-level temporal features, Represents global features of the graph. Indicates the bias term;

[0101] Step 304: Merge features The input is fed into the output layer of the Actor network (a fully connected layer followed by a Softmax function), generating an action probability distribution. .

[0102] Suppose there are M discrete actions (such as "promote the priority of task A" or "allocate 4 cores to task B"), then the output is an M-dimensional vector, where each element represents the probability of choosing the corresponding action, and the sum of all elements is 1.

[0103] ,

[0104]

[0105] in, Indicates weight, Indicates the bias term. Represents a probability distribution. and Let represent the raw output values ​​generated by the agent's policy network for the i-th and j-th actions in the current state s, respectively.

[0106] Step 4: Based on the action probability distribution, introduce hard and soft constraints, construct a reward function, optimize the reinforcement learning agent based on the proximal policy optimization algorithm, and output the action policy based on the optimized reinforcement learning agent;

[0107] Specifically, the steps include the following:

[0108] Step 401: Based on the action probability distribution, introduce hard constraints and soft constraints to construct the reward function;

[0109] To learn better strategies, the agent does not always choose the action with the highest probability, but instead samples based on the probability distribution. For example, if action A has a probability of 0.7 and action B has a probability of 0.3, the system has a 70% chance of choosing A and a 30% chance of choosing B, which ensures sufficient exploration.

[0110] Traditional dependency constraints are absolute hard constraints, which cannot schedule a task with an incomplete prerequisite task. Therefore, in this embodiment, after the agent outputs the action probability distribution, an action masking mechanism is introduced as a hard constraint. The current task dependency graph is analyzed in real time, and an action mask is dynamically generated. This mask will force the probability of all actions that violate the current dependency constraint (such as "start executing a certain incomplete prerequisite task") to zero. Subsequently, the probability distribution of the remaining valid actions is renormalized to ensure that the agent only samples from the valid actions.

[0111] Its innovation lies in the fact that instead of using the penalty of the reward function to "hope" the agent learns to avoid constraints, it fundamentally eliminates illegal actions from the mechanism itself, ensuring 100% scheduling safety. This is the most efficient way to directly inject domain knowledge into the reinforcement learning loop.

[0112] The time window (such as T+5 days to complete) is a soft constraint. Although it can be slightly violated, it will result in huge business losses. Resource constraints (limited manpower and computing power) also need to be used efficiently.

[0113] In this implementation, a multi-objective reward function is designed to guide the agent to satisfy these soft constraints. The specific multi-objective reward function includes time constraints, resource constraints, and stability constraints, expressed as follows:

[0114]

[0115] in, , , This represents the adjustment coefficient. Indicates time constraints. Indicates resource constraints. This indicates a delayed punishment. Indicates a stability reward. , , and Indicates weight;

[0116] The program directly penalizes delays on the critical path. In order to maximize cumulative rewards, the agent will prioritize actions that shorten the critical path time, thus spontaneously striving to meet the final deadline.

[0117] High resource utilization is rewarded, while idle resources are penalized. However, the resource allocation action itself is constrained by the action mask (e.g., it cannot allocate more CPUs than the total number of physical machine cores), ensuring that optimization is performed within the resource limit.

[0118] and This policy penalizes frequent scheduling changes and task restarts, encouraging smooth operation.

[0119] Weight , , and These parameters are not fixed. They can be used as hyperparameters and flexibly configured according to the strategic priorities of different companies and different month-end stages (such as accelerating progress in the early stages and ensuring stability in the later stages). This means that the same system can adapt to different optimization strategies by adjusting the weights, demonstrating strong versatility and adaptability.

[0120] Step 402: Optimize the reinforcement learning agent based on the proximal policy optimization algorithm, and output the action policy based on the optimized reinforcement learning agent;

[0121] In this embodiment, in order to balance sample efficiency and training stability, the Proximal Policy Optimization (PPO-Clip) algorithm is adopted. Its core is to limit the step size of policy updates by using the pruning probability ratio to avoid training crashes.

[0122] The core of PPO is the following improved objective function, used to update the Actor policy network;

[0123] ,

[0124] in, It is the probability ratio of the new strategy to the old strategy. It is the advantage function, which estimates the action. Advantages relative to the average level It is a hyperparameter, usually set to 0.1 or 0.2, which limits... The range of changes is controlled, thus stabilizing policy updates within a reliable interval and effectively preventing performance crashes caused by excessively large single updates. min The operation ensures that the most conservative (pessimistic) update direction is selected even in the event of estimation errors, further enhancing stability.

[0125] To reduce variance, the dominance function Using generalized dominance estimation, it can be expressed as:

[0126] ,

[0127] ,

[0128] Where γ is the discount factor, weighing immediate versus future returns, and λ is the GAE parameter, weighing bias versus variance. Indicates the importance sampling ratio, and This represents the time series difference error between the future and the present. and The value function representing the state and the next state is output by the Critic network. Using the Critic network to estimate the value function, instead of Monte Carlo returns, greatly improves sample efficiency because a trajectory can be used for multiple updates.

[0129] Value function update loss: The update of the Critic network is represented by minimizing the mean squared error loss as follows:

[0130] ,

[0131] in, Represents the initial state value. Indicates the actual cumulative return;

[0132] Entropy Reward Loss: To encourage exploration and prevent the policy from prematurely converging to a local optimum, a reward of policy entropy is added to the objective function, expressed as:

[0133] ,

[0134] in, Represents the policy function. The entropy of the strategy;

[0135] The final total loss function is expressed as:

[0136] ,

[0137] in, and Indicates weight;

[0138] During training, the system samples a mini-batch of data from the experience replay buffer and performs multiple rounds (usually 4-10 rounds) of gradient ascent optimization on the aforementioned objective. This reuse of old data greatly improves sample efficiency. The optimizer is typically Adam, whose adaptive learning rate also helps stabilize training.

[0139] Step 5: Transform the optimized reinforcement learning agent's output action policy into specific scheduling instructions;

[0140] Selected abstract actions (such as action_id = 102) are mapped to one or more specific, executable scheduling instructions, and this mapping is predefined in an instruction library.

[0141] For example, raising the priority of "electricity cost calculation" to P0;

[0142] Allocate 4 CPU cores and 8GB of memory to the "Electricity Bill Calculation" task;

[0143] Assign financial personnel with electricity billing skills.

[0144] The scheduling instructions are generated at a set time, such as every 5 minutes, and the task priority and resource configuration are dynamically modified by calling the scheduling system interface such as Airflow through the REST API.

[0145] Step 6: Execute the scheduling instruction and store the decision result in the experience replay buffer;

[0146] In this embodiment, the final decision result is represented as a quadruple. The data is stored in an experience replay buffer for model retraining. Simultaneously, a visual dashboard displays metrics such as estimated completion time, critical path progress, and resource utilization. It is a time-series state vector. It is a sampling or selection action. It is the actual cumulative return. It is the timing state vector for the next stage.

[0147] Example 2

[0148] like Figure 2 As shown, this embodiment provides an intelligent scheduling system for monthly settlement tasks based on reinforcement learning, including:

[0149] Data acquisition module 201 is used to acquire multi-dimensional status feature data of monthly settlement tasks;

[0150] A time-series state vector is constructed based on the acquired multi-dimensional state feature data of the monthly settlement task;

[0151] Action generation module 202 is used to construct reinforcement learning agent. It uses temporal state vector as state space, encodes temporal state vector to obtain temporal features, constructs task dependency graph, extracts global features of task dependency graph, fuses temporal features and global features of task dependency graph to obtain fused features, and generates action probability distribution based on fused features.

[0152] Action optimization module 203 is used to construct a multi-objective reward function based on action probability distribution, introduce hard and soft constraints, optimize the reinforcement learning agent based on the proximal policy optimization algorithm, and output action policy based on the optimized reinforcement learning agent.

[0153] The task execution module 204 is used to convert the optimized reinforcement learning agent's output action policy into specific monthly task scheduling instructions and execute the monthly task scheduling instructions.

[0154] It should be noted that the specific implementation of the reinforcement learning-based intelligent scheduling system for monthly settlement tasks in this embodiment of the invention is similar to the specific implementation of the reinforcement learning-based intelligent scheduling method for monthly settlement tasks in this embodiment of the invention. Please refer to the description in the method section for details. To reduce redundancy, it will not be repeated here.

[0155] Example 3

[0156] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the reinforcement learning-based intelligent scheduling method for month-end tasks as described above.

[0157] Example 4

[0158] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the reinforcement learning-based intelligent scheduling method for monthly tasks as described above.

[0159] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0160] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0161] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0162] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0163] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0164] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for intelligent scheduling of monthly tasks based on reinforcement learning, characterized in that, The method comprises the following steps: obtaining monthly task multi-dimensional state feature data; the monthly task multi-dimensional state feature data comprises task state, resource state, dependency graph and environment state; the task state comprises task queue, execution duration, remaining task number, priority and task type; the resource state comprises available computing node, CPU / memory / disk I / O load, network bandwidth, personnel to be allocated and skill matrix; the dependency graph comprises dynamically analyzing task DAG dependency relationship; and the environment state comprises time to monthly closing deadline, historical delay condition and system alarm; a time sequence state vector is constructed based on the obtained monthly task multi-dimensional state feature data; The reinforcement learning agent is constructed, a time sequence state vector is taken as a state space, a time sequence feature is obtained by encoding the time sequence state vector, a task dependency graph is constructed, a global feature of the task dependency graph is extracted, a fusion feature is obtained by fusing the time sequence feature and the global feature of the task dependency graph, and an action probability distribution is generated based on the fusion feature; when the task dependency graph is constructed, each node represents a monthly closing task, and an initial feature vector of each node is represented as , and contains state information of the task; a directed edge represents a dependency relationship between tasks, and a direction of the edge indicates a flow direction of the dependency; a multi-objective reward function is constructed based on an action probability distribution, an action mask mechanism hard constraint and a time window soft constraint; a reinforcement learning agent is optimized based on a proximal policy optimization algorithm; and an action policy is output based on the optimized reinforcement learning agent; the multi-objective reward function is as follows: wherein, , , denotes an adjustment coefficient, denotes a time constraint, denotes a resource constraint, denotes a delay penalty, denotes a stability reward, , , and denote weights; The action policy output by the optimized reinforcement learning agent is converted into a specific monthly task scheduling instruction, and the monthly task scheduling instruction is executed. 2.The method of claim 1, wherein, The extracted task dependency graph global feature comprises: For each node i Sample a fixed number of neighbor nodes; a neighbor node feature is aggregated to generate a neighbor message; The current node feature is concatenated with the aggregated neighbor messages and updated by a learnable weight matrix and a nonlinear activation function to get the node i In the first k Layer's new features; after K-layer convolution, embedding vectors of all nodes are obtained, and global average pooling is performed on all node embeddings to obtain a graph global feature. 3.The method of claim 1, wherein, The hard constraint is that an action mask is dynamically generated by introducing an action mask mechanism and analyzing a current task dependency graph in real time; the mask forces the probability of all actions that violate the current dependency constraint to be zero; then, the probability distribution of the remaining valid actions is re-normalized to ensure that the agent only samples from legal actions. 4.The method of claim 1, wherein, When the reinforcement learning agent is optimized based on the proximal policy optimization algorithm, the loss function is as follows: ， ， ， ， ， ， where, denotes the total loss, denotes the update actor policy network loss, denotes the value function update loss, denotes the entropy reward loss, and denotes the weights, is the probability ratio of new and old policy, is the advantage function, is a hyperparameter, γ is the discount factor, and λ is the GAE parameter, denotes the importance sampling ratio, and denotes the future and current time-difference error, and denotes the state and next state value function, denotes the initial state value, denotes the actual cumulative return, denotes the policy function, denotes the entropy of the policy. 5.The intelligent scheduling method for monthly task based on reinforcement learning of claim 1, wherein, When the time sequence state vector is encoded to obtain a time sequence feature, the time sequence state vector is encoded through a fully connected network or a small Transformer encoder to obtain the time sequence feature.

6. The method of claim 1, wherein the method further comprises: When the time sequence state vector is constructed based on the obtained monthly task multi-dimensional state feature data, the multi-dimensional state feature data is normalized, the normalized data is embedded and encoded, and a graph neural network is used to encode a complex graph structure dependency relationship into a fixed-length graph embedding vector.

7. A reinforcement learning-based intelligent scheduling system for monthly tasks, characterized in that, The method comprises the following steps: a data acquisition module is configured to acquire monthly task multi-dimensional state feature data; monthly task multi-dimensional state feature data is acquired; the monthly task multi-dimensional state feature data comprises task state, resource state, dependency graph and environment state; the task state comprises task queue, execution duration, remaining task number, priority and task type; the resource state comprises available computing node, CPU / memory / disk I / O load, network bandwidth, personnel to be allocated and skill matrix; the dependency graph comprises dynamically analyzing task DAG dependency relationship; and the environment state comprises time to monthly closing deadline, historical delay condition and system alarm; a time sequence state vector is constructed based on the obtained monthly task multi-dimensional state feature data; a multi-objective reward function is constructed based on an action probability distribution, an action mask mechanism hard constraint and a time window soft constraint; a reinforcement learning agent is optimized based on a proximal policy optimization algorithm; and an action policy is output based on the optimized reinforcement learning agent; the multi-objective reward function is as follows: An action generation module is configured to build a reinforcement learning agent, take a time sequence state vector as a state space, encode the time sequence state vector to obtain a time sequence feature, build a task dependency graph, extract a global feature of the task dependency graph, fuse the time sequence feature and the global feature of the task dependency graph to obtain a fused feature, and generate an action probability distribution based on the fused feature; when the task dependency graph is built, each node represents a monthly closing task, and an initial feature vector of each node is represented as , and contains state information of the task; a directed edge represents a dependency relationship between tasks, and a direction of the edge indicates a flow direction of the dependency; An action optimization module is configured to introduce an action mask mechanism hard constraint and a time window soft constraint based on an action probability distribution, construct a multi-objective reward function, optimize a reinforcement learning agent based on a proximal policy optimization algorithm, and output an action policy based on the optimized reinforcement learning agent. The multi-objective reward function is: wherein, , , denotes a tuning coefficient, denotes a time constraint, denotes a resource constraint, denotes a delay penalty, denotes a stability reward, , , and denote weights; A task execution module is configured to convert the action policy output by the optimized reinforcement learning agent into a specific monthly task scheduling instruction and execute the monthly task scheduling instruction.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the method when executing the program, and the method is the method of any one of claims 1-6.

Images