Group enterprise resource scheduling optimization method and system based on reinforcement learning

Abstract

This invention discloses a method and system for optimizing resource scheduling in a group enterprise based on reinforcement learning, belonging to the field of scheduling optimization technology. The method involves the headquarters node issuing scheduling action commands to the execution nodes and collecting the observed reward scalar and bidirectional communication delay after execution. A negative exponential delay decay function is used to convert the bidirectional communication delay into a time-dependent confidence weight factor, and the observed reward scalar is denoised to obtain a time-difference denoised reward scalar. A policy gradient is constructed based on this scalar, and a Hessian matrix diagonal sensitivity evaluation operator is introduced to generate a sensitivity mask matrix. The basic gradient tensor is subjected to Hadamard product operation to achieve weight compensation and update. Global delay feature queue filtering and a deadlock early warning mechanism are used. This invention addresses the problem of nonlinear feedback oscillation and policy non-convergence induced by command phase misalignment in distributed hierarchical control by quantifying the evolutionary bias of the reward signal due to physical link delay, fundamentally solving the problem from the underlying algorithm logic.

Description

Technical Field

[0001] This invention relates to the field of scheduling optimization technology, and more specifically, this application relates to a group enterprise resource scheduling optimization method and system based on reinforcement learning. Background Technology

[0002] In the global operations of large corporations, resource allocation across regions and organizations relies on complex distributed, multi-level control networks. Such networks typically involve heterogeneous physical communication links and various financial institution interface protocols, resulting in non-constant physical transmission delays in command transmission and data acquisition between the headquarters control center and branch nodes.

[0003] Conventional technologies for handling resource scheduling tasks generally rely on the idealized assumption of synchronous observation, meaning that the issuance of instructions, physical execution, and state feedback are all completed within the same instantaneous time step. However, in real-world physical conditions, as scheduling instructions are transmitted through hierarchical links, the resource state of the underlying physical system continuously evolves over time. Because this evolution process runs parallel to instruction transmission, by the time an instruction reaches the execution node, the global observation state it depends on has already experienced phase shift. Existing control mechanisms lack methods to quantify the phase shift caused by transmission delays, resulting in execution actions often acting on a physical state that has already deviated from its intended state.

[0004] The problem with existing technologies is that in the scenario of distributed multi-level group fund scheduling with heterogeneous physical transmission delay, ignoring the phase slip caused by instruction transmission leads to phase misalignment of hierarchical control instructions and failure of feedback signals, which in turn causes nonlinear feedback oscillations in the system. Summary of the Invention

[0005] To address the aforementioned technical problems, this technical solution provides a method and system for optimizing resource scheduling in group enterprises based on reinforcement learning. The solution resolves the issues raised in the background section.

[0006] In a first aspect, embodiments of this application provide a group enterprise resource scheduling optimization method based on reinforcement learning, including the following steps: acquiring fund scheduling demand data through the headquarters node, and outputting scheduling action instructions at the first timestamp through the reinforcement learning network of the current data period, and sending them to the execution node; receiving the scheduling action instructions through the execution node and calling the business interface to execute them, collecting the observation reward scalar representing the feedback of the scheduling environment state, and feeding back the observation reward scalar to the headquarters node at the second timestamp after execution completion; calculating the arithmetic difference between the second timestamp and the first timestamp through the headquarters node to obtain the two-way communication delay, and substituting the two-way communication delay into a preset negative exponential delay decay function to calculate the timeliness confidence weight factor. The observation reward scalar is multiplied by the time-difference confidence weight factor to obtain the temporal difference denoised reward scalar. Based on the temporal difference denoised reward scalar, a policy gradient objective function is constructed, and partial derivatives are calculated using the backpropagation algorithm to obtain the basic gradient tensor corresponding to the current weight tensor of the reinforcement learning network. A pre-defined Hessian matrix diagonal sensitivity evaluation operator is invoked to calculate the parameter gradient sensitivity of each dimension in the current weight tensor, generating a sensitivity mask matrix with the same dimensions as the current weight tensor. The sensitivity mask matrix and the basic gradient tensor are then subjected to a Hadamard product to generate a weight compensation difference tensor, which is then superimposed onto the current weight tensor to obtain the updated reinforcement learning network for use in the next data cycle.

[0007] Secondly, this application provides a group enterprise resource scheduling optimization system based on reinforcement learning, including: a scheduling action issuance module: used to obtain fund scheduling demand data through the headquarters node, and output scheduling action instructions through the reinforcement learning network of the current data period at the first timestamp, and issue them to the execution node; a feedback scalar upload module: used to receive scheduling action instructions through the execution node and call the business interface to execute, collect the observation reward scalar representing the feedback of the scheduling environment state, and feed the observation reward scalar back to the headquarters node at the second timestamp after execution; a differential reward processing module: used by the headquarters node to calculate the two-way communication delay based on the arithmetic difference between the second timestamp and the first timestamp, and substitute it into the negative exponential delay decay function to calculate the timeliness confidence weight factor, and feed the observation reward scalar back to the headquarters node. The network performs a multiplication operation between the quantity and the time-difference confidence weight factor to obtain the time-difference denoised reward scalar; the gradient extraction module is used to construct the policy gradient objective function based on the time-difference denoised reward scalar, and calculates the partial derivative through the backpropagation algorithm to obtain the basic gradient tensor corresponding to the current weight tensor of the reinforcement learning network; the mask matrix processing module is used to call the Hessian matrix diagonal sensitivity evaluation operator to calculate the parameter gradient sensitivity of each dimension in the current weight tensor and generate a sensitivity mask matrix with the same dimensions as the current weight tensor; the network update module is used to perform a Hadamard product operation between the sensitivity mask matrix and the basic gradient tensor to generate a weight compensation difference tensor, and then superimposes the weight compensation difference tensor into the current weight tensor to obtain the updated reinforcement learning network for use in the next data cycle.

[0008] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned reinforcement learning-based group enterprise resource scheduling optimization method.

[0009] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0010] 1. By calculating the bidirectional communication delay and substituting it into a negative exponential delay decay function to generate a time-dependent confidence weight factor, the observed reward scalar is transformed into a time-difference denoised reward scalar. This process directly quantifies the "expiration degree" of instructions due to transmission time on the physical link, solving the problem in the background technology where instructions become disconnected from real-time physical rhythms due to physical state evolution. This ensures that the reward signal obtained by the headquarters node can objectively reflect the true contribution of the executed action to the resource balance goal, effectively suppressing systemic feedback oscillations, eliminating feedback oscillations, and improving the accuracy of reward attribution.

[0011] 2. A global latency feature queue is used to store historical bidirectional communication latency, and a historical smooth latency scalar is generated through median filtering, which is then used to calculate latency jump differences. This mechanism can identify and eliminate instantaneous noise generated by heterogeneous interface communication. Combined with the smooth bidirectional communication latency generated by the exponential moving average decay coefficient, it ensures that the timeliness confidence weight factor remains smooth in the face of network dynamic jitter, avoiding the divergence of reinforcement learning network weight updates caused by large latency fluctuations, and enhancing robustness against heterogeneous network jitter.

[0012] 3. A sensitivity mask matrix is ​​generated using the Hessian matrix diagonal sensitivity evaluation operator. Combined with constraints from the same-direction mask matrix and the truncation smoothing threshold, weight compensation is applied to the basic gradient tensor. This mechanism accurately identifies the weight dimensions that contribute significantly to reward enhancement. Coupled with deadlock warning monitoring targeting the L2 norm mean and sliding variance, it can identify and overcome update deadlock states caused by excessive anti-oscillation protection. By restoring network activity through step-size excitation scalars, it significantly improves the closed-loop stability and convergence accuracy of large-scale scheduling networks under complex conditions, resolving update deadlock and achieving accurate compensation for high-dimensional gradients. Attached Figure Description

[0013] Figure 1 A schematic diagram illustrating the steps of the reinforcement learning-based group enterprise resource scheduling optimization method provided in the embodiments of this application;

[0014] Figure 2 A schematic diagram of the logical flow of the group enterprise resource scheduling optimization method based on reinforcement learning provided in the embodiments of this application;

[0015] Figure 3 This is a schematic diagram of the structure of a group enterprise resource scheduling optimization system based on reinforcement learning provided in an embodiment of this application. Detailed Implementation

[0016] This application's embodiments address the technical problem in the prior art where, in a distributed, multi-level group fund scheduling scenario with heterogeneous physical transmission delays, ignoring the phase slip caused by instruction transmission leads to phase misalignment of hierarchical control instructions and failure of feedback signals, thereby causing nonlinear feedback oscillations in the system. This is achieved through a reinforcement learning-based group enterprise resource scheduling optimization method and system.

[0017] This solution starts from the underlying dynamics of distributed asynchronous control and constructs a set of reinforcement learning optimization logic that deeply couples with physical delay characteristics. Considering the differences in physical links between HQ and execution nodes in group resource scheduling, the starting point of the logic is to break the traditional synchronous state assumption and actively capture the timestamp offset of instructions throughout the entire transmission link. By calculating the bidirectional communication delay of each scheduling service at the physical layer, this solution first establishes a confidence evaluation mechanism based on time decay. The core idea of ​​this mechanism is to transform the physical interference term "delay" into a "timeliness confidence weight factor," which directly affects the observation reward scalar of execution feedback, thereby removing the spurious reward component caused by phase slippage at the data source.

[0018] To further address the random jitter generated by heterogeneous networks, this scheme introduces a global delay feature queue for time series modeling. By comparing median filtering with historical smooth delay scalars, outlier spikes in the delay signal are identified and suppressed, ensuring the smoothness of timeliness compensation. Building upon this, the scheme deepens the optimization perspective from simple reward correction to precise gradient allocation. A Hessian matrix diagonal sensitivity evaluation operator quantifies the sensitivity of each weight dimension in the reinforcement learning network to environmental fluctuations. The introduction of this sensitivity mask matrix allows the network to automatically avoid weight directions susceptible to phase misalignment interference during updates.

[0019] Furthermore, to address potential weight update deadlocks in long-latency environments, this scheme constructs an active incentive logic by monitoring the L2 norm of the weight compensation difference tensor and the sliding variance of the observation reward scalar. When a deadlock state caused by overprotection is detected, momentum is injected in real time using the step-size incentive scalar to forcibly restore the strategy's search capability. The entire scheme achieves a complete logical closed loop from "phase awareness" to "reward denoising," then to "gradient mask compensation," and finally to "dynamic deadlock incentives."

[0020] This scheme transforms the unavoidable physical time delay between levels from "random noise" into "prior parameters for adaptive gain control." This logic not only solves the phase misalignment problem mentioned in the background technology, but also achieves superior convergence stability of the global scheduling strategy in a highly disturbed heterogeneous network environment through nonlinear feedback attenuation and sensitivity gradient pruning.

[0021] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0022] like Figure 1The diagram shown illustrates the steps of the reinforcement learning-based group enterprise resource scheduling optimization method provided in this application embodiment. Addressing the technical challenge of feedback signal phase inaccuracies and gradient update oscillations caused by ignoring network bidirectional communication delays in distributed, multi-level group fund scheduling scenarios with heterogeneous physical transmission delays, this embodiment achieves a technical effect of fundamentally eliminating distributed hierarchical control feedback oscillations from the underlying algorithm logic and realizing the stability of large-scale network scheduling closed-loop control.

[0023] This implementation method relates to a group enterprise resource scheduling optimization method based on reinforcement learning, specifically including the following steps:

[0024] The data on fund allocation needs is obtained through the headquarters node, and the reinforcement learning network of the current data period outputs the scheduling action instruction at the first timestamp and sends it to the execution node. Here, the data on fund allocation needs includes the real-time liquidity gap and position allocation requirements of the headquarters for cross-regional fund allocation. The first timestamp records the absolute system clock moment when the reinforcement learning network completes inference and generates a decision.

[0025] The reinforcement learning network is a pre-built deep deterministic policy gradient network or proximal policy optimization network, or other Actor-Critic architecture model. Specifically, the reinforcement learning network includes a state input layer, at least three fully connected hidden layers, and an action output layer. The hidden layers use the ReLU activation function to prevent gradient vanishing during deep forward propagation, and the output layer uses the Tanh activation function to strictly map the output to the normalized action control space of [-1, 1]. Subsequently, it is scaled proportionally to the physical limit range of the actual fund allocation amount of each financial institution through a linear transformation layer.

[0026] The execution node receives scheduling action instructions and calls the business interface to execute them. It collects an observation reward scalar representing the feedback of the scheduling environment's state and feeds it back to the headquarters node at the second timestamp upon completion. In this stage, the business interface serves as an API channel connecting to the underlying data streams of heterogeneous financial institutions. Due to differences in physical distance and processing capacity across institutional links, the second timestamp precisely anchors the moment when the action actually takes effect in the physical world. The frequency of the data collection operation is perfectly aligned with the current data cycle, ensuring that each scheduling action instruction has a corresponding discrete observation node.

[0027] The bidirectional communication delay is obtained by calculating the arithmetic difference between the second and first timestamps at the headquarters node. This bidirectional communication delay is then substituted into a preset negative exponential delay decay function to calculate the timeliness confidence weight factor. The observation reward scalar is multiplied by the timeliness confidence weight factor to obtain the time difference denoising reward scalar. The bidirectional communication delay dynamically quantifies the total time spent on instruction transmission and execution over the wide area network. The preset negative exponential delay decay function is configured by network benchmark measurement experiments to characterize the nonlinear relationship between time delay and the evolution of the underlying physical system state, i.e., the degree of phase drift. The decay function exhibits exponential decay as the bidirectional communication delay increases.

[0028] The specific steps for obtaining the attenuation coefficient in the preset negative exponential delay attenuation function include: collecting physical link delay and state offset logs from the past month before the system goes online; fitting a correlation curve between bidirectional communication delay and physical state evolution error using the least squares method; and extracting the curvature parameter of the fitted curve as a fixed attenuation coefficient to inject into the function. Through multiplication, the system applies a confidence penalty to the scalar of observation reward that is distorted due to time delays at the source.

[0029] The policy gradient objective function is constructed based on the time-difference denoised reward scalar. The partial derivative is calculated using the backpropagation algorithm to obtain the basic gradient tensor corresponding to the current weight tensor of the reinforcement learning network. This step utilizes the chain rule well-known in the field of software communication to backpropagate the denoised reward signal to generate the basic gradient tensor that guides the evolution of the reinforcement learning network, filtering out erroneous partial derivative directions caused by outdated feedback data.

[0030] The pre-defined Hessian matrix diagonal sensitivity evaluation operator is invoked to calculate the gradient sensitivity of parameters in each dimension of the current weight tensor, generating a sensitivity mask matrix consistent with the dimensions of the current weight tensor. This pre-defined Hessian matrix diagonal sensitivity evaluation operator belongs to the second-order optimization estimation module, and its configuration is derived from historical multi-period training trajectory data. This operator quantifies the vulnerability of each parameter dimension in the current weight tensor to overall network fluctuations by calculating the diagonal approximation of the second derivative.

[0031] The specific technical processing steps of the pre-defined Hessian matrix diagonal sensitivity evaluation operator are as follows: First, calculate the first-order fundamental gradient of the policy gradient objective function with respect to the current weight tensor; then, apply a small perturbation vector conforming to a normal distribution to the current weight tensor; calculate the first-order gradient of the objective function after the perturbation; finally, perform element-wise division between the difference between the two first-order gradients and the small perturbation vector, and extract the diagonal elements of the resulting operation matrix as the parameter gradient sensitivity of each dimension. This approximates the high computational complexity of the second-order Hessian matrix diagonal evaluation with low computational cost of two first-order forward derivations. The sensitivity mask matrix limits the update step size of sensitive parameters within a safe range, avoiding catastrophic updates under high-latency interference.

[0032] The sensitivity mask matrix and the basic gradient tensor are subjected to the Hadamard product operation to generate the weight compensation difference tensor. The weight compensation difference tensor is then superimposed on the current weight tensor to obtain the updated reinforcement learning network, which is used in the next data cycle. The Hadamard product operation is the element-wise multiplication of the matrix, which realizes the fine-grained masking and scaling of each dimension in the basic gradient tensor.

[0033] Figure 2 This is a schematic diagram of the logical flow of the reinforcement learning-based resource scheduling optimization method for group enterprises provided in this application embodiment. Through the above overall technical solution, this implementation breaks the erroneous assumption of instantaneous synchronization between command issuance and observation feedback implicitly accepted by existing distributed control systems, accurately assessing and eliminating the phase error contained in the two-way communication delay. Combined with an adaptive sensitivity masking strategy for high-dimensional network parameters, it reverses the passive situation where existing technologies are prone to abnormal oscillations and divergences when facing heterogeneous communication delays. It achieves high robustness of the control closed loop and high convergence accuracy of the scheduling strategy under long-latency, high-jitter network conditions, ensuring the security and optimality of large-scale cross-regional fund scheduling for group enterprises.

[0034] Furthermore, to address the problem of gradient update divergence caused by sudden jitter in heterogeneous networks, this implementation achieves a technical effect of smoothing the network baseline background and eliminating transient delay noise.

[0035] Before calculating the timeliness confidence weight factor, the process includes: storing the bidirectional communication delays collected from multiple consecutive historical data periods in a global delay feature queue of a preset length in chronological order; the preset length is specifically determined by pre-statistically analyzing the decay characteristics of the autocorrelation coefficient of the target network's historical steady-state time series, and the time span corresponding to the first decay of the autocorrelation coefficient to below 0.1 is selected as the preset length. The global delay feature queue of the preset length adopts a first-in-first-out dynamic sliding window update mechanism, with a typical value range of 50 to 200 data periods.

[0036] Median filtering is performed on the global delay feature queue to remove extreme values, and the arithmetic mean is calculated to generate a historical smooth delay scalar that represents the network baseline. This processing method eliminates occasional extremely high delay spikes caused by network congestion by median sorting, ensuring that the historical smooth delay scalar can truly reflect the steady-state transmission capability of the communication physical link.

[0037] Calculate the absolute difference between the bidirectional communication delay of the current data period and the historical smooth delay scalar to obtain the delay jump difference;

[0038] Determine whether the latency jump difference is greater than a preset mutation threshold; the preset mutation threshold is used to distinguish between normal network fluctuations and severe link jitter. The specific dynamic setting steps for the preset mutation threshold are as follows: calculate the standard deviation scalar of all existing data in the global latency feature queue in real time, add a preset multiple, such as 3 times the standard deviation scalar, to the arithmetic mean, and use it as the preset mutation threshold for the current data period.

[0039] If the latency jump difference exceeds the abrupt change threshold, the current bidirectional communication latency is smoothed using a preset exponential moving average decay coefficient to generate a smoothed bidirectional communication latency, which then replaces the original bidirectional communication latency. The preset exponential moving average decay coefficient is typically between 0.1 and 0.3 to give higher weight to the historical smoothed latency scalar, forcibly suppressing the current abnormal bidirectional communication latency. The specific processing logic for smoothing dimensionality reduction is to complete the numerical update using the following standard mathematical formula:

[0040] ,in, The smooth bidirectional communication latency generated for the current data cycle, The preset exponential moving average decay coefficient, For the current two-way communication latency, The smoothed bidirectional communication delay generated for the previous data cycle.

[0041] Through the above technical solutions, this embodiment solves the problem of large fluctuations in the timeliness confidence weight factor caused by instantaneous abnormal noise generated by heterogeneous interface communication by constructing a global time delay feature queue for median filtering and a smooth dimensionality reduction mechanism based on the exponential moving average decay coefficient. It achieves the unique technical effect of smooth timeliness compensation and continuous and stable convergence of reinforcement learning network weight updates under network dynamic extreme jitter.

[0042] Furthermore, to address the problem that gradient compensation under large delay jumps can easily lead to overcorrection, this implementation achieves an unexpected technical effect with adaptive truncation smoothing and unidirectional gradient shading.

[0043] This scheme further performs deep constraints based on the time delay jump difference extracted in the previous implementation method. Specifically, before performing the Hadamard product operation between the sensitivity mask matrix and the basic gradient tensor, it also includes:

[0044] The time delay jump difference is input into a preset linear positive correlation mapping function to generate a truncation smoothing threshold; the specific calculation steps of the preset linear positive correlation mapping function adopt the following standard mathematical formula:

[0045] ,in, To truncate the smoothing threshold, This is the gain coefficient. This represents the time delay jump difference. Based on the smoothing bias constant. Gain coefficient. With the basic smooth bias constant All parameters are pre-fitted and set using a grid search strategy in the offline simulation environment of the system to ensure that the more drastic the time delay jump, the larger the generated truncation smoothing threshold.

[0046] Extract the sign matrix of the underlying gradient tensor; the sign matrix retains only the positive and negative polarity information of each dimension element of the underlying gradient tensor.

[0047] Perform a traversal comparison on the sensitivity mask matrix: force the abnormal mask elements in the sensitivity mask matrix whose signs are opposite to those of the corresponding elements in the sign matrix to be assigned the value zero, and obtain the same-direction mask matrix; this step effectively prevents the sensitivity mask matrix from distorting the original parameter optimization direction in the reverse direction when correcting the gradient, and ensures the monotonicity of the optimization process.

[0048] All mask elements in the same-direction mask matrix whose values ​​are less than the truncation smoothing threshold are uniformly forced to be assigned the truncation smoothing threshold value to obtain a non-negative same-direction sensitivity mask matrix. This operation establishes a bottom line for protecting network parameter updates and prevents gradient vanishing problems caused by extreme scaling.

[0049] The original sensitivity mask matrix is ​​replaced with a non-negative, same-direction sensitivity mask matrix.

[0050] Through the above technical solution, this embodiment generates a truncation smoothing threshold by inputting the time delay jump difference into a preset linear positive correlation mapping function and constructs a unidirectional mask matrix. This solves the parameter solidification problem caused by aggressive gradient masking due to severe network jitter and achieves a unique technical effect of dynamically supporting and protecting the high-dimensional gradient matrix while ensuring the correct update direction.

[0051] Furthermore, addressing the deep-seated flaw of the anti-oscillation overprotection mechanism potentially leading to network update deadlock, this implementation achieves a technical effect of dynamically identifying deadlock states and introducing adaptive momentum excitation to break through stagnation. Given the risk of system freezing under the high-intensity masking mechanism of the previous scheme, before superimposing the weight compensation difference tensor onto the current weight tensor, the implementation further includes: calculating the average L2 norm of the weight compensation difference tensor over a preset number of data periods; the length of the preset number of data periods is typically set to 50 to 100 steps.

[0052] The L2 norm average accurately measures the average movement distance of the model weights in recent times. When the L2 norm average is lower than a preset deadlock warning threshold, the observed reward scalar is obtained, and the sliding variance of the observed reward scalar is calculated over a preset number of consecutive data periods. The preset deadlock warning threshold can be set as the tenth percentile of the total range of the average movement distance of the historical model weights. Obtaining the observed reward scalar and calculating the sliding variance aims to detect whether the external physical execution environment is still fluctuating. Specifically, the preset deadlock warning threshold and the preset environment activity threshold are set as follows: During the system preparation phase, the probability density distribution of the L2 norm of the weights and the sliding variance of the feedback reward within the historical steady-state operating range is statistically analyzed. The lower bound of the probability distribution feature is extracted as the 5th percentile and the upper bound is extracted as the preset environment activity threshold.

[0053] If the sliding variance exceeds a preset environmental activity threshold, the sliding variance is multiplied by a preset momentum scaling factor to generate a step size excitation scalar. The preset environmental activity threshold is used to determine if the external environment has not converged, such as if the funding gap continues to fluctuate frequently. The preset momentum scaling factor, typically 0.01, acts as a dynamic coefficient for escaping local optima. The specific configuration method for the preset momentum scaling factor is as follows: based on the initial learning rate parameter of the underlying optimizer of the reinforcement learning network, such as Adam or SGD, one-tenth of its initial learning rate is taken as the fixed momentum scaling factor. The step size excitation scalar is then superimposed onto the weight compensation difference tensor through tensor addition to obtain the updated weight compensation difference tensor, which replaces the original weight compensation difference tensor.

[0054] Through the above technical solution, this embodiment solves the problem of deadlock in reinforcement learning network updates caused by excessive time delay decay penalty in large-scale scheduling networks by monitoring the average L2 norm of the weight compensation difference tensor and the sliding variance of the observation reward scalar. It achieves the unique technical effect of forcibly restoring the network optimization activity by injecting step size incentive scalar when the environment is active but the policy is stagnant.

[0055] Furthermore, addressing the issue of local execution nodes being unable to accurately quantify the actual resource deviation, this implementation achieves a technical effect that precisely maps physical resource gaps into negatively correlated observation rewards. The specific steps for collecting and representing the observation reward scalar of the scheduling environment status feedback include: obtaining the current resource status vector and a preset resource configuration target vector locally on the execution node; the current resource status vector includes parameters such as the real-time available cash balance and short-term debt of a specific branch; the specific construction rules and generation steps of the preset resource configuration target vector are as follows: combining the group's overall risk control standards, the minimum reserve requirement threshold, expected daily cash flow demand, and credit exposure limit of each branch in different dimensions are constructed and merged into a feature vector with a defined dimension, and this feature vector is stored in the local memory of each execution node as a benchmark reference vector.

[0056] Calculate the Euclidean distance between the current resource state vector and the resource allocation target vector to obtain the resource gap deviation scalar; the Euclidean distance enables dimensionless aggregation of multi-dimensional funding deviations. Substitute the resource gap deviation scalar into a preset negative reward mapping function for negative correlation mapping, and output the observed reward scalar; the specific calculation steps of this preset negative reward mapping function adopt the following standard mathematical formula:

[0057] ,in, To observe the reward scalar, The boundary coefficient for the maximum negative penalty. This is the sensitivity scaling constant. The resource gap deviation scalar. Maximum negative penalty boundary coefficient. The sensitivity scaling constant determines the lower limit and extreme value of the reward. To control the rate at which the hyperbolic tangent function reaches its limit boundary, this mathematical model structure ensures smooth, continuous gradient excitation when the resource gap deviation scalar fluctuates within a small range, while outputting a strong nonlinear negative boundary penalty as the gap widens.

[0058] Through the above technical solution, this embodiment solves the technical problem of lack of a unified metric for the underlying physical execution results of cross-level systems by calculating the Euclidean distance between the current resource state vector and the preset resource configuration target vector and transforming it through a preset negative reward mapping function. This achieves the unique technical effect of objectively and standardizedly constructing an environmental state reward feedback closed loop.

[0059] Furthermore, addressing the challenge of nonlinear error accumulation caused by expired feedback, this implementation achieves a technical effect of accurately quantifying timeliness confidence based on negative exponential decay and performing feedback denoising. The specific process for obtaining the time difference denoising reward scalar is as follows: A preset time delay sensitivity coefficient scalar is obtained; the preset time delay sensitivity coefficient scalar is empirically set based on the tolerance of specific business operations for instruction delays, typically ranging from 0.5 to 2.0. The specific quantification steps for this preset time delay sensitivity coefficient scalar are as follows: The maximum response time to the funding gap in the core business processing within the group is statistically analyzed; a constant 1 is divided by this maximum response time, and the quotient is taken as the preset time delay sensitivity coefficient scalar. The specific calculation formula is as follows:

[0060] , ,in, As a time-sensitive confidence weighting factor, The preset time-delay sensitivity coefficient scalar, For bidirectional communication delay, To observe the reward scalar, The scalar reward is used for time difference denoising.

[0061] Through the above technical solution, this embodiment introduces an exponential negative exponential delay decay function for mathematical mapping, which solves the drawback that the original reinforcement learning framework cannot directly remove the state variation components caused by heterogeneous time delay. It achieves the unique technical effect of strictly quantifying the physical phase misalignment caused by communication lag into a penalty coefficient, thereby extracting a pure and timely usable reward signal.

[0062] To address the challenge of directly mapping gradients in hierarchical networks for discrete control commands, this implementation achieves a technical effect of accurately converting time-delay denoising rewards into network weight update guidance based on a policy gradient mechanism. The specific steps for obtaining the basic gradient tensor are as follows: First, obtain the action probability distribution tensor generated synchronously by the reinforcement learning network when outputting scheduling action commands; use this as the standard output stream for the forward inference stage of deep reinforcement learning. Based on the index position corresponding to the scheduling action command, extract the corresponding action activation probability value from the action probability distribution tensor, and perform a natural logarithmic operation on the action activation probability value to generate a logarithmic probability distribution scalar; the natural logarithmic operation transforms multiplicative probabilities into additive energy representations, facilitating subsequent derivative calculations. Perform an arithmetic multiplication operation between the logarithmic probability distribution scalar and the time-difference denoising reward scalar to obtain the policy gradient objective function value; under this combined logic, actions with positive time-difference denoising reward scalars will have their logarithmic probability amplified, while those with negative scalars will be suppressed. Perform the first-order gradient derivative operation on the policy gradient objective function value with respect to the current weight tensor, and output the basic gradient tensor. The basic gradient tensor is used to characterize the contribution of the current weight tensor to the reward improvement in each dimension.

[0063] Through the above technical solution, this embodiment constructs the objective function by extracting the action probability distribution tensor, performing logarithmic operations, and performing arithmetic multiplication with the time difference denoising reward scalar. This solves the technical problem that complex neural networks cannot update their strategies due to the lack of a directional backpropagation channel with time delay labels, and achieves the unique technical effect of ensuring that the network weights are always optimized in the direction of maximizing the denoising confidence reward.

[0064] Furthermore, to address the problem of excessive computational overhead in evaluating second-order sensitivity parameters of high-dimensional reinforcement learning models under complex conditions, leading to system memory overflow, this implementation achieves an unexpected technical effect: it uses a first-order gradient difference approximation based on small perturbations to calculate the diagonal of the Hessian matrix and adaptively generate constraint masks.

[0065] To replace the massive global second-order Hessian matrix, which cannot be directly solved within an extremely short response period, the specific process of generating a sensitivity mask matrix by calling the preset Hessian matrix diagonal sensitivity evaluation operator is as follows:

[0066] A preset small perturbation tensor is superimposed on the current weight tensor to obtain the perturbed weight tensor. The preset small perturbation tensor is a Gaussian distribution random noise tensor with a mean of zero and a preset minimum variance, and its physical memory dimension is strictly consistent with that of the current weight tensor.

[0067] Calculate the perturbed first-order gradient tensor of the policy gradient objective function with respect to the perturbed weight tensor; this step can be obtained with low latency by performing a single additional forward and backward propagation on the perturbed network.

[0068] The tensor difference between the perturbated first-order gradient tensor and the fundamental gradient tensor is calculated, and then element-wise division is performed between this tensor difference and the preset small perturbation tensor to obtain the Hessian diagonal approximation tensor, which serves as the gradient sensitivity of each dimension. The specific calculation for this step uses the following standard mathematical formula:

[0069] ,in, Let Hessen's diagonal approximation tensor be used. For the first-order gradient tensor after perturbation, Based on the gradient tensor, This is a pre-defined small perturbation tensor. The mathematical model utilizes the difference approximation principle of Taylor expansion. By dividing the difference between two first-order gradient calculations by a small offset, the diagonal elements of the second derivative, which represent the parameter sensitivity, can be estimated with high precision.

[0070] The Hessian diagonal approximation tensor is subjected to maximization-minimization to obtain and output the sensitivity mask matrix. This operation forces the sensitivity coefficients of each dimension of the underlying layer to be mapped to the standard mathematical space of 0 to 1, thus avoiding numerical overflow in the subsequent Hadamard product operation.

[0071] Through the above technical solution, this embodiment solves the computational bottleneck of the classical second-order optimizer in real time calculating the global Hessian matrix in resource scheduling networks with low latency requirements by injecting a preset small perturbation tensor into the current weight tensor and approximating the Hessian diagonal using the difference operation of the first-order gradient. It achieves the unique technical effect of accurately assessing the vulnerability and sensitivity of each parameter dimension and shielding dangerous update directions with extremely low computational overhead, which greatly improves the overall computational efficiency and stability of the network in large-scale scheduling.

[0072] Figure 3 This is a schematic diagram of the structure of a group enterprise resource scheduling optimization system based on reinforcement learning provided in an embodiment of this application. The group enterprise resource scheduling optimization system based on reinforcement learning includes: a scheduling action issuing module: used to obtain fund scheduling demand data through the headquarters node, and output scheduling action instructions at the first timestamp through the reinforcement learning network of the current data period, and issue them to the execution node;

[0073] Feedback Scalar Upload Module: Used to receive scheduling action instructions through the execution node and call the business interface to execute them, collect observation reward scalars that represent the status feedback of the scheduling environment, and feed back the observation reward scalars to the headquarters node at the second timestamp after execution is completed;

[0074] The differential reward processing module calculates the bidirectional communication delay based on the arithmetic difference between the second and first timestamps using the headquarters node, substitutes it into the negative exponential delay decay function to calculate the time-of-fact confidence weight factor, and multiplies the observation reward scalar with the time-of-fact confidence weight factor to obtain the time-difference denoised reward scalar. The gradient extraction module constructs the policy gradient objective function based on the time-difference denoised reward scalar, calculates the partial derivative using the backpropagation algorithm, and obtains the basic gradient tensor corresponding to the current weight tensor of the reinforcement learning network. The mask matrix processing module calls the Hessian matrix diagonal sensitivity evaluation operator to calculate the parameter gradient sensitivity of each dimension in the current weight tensor, generating a sensitivity mask matrix with dimensions consistent with the current weight tensor. The network update module performs a Hadamard product operation on the sensitivity mask matrix and the basic gradient tensor to generate a weight compensation difference tensor, and superimposes the weight compensation difference tensor onto the current weight tensor to obtain the updated reinforcement learning network for use in the next data cycle.

[0075] This application also provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements a group enterprise resource scheduling optimization method based on reinforcement learning.

[0076] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0078] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0080] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0081] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A resource scheduling optimization method for group enterprises based on reinforcement learning, characterized in that, Includes the following steps: The system obtains fund allocation demand data through the headquarters node, and outputs scheduling action instructions at the first time stamp through the reinforcement learning network of the current data period, and sends them to the execution node. The execution node receives scheduling action instructions and calls the business interface to execute them, collects the observation reward scalar that represents the status feedback of the scheduling environment, and feeds the observation reward scalar back to the headquarters node at the second timestamp after the execution is completed. The arithmetic difference between the second timestamp and the first timestamp is calculated by the headquarters node to obtain the two-way communication delay. The two-way communication delay is then substituted into the preset negative exponential delay decay function to calculate the timeliness confidence weight factor. The observation reward scalar and the timeliness confidence weight factor are multiplied to obtain the time difference denoising reward scalar. The policy gradient objective function is constructed based on the temporal difference denoising reward scalar, and the partial derivative is calculated through the backpropagation algorithm to obtain the basic gradient tensor corresponding to the current weight tensor of the reinforcement learning network. Call the preset Hessian matrix diagonal sensitivity evaluation operator to calculate the parameter gradient sensitivity of each dimension in the current weight tensor and generate a sensitivity mask matrix with the same dimensions as the current weight tensor. The sensitivity mask matrix and the basic gradient tensor are subjected to the Hadamard product to generate the weight compensation difference tensor. The weight compensation difference tensor is then superimposed on the current weight tensor to obtain the updated reinforcement learning network, which is then used in the next data cycle.

2. The group enterprise resource scheduling optimization method based on reinforcement learning according to claim 1, characterized in that, Before calculating the timeliness confidence weight factor, the following steps are also included: The bidirectional communication latency collected from multiple consecutive historical data periods is stored in a global latency feature queue of a preset length in chronological order. Median filtering is performed on the global delay feature queue to remove extreme values, and the arithmetic mean is calculated to generate a historical smooth delay scalar that represents the baseline background of the network. Calculate the absolute difference between the bidirectional communication delay of the current data period and the historical smooth delay scalar to obtain the delay jump difference; Determine whether the delay jump difference is greater than the preset jump threshold; If the time delay jump difference is greater than the sudden change threshold, the current bidirectional communication time delay is smoothed and reduced in dimensionality using a preset exponential moving average decay coefficient to generate a smooth bidirectional communication time delay, and the original bidirectional communication time delay is replaced by the smooth bidirectional communication time delay. 3.The group enterprise resource scheduling optimization method based on reinforcement learning according to claim 2, characterized in that, Before performing the Hadamard product operation between the sensitivity mask matrix and the underlying gradient tensor, the following steps are also included: Input the time delay jump difference into the preset linear positive correlation mapping function to generate a truncated smoothing threshold; Extract the sign matrix of the underlying gradient tensor; Perform a traversal comparison on the sensitivity mask matrix: force the abnormal mask elements in the sensitivity mask matrix whose signs are opposite to those of the corresponding elements in the sign matrix to be assigned the value zero, and obtain the same-direction mask matrix; All mask elements in the same-direction mask matrix whose values ​​are less than the truncation smoothing threshold are uniformly forced to be set to the truncation smoothing threshold to obtain the non-negative same-direction sensitivity mask matrix. The original sensitivity mask matrix is ​​replaced with a non-negative, same-direction sensitivity mask matrix. 4.The method of claim 3, wherein, Before adding the weight compensation difference tensor to the current weight tensor, the following steps are also included: Calculate the average L2 norm of the weight compensation difference tensor over a predetermined number of consecutive data periods; When the average L2 norm is lower than the preset deadlock warning threshold, the observation reward scalar is obtained, and the sliding variance of the observation reward scalar is calculated within a data period of a preset number of consecutive times. If the sliding variance is greater than the preset environmental activity threshold, the sliding variance is multiplied by the preset momentum scaling factor to generate a step size excitation scalar. The step size excitation scalar is superimposed onto the weight compensation difference tensor through tensor addition to obtain the updated weight compensation difference tensor, which then replaces the original weight compensation difference tensor. 5.The group enterprise resource scheduling optimization method based on reinforcement learning according to claim 1, wherein, The specific steps for collecting and characterizing the observational reward scalar of the scheduling environment state feedback include: Obtain the current resource status vector and the preset resource configuration target vector on the execution node. Calculate the Euclidean distance between the current resource state vector and the resource allocation target vector to obtain the resource gap deviation scalar; The resource gap deviation scalar is substituted into a preset negative reward mapping function to perform negative correlation mapping, and the observation reward scalar is output. 6.The group enterprise resource scheduling optimization method based on reinforcement learning according to claim 1, wherein, The specific process for obtaining the temporal difference denoising reward scalar is as follows: Obtain the preset latency sensitivity coefficient scalar; The specific calculation formula is as follows: , ,in, As a confidence factor for timeliness, The preset time-delay sensitivity coefficient scalar, For bidirectional communication delay, To observe the reward scalar, The scalar reward is used for time difference denoising.

7. The group enterprise resource scheduling optimization method based on reinforcement learning according to claim 1, characterized in that, The specific steps for obtaining the basic gradient tensor are as follows: Obtain the action probability distribution tensor synchronously generated by the reinforcement learning network when outputting scheduled action instructions; Based on the index position corresponding to the scheduling action instruction, extract the corresponding action activation probability value from the action probability distribution tensor, and perform a natural logarithmic operation on the action activation probability value to generate a logarithmic probability distribution scalar. The logarithmic probability distribution scalar and the temporal difference denoising reward scalar are arithmetically multiplied to obtain the policy gradient objective function value. Perform gradient differentiation on the objective function value of the policy gradient with respect to the current weight tensor, and output the basic gradient tensor. The basic gradient tensor is used to represent the contribution of the current weight tensor to the reward improvement in each dimension.

8. The group enterprise resource scheduling optimization method based on reinforcement learning according to claim 1, characterized in that, The specific process for generating the sensitivity mask matrix is ​​as follows: A preset small perturbation tensor is superimposed on the current weight tensor to obtain the perturbed weight tensor; Calculate the first-order gradient tensor of the policy gradient objective function with respect to the perturbed weight tensor; Calculate the tensor difference between the perturbed first-order gradient tensor and the basic gradient tensor, and perform element-wise division with the preset small perturbation tensor to obtain the Hessian diagonal approximation tensor, which serves as the parameter gradient sensitivity for each dimension. Perform maximization-minimization on the Hessian diagonal approximation tensor to obtain and output the sensitivity mask matrix.

9. A group enterprise resource scheduling optimization system based on reinforcement learning, characterized by, include: The scheduling action issuance module is used to obtain fund scheduling demand data through the headquarters node, and output scheduling action instructions at the first timestamp through the reinforcement learning network of the current data period, and issue them to the execution node. Feedback Scalar Upload Module: Used to receive scheduling action instructions through the execution node and call the business interface to execute them, collect observation reward scalars that represent the status feedback of the scheduling environment, and feed back the observation reward scalars to the headquarters node at the second timestamp after execution is completed; Differential reward processing module: It is used to calculate the two-way communication delay by using the headquarters node based on the arithmetic difference between the second timestamp and the first timestamp, and substitute it into the negative exponential delay decay function to calculate the time confidence weight factor. The observation reward scalar and the time confidence weight factor are multiplied to obtain the time differential denoising reward scalar. Gradient extraction module: used to construct the policy gradient objective function based on the temporal difference denoising reward scalar, and to calculate the partial derivative through the backpropagation algorithm to obtain the basic gradient tensor corresponding to the current weight tensor of the reinforcement learning network; Mask matrix processing module: used to call the Hessian matrix diagonal sensitivity evaluation operator, calculate the parameter gradient sensitivity of each dimension in the current weight tensor, and generate a sensitivity mask matrix with the same dimensions as the current weight tensor; Network update module: This module performs a Hadamard product operation between the sensitivity mask matrix and the basic gradient tensor to generate a weight compensation difference tensor. It then adds the weight compensation difference tensor to the current weight tensor to obtain the updated reinforcement learning network, which is then used in the next data cycle.

10. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images