A multi-task optimization scheduling method and system based on edge computing
By introducing the Hearst exponent and state-space modeling of stochastic differential equations into edge computing, and combining particle filtering and deep reinforcement learning, the tendency to explore and exploit is dynamically adjusted, which solves the problem of policy instability caused by sparse and colored noise reward signals in multi-task scheduling of edge computing, and achieves stable scheduling performance and efficient task scheduling.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- QUANZHOU INST OF INFORMATION ENG
- Filing Date
- 2026-06-05
- Publication Date
- 2026-07-10
AI Technical Summary
Existing edge computing multi-task scheduling methods based on deep reinforcement learning cannot effectively utilize temporal continuity for dynamic estimation when dealing with sparse and colored noise reward signals. This causes the scheduling strategy to jitter during long periods of quiet without events and cannot adjust the exploration intensity when environmental noise increases, resulting in unstable scheduling performance.
By calculating the Hearst exponent of the original observation reward sequence, the state space of the stochastic differential equation is set. By combining particle filtering and deep reinforcement learning, the tendency to explore and exploit is dynamically adjusted. The exploration regulation factor is calculated using the posterior variance and the effective diversity index, and the scheduling strategy is adaptively adjusted.
It effectively suppresses the interference of colored noise and sparsity on strategy updates, maintains the stability of scheduling decisions, and can proactively adjust the exploration probability when the environment changes, thereby achieving a dynamic balance between exploration and utilization and improving scheduling efficiency.
Smart Images
Figure CN122363864A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of edge computing technology, and specifically to a multi-task optimization scheduling method and system based on edge computing. Background Technology
[0002] Edge computing, by deploying computing resources at the network edge close to the data source, can effectively reduce data transmission latency and bandwidth consumption during multi-task scheduling, and is widely used in scenarios with high real-time requirements such as intelligent video surveillance, autonomous driving, and industrial IoT. In intelligent video surveillance security systems, edge nodes typically need to simultaneously schedule video analysis tasks from multiple high-definition cameras, such as intrusion detection, face capture, and behavior recognition, and rationally allocate different tasks to local computing units or adjacent edge nodes for execution.
[0003] Existing edge computing multi-task scheduling methods based on deep reinforcement learning cannot dynamically estimate the implicit true reward when dealing with sparse and colored noise reward signals. They also lack quantitative representation of estimation uncertainty. This causes the scheduling strategy to fall into random jitter due to gradient loss during long event-free quiescent periods. At the same time, it cannot actively adjust the exploration intensity to avoid noise misleading when environmental noise increases. Ultimately, the task offloading decision fluctuates drastically between the cloud and the edge, making it difficult to maintain stable scheduling performance. Summary of the Invention
[0004] The purpose of this invention is to provide a multi-task optimization scheduling method and system based on edge computing to solve the problems mentioned above.
[0005] The objective of this invention can be achieved through the following technical solutions: A multi-task optimization scheduling method based on edge computing includes the following steps: S1, obtain the original observation reward sequence generated after the edge node performs multi-task scheduling. The original observation reward contains sparse scheduling feedback signals with colored noise. S2, calculate the Hearst exponent of the original observed reward sequence, and set the state space of the stochastic differential equation according to the Hearst exponent. The stochastic differential equation is used to describe the mean regression process of the implicit true reward changing continuously over time, and outputs the prior distribution of the implicit true reward at the current time. S3 updates the prior distribution using particle filtering and calculates the effective diversity index of particle weights after each update. The resampling trigger threshold is dynamically adjusted based on the effective diversity index. The posterior distribution of the implicit true reward is recursively calculated using the original observation reward at the current moment. The posterior mean and posterior variance are extracted from the posterior distribution. S4 uses the posterior mean to replace the original observation reward as the training target for updating the deep reinforcement learning strategy. At the same time, it calculates the exploration adjustment factor based on the posterior variance and the effective diversity index to guide the scheduling strategy to adaptively adjust the tendency of exploration and utilization under different degrees of uncertainty. S5, based on the accumulated historical reward trajectory, periodically re-estimates the recovery rate, long-term mean, and diffusion coefficient in the stochastic differential equation, and feeds the updated parameters back to S2, so that the continuous-time dynamic model implying the real reward can continuously adapt to changes in the monitoring scenario.
[0006] As a further aspect of the present invention: the prior distribution specifically includes: The original observation reward sequence is divided into fixed time windows, and the sequence within each time window is used as an analysis unit; Rescaled range transformation is performed on each analysis unit, cumulative deviation and range are calculated, and the Hearst exponent of the analysis unit is obtained by linear regression fitting in logarithmic coordinate system. The median of the Hearst exponents of all analysis units is taken as the final Hearst exponent. When the final Hearst exponent is greater than 0.6, the recovery rate in the stochastic differential equation is set to a high value to enhance mean regression; otherwise, it is set to a low value, and the prior distribution of the implicit true reward is output according to the setting.
[0007] As a further aspect of the present invention: the method of obtaining the Hearst exponent of the analysis unit through linear regression fitting in a logarithmic coordinate system specifically includes: The logarithm of each time window within the same analysis unit is used as the x-axis, and the logarithm of the corresponding rescaled range is used as the y-axis to form a scatter set. Remove the two points corresponding to the minimum and maximum x-coordinates in the scattered point set to obtain the intermediate scattered point set; Perform least-truncation square regression on the intermediate scatter set, calculate the slope of the regression line. The least-truncation square regression adopts an iterative reweighting method, and in each iteration, the point with the largest absolute value of the residual is removed and then refitted. The slope of the final regression line after iterative convergence is used as the Hearst exponent of the analysis unit.
[0008] As a further aspect of the present invention: S3 specifically includes: The normalized entropy value of the current particle set is calculated based on the weight of each particle, and the natural index of the normalized entropy value is used as the effective diversity index. The product of the effective diversity index and the preset base threshold is used as the dynamic trigger threshold for resampling. Calculate the number of effective particles in the current particle set. If the number of effective particles is lower than the dynamic trigger threshold, perform polynomial resampling to update the particle set; otherwise, keep the current particle set unchanged. Using the resampled or unchanged set of particles, combined with the original observed reward at the current moment, the posterior distribution of the implicit true reward is recursively calculated, and the posterior mean and posterior variance are extracted from the posterior distribution.
[0009] As a further aspect of the present invention: the step of calculating the normalized entropy value of the current particle set based on the weight of each particle, and using the natural index of the normalized entropy value as the effective diversity index, specifically includes: Obtain the weights of each particle in the current particle set, and take the natural logarithm of each weight to obtain the logarithmic weight sequence; Calculate the arithmetic mean of the log-weighted sequence, and subtract the arithmetic mean from each log-weight to obtain the centered log-weighted sequence; The normalized entropy value is obtained by multiplying each value in the centralized logarithmic weight sequence by its corresponding original weight, summing the results, and then dividing by the natural logarithm of the total number of particles. Perform a natural index operation on the normalized entropy value and output the result as an effective diversity index.
[0010] As a further aspect of the present invention: S4 specifically includes: The joint uncertainty measure is obtained by multiplying the posterior variance by the effective diversity index and then taking the square root. The joint uncertainty metric is compared with a preset saturation threshold, and the smaller of the two values is taken as the input of the nonlinear transformation. The hyperbolic tangent operation is then performed on the input to obtain an adjustment base value between 0 and 1. Multiply the maximum exploration probability by the adjustment base value to obtain the current exploration adjustment factor, and use the exploration adjustment factor as the upper limit of the probability of random exploration when the scheduling strategy executes the action; Based on the exploration of the adjustment factor to control the update process of the deep reinforcement learning strategy, the posterior mean is directly used as the immediate benefit term in the temporal difference error.
[0011] As a further aspect of the present invention: multiplying the maximum exploration probability by the adjustment base value to obtain the current exploration adjustment factor, and using the exploration adjustment factor as the upper limit of the probability of random exploration when the scheduling strategy executes an action, specifically includes: Obtain the sequence number of the current scheduling cycle and calculate the ratio of the sequence number to the preset total cycle to obtain the cycle progress ratio; The progress decay factor is obtained by squaring the periodic progress ratio using a sine function. Multiply the preset maximum exploration probability by the progress decay factor to obtain the periodically adaptive maximum exploration probability; Multiply the maximum exploration probability of the periodic adaptive method by the adjustment base value to output the current exploration adjustment factor.
[0012] As a further aspect of the present invention: S5 specifically includes: Extract the posterior mean sequence of the latent true reward within the most recent re-estimation period from the historical reward trajectory, and use the posterior mean sequence as the sample to be fitted. Perform maximum likelihood estimation on the fitted sample and iteratively solve for the optimal estimates of the recovery rate, long-term mean, and diffusion coefficient. The diffusion coefficient estimated in the previous period and the diffusion coefficient estimated in the current period are subjected to an exponentially weighted moving average to obtain a smoothed diffusion coefficient. The smoothed diffusion coefficient, the recovery rate estimated in the current period, and the long-term mean are then used to replace the original parameters. The updated response rate, long-term mean, and diffusion coefficient are fed back into the state space of S2 for use in calculating the prior distribution at subsequent time steps.
[0013] As a further aspect of the present invention: the step of performing maximum likelihood estimation on the sample to be fitted, and iteratively solving for the optimal estimates of the recovery rate, long-term mean, and diffusion coefficient, specifically includes: Use the recovery rate, long-term mean, and diffusion coefficient estimated in the previous cycle as the initial values for the current iteration; With the current diffusion coefficient fixed, perform linear fitting on the residuals constructed from the sample to be fitted and the current recovery rate and long-term mean, and update the recovery rate and long-term mean. The diffusion coefficient is updated based on the moment estimation conditions of the residual sum of squares, with the updated recovery rate and long-term mean fixed. The current iteration value is output as the optimal estimate when the relative changes in the recovery rate, long-term mean, and diffusion coefficient obtained in two consecutive iterations are all lower than the preset convergence threshold.
[0014] A multi-task optimization scheduling system based on edge computing includes: The reward sequence acquisition module acquires the original observation reward sequence generated after the edge node performs multi-task scheduling. The original observation reward contains sparse scheduling feedback signals with colored noise. The state space construction module calculates the Hearst exponent of the original observed reward sequence and sets the state space of the stochastic differential equation based on the Hearst exponent. The stochastic differential equation is used to describe the mean regression process of the implicit true reward that changes continuously over time and outputs the prior distribution of the implicit true reward at the current time. The particle filtering and posterior estimation module updates the prior distribution using particle filtering and calculates the effective diversity index of particle weights after each update. It dynamically adjusts the resampling trigger threshold based on the effective diversity index and recursively calculates the posterior distribution of the implicit true reward using the original observation reward at the current moment. It then extracts the posterior mean and posterior variance from the posterior distribution. The strategy training and exploration regulation module replaces the original observation reward with the posterior mean as the training target for updating the deep reinforcement learning strategy. At the same time, it calculates the exploration regulation factor based on the posterior variance and the effective diversity index to guide the scheduling strategy to adaptively adjust the tendency of exploration and utilization under different degrees of uncertainty. The adaptive update module for model parameters periodically re-estimates the recovery rate, long-term mean, and diffusion coefficient in the stochastic differential equation based on the accumulated historical reward trajectory, and feeds the updated parameters back to S2 so that the continuous-time dynamic model implying the real reward can continuously adapt to changes in the monitoring scenario.
[0015] The beneficial effects of this invention are: (1) By introducing state space modeling based on Hearst exponent and stochastic differential equations into the deep reinforcement learning scheduling strategy, and combining particle filtering to perform online posterior estimation of the hidden real reward, the interference of colored noise and sparsity in the original observed reward on the policy update is effectively suppressed, so that the scheduling decision can remain stable during the long-term silent period without intrusion events, and avoids the policy blind jitter caused by the lack or jump of reward signal. (2) By using the posterior variance and the effective diversity index to jointly calculate the exploration adjustment factor, the scheduling strategy can autonomously adjust the upper limit of the probability of random exploration according to the uncertainty of the current reward estimation. In the event of sudden changes in ambient light or increased detection noise caused by camera jitter, the exploration can be actively increased to avoid local optima. When the environment is stable, the exploration can be reduced to ensure scheduling efficiency, thereby achieving a dynamic balance between exploration and utilization without relying on manual parameter adjustment. Attached Figure Description
[0016] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system block diagram of the present invention. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] Please see Figure 1 As shown, this invention is a multi-task optimization scheduling method based on edge computing, comprising the following steps: S1, obtain the original observation reward sequence generated after the edge node performs multi-task scheduling. The original observation reward contains sparse scheduling feedback signals with colored noise. S2, calculate the Hearst exponent of the original observed reward sequence, and set the state space of the stochastic differential equation according to the Hearst exponent. The stochastic differential equation is used to describe the mean regression process of the implicit true reward changing continuously over time, and outputs the prior distribution of the implicit true reward at the current time. S3 updates the prior distribution using particle filtering and calculates the effective diversity index of particle weights after each update. The resampling trigger threshold is dynamically adjusted based on the effective diversity index. The posterior distribution of the implicit true reward is recursively calculated using the original observation reward at the current moment. The posterior mean and posterior variance are extracted from the posterior distribution. S4 uses the posterior mean to replace the original observation reward as the training target for updating the deep reinforcement learning strategy. At the same time, it calculates the exploration adjustment factor based on the posterior variance and the effective diversity index to guide the scheduling strategy to adaptively adjust the tendency of exploration and utilization under different degrees of uncertainty. S5, based on the accumulated historical reward trajectory, periodically re-estimates the recovery rate, long-term mean, and diffusion coefficient in the stochastic differential equation, and feeds the updated parameters back to S2, so that the continuous-time dynamic model implying the real reward can continuously adapt to changes in the monitoring scenario.
[0019] In S1, the raw observation reward sequence generated after the edge nodes perform multi-task scheduling is obtained. The raw observation reward contains sparse scheduling feedback information with colored noise, specifically including: This embodiment takes an intelligent video surveillance security scenario as an example. In the intelligent video surveillance security scenario, edge nodes are deployed at each monitoring point. Each edge node uses its internal task scheduling executor to allocate the received video analysis tasks (such as intrusion detection and face capture) to the local computing unit or offload them to adjacent nodes.
[0020] After each scheduling action is completed, the performance monitoring component built into the edge node collects feedback data for this scheduling. The performance monitoring component consists of event hooks in the operating system kernel and a lightweight statistics program, and runs in the real-time operating system of the edge node.
[0021] Specifically, once a scheduling decision is executed, the monitoring component records the actual completion time, task success flag, and resource usage for that scheduling, and calculates a scalar value as the initial observation reward based on a predefined reward function. The definition of the reward function depends on the business objective: for intrusion detection tasks, if the target is correctly identified and the delay is less than the preset deadline, the reward value is positive; if a false negative or timeout occurs, the reward value is negative; for face capture tasks, the reward is based on a comprehensive score of capture clarity and response speed. Since no intrusion events occur more than 90% of the time in a security environment, the calculated reward values are mostly zero or close to zero, forming a sparse distribution. At the same time, sudden changes in ambient light, camera jitter, and instantaneous network fluctuations can cause abnormal jumps in the completion time or success flag collected by the performance monitoring component, thus mixing non-Gaussian colored noise into the initial observation reward.
[0022] Edge nodes store the raw observation rewards generated in each scheduling cycle into a local cache queue in chronological order, forming a raw observation reward sequence for subsequent steps. The length of this sequence is determined by a preset time window or the maximum cache depth, typically 200 sampling points.
[0023] In S2, the Hearst exponent of the original observed reward sequence is calculated, and the state space of the stochastic differential equation is set according to the Hearst exponent. The stochastic differential equation is used to describe the mean regression process of the implicit true reward that changes continuously over time, and outputs the prior distribution of the implicit true reward at the current time, specifically including: The acquired raw observation reward sequences were divided into fixed time windows. Each time window was set to a width of 50 sampling points, with 20 sampling points overlapping between adjacent time windows to ensure continuity of analysis. The reward value sequence within each time window was treated as an independent analysis unit.
[0024] For each analysis unit, a rescaled range transformation is performed. The specific process is as follows: For a given time window length, the sequence within the unit is divided into several adjacent and non-overlapping sub-intervals, each with the same length as the window. The cumulative deviation of the reward values within each sub-interval is calculated, which is the cumulative sum from the start point of the sub-interval to the current point minus the arithmetic mean of the reward values within that sub-interval multiplied by the number of points. Within each sub-interval, the maximum and minimum cumulative deviations are found; the difference between these two values is the range of that sub-interval. Simultaneously, the standard deviation of the reward values within each sub-interval is calculated. The rescaled range value of that sub-interval is obtained by dividing the range by the standard deviation. The arithmetic mean of the rescaled range values for all sub-intervals is calculated to obtain the average rescaled range corresponding to that time window length. Then, the time window length is changed (e.g., using different integers from 10 to 100), and the above process is repeated to obtain a series of time window lengths and their corresponding average rescaled ranges.
[0025] Using the logarithm of each time window as the x-axis and the logarithm of the corresponding average rescaled range as the y-axis, a scatter plot is formed in a two-dimensional coordinate system. The two points with the smallest and largest x-coordinates in this scatter plot are removed to obtain an intermediate scatter plot, thus eliminating the endpoint effect.
[0026] Perform least-truncated squares regression on the intermediate scatter set and calculate the slope of the regression line. This regression method employs an iterative reweighting approach: fit an initial line to the intermediate scatter set using ordinary least squares, and calculate the absolute value of the residual from each scatter point to that line. Remove the point with the largest absolute residual value, and perform ordinary least squares fitting again with the remaining scatter points. Repeat the process of removing the point with the largest absolute residual value and refitting until the slope obtained from three consecutive fittings does not change by more than 2%. Use the slope of the final fitted line as the Hearst exponent for this analysis unit.
[0027] Repeat the above calculation for each analysis unit to obtain multiple Hearst exponents. Take the median of the Hearst exponents of all analysis units as the final Hearst exponent of the original observed reward sequence.
[0028] The state-space parameters of the stochastic differential equation are set based on the final Hearst exponent. This stochastic differential equation describes the mean regression process of the implicit true reward, which changes continuously over time. Specifically, the derivative of the implicit true reward equals the recovery rate multiplied by the difference between the long-term mean and the current implicit true reward, multiplied by the time derivative, plus the diffusion coefficient multiplied by the Brownian motion derivative. The recovery rate controls the strength of the mean regression. When the final Hearst exponent is greater than 0.6, it indicates that the reward sequence has long-term memory and a trend; in this case, the recovery rate is set to a high value, 0.8. Conversely, when the final Hearst exponent is less than or equal to 0.6, the recovery rate is set to a low value, 0.2. The initial value of the long-term mean is set to 0, and the initial value of the diffusion coefficient is set to 1. Based on these settings, substituting the estimated implicit true reward before the current time into the mean regression equation yields the prior distribution of the implicit true reward at the current time. This prior distribution follows a normal distribution, with its mean being the regression value determined by the recovery rate and the long-term mean, and its variance being the square of the diffusion coefficient multiplied by the time derivative.
[0029] In S3, the prior distribution is updated using particle filtering, and the effective diversity index of the particle weights is calculated after each update. The resampling trigger threshold is dynamically adjusted based on the effective diversity index. The posterior distribution of the implicit true reward is recursively calculated using the original observed reward at the current moment. The posterior mean and posterior variance are extracted from the posterior distribution, specifically including: Initialize the particle filter. Set the total number of particles to 200, with each particle carrying an estimate of the implicit true reward at the current time step. At the first time step, all particle estimates are randomly sampled from the prior distribution, and each particle has an equal initial weight of 1 / 200.
[0030] For each subsequent time step, the following prediction and update operations are performed. The state of each particle is predicted according to the mean regression process described by the stochastic differential equation. The prediction calculation process is as follows: the predicted state of the current particle equals the state of the particle at the previous time step plus the recovery rate multiplied by the difference between the long-term mean and the state at the previous time step, plus the diffusion coefficient multiplied by the square root of the time step, and then multiplied by a standard normal random number. The recovery rate, long-term mean, and diffusion coefficient are all taken from the parameter values updated in step S2 or S5. The time step is 0.1 seconds, and the standard normal random number is generated by the Mason rotation algorithm built into the edge nodes. After all particles have been predicted, a set of predicted states is obtained, constituting the particle representation of the prior distribution.
[0031] The particle weights are updated using the original observation reward obtained at the current time step. For each particle, the likelihood between its predicted state and the original observation reward is calculated. The likelihood is defined as the value of the normal distribution density function with the predicted state as the mean and the observation noise standard deviation as 0.5 at the original observation reward. This likelihood is multiplied by the particle's weight at the previous time step to obtain the unnormalized new weight. The unnormalized new weights of all particles are summed, and then the unnormalized new weight of each particle is divided by the sum to obtain the normalized weight. At this point, each particle obtains a weight value between 0 and 1, with a total sum of 1.
[0032] After each weight update, calculate the normalized entropy value of the current particle set. Let the total number of particles be... , No. The normalized weights of the particles are Then the normalized entropy value The calculation expression is: ; in, Represents the natural logarithm. The value is set to 200. The calculation process is as follows: for each particle, assign its weight... Taking the natural logarithm yields Then multiply by , to obtain the product Summing up such products of all particles; dividing the sum by Finally, take the negative value to obtain the normalized entropy value. The normalized entropy value ranges from 0 to 1. H reaches its maximum value of 1 when all weights are equal, and its maximum value of 1 when a certain weight is 1 and the rest are 0. Take the minimum value of 0.
[0033] The natural index of the above normalized entropy value is used as the effective diversity index. Effective diversity index The calculation formula is: Where e is the natural constant, with a value of 2.71828. In the calculation, the normalized entropy value H is used as the exponent to calculate e. The power of this power yields the effective diversity index D. Because Since it is between 0 and 1, D is between 1 and e, that is, between 1 and 2.71828.
[0034] The resampling trigger threshold is dynamically adjusted based on the effective diversity index. A pre-set base threshold of 0.5 is used. The effective diversity index D is multiplied by the base threshold of 0.5 to obtain the dynamic trigger threshold T.
[0035] Calculate the effective number of particles in the current particle set. The effective number of particles is calculated as follows: first, square the weight of each particle, then sum all the squared values, and finally take the reciprocal of the sum. For example, if all weights are 1 / 200, then the sum of squares is 200 multiplied by the square of (0.005) equals 0.005, and its reciprocal is 200, indicating that the effective number of particles equals the total number of particles; if a particle has a weight of 1 while the rest have a weight of 0, then the sum of squares is 1, and its reciprocal is 1, indicating that the effective number of particles is only 1.
[0036] The calculated number of effective particles is compared with the dynamic trigger threshold T. If the number of effective particles is less than T, polynomial resampling is performed. The specific steps of polynomial resampling are as follows: generate a random number that follows a uniform distribution between 0 and 1; determine which particle to copy according to the cumulative distribution function of the particle weights; repeat this process 200 times to obtain 200 new particles, and reset the weight of each new particle to 1 / 200. If the number of effective particles is greater than or equal to T, the current particle set remains unchanged, and resampling is not performed.
[0037] Using the resampled or unchanged particle set, combined with the original observed reward at the current moment, the posterior distribution of the implicit true reward is recursively calculated. The posterior distribution is characterized by the states and weights of all particles in the particle set. The posterior mean and posterior variance are extracted from this posterior distribution: the posterior mean is the sum of the products of all particle states and their corresponding weights; the posterior variance is the sum of the squares of the differences between each particle state and the posterior mean multiplied by its weight. Since the weights are equal after resampling, the posterior mean is the arithmetic mean of all particle states, and the posterior variance is the sum of the squares of the differences between each particle state and the mean divided by the total number of particles.
[0038] The calculated posterior mean and posterior variance are output. The entire particle filter update process is repeated once in each scheduling cycle, thereby achieving real-time online estimation of the implicit true reward.
[0039] In S4, the posterior mean replaces the original observed reward as the training objective for updating the deep reinforcement learning policy. Simultaneously, an exploration moderating factor is jointly calculated based on the posterior variance and the effective diversity index to guide the scheduling policy to adaptively adjust its exploration and utilization tendencies under different levels of uncertainty. Specifically, this includes: The obtained posterior variance and the effective diversity index are jointly calculated to obtain a joint uncertainty measure. Specifically, the posterior variance and the effective diversity index are multiplied to obtain a product; then, the square root of this product is taken, and the square root value is used as the joint uncertainty measure. The posterior variance reflects the dispersion of the current implicit true reward estimate, while the effective diversity index reflects the uniformity of the particle set weights. Multiplying the two and taking the square root comprehensively characterizes the uncertainty level of the current scheduling decision-making environment.
[0040] The joint uncertainty metric is compared with a preset saturation threshold, which is set to 0.95. The comparison rule is to take the smaller value between the joint uncertainty metric and the saturation threshold as the input for the subsequent nonlinear transformation. This is to prevent the nonlinear transformation output from becoming oversaturated and losing its discriminative power due to excessively large input values. The smaller value is then input into the hyperbolic tangent function for calculation. The mathematical definition of the hyperbolic tangent function is to map the input value to the interval between -1 and +1. However, since the input value in this step is always non-negative and does not exceed 0.95, the output of the hyperbolic tangent will fall between 0 and 0.74. To adjust the output range to the interval between 0 and 1, the result of the hyperbolic tangent calculation is further increased by +1 and then divided by 2. After the above processing, an adjustment base value is obtained, which ranges from 0 to 1.
[0041] Specifically, a maximum exploration probability is preset, with an initial value of 0.8. The maximum exploration probability is multiplied by the adjustment base value to obtain the current exploration adjustment factor. The product directly serves as the upper limit of the probability of random exploration when the scheduling policy executes an action. That is, before each scheduling decision, a uniformly distributed random number between 0 and 1 is generated. If this random number is less than the exploration adjustment factor, an action is randomly selected for execution; otherwise, the optimal action output by the current deep reinforcement learning policy network is selected.
[0042] Based on the above exploration of the adjustment factor, the update process of the deep reinforcement learning strategy is controlled. The deep reinforcement learning strategy adopts the standard temporal difference learning method. After each scheduling action is executed, the posterior mean calculated in step S3 is directly used as the immediate reward term in the temporal difference error of that scheduling, replacing the original observation reward. The calculation process of the temporal difference error is as follows: the immediate reward term is added to the discount factor (with a value of 0.9), multiplied by the maximum action value of the next state, and then the action value of the current state is subtracted. This temporal difference error is used to update the parameters of the deep reinforcement learning strategy network. The entire update process is executed in real time on the edge nodes, and the above calculation and update steps are repeated once in each scheduling cycle, so that the scheduling strategy can adaptively adjust the exploration and exploitation tendencies according to the dynamic changes of the joint uncertainty metric.
[0043] In S5, based on the accumulated historical reward trajectory, the recovery rate, long-term mean, and diffusion coefficient in the stochastic differential equation are periodically re-estimated, and the updated parameters are fed back to S2 to ensure that the continuous-time dynamic model implying the true reward continuously adapts to changes in the monitoring scenario. Specifically, this includes: A re-estimation period of 200 scheduling steps is defined. Parameter re-estimation is performed every 200 scheduling steps. The implicit true reward posterior mean sequence for the most recent re-estimation period is extracted from the historical reward trajectory. This sequence, composed of 200 values, is formed by arranging the posterior means output in step S3 for each scheduling period in chronological order. This sequence is used as the sample to be fitted for subsequent parameter estimation.
[0044] Maximum likelihood estimation is performed on the samples to be fitted, iteratively solving for the optimal estimates of the recovery rate, long-term mean, and diffusion coefficient. The recovery rate, long-term mean, and diffusion coefficient obtained at the end of the previous reestimation cycle are used as the initial values for the current iteration. In the first reestimation (i.e., the initial cycle), the recovery rate is initialized to 0.5, the long-term mean to 0, and the diffusion coefficient to 1.
[0045] The first step in the iterative solution is to fix the current diffusion coefficient and perform a linear fit on the residuals constructed from the sample to be fitted, the current recovery rate, and the current long-term mean. Specifically, for each posterior mean in the sample to be fitted, a linear relationship can be established between the current posterior mean and the previous posterior mean based on the discretized form of the stochastic differential equation. The slope term in this linear relationship is related to the recovery rate, and the intercept term is related to the product of the recovery rate and the long-term mean. Ordinary least squares is used to perform linear regression on all adjacent time points in the sample to be fitted, calculating the updated slope and intercept. Then, the updated recovery rate and long-term mean are derived from the slope and intercept.
[0046] The second step of the iterative solution involves fixing the updated recovery rate and long-term mean from the previous step, and updating the diffusion coefficient based on the moment estimation condition of the sum of squared residuals. Specifically, using the updated recovery rate and long-term mean, the residual between the predicted and actual values of each posterior mean in the sample to be fitted is calculated. Each residual is squared, summed, and then divided by the number of sample points minus one to obtain the residual variance. This residual variance is used as an approximate estimate of the diffusion coefficient by multiplying the square of the diffusion coefficient by the time step, thus calculating the updated diffusion coefficient.
[0047] Repeat steps one and two above. After each iteration, calculate the relative changes in the recovery rate, long-term mean, and diffusion coefficient obtained in the current iteration compared to the corresponding parameters in the previous iteration. The relative changes are calculated by dividing the absolute value of the difference between the two values by the value from the previous iteration. When the relative changes of all three parameters are lower than a preset convergence threshold (which is 2%), stop the iteration and output the recovery rate, long-term mean, and diffusion coefficient obtained in the current iteration as the optimal estimates.
[0048] The diffusion coefficient is smoothed. An exponentially weighted moving average is calculated between the diffusion coefficient estimated in the previous re-estimation period and the diffusion coefficient estimated in the current period to obtain the smoothed diffusion coefficient. Specifically, the smoothing factor is set to 0.8, and the smoothed diffusion coefficient equals the smoothing factor multiplied by the diffusion coefficient of the previous period, plus one minus the smoothing factor multiplied by the diffusion coefficient of the current period. The recovery rate and long-term mean are not smoothed; the optimal estimate from the current period is used directly.
[0049] The updated response rate, long-term mean, and smoothed diffusion coefficient are fed back into the S2 state space to replace the original parameters. In subsequent time steps, when calculating the prior distribution, step S2 uses the updated response rate, long-term mean, and diffusion coefficient, thus enabling the implicit true reward continuous-time dynamic model to continuously adapt to changes in the monitoring scenario. The entire re-estimation process is automatically triggered every 200 scheduling steps at the edge nodes, without relying on external intervention.
[0050] Please see Figure 2 As shown, a multi-task optimization scheduling system based on edge computing includes: The reward sequence acquisition module acquires the original observation reward sequence generated after the edge node performs multi-task scheduling. The original observation reward contains sparse scheduling feedback signals with colored noise. The state space construction module calculates the Hearst exponent of the original observed reward sequence and sets the state space of the stochastic differential equation based on the Hearst exponent. The stochastic differential equation is used to describe the mean regression process of the implicit true reward that changes continuously over time and outputs the prior distribution of the implicit true reward at the current time. The particle filtering and posterior estimation module updates the prior distribution using particle filtering and calculates the effective diversity index of particle weights after each update. It dynamically adjusts the resampling trigger threshold based on the effective diversity index and recursively calculates the posterior distribution of the implicit true reward using the original observation reward at the current moment. It then extracts the posterior mean and posterior variance from the posterior distribution. The strategy training and exploration regulation module replaces the original observation reward with the posterior mean as the training target for updating the deep reinforcement learning strategy. At the same time, it calculates the exploration regulation factor based on the posterior variance and the effective diversity index to guide the scheduling strategy to adaptively adjust the tendency of exploration and utilization under different degrees of uncertainty. The adaptive update module for model parameters periodically re-estimates the recovery rate, long-term mean, and diffusion coefficient in the stochastic differential equation based on the accumulated historical reward trajectory, and feeds the updated parameters back to S2 so that the continuous-time dynamic model implying the real reward can continuously adapt to changes in the monitoring scenario.
[0051] The working principle of this invention is as follows: A sparse and colored noise-laden original observation reward sequence is obtained after edge nodes perform multi-task scheduling. The Hearst exponent of this sequence is calculated, and the state space of the stochastic differential equation is set according to this exponent to describe the mean regression process of the implicit true reward and output the prior distribution at the current time. This prior distribution is updated using particle filtering. After each update, the effective diversity index of the particle weights is calculated, and the resampling trigger threshold is dynamically adjusted accordingly. The posterior distribution of the implicit true reward is recursively calculated using the current original observation reward, and the posterior mean and posterior variance are extracted from it. The posterior mean is used to replace the original observation reward as the training target for updating the deep reinforcement learning strategy. Simultaneously, an exploration adjustment factor is jointly calculated based on the posterior variance and the effective diversity index to adaptively adjust the exploration and utilization tendency of the scheduling strategy. Based on the accumulated historical reward trajectory, the recovery rate, long-term mean, and diffusion coefficient in the stochastic differential equation are periodically re-estimated, and the updated parameters are fed back to the state space, enabling the continuous-time dynamic model of the implicit true reward to continuously adapt to changes in the monitoring scenario.
[0052] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.
Claims
1. A multi-task optimization scheduling method based on edge computing, characterized in that, Includes the following steps: S1, obtain the original observation reward sequence generated after the edge node performs multi-task scheduling. The original observation reward contains sparse scheduling feedback signals with colored noise. S2, calculate the Hearst exponent of the original observed reward sequence, and set the state space of the stochastic differential equation according to the Hearst exponent. The stochastic differential equation is used to describe the mean regression process of the implicit true reward changing continuously over time, and outputs the prior distribution of the implicit true reward at the current time. S3 updates the prior distribution using particle filtering and calculates the effective diversity index of particle weights after each update. The resampling trigger threshold is dynamically adjusted based on the effective diversity index. The posterior distribution of the implicit true reward is recursively calculated using the original observation reward at the current moment. The posterior mean and posterior variance are extracted from the posterior distribution. S4 uses the posterior mean to replace the original observation reward as the training target for updating the deep reinforcement learning strategy. At the same time, it calculates the exploration adjustment factor based on the posterior variance and the effective diversity index to guide the scheduling strategy to adaptively adjust the tendency of exploration and utilization under different degrees of uncertainty. S5, based on the accumulated historical reward trajectory, periodically re-estimates the recovery rate, long-term mean, and diffusion coefficient in the stochastic differential equation, and feeds the updated parameters back to S2, so that the continuous-time dynamic model implying the real reward can continuously adapt to changes in the monitoring scenario.
2. The multi-task optimization scheduling method based on edge computing according to claim 1, characterized in that, The prior distribution specifically includes: The original observation reward sequence is divided into fixed time windows, and the sequence within each time window is used as an analysis unit; Rescaled range transformation is performed on each analysis unit, cumulative deviation and range are calculated, and the Hearst exponent of the analysis unit is obtained by linear regression fitting in logarithmic coordinate system. The median of the Hearst exponents of all analysis units is taken as the final Hearst exponent. When the final Hearst exponent is greater than 0.6, the recovery rate in the stochastic differential equation is set to a high value to enhance mean regression; otherwise, it is set to a low value, and the prior distribution of the implicit true reward is output according to the setting.
3. The multi-task optimization scheduling method based on edge computing according to claim 2, characterized in that, The method of obtaining the Hearst exponent of the analysis unit through linear regression fitting in a logarithmic coordinate system specifically includes: The logarithm of each time window within the same analysis unit is used as the x-axis, and the logarithm of the corresponding rescaled range is used as the y-axis to form a scatter set. Remove the two points corresponding to the minimum and maximum x-coordinates in the scattered point set to obtain the intermediate scattered point set; Perform least-truncation square regression on the intermediate scatter set, calculate the slope of the regression line. The least-truncation square regression adopts an iterative reweighting method, and in each iteration, the point with the largest absolute value of the residual is removed and then refitted. The slope of the final regression line after iterative convergence is used as the Hearst exponent of the analysis unit.
4. The multi-task optimization scheduling method based on edge computing according to claim 1, characterized in that, S3 specifically includes: The normalized entropy value of the current particle set is calculated based on the weight of each particle, and the natural index of the normalized entropy value is used as the effective diversity index. The product of the effective diversity index and the preset base threshold is used as the dynamic trigger threshold for resampling. Calculate the number of effective particles in the current particle set. If the number of effective particles is lower than the dynamic trigger threshold, perform polynomial resampling to update the particle set; otherwise, keep the current particle set unchanged. Using the resampled or unchanged set of particles, combined with the original observed reward at the current moment, the posterior distribution of the implicit true reward is recursively calculated, and the posterior mean and posterior variance are extracted from the posterior distribution.
5. The multi-task optimization scheduling method based on edge computing according to claim 4, characterized in that, The process of calculating the normalized entropy value of the current particle set based on the weights of each particle, and using the natural index of the normalized entropy value as the effective diversity index, specifically includes: Obtain the weights of each particle in the current particle set, and take the natural logarithm of each weight to obtain the logarithmic weight sequence; Calculate the arithmetic mean of the log-weighted sequence, and subtract the arithmetic mean from each log-weight to obtain the centered log-weighted sequence; The normalized entropy value is obtained by multiplying each value in the centralized logarithmic weight sequence by its corresponding original weight, summing the results, and then dividing by the natural logarithm of the total number of particles. Perform a natural index operation on the normalized entropy value and output the result as an effective diversity index.
6. The multi-task optimization scheduling method based on edge computing according to claim 1, characterized in that, S4 specifically includes: The joint uncertainty measure is obtained by multiplying the posterior variance by the effective diversity index and then taking the square root. The joint uncertainty metric is compared with a preset saturation threshold, and the smaller of the two values is taken as the input of the nonlinear transformation. The hyperbolic tangent operation is then performed on the input to obtain an adjustment base value between 0 and 1. Multiply the maximum exploration probability by the adjustment base value to obtain the current exploration adjustment factor, and use the exploration adjustment factor as the upper limit of the probability of random exploration when the scheduling strategy executes the action; Based on the exploration of the adjustment factor to control the update process of the deep reinforcement learning strategy, the posterior mean is directly used as the immediate benefit term in the temporal difference error.
7. The multi-task optimization scheduling method based on edge computing according to claim 6, characterized in that, The process of multiplying the maximum exploration probability by the adjustment base value to obtain the current exploration adjustment factor, and using the exploration adjustment factor as the upper limit of the probability of random exploration when the scheduling strategy executes an action, specifically includes: Obtain the sequence number of the current scheduling cycle and calculate the ratio of the sequence number to the preset total cycle to obtain the cycle progress ratio; The progress decay factor is obtained by squaring the periodic progress ratio using a sine function. Multiply the preset maximum exploration probability by the progress decay factor to obtain the periodically adaptive maximum exploration probability; Multiply the maximum exploration probability of the periodic adaptive method by the adjustment base value to output the current exploration adjustment factor.
8. The multi-task optimization scheduling method based on edge computing according to claim 1, characterized in that, S5 specifically includes: Extract the posterior mean sequence of the latent true reward within the most recent re-estimation period from the historical reward trajectory, and use the posterior mean sequence as the sample to be fitted. Perform maximum likelihood estimation on the fitted sample and iteratively solve for the optimal estimates of the recovery rate, long-term mean, and diffusion coefficient. The diffusion coefficient estimated in the previous period and the diffusion coefficient estimated in the current period are subjected to an exponentially weighted moving average to obtain a smoothed diffusion coefficient. The smoothed diffusion coefficient, the recovery rate estimated in the current period, and the long-term mean are then used to replace the original parameters. The updated response rate, long-term mean, and diffusion coefficient are fed back into the state space of S2 for use in calculating the prior distribution at subsequent time steps.
9. The multi-task optimization scheduling method based on edge computing according to claim 8, characterized in that, The process of performing maximum likelihood estimation on the samples to be fitted, and iteratively solving for the optimal estimates of the recovery rate, long-term mean, and diffusion coefficient, specifically includes: Use the recovery rate, long-term mean, and diffusion coefficient estimated in the previous cycle as the initial values for the current iteration; With the current diffusion coefficient fixed, perform linear fitting on the residuals constructed from the sample to be fitted and the current recovery rate and long-term mean, and update the recovery rate and long-term mean. The diffusion coefficient is updated based on the moment estimation conditions of the residual sum of squares, with the updated recovery rate and long-term mean fixed. The current iteration value is output as the optimal estimate when the relative changes in the recovery rate, long-term mean, and diffusion coefficient obtained in two consecutive iterations are all lower than the preset convergence threshold.
10. A multi-task optimization scheduling system based on edge computing, characterized in that, A method for executing a multi-task optimization scheduling method based on edge computing as described in any one of claims 1-9, comprising: The reward sequence acquisition module acquires the original observation reward sequence generated after the edge node performs multi-task scheduling. The original observation reward contains sparse scheduling feedback signals with colored noise. The state space construction module calculates the Hearst exponent of the original observed reward sequence and sets the state space of the stochastic differential equation based on the Hearst exponent. The stochastic differential equation is used to describe the mean regression process of the implicit true reward that changes continuously over time and outputs the prior distribution of the implicit true reward at the current time. The particle filtering and posterior estimation module updates the prior distribution using particle filtering and calculates the effective diversity index of particle weights after each update. It dynamically adjusts the resampling trigger threshold based on the effective diversity index and recursively calculates the posterior distribution of the implicit true reward using the original observation reward at the current moment. It then extracts the posterior mean and posterior variance from the posterior distribution. The strategy training and exploration regulation module replaces the original observation reward with the posterior mean as the training target for updating the deep reinforcement learning strategy. At the same time, it calculates the exploration regulation factor based on the posterior variance and the effective diversity index to guide the scheduling strategy to adaptively adjust the tendency of exploration and utilization under different degrees of uncertainty. The adaptive update module for model parameters periodically re-estimates the recovery rate, long-term mean, and diffusion coefficient in the stochastic differential equation based on the accumulated historical reward trajectory, and feeds the updated parameters back to S2 so that the continuous-time dynamic model implying the real reward can continuously adapt to changes in the monitoring scenario.