Industrial operating system resource instance scheduling optimization method

Abstract

The invention discloses an industrial operating system resource instance scheduling optimization method, which comprises the following steps of: firstly, learning experience related to exploration and development in a WOA search process from historical data in combination with a DRL algorithm, and training an optimal decision model; and then, by utilizing the trained model, the search agent can realize the optimal action selection in each iteration, so that the search agent can be effectively guided to realize better balance between global exploration and local development, the convergence speed and the solving quality of the search agent are improved, and finally, the optimal solution of the RIS problem is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of industrial operating systems, and specifically relates to an industrial operating system resource instantiation scheduling optimization algorithm that integrates deep reinforcement learning (DRL) and a whale optimization algorithm (WOA). Background Art

[0002] The advantage of intelligent manufacturing is that it can meet personalized and differentiated market demands more efficiently and flexibly. To realize intelligent manufacturing, it is necessary to uniformly and efficiently manage heterogeneous resources in industrial production. In intelligent manufacturing, the realization of resource services depends on the abstraction and pooling of industrial production resources to achieve maximum resource reuse. Industrial operating systems play an important role in this process, responsible for key tasks such as management, scheduling and utilization of heterogeneous production resources. The effective management and scheduling of heterogeneous resources directly affects the overall performance of intelligent manufacturing systems, and at the same time poses challenges to the implementation of industrial operating systems.

[0003] In traditional manufacturing, the software control of heterogeneous resources such as sensors, robots, and conveyor belts is tightly coupled with physical hardware, which has disadvantages such as specialization and difficulty in scalability. This makes resource management and scheduling more complex and difficult, and it takes a lot of time and resources to reconfigure, deploy, and optimize various resources to adapt to large-variety and small-batch manufacturing. In order to optimize resource management and improve flexible and efficient resource utilization capabilities, the application of software-defined paradigm has become the mainstream choice for current industrial operating systems. In the software-defined paradigm, semantic description methods such as ontology modeling technology are used to virtualize, abstract, and pool physical hardware devices to achieve interconnection and interoperability of heterogeneous resources in physical space and cyberspace. This approach enables various resources to be subdivided and recombined on demand to meet diverse needs. In addition, the control and management of production resources are performed by a logically centralized controller, and the control and management functions are separated from the hardware devices and run in the form of software. Therefore, software-defined industrial operating systems can easily use software control modes to achieve flexible resource sharing, scheduling, and reuse. In this case, industrial application software can focus on the implementation of business logic without worrying about the details of the underlying physical devices. Therefore, when executing industrial application software, the operating system needs to perform a reverse mapping from network space to physical space through resource instantiation scheduling (RIS) based on the specific resources and service quality (QoS) involved in business requirements. This approach enables the operating system to dynamically allocate and schedule resources to meet the needs of different applications and provide the required quality of service.

[0004] In the production process, a large number of available heterogeneous production resources are involved. However, the service quality constraints of these resources have become a huge challenge for optimizing RIS. In industrial applications, a well-orchestrated production workflow contains a series of production tasks, each of which usually relies on the operating system to schedule and instantiate specific production resources for execution. These resources are classified according to their production capabilities and have various QoS attributes, including time, cost, reliability, and availability. In industrial manufacturing systems, given that production tasks are often accompanied by deadlines or periodic time constraints, real-time execution capabilities are particularly critical and important. If production tasks are executed overtime, their due value will be reduced or lost, which will affect production quality and efficiency. Therefore, resource instantiation scheduling not only needs to meet specific production functions, but also needs to emphasize QoS requirements and put timeliness first. In addition, if production equipment or services fail or performance degrades, resources need to be dynamically and quickly rescheduled during the production process, which can improve user experience and service quality.

[0005] In industrial operating systems, RIS, as a basic service, plays an important role in supporting the execution of industrial applications. The frequent invocation of RIS services and the need for high QoS standards and strict timeliness requirements highlight its importance in industrial environments, which has a huge impact on the overall efficiency, performance and cost-effectiveness of manufacturing systems. Implementing efficient resource instantiation scheduling has direct benefits for optimizing the utilization of available resources, minimizing idle time and improving production performance. Through reasonable resource scheduling, it can ensure that industrial applications are executed at the right time and on the right resources, thereby improving overall production efficiency. In addition, through effective resource utilization, energy consumption, equipment wear and maintenance costs can be reduced, further improving the cost-effectiveness of manufacturing systems. However, efficient methods for implementing RIS face complex optimization problems. Due to the complexity and diversity of industrial environments, resource instantiation scheduling needs to take into account multiple factors, such as task priority, resource constraints, communication delays, etc. How to implement RIS efficiently and reasonably under these complex constraints has become an urgent problem to be solved. Summary of the invention

[0006] In view of the shortcomings of the background technology, the present invention proposes an industrial operating system resource instantiation scheduling optimization method that integrates deep reinforcement learning (DRL) and whale optimization algorithm (WOA). Firstly, the DRL algorithm is combined to learn the experience of exploration and development in the WOA search process from historical data, and an optimal decision model is trained; then, the trained model is used to effectively guide the search agent to achieve a better balance between global exploration and local development, improve its convergence speed and solution quality, and finally obtain the optimal solution to the RIS problem.

[0007] The present invention provides an industrial operating system resource instance scheduling optimization method (DWOA), and the specific steps are as follows:

[0008] (1) Constructing a WOA search agent decision model based on DRL: The DRL agent is directly involved in the WOA search process as a single search agent in WOA, and all WOA search agents share a DRL agent, which is conducive to the efficient learning of the decision model; the DRL agent learns the decision model of the WOA search agent action through iterative interaction with the WOA environment. In each interactive learning, the DRL agent selects an action from the action space to execute. The reward of the selected action is obtained by quantifying the evolutionary quality of the obtained candidate solution. If the newly obtained candidate solution is improved over the original solution, it will receive a positive reward. If the newly obtained candidate solution is the original solution, no reward will be given. Otherwise, the WOA environment will give a penalty. When training the decision model, it is necessary to provide the DRL agent with sufficient information to support the action decision in each iteration, and to mine the final decision model under a given training budget. It is worth noting that once the decision model is trained, its relevant parameters will be fixed, and the offline training of the decision model only needs to be performed once. The decision model provides support for the search agent to select the most appropriate action in DWOA.

[0009] (2) Online solution optimization: DWOA optimizes the solution in a similar framework to WOA. However, unlike WOA, the original WOA uses random parameters p and parameters A in the state space to select actions. DWOA replaces this with first constructing a state space related to the WOA environment, and then relying on the trained decision model to determine the actions of the search agent in each iteration, iteratively optimizing the RIS problem, and finally obtaining the optimal solution. Since the decision model is an optimized WOA search mechanism, the performance of the DWOA algorithm will be significantly improved.

[0010] Preferably, the design of the action space in step (1) is as follows:

[0011] Action = {SE, SU, RS}

[0012] Among them, SE is the shrinking and enclosing formula in the original WOA, SU is the spiral update formula in the original WOA, and RS is the random search formula in the original WOA. The original WOA determines whether the search agent uses spiral motion or circular motion according to the value of the random parameter p. The decision model of the present invention is replaced by using the DRL agent to decide the action in the action space.

[0013] Preferably, the reward in step (1) is designed as follows:

[0014]

[0015] in, represents the value of the current solution (i.e., state) at the tth iteration, It is the fitness value of the new solution obtained after executing the update operation (i.e., the current state is converted to the next new state). f is the fitness function. The goal of the iterative optimization of the RIS problem in the present invention is to find the minimum value, that is, the smaller the fitness value, the better the solution. Therefore, when the update operation selected by the DRL agent causes the fitness value to decrease, it should be given a positive reward, that is, the reward value is set to 1; if the update operation increases the fitness value, it should be given a negative reward, that is, the reward value is set to -1; if the update operation makes the fitness value remain unchanged, the reward value is set to 0.

[0016] Preferably, the training of the decision model is based on the perspective of a single DRL agent, and the deep neural network is trained in combination with Dueling DQN to approximate the optimal strategy; wherein the historical actions selected by the DRL agent in the iterative process of optimizing the training function are used as training data, the training function comes from the classic benchmark function (also called the benchmark test function), the training function is collected into a queue, and at each stage of the training, a training function is selected from the queue as the training target; each round of training is cyclically selected from the training function queue in a round-robin mode, the DRL agent obtains the observation information of the WOA environment according to the state space designed in step (2), that is, the current state, and then selects an action from the action space in step (1) and executes it according to the training experience (the experience in the first training round is 0, and as the number of training rounds increases, the DRL agent has richer experience), and then according to the WOA environment design in step (1), the current state is converted to the next new state, and for the new state, the WOA environment obtains the reward feedback of the DRL agent for this action according to the reward design in step (1). Repeat the training, through continuous trial and error and interaction, and eventually obtain a decision-making model.

[0017] Preferably, the design of the state space in step (2) is as follows:

[0018] State={NDIV,∣A∣ / 2,∣C∣ / 2}

[0019] Among them, NDIV is the normalized population diversity index, A and C are two parameters in WOA, and their initial value ranges are [-2, 2] and [0, 1] respectively. After normalization, the value ranges of these three features are converted to [0, 1].

[0020] More preferably, the normalized population diversity index NDIV is designed as follows:

[0021]

[0022] in, represents the value of search agent i in the current population on dimension j, median(x j ) is the median value of all search agents in the current population on the j dimension, n represents the total number of search agents in the current population, Div j It represents the mean distance between the value of each search agent in the j dimension and the median of all search agents in the j dimension, which is used to measure the diversity of the population in terms of optimization variables. Div is the Div of the candidate solution in each dimension j The average value of , m represents the number of variables contained in the candidate solution of the optimization problem, that is, the total dimension. It is worth noting that the population diversity needs to be calculated separately in each iteration. ub and lb represent the upper and lower bounds of Div, respectively.

[0023] Preferably, the step (2) is specifically as follows: first, the decision model trained in step (1) is imported, and the relevant parameters are initialized, wherein the relevant parameters include the population size, the problem scale dimension, the maximum number of iteration steps, the hyperparameters in the DRL agent, and the hyperparameters in the WOA; then, according to the set maximum number of iterations, the optimal solution is found in a framework similar to WOA. Unlike WOA, DWOA constructs a state space related to the WOA environment, and then uses the decision model to guide the search agent to select the optimal action from the action space in step (1) to iteratively optimize the RIS problem to update the solution; finally, when the iterative search terminates, the optimal solution is obtained.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The present invention combines deep reinforcement learning technology to improve the exploration and development mechanism of the WOA algorithm, thereby proposing an industrial operating system resource instance scheduling optimization method (DWOA), which more effectively solves the RIS problem. Specifically, the present invention can automatically utilize a large amount of data generated during the optimization process of the WOA algorithm, including solution update operations, solution status, target values, etc., and use DRL technology to extract hidden knowledge or experience, and derive a decision model from it; through the optimal decision model obtained through training, the search agent can achieve the best action selection in each iteration, thereby improving the performance of the algorithm, preventing the algorithm from falling into local optimality too early, and accelerating the convergence speed. Therefore, thanks to the learning and decision-making capabilities of the DRL technology, based on the decision model, WOA can accurately guide the search agent to take appropriate actions in each iteration, achieving a smarter and more reasonable balance of exploration and development, so that the present invention can efficiently solve the RIS problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the overall framework of DWOA of the present invention.

[0027] Figure 2 (a), (b), (c) and (d) are respectively schematic diagrams of the comparison of convergence curves when the present invention uses DWOA, genetic algorithm and whale optimization algorithm to solve the RIS problem of network physical components with a number scale of 60, 90, 120 and 150. DETAILED DESCRIPTION

[0028] See also Figure 1 The present invention provides an industrial operating system resource instance scheduling optimization method (hereinafter referred to as DWOA) mainly composed of two parts: the first part is to construct a WOA search agent decision model based on DRL; the DRL agent is directly involved in the WOA search process as a single search agent in WOA, and all WOA search agents share a DRL agent, which is conducive to the efficient learning of the decision model; the DRL agent learns the decision model of the WOA search agent action through iterative interaction with the WOA environment. In each interactive learning, the DRL agent selects an action from SE, SU and RS in the action space to execute. The reward of the selected action is obtained by quantifying the evolutionary quality of the obtained candidate solution. If the newly obtained candidate solution (i.e., the new state) is improved over the original solution (i.e., the current state), a positive reward is obtained. If the newly obtained candidate solution is the original solution, no reward is given, otherwise the WOA environment gives a penalty (i.e., a negative reward). When training the decision model, it is necessary to provide the DRL agent with sufficient information to support the action decision in each iteration, and to mine the final decision model under a given training budget (or termination condition). It is worth noting that once the decision model is trained, its related parameters will be fixed, and the offline training of the decision model only needs to be performed once. The decision model provides support for the search agent to select the most appropriate action in DWOA. The second part is online solution optimization: DWOA optimizes the solution in a framework similar to WOA, but unlike WOA, the original WOA uses random parameters p and parameters A in the state space to select actions. DWOA replaces this with first constructing a state space related to the WOA environment, and then relying on the trained decision model to determine the actions of the search agent in each iteration, iteratively optimizing the RIS problem, and finally obtaining the optimal solution. Since the decision model is an optimized WOA search mechanism, the performance of the DWOA algorithm will be significantly improved.

[0029] Industrial applications are usually composed of a set of cyber-physical components (CPCs). When industrial applications are executed, the industrial operating system needs to perform RIS services, allocate manufacturing resources (MRs) to each CPC, and guarantee the required quality of service (QoS). Therefore, there are theoretically combinations of MRs, and finding an optimal combination from these reasonable alternatives is the goal of the RIS QoS optimization problem.

[0030] Table 1 shows the aggregate value of QoS calculated by the aggregation function, where mr j Represents the manufacturing resource j of the selected instantiated component (cyber-physical component).

[0031] Table 1 QoS aggregation function

[0032]

[0033] The six widely used evaluation indicators for measuring RIS from the perspective of QoS are: t ,Q c ,Q a ,Q r ,Q ec ,Q ft Specifically, q t represents the estimated execution time required for manufacturing resource j to complete the specified task, q c represents the cost of invoking the service of manufacturing resource j, including possible overhead, q a represents the probability of manufacturing resource j being accessed, q r represents the ability of manufacturing resource j to correctly perform its function, q ec represents the consumption of manufacturing resource j when executing the task, q ft represents the hardware and software fault tolerance of manufacturing resource j, Q t represents the total estimated execution time required for each manufacturing resource service of the selected instantiated component to complete the specified task, q represents the total number of manufacturing resources of the selected instantiated component, Q c represents the sum of the costs of invoking the services of each manufacturing resource of the selected instantiated component, including possible overhead, Q a represents the product of the access probabilities of each manufacturing resource of the selected instantiated component, Q r represents the product of the capabilities of each manufacturing resource of the selected instantiated component to correctly perform its function, Q ec represents the total consumption of each manufacturing resource of the selected instantiated component when executing the task, Q ft Represents the product of the hardware and software fault tolerance of each manufacturing resource of the selected instantiated component. Each individual QoS value involved in the aggregation function is normalized at initialization, and they all have the same value range [0,1]. Therefore, the QoS value of the selected instantiated component can be evaluated as follows:

[0034]

[0035] Among them, Agg i () is the normalized metric Q i ∈QS corresponding to the aggregation function, Agg i (Q i ) returns its aggregate value. i Indicates Q i The weight, w i ∈[0,1],∑w i =1. QS + ∪QS - =QS, QS + Indicates a positive indicator (i.e. Q a , Q r and Q ft ), its value should be as high as possible. On the contrary, QS - Indicates a negative indicator (i.e. Q t , Q c and Q ec ), its value should be as small as possible. + When Agg i The output value of () is usually very small. In order to facilitate calculation and comparison, it is logarithmically transformed and then normalized to the interval [0,1].

[0036] Real-time performance is one of the key requirements for industrial production. Therefore, RIS needs to consider both quality of service and time constraints to support the operating efficiency and deterministic latency of industrial operating systems. The optimization problem of RIS is stated as follows:

[0037] min f qos (MRC) = 1-f(MRC)

[0038]

[0039] Among them, f qos (MRC) is the QoS fitness function expression, T deadline Indicates the maximum time delay allowed for each manufacturing resource service of the selected instantiated component to complete the specified task.

[0040] In order to verify the effectiveness and feasibility of the proposed method, experiments were designed to comprehensively explore the performance of DWOA in solving RIS problems of different scales. In the experiment, the number of cyber-physical components (CPCs) was 60, 90, 120, and 150, and the number of manufacturing resources (MRs) corresponding to each CPC was 150. 30 groups of experiments were performed when the number of cyber-physical components (CPCs) was 60, 90, 120, and 150. For example, RIS-60-150 means that the number of CPCs is 60 and the number of candidate MRs for each CPC is 150. In the case of the present invention, it is assumed that each MR considers the above six popular QoS evaluation indicators. The values ​​of these indicators are randomly derived within a certain range, and each considered QoS normalized metric is a random value generated in the interval [0.6, 0.96]. The weights of these six QoS indicators are as follows: w t =0.3, w c =0.2, w ec =0.2, w a =0.1, w r = 0.1 and w ft = 0.1. In addition, the maximum time delay allowed for completing a specified task is T deadline is set to 0.8*q.

[0041] The DWOA of the present invention is compared with the whale optimization algorithm (WOA) and the genetic algorithm (GA) below, and the performance of the DWOA algorithm is measured from three aspects: solution accuracy, convergence curve and time efficiency.

[0042] Table 2 Comparison of solution accuracy of RIS problems of different scales

[0043]

[0044]

[0045] Table 2 records the experimental comparison results of DWOA and other two algorithms on the RIS optimization problem. It can be clearly seen from Table 2 that in terms of RIS fitness value (min f of network physical components), qos In terms of the average value of the (MRC) calculation value) (i.e., the average value of each RIS fitness value in 30 experiments with the same number of network physical components), DWOA is better than the other two algorithms, and the average improvement is about 3.1% when all the numbers of network physical components are compared. From the standard deviation (i.e., the standard deviation of each RIS fitness value in 30 experiments with the same number of network physical components), it can be seen that the DWOA results are also relatively stable. In addition, the optimal value in Table 2 is the minimum value of each RIS fitness value in the 30 experiments with the same number of network physical components.

[0046] Figure 2 The convergence curve of the algorithm when solving the RIS problem is shown. It can be seen that DWOA has always obtained the best average RIS fitness value in all four scenarios. In addition, DWOA has shown obvious advantages in all stages of iterative search, with faster convergence speed and higher convergence accuracy. It is worth noting that when the number of iterations is less than 200 steps, the advantage of DWOA is already very obvious. This experimental result shows that DWOA has the ability to quickly find the optimal manufacturing resource combination under a certain time limit.

[0047] Table 3 shows the time efficiency of DWOA and the comparison algorithm on RIS problems of different scales. The “-” in the table indicates that the termination fitness value is not reached within the corresponding time limit. It can be seen that for RIS problems of different scales, DWOA can successfully generate all valid solutions, and the required solution time is the least, which is 2.18 times higher than other algorithms in average time efficiency. Among them, in the RIS problems of different scales, the termination fitness value of the first row directly selects the optimal value of the RIS fitness value of the GA algorithm corresponding to the scale RIS problem in Table 2, the termination fitness value of the second row is based on the termination fitness value of the first row minus 0.01, and the termination fitness value of the third row is based on the termination fitness value of the second row minus 0.01, and the corresponding time limits of the termination fitness values ​​of each row are 300 seconds, 400 seconds and 500 seconds respectively.

[0048] Table 3 Comparison of time efficiency (time cost) of RIS problems

[0049]

[0050]

[0051] In summary, the present invention can effectively improve the ability to solve RIS problems in industrial operating systems and reduce time overhead. Compared with the widely used genetic algorithm and whale optimization algorithm, the present invention can learn the action update operation, solution status, etc. in the WOA optimization process through the historical experience reuse mechanism of DRL, and construct a reusable optimal decision model to help solve unseen optimization problems.

Claims

1. A method for optimizing resource instance scheduling of an industrial operating system, characterized in that: The specific steps are as follows: (1) Construct a WOA search agent decision model based on DRL: The DRL agent is directly involved in the WOA search process as a single search agent in WOA, and all WOA search agents share a DRL agent; the DRL agent learns the decision model of the WOA search agent action through iterative interaction with the WOA environment. In each interactive learning, the DRL agent selects an action from the action space to execute. The reward of the selected action is obtained by quantifying the evolutionary quality of the obtained candidate solution. If the newly obtained candidate solution is improved over the original solution, it will receive a positive reward. If the newly obtained candidate solution is the original solution, no reward will be given. Otherwise, the WOA environment will impose a penalty. When training the decision model, the final decision model is mined under a given training budget. (2) Online solution optimization: First, construct the state space related to the WOA environment, and then rely on the trained decision model to determine the actions of the search agent in each iteration, iteratively optimize the RIS problem, and finally obtain the optimal solution.

2. The method for optimizing resource instance scheduling of an industrial operating system according to claim 1, characterized in that: The design of the action space in step (1) is as follows: Action = {SE, SU, RS} Among them, SE is the shrinking and surrounding formula in the original WOA, SU is the spiral update formula in the original WOA, and RS is the random search formula in the original WOA; the decision model uses the DRL agent to decide and select actions in the action space.

3. The method for optimizing resource instance scheduling of an industrial operating system according to claim 1, characterized in that: The reward design in step (1) is as follows: in, represents the value of the current solution at the tth iteration, is the fitness value of the new solution obtained after the update operation; f is the fitness function.

4. The method for optimizing resource instance scheduling of an industrial operating system according to claim 1, characterized in that: The training of the decision model starts from the DRL agent and combines Dueling DQN to train the deep neural network to approximate the optimal strategy; wherein the historical actions selected by the DRL agent in the iterative process of optimizing the training function are used as training data, the training function comes from the benchmark function, the training function is collected into a queue, and at each stage of the training, a training function is selected from the queue as the training target; each round of training is cyclically selected from the training function queue according to the polling scheduling mode, the DRL agent obtains the current state of the WOA environment according to the state space designed in step (2), and then selects an action from the action space in step (1) according to the training experience and executes it, and then according to the WOA environment design in step (1), it switches from the current state to the next new state, and for the new state, the WOA environment obtains the reward feedback of the DRL agent for this action according to the reward design in step (1); the training is repeated, and the decision model is finally obtained through continuous trial and error and interaction.

5. The method for optimizing resource instance scheduling of an industrial operating system according to claim 1, characterized in that: The design of the state space in step (2) is as follows: State={NDIV, |A| / 2, |C| / 2} Among them, NDIV is the normalized population diversity index, A and C are two parameters in WOA, and both are normalized.

6. The method for optimizing resource instance scheduling of an industrial operating system according to claim 5, characterized in that: The design of the normalized population diversity index NDIV is as follows: in, represents the value of search agent i in the current population on dimension j, median(x j ) represents the median value of all search agents in the current population on the j dimension, n represents the total number of search agents in the current population, Div j represents the mean distance between the value of each search agent in the j dimension and the median of all search agents in the j dimension; Div is the Div of the candidate solution in each dimension j , m represents the total dimension; ub and lb represent the upper and lower bounds of Div respectively.

7. The method for optimizing resource instance scheduling of an industrial operating system according to claim 1, characterized in that: The step (2) is specifically as follows: first, the decision model trained in step (1) is imported, and the relevant parameters are initialized, wherein the relevant parameters include the population size, the problem scale dimension, the maximum number of iteration steps, the hyperparameters in the DRL agent, and the hyperparameters in the WOA; then, according to the set maximum number of iterations, a state space related to the WOA environment is constructed, and then the decision model is used to guide the search agent to select the optimal action from the action space in step (1) to iteratively optimize the RIS problem to update the solution; finally, when the iterative search is terminated, the optimal solution is obtained.

Images