Improved PPO-RND algorithm-based machine tool equipment resource adaptive optimization method in industrial interconnection environment

Abstract

A machine tool equipment resource adaptive optimization method in an industrial interconnection environment based on an improved PPO-RND algorithm comprises the following steps: 1) modeling machine tool equipment resources, and establishing a production disturbance model; 2) modeling a machine tool equipment resource optimization process as a six-tuple MR-MDP; 3) designing a reward function of the machine tool equipment resource optimization process MR-MDP; 4) constructing an improved PPO-RND algorithm framework; 5) solving the machine tool equipment resource optimization process MR-MDP by using an improved PPO-RND algorithm framework to obtain each machine tool equipment resource combination and corresponding action; and after production disturbance occurs, re-optimization is carried out on each machine tool equipment resource combination and corresponding actions. The method has excellent robustness, and provides certain theoretical method support for stable and reliable operation of industrial interconnection manufacturing service.

Description

Technical Field

[0001] This invention relates to the field of industrial internet manufacturing technology, specifically to an adaptive optimization method for machine tool equipment resources in an industrial internet environment based on an improved PPO-RND algorithm. Background Technology

[0002] With the advancement of information technology, industrial internet manufacturing has gradually become one of the core driving forces for the transformation of modern manufacturing systems. Industrial internet, by combining industrial IoT, cyber-physical systems, and artificial intelligence technologies, encapsulates geographically dispersed machine tool resources into virtualized manufacturing services within an industrial internet platform. Based on the personalized needs of customers on the platform, it optimizes the combination of machine tool resource services, thereby providing corresponding solutions for customers' complex manufacturing tasks. However, in actual production environments, production disturbances such as demand fluctuations, equipment failures, and resource shortages often impact the optimization environment of machine tool resource services, leading to the invalidation of original optimization results. Furthermore, the machine tool resource optimization problem is an NP-hard problem, making it highly difficult to solve. Therefore, the machine tool resource optimization problem under dynamic disturbances places more stringent performance requirements on the computational efficiency and adaptability of the optimization method.

[0003] Proximal Policy Optimization (PPO), proposed by Schulman et al. in 2017, aims to improve the stability and efficiency of policy gradient methods. PPO introduces a clipping mechanism during policy updates to limit the magnitude of changes between the old and new policies, avoiding instability caused by excessively large policy updates. PPO has demonstrated good performance in practical applications and is relatively simple to implement, making it a mainstream algorithm in reinforcement learning.

[0004] Random Network Distillation (RND), proposed by Burda et al. in 2018, aims to enhance the exploration capabilities of reinforcement learning agents by introducing an intrinsic reward mechanism. RND uses prediction errors as reward signals to encourage agents to explore novel states in the environment, thus overcoming the problem of insufficient exploration in traditional reinforcement learning methods. This characteristic allows the RND algorithm to address, to some extent, the problem of insufficient exploration caused by sparse reward environments.

[0005] In summary, although the problem of optimal resource selection for machine tool equipment in an industrial internet environment has a certain research foundation, there is still room for exploration in the field of adaptive resource selection under production disturbances. Summary of the Invention

[0006] The purpose of this invention is to provide an adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on an improved PPO-RND algorithm, comprising the following steps:

[0007] 1) Model the machine tool equipment resources and establish a production disturbance model.

[0008] 2) The process of optimizing machine tool equipment resources is modeled as a six-tuple MR-MDP.

[0009] 3) Design the reward function for the machine tool equipment resource optimization process MR-MDP.

[0010] 4) Construct an improved PPO-RND algorithm framework.

[0011] 5) The improved PPO-RND algorithm framework is used to solve the machine tool equipment resource optimization process MR-MDP to obtain the resource combination of each machine tool equipment and the corresponding actions.

[0012] After a production disturbance occurs, the resource combination of each machine tool and the corresponding actions are re-optimized.

[0013] Furthermore, when establishing a machine tool equipment resource model, a machine tool equipment resource is modeled as a quadruple MR, that is:

[0014] MR = <ID,In,Out,Qos> (1)

[0015] Qos=(T,C,P,R) (2)

[0016] In the formula, ID is the identifier of the machine tool resource (MR), In is the input part before processing by the machine tool resource MR, Out is the output part after processing by the machine tool resource MR, and Qos is the Qos attribute of the machine tool resource MR. T represents service time. C represents service cost. P represents the product qualification rate. R represents service reliability.

[0017] Production disturbances are modeled as quadruples ED, i.e.:

[0018] ED = < D r D p D o D d > (3)

[0019] D r = <d rt ,d rc ,d rp ,d rr > (4)

[0020] D p = <d p1 ,d p2,d p3 > (5)

[0021] D o = <d ot ,d oc ,d op ,d or > (6)

[0022] D d = <d d1 ,d d2 ,d d3 ,…,d dz > (7)

[0023] In the formula, ED represents the production disturbance. D r D p D o D d Let d represent the sets of disturbances related to service demand, service provision, platform operation, and logistics transportation, respectively. rt d rc d rp d rr These represent the changes in the customer's minimum requirements for service time T, service cost C, product qualification rate P, and service reliability R, respectively. p1 d p2 d p3 These represent disturbances caused by increased machine resources, disturbances caused by decreased machine resources, and changes in QoS attributes, respectively. ot d oc d op d or These represent the weighting factors for service time T, service cost C, product qualification rate P, and service reliability R, respectively. d1 ,d d2 ,d d3 ,…,d dz This represents z types of logistics path disturbances for both the service provider and the service demander.

[0024] Furthermore, the machine tool equipment resource optimization process MR-MDP is as follows:

[0025] MR-MDP=<S,s0,s τ A, P sa ,Reward> (8)

[0026] In the formula, MR-MDP represents the machine tool equipment resource optimization process (MR-MDP). S represents the discrete set of environmental states. s0 represents the initial state. τ This indicates the termination state. A is the action space set. P saThis represents the state transition probability. Reward represents the reward function.

[0027] Among them, the state elements s in the discrete set S of environmental states t As shown below:

[0028]

[0029] In the formula, t represents time, s t This represents the state at time t. i represents the subtask index, l represents the total number of subtasks, j represents the task sequence index, and h represents the total number of task sequences. This represents the ID identifier of the machine tool equipment resource matched by the i-th subtask in the j-th task sequence.

[0030] The reward function Reward is as follows:

[0031] r = Reward(s'|s,a) (10)

[0032] In the formula, a represents the action, s and s' represent the states before and after the action a is performed, respectively, and r represents the reward after the state s transitions to the state s'.

[0033] Furthermore, the reward function steps of the MR-MDP process for optimizing machine tool equipment resources are as follows:

[0034] 3.1) Calculate the comprehensive QoS index corresponding to the historical task sequence through the QoS attribute, and establish reward function branches corresponding to different service quality indicators.

[0035] The service quality indicators include service time, service cost, product qualification rate, and service reliability.

[0036] 3.2) The external reward function is obtained by weighting different reward function branches using a linear weighting method.

[0037] Furthermore, the reward function branches corresponding to the different service quality indicators are as follows:

[0038]

[0039] In the formula, j represents the task sequence index. Let T, C, P, and R represent the reward function branches corresponding to the service time T, service cost C, qualified product rate P, and service reliability R of the j-th task sequence, respectively. These represent the maximum and minimum values ​​of the service time T, respectively. These represent the maximum and minimum values ​​of the service cost C, respectively. These represent the maximum and minimum values ​​of the qualified product rate P, respectively. Let T represent the maximum and minimum values ​​of service reliability R, respectively.(j) C (j) P (j) R (j) Let T, C, P, and R represent the service time, service cost, product qualification rate, and service reliability values ​​for the j-th task sequence, respectively. Let a1 represent the decay rate. These represent the upper limit constraints on service time T and service cost C, respectively. These represent the lower limits of the qualified product rate P and service reliability R, respectively. B represents the baseline number. δ represents the deviation value.

[0040] Furthermore, the external reward function is as follows:

[0041]

[0042] In the formula, r t ext This represents the external reward function. j represents the task sequence index, and h represents the total number of task sequences. (j) Let represent the external reward of the j-th task sequence. Let Reward represent the reward function. Let 'a' represent the action, and 's' and 's' represent the states before and after executing action 'a', respectively. τ Indicates a terminated state. ω T ω C ω P ω R These represent the weights of service time T, service cost C, qualified product rate P, and service reliability R, respectively. Let T, C, P, and R represent the reward branches corresponding to the service time T, service cost C, qualified product rate P, and service reliability R of the j-th task sequence, respectively.

[0043] Furthermore, the steps for solving the machine tool equipment resource optimization process MR-MDP using the improved PPO-RND algorithm framework are as follows:

[0044] 5.1) The comprehensive reward function of the machine tool equipment resource optimization process MR-MDP is calculated based on the improved PPO-RND algorithm.

[0045] 5.2) Filter out illegal actions in the machine tool equipment resource optimization process MR-MDP based on dynamic action masking mechanism.

[0046] 5.3) Input the current task sequence into the machine tool equipment resource optimization process MR-MDP after filtering out illegal actions to obtain the combination of machine tool equipment resources and corresponding actions at the next moment.

[0047] Furthermore, in step 5.1), the steps for calculating the comprehensive reward function of the machine tool equipment resource optimization process MR-MDP are as follows:

[0048] 5.1.1) Define the strategy ratio r t (θ) is shown below:

[0049]

[0050] In the formula, t represents time, s t This represents the state at time t. t θ represents the action at time t. old θ and θ represent the fixed PPO strategy parameters and the PPO strategy parameters to be optimized, respectively. π θ (a t |s t ) represent the fixed PPO strategy parameters θ old The state s under the PPO strategy parameter θ to be optimized t Select action a t The probability of.

[0051] 5.1.2) Define the shearing objective function of the PPO algorithm, and update the PPO policy parameter θ to be optimized by solving the shearing objective function of the PPO algorithm.

[0052] The shearing objective function of the PPO algorithm is as follows:

[0053]

[0054] In the formula, L CLIP (θ) is the shearing objective function of PPO. For advantage estimation. E t Let clip(r) be the expected function. t (θ), 1-ε, 1+ε) are the constraint functions, and the policy ratio r t (θ) is restricted to the range [1-ε, 1+ε]. ε is a hyperparameter.

[0055] 5.1.3) By introducing a target network and a prediction network, an intrinsic reward r is generated. int .

[0056] 5.1.4) Combining external rewards r t ext and intrinsic reward r t int Receive a comprehensive reward r t As shown below:

[0057] r t =r t ext +βr t int (16)

[0058] In the formula, β is the weight hyperparameter.

[0059] Furthermore, the intrinsic reward is generated by introducing a target network and a prediction network. The steps are as follows:

[0060] 5.1.3.1) Constructing the target network f target and prediction network f pred .

[0061] The target network f target and prediction network f pred Both are feedforward neural networks, and both take the state as input and the state features as output.

[0062] 5.1.3.2) Define intrinsic reward r int As shown below:

[0063]

[0064] In the formula, t represents time, o target o pred They represent the target network f respectively. target Predictive network f pred The output characteristics. This represents the mean square error function.

[0065] 5.1.3.3) Construct the loss function for the prediction network as follows:

[0066]

[0067] In the formula, L RND To predict the loss function of the network, s t Let represent the state at time t. Let D represent the state sample set. Let be the expected function.

[0068] 5.1.3.4) Update the parameters of the prediction network through the backpropagation algorithm to minimize the loss function of the prediction network.

[0069] The parameters of the prediction network include the prediction network f. pred Output feature o pred Target network f target Output feature o target .

[0070] 5.1.3.5) Based on the updated prediction network parameters, the intrinsic reward r is calculated. t int .

[0071] Furthermore, the steps for filtering out illegal actions in the machine tool equipment resource optimization process MR-MDP based on the dynamic action masking mechanism are as follows:

[0072] 5.2.1) Establish an effective mapping between machine tool equipment resources and actions, as shown below:

[0073] g:A(s t )×MRS valid (t)→{0,1} (19)

[0074] In the formula, t represents time, s t This represents the state at time t. g represents the mapping relationship between effective machine tool equipment resources and actions. A(s) t () represents the set of actions at time t. MRS valid (t) represents the set of effective machine tool equipment resources at time t.

[0075] 5.2.2) Based on the mapping relationship between effective machine tool equipment resources and actions, establish the legal action set and the illegal action set, as shown below:

[0076]

[0077] A illegal (t)=A(s t )\A legal (t) (21)

[0078] In the formula, A legal (t), A illegal (t) represents the set of legal actions and the set of illegal actions at time t, respectively. a represents an action, and MR represents machine tool resources. g(a,MR) represents the mapping relationship between action a and machine tool resources MR.

[0079] 5.2.3) Set the probability of all illegal actions in the illegal action set to zero, and filter out illegal actions from the MR-MDP optimization process model, as shown below:

[0080]

[0081] In the formula, s represents the state, a represents the action, and a illegal Indicates an illegal action, π(a) illegal |s) represents the probability of selecting an illegal action. Logit(·) is the Logit value of the action. A legal (t), A illegal (t) represents the set of legal actions and the set of illegal actions at time t, respectively.

[0082] The technical effectiveness of this invention is undeniable. This invention proposes an adaptive intelligent optimization method for machine tool equipment resources. Based on the analysis of the machine tool equipment optimization process and characteristics in industrial interconnected manufacturing scenarios, and combined with Markov decision process theory, an MR-MDP optimization model is established, and a reward function considering Qos multi-objective optimization and constraints is constructed. To solve this model, an improved PPO-RND algorithm incorporating a dynamic masking mechanism is proposed. Experimental results show that, under the influence of production disturbances, compared with benchmark algorithms such as PPO and Parallel DQN, the proposed method achieves the best overall solution results across 16 task sequences and exhibits superior robustness. This invention provides theoretical and methodological support for the stable and reliable operation of industrial interconnected manufacturing services.

[0083] This invention addresses the problem of environmental dynamics caused by production disturbances during the optimal selection process of machine tool equipment resources in an industrial interconnected environment, which can lead to the invalidation of the selection results. It researches an adaptive optimization method for machine tool equipment resources in the dynamic re-optimization stage. This invention establishes a machine tool equipment resource and production disturbance model, and further establishes an MR-MDP machine tool equipment optimization process model, constructing a reward function that considers multi-objective optimization and constraints. For solving the model, this invention proposes a PPO-RND algorithm that integrates a dynamic masking mechanism. The PPO algorithm serves as the main algorithm to improve its stability, while the RND algorithm enhances the exploration mechanism. The curiosity reward mechanism of the RND algorithm drives the agent to search for more unexplored strategies, thereby improving the exploration efficiency of the algorithm, reducing the probability of getting trapped in local optima during the optimization process, and also solving the sparse reward problem inherent in machine tool equipment resource optimization. Finally, a dynamic masking mechanism is established to construct a dynamically variable action space, thereby reducing the impact of production disturbances such as machine tool failures on the stability of the algorithm. The improved PPO-RND algorithm of this invention exhibits a more stable training process and more robust perturbation adaptation ability compared with the two baseline algorithms PPO and Parallel DQN and their HER variants in the dynamic reselection stage. In the parallel optimization task of multiple task sequences, its solution result is generally optimal. Attached Figure Description

[0084] Figure 1 A framework diagram of the overall process of industrial interconnection manufacturing;

[0085] Figure 2 This is a diagram illustrating the state transition;

[0086] Figure 3 This is a schematic diagram of a sparse reward environment.

[0087] Figure 4 This is a schematic diagram of the adaptive optimization method. Detailed Implementation

[0088] The present invention will be further described below with reference to embodiments, but it should not be construed that the scope of the present invention is limited to the following embodiments. Various substitutions and modifications made based on ordinary technical knowledge and common practices in the art without departing from the above-described technical concept of the present invention should be included within the scope of protection of the present invention.

[0089] Example 1:

[0090] See Figures 1 to 4 An adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on an improved PPO-RND algorithm includes the following steps:

[0091] 1) Model the machine tool equipment resources and establish a production disturbance model.

[0092] 2) The process of optimizing machine tool equipment resources is modeled as a six-tuple MR-MDP.

[0093] 3) Design the reward function for the machine tool equipment resource optimization process MR-MDP.

[0094] 4) Construct an improved PPO-RND algorithm framework.

[0095] 5) The improved PPO-RND algorithm framework is used to solve the machine tool equipment resource optimization process MR-MDP to obtain the resource combination of each machine tool equipment and the corresponding actions.

[0096] After a production disturbance occurs, the resource combination of each machine tool and the corresponding actions are re-optimized.

[0097] Example 2:

[0098] An adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on an improved PPO-RND algorithm is described in Example 1. Further, when establishing the machine tool equipment resource model, a machine tool equipment resource is modeled as a quadruple MR, i.e.:

[0099] MR = <ID,In,Out,Qos> (1)

[0100] Qos=(T,C,P,R) (2)

[0101] In the formula, ID is the identifier of the machine tool resource (MR), In is the input part before processing by the machine tool resource MR, Out is the output part after processing by the machine tool resource MR, and Qos is the Qos attribute of the machine tool resource MR. T represents service time. C represents service cost. P represents the product qualification rate. R represents service reliability.

[0102] Production disturbances are modeled as quadruples ED, i.e.:

[0103] ED = < D r D p D o D d > (3)

[0104] D r = <d rt ,d rc ,d rp ,d rr > (4)

[0105] D p = <d p1 ,d p2 ,d p3 > (5)

[0106] D o = <d ot ,d oc ,d op ,d or > (6)

[0107] D d = <d d1 ,d d2 ,d d3 ,…,d dz > (7)

[0108] In the formula, ED represents the production disturbance. D r D p D o D d Let d represent the sets of disturbances related to service demand, service provision, platform operation, and logistics transportation, respectively. rt d rc d rp d rr These represent the changes in the customer's minimum requirements for service time T, service cost C, product qualification rate P, and service reliability R, respectively. p1 d p2 d p3 These represent disturbances caused by increased machine resources, disturbances caused by decreased machine resources, and changes in QoS attributes, respectively. ot d oc d op d or These represent the weighting factors for service time T, service cost C, product qualification rate P, and service reliability R, respectively. d1 ,d d2 ,d d3 ,…,d dz This represents z types of logistics path disturbances for both the service provider and the service demander.

[0109] Example 3:

[0110] An adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on an improved PPO-RND algorithm is described in any one of Examples 1 and 2. Further, the machine tool equipment resource optimization process MR-MDP is as follows:

[0111] MR-MDP=<S,s0,s τ A, P sa ,Reward> (8)

[0112] In the formula, MR-MDP represents the machine tool equipment resource optimization process (MR-MDP). S represents the discrete set of environmental states. s0 represents the initial state. τ This indicates the termination state. A is the action space set. P sa This represents the state transition probability. Reward represents the reward function.

[0113] Among them, the state elements s in the discrete set S of environmental states t As shown below:

[0114]

[0115] In the formula, t represents time, s t This represents the state at time t. i represents the subtask index, l represents the total number of subtasks, j represents the task sequence index, and h represents the total number of task sequences. This represents the ID identifier of the machine tool equipment resource matched by the i-th subtask in the j-th task sequence.

[0116] The reward function Reward is as follows:

[0117] r = Reward(s'|s,a) (10)

[0118] In the formula, a represents the action, s and s' represent the states before and after the action a is performed, respectively, and r represents the reward after the state s transitions to the state s'.

[0119] Example 4:

[0120] An adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on an improved PPO-RND algorithm is provided. The main technical content is described in any one of Examples 1 to 3. Furthermore, the steps for designing the reward function of the MR-MDP in the machine tool equipment resource optimization process are as follows:

[0121] 3.1) Calculate the comprehensive QoS index corresponding to the historical task sequence through the QoS attribute, and establish reward function branches corresponding to different service quality indicators.

[0122] The service quality indicators include service time, service cost, product qualification rate, and service reliability.

[0123] 3.2) The external reward function is obtained by weighting different reward function branches using a linear weighting method.

[0124] Example 5:

[0125] An adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on an improved PPO-RND algorithm is described in any one of Examples 1 to 4. Further, the reward function branches corresponding to the different service quality indicators are as follows:

[0126]

[0127] In the formula, j represents the task sequence index. Let T, C, P, and R represent the reward function branches corresponding to the service time T, service cost C, qualified product rate P, and service reliability R of the j-th task sequence, respectively. These represent the maximum and minimum values ​​of the service time T, respectively. These represent the maximum and minimum values ​​of the service cost C, respectively. These represent the maximum and minimum values ​​of the qualified product rate P, respectively. Let T represent the maximum and minimum values ​​of service reliability R, respectively. (j) C (j) P (j) R (j) Let T, C, P, and R represent the service time, service cost, product qualification rate, and service reliability values ​​for the j-th task sequence, respectively. Let a1 represent the decay rate. These represent the upper limit constraints on service time T and service cost C, respectively. These represent the lower limits of the qualified product rate P and service reliability R, respectively. B represents the baseline number. δ represents the deviation value.

[0128] Example 6:

[0129] An adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on an improved PPO-RND algorithm, the main technical contents of which are described in any one of Examples 1 to 5, and further, the external reward function is as follows:

[0130]

[0131] In the formula, r t ext This represents the external reward function. j represents the task sequence index, and h represents the total number of task sequences. (j)Let represent the external reward of the j-th task sequence. Let Reward represent the reward function. Let 'a' represent the action, and 's' and 's' represent the states before and after executing action 'a', respectively. τ Indicates a terminated state. ω T ω C ω P ω R These represent the weights of service time T, service cost C, qualified product rate P, and service reliability R, respectively. Let T, C, P, and R represent the reward branches corresponding to the service time T, service cost C, qualified product rate P, and service reliability R of the j-th task sequence, respectively.

[0132] Example 7:

[0133] An adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on an improved PPO-RND algorithm is described in any one of Examples 1 to 6. Further, the steps for solving the MR-MDP process of machine tool equipment resource optimization using the improved PPO-RND algorithm framework are as follows:

[0134] 5.1) The comprehensive reward function of the machine tool equipment resource optimization process MR-MDP is calculated based on the improved PPO-RND algorithm.

[0135] 5.2) Filter out illegal actions in the machine tool equipment resource optimization process MR-MDP based on dynamic action masking mechanism.

[0136] 5.3) Input the current task sequence into the machine tool equipment resource optimization process MR-MDP after filtering out illegal actions to obtain the combination of machine tool equipment resources and corresponding actions at the next moment.

[0137] Example 8:

[0138] An adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on an improved PPO-RND algorithm, the main technical contents of which are described in any one of Examples 1 to 7. Further, in step 5.1), the step of calculating the comprehensive reward function of the MR-MDP process for optimizing machine tool equipment resources is as follows:

[0139] 5.1.1) Define the strategy ratio r t (θ) is shown below:

[0140]

[0141] In the formula, t represents time, s t This represents the state at time t. t θ represents the action at time t. old θ and θ represent the fixed PPO strategy parameters and the PPO strategy parameters to be optimized, respectively. πθ (a t |s t ) represent the fixed PPO strategy parameters θ old The state s under the PPO strategy parameter θ to be optimized t Select action a t The probability of.

[0142] 5.1.2) Define the shearing objective function of the PPO algorithm, and update the PPO policy parameter θ to be optimized by solving the shearing objective function of the PPO algorithm.

[0143] The shearing objective function of the PPO algorithm is as follows:

[0144]

[0145] In the formula, L CLIP (θ) is the shearing objective function of PPO. For advantage estimation. E t Let clip(r) be the expected function. t (θ), 1-ε, 1+ε) are the constraint functions, and the policy ratio r t (θ) is restricted to the range [1-ε, 1+ε]. ε is a hyperparameter.

[0146] 5.1.3) By introducing a target network and a prediction network, an intrinsic reward r is generated. t int .

[0147] 5.1.4) Combining external rewards r t ext and intrinsic reward r t int Receive a comprehensive reward r t As shown below:

[0148] r t =r t ext +βr t int (16)

[0149] In the formula, β is the weight hyperparameter.

[0150] Example 9:

[0151] An adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on an improved PPO-RND algorithm is described in any one of Examples 1 to 8. Further, the method generates intrinsic rewards by introducing a target network and a prediction network. int

[0152] r tThe steps are as follows:

[0153] 5.1.3.1) Constructing the target network f target and prediction network f pred .

[0154] The target network f target and prediction network f pred Both are feedforward neural networks, and both take the state as input and the state features as output.

[0155] 5.1.3.2) Define intrinsic reward r t int As shown below:

[0156]

[0157] In the formula, t represents time, o target o pred They represent the target network f respectively. target Predictive network f pred The output characteristics. This represents the mean square error function.

[0158] 5.1.3.3) Construct the loss function for the prediction network as follows:

[0159]

[0160] In the formula, L RND To predict the loss function of the network, s t Let represent the state at time t. Let D represent the state sample set. Let be the expected function.

[0161] 5.1.3.4) Update the parameters of the prediction network through the backpropagation algorithm to minimize the loss function of the prediction network.

[0162] The parameters of the prediction network include the prediction network f. pred Output feature o pred Target network f target Output feature o target .

[0163] 5.1.3.5) Based on the updated prediction network parameters, the intrinsic reward r is calculated. t int .

[0164] Example 10:

[0165] An adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on an improved PPO-RND algorithm is provided. The main technical content is detailed in any one of Examples 1 to 9. Further, the steps for filtering illegal actions in the machine tool equipment resource optimization process using a dynamic action masking mechanism (MR-MDP) are as follows:

[0166] 5.2.1) Establish an effective mapping between machine tool equipment resources and actions, as shown below:

[0167] g:A(s t )×MRS valid (t)→{0,1} (19)

[0168] In the formula, t represents time, s t This represents the state at time t. g represents the mapping relationship between effective machine tool equipment resources and actions. A(s) t () represents the set of actions at time t. MRS valid (t) represents the set of effective machine tool equipment resources at time t.

[0169] 5.2.2) Based on the mapping relationship between effective machine tool equipment resources and actions, establish the legal action set and the illegal action set, as shown below:

[0170]

[0171] A illegal (t)=A(s t )\A legal (t) (21)

[0172] In the formula, A legal (t), A illegal (t) represents the set of legal actions and the set of illegal actions at time t, respectively. a represents an action, and MR represents machine tool resources. g(a,MR) represents the mapping relationship between action a and machine tool resources MR.

[0173] 5.2.3) Set the probability of all illegal actions in the illegal action set to zero, and filter out illegal actions from the MR-MDP optimization process model, as shown below:

[0174]

[0175] In the formula, s represents the state, a represents the action, and a illegal Indicates an illegal action, π(a) illegal |s) represents the probability of selecting an illegal action. Logit(·) is the Logit value of the action. A legal (t), A illegal (t) represents the set of legal actions and the set of illegal actions at time t, respectively.

[0176] Example 11:

[0177] See Figures 1 to 4 An adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on an improved PPO-RND algorithm includes the following steps:

[0178] S1: Construct a model of machine tool equipment resources and production disturbances.

[0179] S2: Construct an MR-MDP optimization process model.

[0180] S3: Design the reward function. Calculate the comprehensive score of the task sequence based on QoS metrics. Introduce a sigmoid window function to implement out-of-bounds penalties, ensuring the agent explores solutions that meet the constraints. Construct the final reward function using a linear weighted method.

[0181] S4: Propose an improved PPO-RND algorithm. Based on the PPO algorithm, improve the stability of policy updates. Integrate the RND algorithm to enhance the agent's exploration ability through an intrinsic reward mechanism. Introduce a dynamic masking mechanism to dynamically filter illegal actions and reduce the impact of perturbations on the algorithm.

[0182] S5: Implement dynamic re-optimization. After a production disturbance occurs, re-optimize based on the current optimization results. The pre-trained model continues to train in the new environment, quickly adapting to the changes brought about by the disturbance.

[0183] In step S1, the model is constructed through the following steps:

[0184] S101: A machine tool equipment resource can be modeled as a quadruple MR =<ID,In,Out,Qos> Where ID is the identifier of a machine tool resource, In is the input part before processing by the machine tool resource, Out is the output part after processing by the machine tool resource, and Qos is an n-tuple. <attr1,attr2,attr3,…,attr n >, where each attr i A Qos attribute representing machine tool equipment resources. The relationships between related Qos attributes can be represented by a quadruple:

[0185] Qos = (T, C, P, R)

[0186] T represents service time. C represents service cost. P represents the product qualification rate. R represents service reliability.

[0187] S102: Production disturbances are primarily defined as a quadruple ED=<D r D p D o D d >.

[0188] Where D r It is a set of service demand-type perturbations, and D r It is a quadruple D r = <d rt ,d rc ,d rp ,d rr > represents the changes in the customer's minimum requirements for four QoS indicators: service time, service cost, product qualification rate, and service reliability. D p It is a set of service-providing disturbances, representing the increase or decrease of a certain machine tool resource, D. p It can be represented as a triple D p = <d p1 ,d p2 ,d p3 >, d p1 This represents a resource-increasing disturbance in a certain machine tool, d p2 This represents a resource reduction disturbance in a certain machine tool, d p3 This indicates a change in the QOS attribute of a certain machine tool. (D) o Do is a set of platform operation-related perturbations, which includes perturbations to the weighting factors of various QoS attributes. Therefore, D o For a quadruple D o = <d ot ,d oc ,d op ,d or >, d ot Represents the service time weighting factor, d oc The service cost weighting factor, d op The weighting factor representing the qualified product rate, d or D represents the service reliability weighting factor. d It is a set of logistics and transportation disturbances, including all logistics path disturbances from both service providers and service demanders. Assuming there are z logistics paths, then D... d For a z-tuple D d = <d d1 ,d d2 ,d d3 ,…,d dz >.

[0189] In step S2, the model is constructed through the following steps:

[0190] S201: The MR-MDP-oriented Markov decision process model for machine tool equipment resource optimization is a six-tuple MR-MDP =<S,s0,s τ A, P sa ,Reward>.

[0191] Here, s represents the discrete set of environmental states S. t ∈S, s t Record the machine tool resource matching information for all task sequences at time t. Assume the longest task sequence has l sub-tasks, and there are h task sequences in total, then s t It can be represented by the following matrix:

[0192]

[0193] This is the ID of the machine tool resource matched with the i-th subtask MT in the j-th task sequence. If it is 0, it means that there is no match or no corresponding subtask.

[0194] Let s0∈S be the initial state. At the start of each training round, the agent will begin executing service composition from this state. Generally, at the start of a training round, all subtasks in the task sequence are in an unmatched state, therefore:

[0195] s0 = 0 h×l

[0196] s τ For the set of terminating states, when the agent's state reaches s τ When any state in the set is reached, the training round ends and the service composition terminates. τ Related to the resource matching status in the task sequence, the current state is s when all task sequences have completed resource matching. τ .

[0197] S202: A is the action space set, A = A(s0)∪A(s1)∪A(s2)…∪A(s... n ), where A(s) t ) represents state s t The set of actions that an intelligent agent can take, and this set is related to s t Candidate Machine Tool Equipment Resource Set (MRS) in State t Form a mapping. Assume state s t When the matching begins for the i-th subtask MT in the j-th task sequence, the corresponding machine tool equipment resource set is MRS. (j) (s t )={MR (j) i_1,MR (j) i_2,…,MR (j) Let i_k} be the number of machine tools, then MRS t ={MRS (1) (s t ), MRS (2) (s t ),…,MRS (h) (st MRS under the influence of production disturbances t This may change, thereby causing A(s) t ) has changed.

[0198] Therefore, at time t, the agent, based on the current state s t Execute action a t After matching each task sequence once, it enters state s. t+1 .

[0199] S203:P sa Let P be the state transition probability, which represents the probability distribution of transitioning to another state s' after performing action a in the current state s. It is generally denoted as p(s'|s,a). In the context of the machine tool resource optimization problem, since the agent receives feedback based on the resource allocation result at each decision moment, and calculates the reward r through this feedback, P... sa It is denoted as p(s',r|s,a).

[0200] S204: Reward is the reward function. When action a is performed in state s and the state transitions to another state s', the reward r = Reward(s'|s,a) can be calculated.

[0201] In step S3, the reward function is designed through the following steps:

[0202] S301: First, calculate the comprehensive QoS index corresponding to the task sequence through the QoS attribute, and establish the corresponding reward function branch considering the multi-objective requirements and constraints of the four indicators: service time, service cost, product qualification rate, and service reliability.

[0203] S302: The reward function is constructed using a linear weighting method.

[0204] In step S301, the QoS attribute is calculated through the following steps:

[0205] S3011: The calculation of the four service quality indicators of service time, service cost, product qualification rate and service reliability of subtasks in the task sequence should follow the calculation method of the corresponding structure. The specific calculation method is shown in Table 1.

[0206] S3012: Based on the table above, the comprehensive index of a certain task sequence regarding service time, service cost, product qualification rate, and service reliability can be calculated as: QoS (j) =(T (j) C (j) ,P (j) ,R (j) )

[0207] S3013: Considering the significant differences in dimensions and magnitudes among different indicators, direct analysis and comparison are not possible; therefore, normalization is required for each indicator.

[0208]

[0209] After normalization, all QoS indicators can be limited to the range of [0,1], and the monotonicity of different QoS indicators can reflect the multi-objective optimization requirements of minimizing service time, minimizing service cost, maximizing qualified products, and maximizing service reliability.

[0210] Customer-specific needs include both multi-objective optimization requirements and constraints, and the mathematical definition of the constraints is as follows:

[0211] T≤T max

[0212] C≤C max

[0213] P≥P min

[0214] R≥R min

[0215] S3014: To guide the agent in exploring optimal results that better meet the constraints, an appropriate penalty for exceeding the limits can be introduced into the reward function. Specifically, a smooth window function can be constructed to achieve a sharp reduction in rewards when constraints are exceeded.

[0216] Suppose there is an upper limit constraint q for a certain Qos metric q. upper Or lower bound constraint q lower A window function can then be constructed using the Sigmoid function:

[0217]

[0218] In this window function, 'a' represents the decay rate, which is primarily determined by customer requirements.

[0219] S3015: Construct a reward function branch for each Qos metric:

[0220]

[0221] Where B is the baseline value, used to adjust the overall difference between function values ​​inside and outside the target range. When B is large, the function values ​​within the target range are generally higher than those outside the target range. δ is the deviation value, a constant, used to adjust the overall deviation of the function.

[0222] Table 1. Calculation Methods for the Four Service Quality Indicators

[0223]

[0224] Note: W stands for Machine Tool Equipment Resource Service Number of times it is looped

[0225] In step S302, the reward function is constructed through the following steps:

[0226] S3021: A reward function for a given task sequence can be constructed using a linear weighting method.

[0227]

[0228] S3022: By combining the rewards of all task sequences, a vector representation of the current reward can be constructed:

[0229]

[0230] In step S4, the steps for improving the PPO-RND algorithm and incorporating the dynamic masking mechanism are as follows:

[0231] S401: PPO is the main algorithm applicable to both discrete and continuous action spaces. In addition, the RND algorithm is used to reduce the negative impact of sparse reward environment and further enhance the algorithm's exploration ability in high-dimensional space.

[0232] S402: Construct a dynamic action masking mechanism to further improve the algorithm.

[0233] In step S401, the PPO-RND algorithm is improved through the following steps:

[0234] S4011: PPO defines a policy ratio r t (θ):

[0235]

[0236] Where θ old These are the policy parameters before the update. To prevent r t (θ) deviates too much, so PPO uses a shearing method to restrict the objective function to a small range, usually set to [1-ε, 1+ε], where ε is a small hyperparameter.

[0237] The objective function for PPO shearing is defined as:

[0238]

[0239] in, The dominance estimate is usually calculated using the generalized dominance estimate. clip(r) t (θ), 1-ε, 1+ε) will be the ratio r t(θ) is restricted to the range [1-ε, 1+ε]. This objective function ensures that when r... t When (θ) exceeds the shearing range, the gradient of the objective function will not push the policy to deviate excessively, thus achieving effective control over the policy update magnitude.

[0240] S4012: RND generates intrinsic reward signals by introducing a target network and a prediction network. The target network is a fixed and randomly initialized feedforward neural network f. target Its parameters remain unchanged during training. The prediction network is a trainable feedforward neural network f pred Its goal is to predict the target network's output for the same input state. At each time step t, the agent receives state s. t The target network and the prediction network generate the corresponding outputs respectively:

[0241] o target =f target (s t ),o pred =f pred (s t )

[0242] Intrinsic reward r t int Defined as the mean square error between the two outputs:

[0243]

[0244] The training objective of RND is to minimize the error between the output of the prediction network and the target network, thereby improving the prediction network's ability to predict environmental states. The loss function of the prediction network is L. RND Defined as:

[0245]

[0246] Here, D represents the set of state samples collected from the environment. The parameters of the prediction network are continuously updated using the backpropagation algorithm to minimize the aforementioned loss function.

[0247] Finally, RND and external reward r t ext Combined, forming a comprehensive reward r t :

[0248] r t =r t ext +βr t int

[0249] Here, β is a weighted hyperparameter used to balance the influence of external and internal rewards.

[0250] In step S402, a dynamic action masking mechanism is constructed through the following steps:

[0251] S4021: Dynamically maintain an effective Machine Tool Equipment Resource Service Set (MRS) valid (t), for the action set A(s) at time t t Establish a mapping:

[0252] g:A(s t )×MRS valid (t)→{0,1}

[0253] This mapping represents A(s) t Does action 'a' in the context of a given machine tool resource (MR) relate to a specific valid machine tool resource (MR)? valid If correlated, then g(a,MR) valid ) = 1, otherwise g(a,MR valid ) = 0.

[0254] S4022: Establish a set of legal actions based on the mapping relationship:

[0255]

[0256] The set of illegal actions is:

[0257] A illegal (t)=A(t)\A legal (t)

[0258] S4023: For all illegal actions a∈A illegal (t), set the corresponding Logit value to negative infinity:

[0259]

[0260] S4024: In the Softmax calculation, the probability of an illegal action will be set to zero so that it will not be selected.

[0261]

[0262] In step S5, the steps for implementing dynamic re-optimization are as follows:

[0263] S501: The PPO-RND algorithm adaptively optimizes the overall frame using four perturbations, making the overall environment dynamic. Among them, the production perturbation D... p With D d Primarily used in machine tool databases, to disturb D r It mainly acts on constraints, while the disturbance D oThe four perturbations directly affect the reward function. All four perturbations have some impact on the reward function, and in addition, perturbation D... p For MRS t Construction and a t The selection of [aspect] also has a certain impact.

[0264] S502: When a perturbation occurs, its impact is primarily transmitted to the reward function. Specifically, the agent cannot obtain the original high reward value under the original optimal solution. However, through exploration, the agent will discover that the originally low-reward optimal solution becomes a high-reward optimal solution under the influence of the perturbation. The agent then stores the experience of the new environment after the perturbation in the storage area and further updates the network structure of the PPO and RND algorithms, thus beginning the exploration of the optimal solution in the new environment after the perturbation.

[0265] Example 12:

[0266] See Figures 1 to 4 An adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on an improved PPO-RND algorithm, the main technical contents of which include:

[0267] like Figure 1 As shown, the adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on the improved PPO-RND algorithm includes:

[0268] S1: Construct a model of machine tool equipment resources and production disturbances.

[0269] S2: Construct an MR-MDP optimization process model.

[0270] S3: Design the reward function. Calculate the comprehensive score of the task sequence based on QoS metrics. Introduce a sigmoid window function to implement out-of-bounds penalties, ensuring the agent explores solutions that meet the constraints. Construct the final reward function using a linear weighted method.

[0271] S4: Propose an improved PPO-RND algorithm. Based on the PPO algorithm, improve the stability of policy updates. Integrate the RND algorithm to enhance the agent's exploration ability through an intrinsic reward mechanism. Introduce a dynamic masking mechanism to dynamically filter illegal actions and reduce the impact of perturbations on the algorithm.

[0272] S5: Implement dynamic re-optimization. After a production disturbance occurs, re-optimize based on the current optimization results. The pre-trained model continues to train in the new environment, quickly adapting to the changes brought about by the disturbance.

[0273] This invention addresses the problem of environmental dynamics caused by production disturbances during the optimal selection process of machine tool equipment in an industrial interconnected environment, which can lead to the invalidation of the selection results. It studies an adaptive optimization method for machine tool equipment resources in the dynamic re-optimization stage. The study establishes a model of machine tool equipment resources and production disturbances, and further establishes an MR-MDP machine tool equipment optimization process model, constructing a reward function that considers multi-objective optimization and constraints. For solving the model, this invention proposes a PPO-RND algorithm that integrates a dynamic masking mechanism. The PPO algorithm serves as the main algorithm to improve its stability, while the RND algorithm enhances the exploration mechanism. The curiosity reward mechanism of the RND algorithm drives the agent to search for more unexplored strategies, thereby improving the exploration efficiency of the algorithm, reducing the probability of getting trapped in local optima during the optimization process, and also solving the sparse reward problem inherent in machine tool equipment resource optimization. Finally, a dynamic masking mechanism is established to construct a dynamically variable action space, thereby reducing the impact of production disturbances such as machine tool failures on the stability of the algorithm. The improved PPO-RND algorithm of this invention exhibits a more stable training process and more robust perturbation adaptation ability compared with the two baseline algorithms PPO and Parallel DQN and their HER variants in the dynamic reselection stage. In the parallel optimization task of multiple task sequences, its solution result is generally optimal.

[0274] To better illustrate the technical solution of the present invention, this embodiment is described in the following parts.

[0275] I. Constructing the MR-MDP Optimization Process Model

[0276] In this experimental example, combined with Figure 2 As shown, the MR-MDP optimization process model is constructed through the following steps:

[0277] S201: The MR-MDP-oriented Markov decision process model for machine tool equipment resource optimization is a six-tuple MR-MDP =<S,s0,s τ A, P sa ,Reward>.

[0278] Here, s represents the discrete set of environmental states S. t ∈S, s t Record the machine tool resource matching information for all task sequences at time t. Assume the longest task sequence has l sub-tasks, and there are h task sequences in total, then s t It can be represented by the following matrix:

[0279]

[0280] This is the ID of the machine tool resource matched with the i-th subtask MT in the j-th task sequence. If it is 0, it means that there is no match or no corresponding subtask.

[0281] Let s0∈S be the initial state. At the start of each training round, the agent will begin executing service composition from this state. Generally, at the start of a training round, all subtasks in the task sequence are in an unmatched state, therefore:

[0282] s0 = 0 h×l

[0283] s τ For the set of terminating states, when the agent's state reaches s τ When any state in the set is reached, the training round ends and the service composition terminates. τ Related to the resource matching status in the task sequence, the current state is s when all task sequences have completed resource matching. τ .

[0284] S202: A is the action space set, A = A(s0)∪A(s1)∪A(s2)…∪A(s... n ), where A(s) t ) represents state s t The set of actions that an intelligent agent can take, and this set is related to s t Candidate Machine Tool Equipment Resource Set (MRS) in State t Form a mapping. Assume state s t When the matching begins for the i-th subtask MT in the j-th task sequence, the corresponding machine tool equipment resource set is MRS. (j) (s t )={MR (j) i_1,MR (j) i_2,…,MR (j) Let i_k} be the number of machine tools, then MRS t ={MRS (1) (s t ), MRS (2) (s t ),…,MRS (h) (s t MRS under the influence of production disturbances t This may change, thereby causing A(s) t ) has changed.

[0285] Therefore, at time t, the agent, based on the current state s t Execute action a t After matching each task sequence once, it enters state s. t+1 .

[0286] S203:P sa Let P be the state transition probability, which represents the probability distribution of transitioning to another state s' after performing action a in the current state s. It is generally denoted as p(s'|s,a). In the context of the machine tool resource optimization problem, since the agent receives feedback based on the resource allocation result at each decision moment, and calculates the reward r through this feedback, P... sa It is denoted as p(s',r|s,a).

[0287] S204: Reward is the reward function. When action a is performed in state s and the state transitions to another state s', the reward r = Reward(s'|s,a) can be calculated.

[0288] II. Reward Function

[0289] In this embodiment, the reward function is constructed through the following steps:

[0290] S301: First, calculate the comprehensive QoS index corresponding to the task sequence through the QoS attribute, and establish the corresponding reward function branch considering the multi-objective requirements and constraints of the four indicators: service time, service cost, product qualification rate, and service reliability.

[0291] S302: The reward function is constructed using a linear weighting method.

[0292] Combination Figure 3 As shown, the reward function is constructed through the following steps:

[0293] S3011: The calculation of the four service quality indicators of service time, service cost, product qualification rate and service reliability of subtasks in the task sequence should follow the calculation method of the corresponding structure. The specific calculation method is shown in Table 1.

[0294] S3012: Based on the table above, the comprehensive index of a certain task sequence regarding service time, service cost, product qualification rate, and service reliability can be calculated as: QoS (j) =(T (j) C (j) ,P (j) ,R (j) )

[0295] S3013: Considering the significant differences in dimensions and magnitudes among different indicators, direct analysis and comparison are not possible; therefore, normalization is required for each indicator.

[0296]

[0297] After normalization, all QoS indicators can be limited to the range of [0,1], and the monotonicity of different QoS indicators can reflect the multi-objective optimization requirements of minimizing service time, minimizing service cost, maximizing qualified products, and maximizing service reliability.

[0298] Customer-specific needs include both multi-objective optimization requirements and constraints, and the mathematical definition of the constraints is as follows:

[0299] T≤T max

[0300] C≤C max

[0301] P≥P min

[0302] R≥R min

[0303] S3014: To guide the agent in exploring optimal results that better meet the constraints, an appropriate penalty for exceeding the limits can be introduced into the reward function. Specifically, a smooth window function can be constructed to achieve a sharp reduction in rewards when constraints are exceeded.

[0304] Suppose there is an upper limit constraint q for a certain Qos metric q. upper Or lower bound constraint q lower A window function can then be constructed using the Sigmoid function:

[0305]

[0306] In this window function, 'a' represents the decay rate, which is primarily determined by customer requirements.

[0307] S3014: Construct a reward function branch for each Qos metric:

[0308]

[0309] Where B is the baseline value, used to adjust the overall difference between function values ​​inside and outside the target range. When B is large, the function values ​​within the target range are generally higher than those outside the target range. δ is the deviation value, a constant, used to adjust the overall deviation of the function.

[0310] Table 1. Calculation Methods for the Four Service Quality Indicators

[0311]

[0312] Note: W stands for Machine Tool Equipment Resource Service Number of times it is looped

[0313] In step S302, the reward function is constructed through the following steps:

[0314] S3021: A reward function for a given task sequence can be constructed using a linear weighting method.

[0315]

[0316] S3022: By combining the rewards of all task sequences, a vector representation of the current reward can be constructed:

[0317]

[0318] III. Improvements to the PPO-RND Algorithm

[0319] In this embodiment, the PPO-RND algorithm is improved through the following steps:

[0320] S401: PPO is the main algorithm applicable to both discrete and continuous action spaces. In addition, the RND algorithm is used to reduce the negative impact of sparse reward environment and further enhance the algorithm's exploration ability in high-dimensional space.

[0321] S402: Construct a dynamic action masking mechanism to further improve the algorithm.

[0322] Combination Figure 4 As shown, the steps to improve the PPO-RND algorithm are as follows:

[0323] S4011: PPO defines a policy ratio r t (θ):

[0324]

[0325] Where θ old These are the policy parameters before the update. To prevent r t (θ) deviates too much, so PPO uses a shearing method to restrict the objective function to a small range, usually set to [1-ε, 1+ε], where ε is a small hyperparameter.

[0326] The objective function for PPO shearing is defined as:

[0327]

[0328] in, The dominance estimate is usually calculated using the generalized dominance estimate. clip(r) t (θ), 1-ε, 1+ε) will be the ratio r t (θ) is restricted to the range [1-ε, 1+ε]. This objective function ensures that when r... t When (θ) exceeds the shearing range, the gradient of the objective function will not push the policy to deviate excessively, thus achieving effective control over the policy update magnitude.

[0329] S4012: RND generates intrinsic reward signals by introducing a target network and a prediction network. The target network is a fixed and randomly initialized feedforward neural network f. target Its parameters remain unchanged during training. The prediction network is a trainable feedforward neural network f pred Its goal is to predict the target network's output for the same input state. At each time step t, the agent receives state s. t The target network and the prediction network generate the corresponding outputs respectively:

[0330] o target =f target (s t ),o pred =f pred (s t )

[0331] Intrinsic reward r t int Defined as the mean square error between the two outputs:

[0332]

[0333] The training objective of RND is to minimize the error between the output of the prediction network and the target network, thereby improving the prediction network's ability to predict environmental states. The loss function of the prediction network is L. RND Defined as:

[0334]

[0335] Here, D represents the set of state samples collected from the environment. The parameters of the prediction network are continuously updated using the backpropagation algorithm to minimize the aforementioned loss function.

[0336] Finally, RND and external reward r t ext Combined, forming a comprehensive reward r t :

[0337] r t =r t ext +βr t int

[0338] Here, β is a weighted hyperparameter used to balance the influence of external and internal rewards.

[0339] In step S402, a dynamic action masking mechanism is constructed through the following steps:

[0340] S4021: Dynamically maintain an effective Machine Tool Equipment Resource Service Set (MRS) valid (t), for the action set A(s) at time t t Establish a mapping:

[0341] g:A(s t )×MRS valid (t)→{0,1}

[0342] This mapping represents A(s) t Does action 'a' in the context of a given machine tool resource (MR) relate to a specific valid machine tool resource (MR)? valid If correlated, then g(a,MR) valid ) = 1, otherwise g(a,MR valid ) = 0.

[0343] S4022: Establish a set of legal actions based on the mapping relationship:

[0344]

[0345] The set of illegal actions is:

[0346] A illegal (t)=A(t)\A legal (t)

[0347] S4023: For all illegal actions a∈A illegal (t), set the corresponding Logit value to negative infinity:

[0348]

[0349] S4024: In the Softmax calculation, the probability of an illegal action will be set to zero so that it will not be selected.

[0350]

Claims

1. A method for adaptive optimization of machine tool equipment resources in an industrial interconnected environment based on an improved PPO-RND algorithm, characterized in that, Includes the following steps: 1) Model the machine tool equipment resources and establish a production disturbance model; 2) The process of optimizing machine tool equipment resources is modeled as a six-tuple MR-MDP; 3) Design the reward function for the MR-MDP process of machine tool equipment resource optimization; 4) Construct an improved PPO-RND algorithm framework; 5) The improved PPO-RND algorithm framework is used to solve the machine tool equipment resource optimization process MR-MDP to obtain the resource combination of each machine tool equipment and the corresponding actions. After a production disturbance occurs, the resource combination of each machine tool and the corresponding actions are re-optimized.

2. The adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on the improved PPO-RND algorithm as described in claim 1, characterized in that, When establishing a machine tool equipment resource model, a machine tool equipment resource is modeled as a quadruple (MR), that is: MR = <ID,In,Out,Qos> (1) Qos=(T,C,P,R) (2) In the formula, ID is the identifier of the machine tool equipment resource (MR), In is the input part before processing by the machine tool equipment resource (MR), Out is the output part after processing by the machine tool equipment resource (MR), Qos is the Qos attribute of the machine tool equipment resource (MR); T represents service time; C represents service cost; P represents the pass rate; and R represents service reliability. Production disturbances are modeled as quadruples ED, i.e.: ED=<D r ,D p ,D o ,D d > (3) D r =<d rt ,d rc ,d rp ,d rr > (4) D p =<d p1 ,d p2 ,d p3 > (5) D o =<d ot ,d oc ,d op ,d or > (6) D d =<d d1 ,d d2 ,d d3 ,…,d dz > (7) In the formula, ED represents production disturbance; D r D p D o D d These represent the sets of disturbances related to service demand, service provision, platform operation, and logistics / transportation, respectively; d rt d rc d rp d rr These represent the customer's changes in minimum requirements for service time T, service cost C, product qualification rate P, and service reliability R, respectively; d p1 d p2 d p3 These represent disturbances related to increased machine resources, decreased machine resources, and changes in QoS attributes, respectively; d ot d oc d op d or These represent the weighting factors for service time T, service cost C, qualified product rate P, and service reliability R, respectively; d d1 ,d d2 ,d d3 ,…,d dz This represents z types of logistics path disturbances for both the service provider and the service demander.

3. The adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on the improved PPO-RND algorithm according to claim 1, characterized in that, The machine tool equipment resource optimization process (MR-MDP) is as follows: MR-MDP=<S,s0, s τ ,A,P sa ,Reward> (8) In the formula, MR-MDP represents the machine tool equipment resource optimization process; S represents the discrete set of environmental states; s0 represents the initial state; s τ Indicates the termination state; A is the action space set; P sa Represents the state transition probability; Reward represents the reward function; Among them, the state elements s in the discrete set S of environmental states t As shown below: In the formula, t represents time, s t This represents the state at time t; i represents the subtask index, l represents the total number of subtasks; j represents the task sequence index, and h represents the total number of task sequences. The ID identifier code represents the machine tool equipment resource matched by the i-th subtask in the j-th task sequence; The reward function Reward is as follows: r = Reward(s'|s,a) (10) In the formula, a represents the action, s and s' represent the states before and after the action a is performed, respectively, and r represents the reward after the state s transitions to the state s'.

4. The adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on the improved PPO-RND algorithm according to claim 1, characterized in that, The reward function steps of the MR-MDP process for optimizing machine tool equipment resources are as follows: 3.1) Calculate the comprehensive QoS index corresponding to the historical task sequence through the QoS attribute, and establish reward function branches corresponding to different service quality indicators; The service quality indicators include service time, service cost, product qualification rate, and service reliability. 3.2) The external reward function is obtained by weighting different reward function branches using a linear weighting method.

5. The adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on the improved PPO-RND algorithm according to claim 4, characterized in that, The reward function branches corresponding to the different service quality indicators are shown below: In the formula, j represents the task sequence index; Let T, C, P, and R represent the reward function branches corresponding to the service time T, service cost C, qualified product rate P, and service reliability R of the j-th task sequence, respectively. These represent the maximum and minimum values ​​of the service time T, respectively. These represent the maximum and minimum values ​​of the service cost C, respectively. These represent the maximum and minimum values ​​of the qualified product rate P, respectively; These represent the maximum and minimum values ​​of service reliability R, respectively; T (j) C (j) P (j) R (j) Let T, C, P (service cost), and R (product qualification rate) represent the service time, service cost, and service reliability values ​​of the j-th task sequence, respectively; a1 represents the decay rate. Let T and C represent the upper limit constraints for service time and service cost, respectively. δ represents the lower limit constraints of the qualified product rate P and service reliability R, respectively; B represents the benchmark number; and δ represents the deviation value.

6. The adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on the improved PPO-RND algorithm according to claim 4, characterized in that, The external reward function is as follows: In the formula, Represents the external reward function; j represents the task sequence index, h represents the total number of task sequences; r (j) Represents the external reward of the j-th task sequence; Reward represents the reward function; a represents the action, and s and s' represent the states before and after executing action a, respectively; s τ Indicates the termination state; ω T ω C ω P ω R These represent the weights of service time T, service cost C, qualified product rate P, and service reliability R, respectively. Let T, C, P, and R represent the reward branches corresponding to the service time T, service cost C, qualified product rate P, and service reliability R of the j-th task sequence, respectively.

7. The adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on the improved PPO-RND algorithm according to claim 1, characterized in that, The steps for solving the machine tool equipment resource optimization process MR-MDP using the improved PPO-RND algorithm framework are as follows: 5.1) The comprehensive reward function of the machine tool equipment resource optimization process MR-MDP is calculated based on the improved PPO-RND algorithm; 5.2) Filter out illegal actions in the machine tool equipment resource optimization process MR-MDP based on a dynamic action masking mechanism; 5.3) Input the current task sequence into the machine tool equipment resource optimization process MR-MDP after filtering out illegal actions to obtain the combination of machine tool equipment resources and corresponding actions at the next moment.

8. The adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on the improved PPO-RND algorithm according to claim 7, characterized in that, In step 5.1), the steps for calculating the comprehensive reward function of the machine tool equipment resource optimization process MR-MDP are as follows: 5.1.1) Define the strategy ratio r t (θ) is shown below: In the formula, t represents time, s t Indicates the state at time t; a t θ represents the action at time t; old θ and θ represent the fixed PPO strategy parameters and the PPO strategy parameters to be optimized, respectively. π θ (a t |s t ) represent the fixed PPO strategy parameters θ old The state s under the PPO strategy parameter θ to be optimized t Select action a t The probability of; 5.1.2) Define the shearing objective function of the PPO algorithm, and update the PPO policy parameter θ to be optimized by solving the shearing objective function of the PPO algorithm; The shearing objective function of the PPO algorithm is as follows: In the formula, L CLIP (θ) is the shearing objective function of PPO; For advantage estimation; E t For a period of time Clip function; clip(r) t (θ), 1-ε, 1+ε) are the constraint functions, and the policy ratio r t (θ) is restricted to the range [1-ε, 1+ε]; ε is a hyperparameter; 5.1.3) By introducing a target network and a prediction network, intrinsic rewards are generated. 5.1.4) Combine with external rewards and intrinsic rewards Receive comprehensive reward r t As shown below: r t ＝r t ext +βr t int (16) In the formula, β is the weight hyperparameter.

9. The adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on the improved PPO-RND algorithm according to claim 8, characterized in that, The process involves introducing a target network and a prediction network to generate intrinsic rewards. The steps are as follows: 5.1.3.1) Constructing the target network f target and prediction network f pred ; The target network f target and prediction network f pred Both are feedforward neural networks, and both take the state as input and the state features as output; 5.1.3.2) Define intrinsic reward As shown below: In the formula, t represents time, o target o pred They represent the target network f respectively. target Predictive network f pred The output characteristics; Represents the mean square error function; 5.1.3.3) Construct the loss function for the prediction network as follows: In the formula, L RND To predict the loss function of the network, s t D represents the state at time t; D represents the state sample set. Let it be the expected function; 5.1.3.4) Update the parameters of the prediction network using the backpropagation algorithm to minimize the loss function of the prediction network; The parameters of the prediction network include the prediction network f. pred Output feature o pred Target network f target Output feature o target ; 5.1.3.5) Based on the updated prediction network parameters, the intrinsic reward is calculated.

10. The adaptive optimization method for machine tool equipment resources in an industrial interconnected environment based on the improved PPO-RND algorithm according to claim 7, characterized in that, The steps for filtering out illegal actions in the machine tool equipment resource optimization process (MR-MDP) based on the dynamic action masking mechanism are as follows: 5.2.1) Establish an effective mapping between machine tool equipment resources and actions, as shown below: g:A(s t )×MRS valid (t)→{0,1} (19) In the formula, t represents time, s t Represents the state at time t; g is the mapping relationship between effective machine tool equipment resources and actions; A(s) t ) represents the action set at time t; MRS valid (t) represents the set of effective machine tool equipment resources at time t; 5.2.2) Based on the mapping relationship between effective machine tool equipment resources and actions, establish the legal action set and the illegal action set, as shown below: A illegal (t)=A(s t )\A legal (t) (21) In the formula, A legal (t), A illegal (t) represents the set of legal actions and the set of illegal actions at time t, respectively; a represents an action; MR represents machine tool equipment resources; g(a,MR) represents the mapping relationship between action a and machine tool equipment resources MR. 5.2.3) Set the probability of all illegal actions in the illegal action set to zero, and filter out illegal actions from the MR-MDP optimization process model, as shown below: In the formula, s represents the state, a represents the action, and a illegal Indicates an illegal action, π(a) illegal |s) represents the probability of choosing an illegal action; Logit(·) is the Logit value of the action; A legal (t), A illegal (t) represents the set of legal actions and the set of illegal actions at time t, respectively.

Images