A self-similar fractal reconstruction system for unmanned production lines based on CPS manufacturing component self-similar modeling and self-organizing hierarchical aggregation methods.

Abstract

The present invention discloses a self-similar modeling and self-organizing hierarchical aggregation method based on CPS manufacturing components and a self-similar fractal reconstruction system for unmanned production lines. First, order task information is obtained; when the current production line state cannot complete the order task, the current unmanned production line is decoupled to obtain CPS manufacturing components of different levels; the order task is decomposed into several sub-processing tasks; a multi-level and multi-granular CPS manufacturing component that can complete the order sub-task is formed; the order task and the CPS manufacturing component are matched and combined to obtain a set of CPS manufacturing components that meet the requirements in the current state, forming an order-customized reconstruction production line. The method provided by the present invention is a method and system based on hierarchical digital twins, which realizes a self-similar fractal reconstruction manufacturing system driven by virtual-real fusion of man, machine, and object according to self-similar fractal theory and digital twin technology, so as to realize adaptive and rapid reconstruction of the production line when the current production line cannot meet production needs.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing technology, and in particular to a self-similar modeling and self-organizing hierarchical aggregation method based on CPS manufacturing components and a self-similar fractal reconstruction system for an unmanned production line. Background Art

[0002] With the development of social economy, the changes of diversified and personalized customer demands (cars, water heaters, etc.) are also getting faster and faster, which makes the types and quantities of product market demands grow, fluctuate and change rapidly. However, the existing manufacturing system can no longer adapt to the current rapidly changing market situation. In addition, the current production line that lacks autonomous reconstruction capabilities will inevitably have excess or insufficient production capacity when processing production tasks, resulting in unreasonable use of production costs and unnecessary idle waste. In addition, with the skyrocketing increase in labor costs and the consideration of personnel safety factors in extreme working environments (high temperature, high pressure, hypoxia, etc.), production lines are gradually deploying more information technology and AI technology, showing a trend of unmanned development driven by big data. In this context, in order to achieve refined perception and control of the production process, it is essential to consider the virtual and real fusion information of people (expert knowledge information), machines (process information model in computers), and objects (production equipment information) in the reconstruction process. Therefore, the autonomous dynamic reconstruction capability of unmanned production lines driven by the virtual and real fusion of man, machine and object is crucial.

[0003] Therefore, in order to quickly meet the personalized and diversified needs of customers and improve the market competitiveness of enterprises, it is imperative to study reconfigurable unmanned autonomous manufacturing systems.

[0004] However, there are some manufacturing systems with reconfigurable capabilities, such as reconfigurable manufacturing systems (RMS), daughter cell manufacturing systems (HMS), fractal manufacturing systems (FrMS), and virtual cell manufacturing systems (VCM). However, these systems do not consider the virtual-real fusion information of man, machine, and object in the unmanned production process, and cannot be applied to the autonomous reconstruction process of unmanned production lines with virtual-real fusion of man, machine, and object, let alone achieve high abstraction and natural decoupling of unmanned processing production lines. Summary of the invention

[0005] In view of this, the object of the present invention is to provide a self-similar modeling and self-organizing hierarchical aggregation method based on CPS manufacturing components and a self-similar fractal reconstruction system for unmanned production lines.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] The CPS-based self-similar modeling method for manufacturing components provided by the present invention comprises the following steps:

[0008] In the digital twin model, the unmanned production line is deconstructed according to the fractal principle and self-similarity to form a set of basic fractal components, and the basic fractal components are autonomously hierarchically aggregated to obtain CPS manufacturing components;

[0009] Obtaining a physical model corresponding to a CPS manufacturing component and storing the physical model in a database;

[0010] Set the spatiotemporal information update factor γ of the CPS manufacturing component information model to update the model attributes in a timely manner;

[0011] Setting the attribute interface of the CPS manufacturing component; the attribute interface includes any one or a combination of input attributes, model attributes, and output attributes;

[0012] The input attribute is used to determine the input information;

[0013] The model attributes are used to determine model parameters;

[0014] The output attribute is used to determine the output information.

[0015] Further, the model attributes include any one or a combination of multiple items of model type, mechanism, constraint, and control;

[0016] The model type is used to determine any one or a combination of a three-dimensional model, an associated model, and an environmental model of the device;

[0017] The mechanism is used to determine the processing mechanism and / or evolution law;

[0018] The constraint is used to determine any one or a combination of time constraints, environmental constraints, logical constraints, and relational constraints;

[0019] The control is used to determine any one or a combination of multiple items among a control model, a control algorithm, and a control accuracy.

[0020] The present invention provides a self-similar fractal architecture method for an unmanned production line implemented by using the CPS manufacturing component service combination described in any one of claims 1 to 2, wherein the formation mode of the CPS manufacturing component and its self-organization aggregation mode are both self-similar; the method comprises the following steps:

[0021] Get order task information;

[0022] When the current production line status cannot complete the order task, the processing tasks at each level are decomposed into the next level of sub-processing task sets based on the order information;

[0023] Decouple the current unmanned production line to obtain the lowest granularity CPS manufacturing components;

[0024] Used to form a candidate service set of CPS manufacturing components for a specific processing task based on the sub-processing task sets of each layer;

[0025] Each CPS manufacturing component aggregation unit corresponds to a processing task of the previous layer; according to the processing task of the previous layer, the current layer CPS manufacturing component self-organizes multiple times based on DPD-DQN to form a current layer CPS manufacturing component service combination that completes the processing task of the previous layer, that is, the CPS manufacturing component of the previous layer; after multiple aggregations of the current layer CPS manufacturing components self-organize and aggregate, a CPS manufacturing component candidate service set of a processing task of the previous layer is formed; the self-organization aggregation methods of the CPS manufacturing components of each layer are self-similar;

[0026] Match the processing tasks at each level and their required CPS manufacturing components to obtain a set of CPS manufacturing components that meet the requirements in the current state, forming an order-customized reconstruction production line;

[0027] Order-customized reconfiguration of production line generation units.

[0028] Furthermore, the CPS manufacturing component layer-by-layer self-organization aggregation process is carried out according to the following steps:

[0029] A low-level self-organizing aggregation model of CPS manufacturing components is constructed. Optimization strategies are introduced to address the overfitting, simple random sampling, and poor learning effects of DQN. DPD-DQN is proposed to solve the single-layer self-organizing aggregation model of CPS manufacturing components. The self-organizing aggregation model of CPS manufacturing components is solved layer by layer using DPD-DQN until a hierarchical CPS manufacturing component that can complete the order subtask is obtained. A multi-level autonomous aggregation model of CPS manufacturing components is formed.

[0030] The parameters in the DPD-DQN are defined as follows:

[0031] In the traditional DQN, there are two neural networks with the same structure and parameter updates with time difference, namely:

[0032] Q_eval(Q 估计 ) and Q_next(Q 现实 );

[0033] Neural network hyperparameters:

[0034] Learning rate α, minimum batch size b, sampling weight β, discount factor γ, ε;

[0035] Indicates the Q in the original DQN 现实 Neural network update methods; represents Q using optimization strategy 1 现实Neural network update method; S represents the state set of the environment; s represents a specific state; a represents a specific action in the action set taken by the agent; r represents the immediate return reward; γ represents the discount factor; R t+1 represents the reword value of the t+1th generation; θ represents the neural network parameter; Indicates the selection of the next state S t+1 The maximum Q value of an action; Q(s,a;θ,α,β) represents the Q value of state s when taking action a and the neural network parameters are (θ,α,β); V(s;θ,β) represents the state value of state s with (θ,β) neural network parameters; A(s,a,θ,α) represents the additional reward for state s to perform a specific action a.

[0036] Furthermore, the traditional DQN uses a deep network to approximate the current value function and applies another deep network with the same structure but different parameters to generate the target Q value (Q 现实 / Q_target), the Q value is calculated and updated according to the following formula:

[0037]

[0038] The DPD-DQN adopts multiple optimization strategies to prevent overfitting. 估计 The neural network computes action a.

[0039] Furthermore, the optimization strategy includes any one or a combination of any two or more of the following strategies:

[0040] Optimization strategy 1: Use Q to prevent overfitting 估计 The neural network calculates action a, and the Q value calculation formula is revised as follows.

[0041]

[0042] Optimization strategy 2: Sampling strategy considering the importance of experience data Extract experience data according to the importance of DQN experience data, as follows: Apply the SumTree method to form a tree structure of experience data, each leaf stores the priority of each sample, each branch node has only two branches, and the node value is the sum of the two branch values; when sampling, divide the sum of p into batch_size intervals (n=sum(p) / batch_size), and then randomly select a number in each interval;

[0043] Optimization strategy 3: During the model optimization process, the Q value update method of each action of DQN is revised to the following formula:

[0044] Q(s,a;θ,α,β)=V(s;θ,β)+A(s,a;θ,α).

[0045] Furthermore, the DPD-DQN solves the self-organizing aggregation model of each layer of CPS manufacturing components according to the following steps:

[0046] Initialize the DPD-DQN neural network parameters, including the learning rate α, the minimum batch size b, the sampling weight β, the discount coefficient γ, and the greedy strategy (ε-greedy) probability ε; input the TQCS data of the bottom-level CPS manufacturing component set;

[0047] Enter the training process:

[0048] In each training round, each agent selects a behavior; multiple agents form a joint action a; the environment executes the behaviors in the joint action set and updates the state for each action; after all actions are executed, the total state of the system s is updated; the immediate reward r is calculated; the storage s, a, r, s is updated; when the experience pool overflows, the learning mechanism is started; and Q is updated in a fixed step period 现实 (Q_target) neural network parameters;

[0049] After a fixed number of training times, the optimal CPS manufacturing component combination of a subtask in this layer is obtained. Then, according to the Furutaguchi experimental method, the target weight value is changed according to the four levels of multiple factors, and retraining is performed to obtain the corresponding optimal CPS manufacturing component combination; and so on, the candidate set of CPS manufacturing components for a subtask in the upper layer is obtained;

[0050] DPD-DQN is applied to obtain the candidate service set of CPS manufacturing components for each subtask of the upper level, and so on, until a hierarchical combination of CPS manufacturing components that can complete the order production task is obtained.

[0051] The self-similar fractal reconstruction system for an unmanned production line provided by the present invention utilizes the self-similarity of the formation mode and self-organization aggregation mode of CPS manufacturing components to construct a self-similar fractal reconstruction system for an unmanned production line, specifically including

[0052] An order management unit, used to obtain order task information;

[0053] When the current production line status cannot complete the order task, the order processing task is decomposed into units to decompose each layer of processing tasks into the next level of sub-processing task sets;

[0054] The CPS manufacturing component generation unit is used to decouple the current unmanned production line to obtain low-granularity CPS manufacturing components;

[0055] The CPS manufacturing component classification unit is used to form a candidate service set of CPS manufacturing components for a specific processing task based on the sub-processing task sets of each layer;

[0056] CPS manufacturing component aggregation unit, each CPS manufacturing component aggregation unit corresponds to a processing task of the previous layer; according to the processing task of the previous layer, the current layer CPS manufacturing component self-organizes multiple times based on DPD-DQN to form a current layer CPS manufacturing component service combination that completes the processing task of the previous layer, that is, the CPS manufacturing component of the previous layer; after multiple aggregations of the current layer CPS manufacturing components self-organize and aggregate, a CPS manufacturing component candidate service set of a processing task of the previous layer is formed; obviously, the self-organization aggregation methods of CPS manufacturing components at each layer are self-similar;

[0057] The order-customized reconstruction production line generation unit is used to match the processing tasks at each level and the CPS manufacturing components they require to obtain a hierarchical and multi-granular set of CPS manufacturing components that meet the requirements in the current state, forming an order-customized reconstruction production line.

[0058] Further, the CPS manufacturing component aggregation unit includes a CPS manufacturing component aggregation fractal model based on DPD-DQN and a weight parameter adjustment method based on Furutaguchi;

[0059] The improved DQN self-organizing hierarchical aggregation model generation unit is used to construct a self-organizing aggregation model of CPS manufacturing components at each layer. The DPD-DQN-based CPS manufacturing component self-organizing aggregation fractal model introduces optimization strategies for the overfitting, simple random sampling, and poor learning effect of DQN.

[0060] The parameters in the DPD-DQN are defined as follows:

[0061] The traditional DQN has two neural networks with the same structure and time difference in parameter update:

[0062] Q_eval(Q 估计 ) and Q_next(Q 现实 );

[0063] Neural network hyperparameters: learning rate α, minimum batch size b, sampling weight β, discount factor γ, ε; Indicates the Q in the original DQN 现实 Neural network update methods; represents Q using optimization strategy 1 现实 Neural network update method; S represents the state set of the environment; s represents a specific state; a represents a specific action in the action set taken by the agent; r represents the immediate return reward; γ represents the discount factor; R t+1 represents the reword value of the t+1th generation; θ represents the neural network parameter; Indicates the selection of the next state S t+1Q(s,a;θ,α,β) represents the Q value of state s when taking action a and the neural network parameters are (θ,α,β); V(s;θ,β) represents the state value of state s with (θ,β) neural network parameters; A(s,a,θ,α) represents the additional reward for state s to perform a specific action a;

[0064] The traditional DQN uses a deep network to approximate the current value function and applies another deep network with the same structure but different parameters to generate the target Q value, i.e., Q 现实 / Q_target, the Q value is calculated and updated according to the following formula:

[0065]

[0066] The DPD-DQN adopts multiple optimization strategies to prevent overfitting. 估计 The neural network computes action a.

[0067] Furthermore, the optimization strategy includes any one or a combination of any two or more of the following strategies:

[0068] Optimization strategy 1: Leverage Q 估计 The neural network calculates action a, and the Q value calculation formula is revised as follows.

[0069]

[0070] Optimization strategy 2: Sampling strategy based on the importance of empirical data Extract empirical data according to the importance of DQN empirical data, as follows: Apply the SumTree method to form a tree structure of empirical data, each leaf stores the priority of each sample, each branch node has only two branches, and the node value is the sum of the two branch values; when sampling, divide the sum of p into batch_size intervals, that is, n = sum(p) / batch_size, and then randomly select a number in each interval;

[0071] Optimization Strategy 3: During the model optimization process, the Q value update method for each action of DQN is revised to the following formula:

[0072] Q(s,a;θ,α,β)=V(s;θ,β)+A(s,a;θ,α);

[0073] The DPD-DQN solves the self-organizing aggregation model of each layer of CPS manufacturing components according to the following steps:

[0074] Initialize the DPD-DQN neural network parameters, including the learning rate α, the minimum batch size b, the sampling weight β, the discount factor γ, the greedy strategy (ε-greedy) probability ε, etc.; input the TQCS data of the bottom-level CPS manufacturing component set;

[0075] Enter the training process:

[0076] In each training round, each agent selects a behavior; multiple agents form a joint action a; the environment executes the behaviors in the joint action set and updates the state for each action; after all actions are executed, the total state of the system s is updated; the immediate reward r is calculated; the storage s, a, r, s is updated; when the experience pool overflows, the learning mechanism is started; and Q is updated in a fixed step period 现实 (Q_target) neural network parameters;

[0077] After a fixed number of training times, the optimal CPS manufacturing component combination of a subtask in this layer is obtained. Then, according to the Furutaguchi experimental method, the target weight value is changed according to the four levels of multiple factors, and retraining is performed to obtain the corresponding optimal CPS manufacturing component combination; and so on, the candidate set of CPS manufacturing components for a subtask in the upper layer is obtained;

[0078] DPD-DQN is applied to obtain the candidate service set of CPS manufacturing components for each subtask of the upper level; based on the self-similar fractal characteristics of the self-organizing aggregation model of CPS manufacturing components, and so on, until a hierarchical CPS manufacturing component combination that can complete the order production task is obtained.

[0079] The beneficial effects of the present invention are:

[0080] The method and system based on self-similar modeling and fractal architecture of CPS manufacturing components provided by the present invention is a self-organizing self-similar fractal reconstruction method and system for unmanned production lines based on hierarchical digital twins. According to the self-similar fractal theory and digital twin technology, a self-similar fractal reconstruction manufacturing system (DTFRMS) for unmanned production lines based on hierarchical digital twins driven by the virtual-real fusion of man, machine and object is proposed to realize adaptive and rapid reconstruction of the production line when the current production line cannot meet production needs.

[0081] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0083] Figure 1 Schematic diagram of the self-similar fractal architecture of the unmanned production line.

[0084] Figure 2 Schematic diagram of DPD-DQN solving the autonomous hierarchical aggregation model of CPS manufacturing components.

[0085] Figure 3 Schematic diagram of the graph training process. DETAILED DESCRIPTION

[0086] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0087] Example 1

[0088] The method for rapid homogeneous organic modeling of self-similar CPS manufacturing components of unmanned production lines provided in this implementation case (when the current production line status cannot complete the order task) is based on fractal theory and the self-similarity of unmanned production lines / equipment / production processes. The equipment, manufacturing process, collaboration process, and control process of unmanned production lines are highly abstracted into self-similar manufacturing fractal units; wherein the fractal unit is provided with three attributes: input attributes (input objects, process parameters, auxiliary parameters, input actions, etc.), model attributes, and output attributes (completion time, performance, output objects, etc.). The model attributes include four sub-attributes, as follows: model (equipment 3D model, association model, environmental model, etc.), mechanism (processing mechanism, evolution law, etc.), constraint (time constraint, environmental constraint, logical constraint, relationship constraint, etc.), control (control model, control algorithm, control accuracy, etc.); as long as the three attributes of the fractal unit are determined, the heterogeneous fractal unit can be specifically constructed. In addition, the fractal unit can also be evolved and expanded to the upper layer, current layer, and lower layer through the fusion of the three attributes to form CPS manufacturing component models of different granularities, that is, multi-level and multi-granular CPS manufacturing components.

[0089] The physical model and information model in the CPS manufacturing component in this embodiment can be interactively controlled in real time and bidirectionally. The method for rapid homogeneous organic modeling of self-similar CPS manufacturing components in an unmanned production line includes the following steps:

[0090] Step 1: In the digital twin model, the unmanned production line is deconstructed at the component level according to the fractal principle to form a basic fractal component set, and a multi-agent deep reinforcement learning method is used to realize the hierarchical and autonomous aggregation of CPS manufacturing components;

[0091] Step 2: Map the CPS manufacturing components in the digital twin model to their physical models one by one, and then collect real-time signals from the physical model of the fractal components based on protocols such as PLC / WLAN / 5G / PROFINET, and then store them in the database;

[0092] Step 3: Use Navicat to dynamically manage the database. Based on the data in the database, update the model properties in Unity in a timely manner to dynamically display the CPS manufacturing components in real time. Set the spatiotemporal information update factor γ (γ < 5) of the CPS manufacturing component information model to represent the γth CPS manufacturing component input information update.

[0093] Step 4: Build real-time communication between the matlab / pycharm platform and the database, and build a CPS manufacturing component information model including model principles, control methods, processing mechanisms, constraints, etc. in the matlab / pycharm environment;

[0094] Step 5: Based on the model in step 4, determine the input and output of the CPS manufacturing component and read and write database data as appropriate;

[0095] Step 6: Update the factor γ according to the spatiotemporal information, record all CPS manufacturing component information, and if the spatiotemporal information update factor γ = 5, overwrite the previous information in sequence;

[0096] According to the above CPS manufacturing component modeling method, the current fractal can be constructed downward. By analogy, the current fractal can still be divided into several CPS manufacturing component processing sub-fractal units. Through the rapid bridging combination of three attribute interfaces, the fractal deconstruction of the unmanned processing production line can be quickly completed, and a hierarchical CPS manufacturing component model of different granularities can be constructed.

[0097] like Figure 1 As shown, Figure 1 It is a self-similar fractal architecture of unmanned production line. Since the production type of personalized orders in manufacturing industry is usually single-piece small batch or multi-variety small batch production. This type of order is either a special product with one-time demand, such as heavy machinery manufacturing, large power generation equipment manufacturing, ocean-going ships, etc., or a small batch production order with many product varieties. These cannot maintain long-term contact production. Therefore, it is not worthwhile to spend a lot of time and energy to build a special unmanned production line.

[0098] The self-similar fractal architecture method for the unmanned production line provided in this embodiment is a self-similar fractal architecture for the unmanned production line driven by personalized order tasks, which completes production tasks in a dynamic and adaptive manner. After receiving a personalized order, the self-organizing self-similar fractal reconstruction system of the unmanned production line will first analyze whether the current production line production capacity meets the production task of the order. If it does not meet the requirements, it is necessary to organically and homogeneously decouple the unmanned production line according to the above-mentioned fractal method to form multi-granular CPS manufacturing components; then the system decomposes the order production task into sub-processing tasks; then based on the self-similar fractional characteristics of the CPS manufacturing component self-organizing aggregation method, starting from the lowest-level CPS manufacturing component (fractal unit), self-organize and aggregate upward layer by layer to form CPS manufacturing components that can complete the order sub-tasks; then the self-organizing self-similar fractal reconstruction system of the unmanned production line matches and combines and optimizes the order processing tasks and CPS manufacturing component services to obtain the optimal customized CPS manufacturing component set in the current state, that is, the production line customized and reconstructed for the personalized order.

[0099] Self-organizing hierarchical aggregation model of CPS manufacturing components

[0100] (1) A large number of multi-granularity and multi-level CPS manufacturing component services are constructed through the rapid homogeneous organic modeling method of self-similar CPS manufacturing components in unmanned production lines. How to autonomously aggregate a large number of small-granularity CPS manufacturing components in each layer to form larger-granularity CPS manufacturing components is crucial for the self-organizing self-similar fractal reconstruction system of unmanned production lines; obviously, this is a multi-level combinatorial optimization problem. In addition, the self-organizing aggregation method of CPS manufacturing components in each layer has self-similarity. The parameters of the single-layer CPS manufacturing component self-organizing aggregation model based on the multi-agent reinforcement learning model (DPD-DQN) are defined below, mainly including multi-agent, environment, state set, and action set.

[0101] 1) Environment variables

[0102] Important parameters:

[0103] T ij : represents the processing time of the i-th CPS manufacturing component in the candidate service set of the j-th processing task;

[0104] Q ij : represents the processing quality of the i-th CPS manufacturing component in the candidate service set of the j-th processing task;

[0105] C ij : represents the processing cost of the i-th CPS manufacturing component in the candidate service set of the j-th processing task;

[0106] S ij: represents the service quality evaluation of the i-th CPS manufacturing component in the candidate service set of the j-th processing task;

[0107] LC ij : represents the logistics cost from the i-th CPS manufacturing component location to the j-th CPS manufacturing component location;

[0108] LT ij : represents the logistics time from the i-th CPS manufacturing component location to the j-th CPS manufacturing component location;

[0109] Location ij : The physical location of the i-th CPS manufacturing component in the candidate service set of the j-th processing task;

[0110] Optimization goal:

[0111] In the DPD-DQN model, TQCS is used as the optimization target;

[0112] Obj=w1*T n +w2*Q n +w3*C n +w4*S n

[0113] is the total number of processing tasks

[0114] is the total number of processing tasks

[0115] is the total number of processing tasks

[0116] is the total number of processing tasks

[0117] Where Obj is the target value; T n , Q n , C n and S n They are the total time, total processing quality, total processing cost and total service quality corresponding to the CPS manufacturing component service combination. w1, w2 and w3 are the weight coefficients of total time, total processing quality, total processing cost and total service quality respectively;

[0118] State space (State) and action space (Action) settings:

[0119] The CPS manufacturing component aggregation model at the layer level can be modeled as a Markov process, in which a CPS manufacturing component service is selected for each subtask until the CPS manufacturing component service selection for all subtasks is completed.

[0120] In this embodiment, each process of selecting a CPS manufacturing component for a subtask is regarded as an agent. Obviously, each agent has two states, one in which an action has occurred and the other in which an action has not occurred. There are n agents at the bottom layer. And each agent selects a CPS manufacturing component from the CPS candidate service set, which is regarded as the action of the agent. The execution of an action by the agent will change the corresponding CPS manufacturing component state (select 1 and not select 0). As the CPS manufacturing component service combination proceeds (for example, the CPS manufacturing component service selection moves from one agent to another agent decision), the state of the subtask and the state of the CPS manufacturing component service will change. All of the above process states constitute the state Sn+1,m of the entire CPS manufacturing component service combination. The system state space consists of the agent state space (the first row of Sn+1,m) and the state space of all CPS manufacturing component services (Sn+1,m non-first row).

[0121] The agent responsible for selecting the CPS manufacturing component for each subtask has an action space action, the size of which is consistent with the size of the candidate service set of the CPS manufacturing component. That is, the action taken by the agent is to select the CPS manufacturing component in the corresponding candidate set (selection is 1, otherwise it is 0).

[0122] Reward function: The reward function plays a key role in guiding DRL to search for the best CPS manufacturing component service composition solution. The reward is calculated based on the standardized QoS value as shown in the following formula.

[0123] reward=w1*Tnor+w2*Qnor+w3*Cnor+w4*Snor;

[0124] Among them, Tnor, Qnor, Cnor and Snor are the normalized total processing time, processing quality, processing cost and service quality of the subtask respectively.

[0125] 4) Improvement of DQN algorithm

[0126] Metaheuristic algorithms (such as genetic algorithms and whale optimization algorithms) are commonly used algorithms to solve service composition problems. However, these algorithms not only require the design of complex framework processes, but also require difficult parameter adjustments. In addition, metaheuristic algorithm models lack adaptability to dynamic environments. Multi-agent deep reinforcement learning (MADRL) provides a dynamic robustness approach to solving the CPS manufacturing component service composition (FMfg-CPSMCSC) problem under the fog manufacturing paradigm. Deep reinforcement learning is a model-free artificial intelligence method that enables the system to learn the optimal CPS manufacturing component service composition solution through training.

[0127] This embodiment is dedicated to the application exploration of multi-agent reinforcement learning MARDRL in FMfg-CPSMCSC, and designs a double dueling deep Q-network (Double Priortized-replay Durling DQN, DPD-DNQ) algorithm with prioritized replay based on multi-agent joint decision-making.

[0128] Traditional DQN uses deep neural networks to approximate the q function, and the agent interacts with the environment to maximize future rewards. DQN solves this problem by introducing an experience replay mechanism and a separate target network. The introduction of the former breaks the correlation between samples. At each time step, the transformation obtained by the interaction between the agent and the environment is stored in an experience replay buffer. During the training process, small batches of samples are randomly extracted from the experience replay buffer, and the q-network parameters θ are updated using the stochastic gradient descent (SGD) algorithm. In addition, DQN uses a deep network to approximate the current value function, and another deep network with the same structure but different parameters to generate the target Q value (Q_Target). The DPD-DQN adopts multiple optimization strategies to prevent overfitting, and calculates the action a through the Q estimation neural network; the importance of data is considered in the sampling process of the traditional DQN experience replay mechanism; at the same time, the influence of the state and the action taken is emphasized; the Q reality calculation formula in DQN is as follows.

[0129]

[0130] Optimization strategy 1: Since the neural network predicts Qmax with errors, and each time the network is improved towards the Q reality with the maximum error, overfitting will occur. To avoid this, Double-DQN introduces another neural network to offset the impact of the maximum error. 估计 The neural network estimates the action a in equation (1), i.e., Q 现实 The calculation formula is revised as follows.

[0131]

[0132] Optimization strategy 2: Sampling strategy that considers the importance of experience data. Just like human memory has different levels of importance, DQN experience data also has different levels of importance. That is, when extracting experience data, extract it according to its importance (Prioritized-replay) to replace the previous random sampling, give priority to learning important experience data, and effectively improve learning efficiency. The SumTree method is used to form a tree structure of experience data. Each leaf stores the priority of each sample. Each branch node has only two branches, and the node value is the sum of the two branch values. When sampling, the sum of p is divided into batch_size intervals (n=sum(p) / batch_size). Then a number is randomly selected in each interval.

[0133] Optimization Strategy 3: The core idea of ​​During-DQN is that the Q value Q(s,a) can be divided into the value of this state and the advantage of each action in this state. In the process of model optimization, there are some states where no matter what action is taken, it will not affect the next state. To simulate this situation, the influence of the state and the action taken are considered separately, as shown in the following formula. Because sometimes in a certain state, no matter what action is taken, it will not have much impact on the next state.

[0134] Q(s,a;θ,α,β)=V(s;θ,β)+A(s,a;θ,α) (3)

[0135] From this formula, we can see that changes in either the state value or the action value of this state will affect the Q value, which helps improve the learning efficiency of the model; the advantage can be positive or negative, and in general, the size of the advantage can be arbitrary. The obvious separation of Q values ​​and the advantages in network structure make the training more stable and converge faster.

[0136] 5) Propose DPD-DQN

[0137] like Figure 2 As shown, Figure 2 In order to solve the CPS manufacturing component autonomous hierarchical aggregation model with self-similar fractal characteristics by DPD-DQN (having self-similar fractal characteristics), based on the above three DQN optimization strategies, this embodiment designs a DPD-DQN to solve the CPS manufacturing component autonomous hierarchical aggregation model with self-similar fractal characteristics, including the following steps:

[0138] Step 1: Initialize the parameters of the DPD-DQN neural network, including the learning rate α, the minimum batch size b, the sampling weight β, the discount coefficient γ, the greedy strategy (ε-greedy) probability ε, etc.; input the TQCS data of the bottom-level CPS manufacturing component set;

[0139] Step 2: Enter training;

[0140] Step 3: In each training round, each agent selects a behavior; multiple agents form a joint action a; the environment executes the behaviors in the joint action set and updates the state for each action; after all actions are executed, the total state of the system s is updated; the immediate reward r is calculated; the storage s, a, r, s is updated; the learning mechanism is started when the experience pool overflows; and Q is updated in a fixed step period 现实 (Q_target) neural network parameters;

[0141] Step 4: After a fixed number of training times, the optimal CPS manufacturing component combination of a subtask in this layer is obtained. Then, according to the Furutaguchi experimental method, the target weight value is changed according to the four levels of multiple factors, and retraining is performed to obtain the corresponding optimal CPS manufacturing component combination; and so on, the candidate set of CPS manufacturing components for a subtask in the upper layer is obtained;

[0142] Step 5: Apply DPD-DQN to obtain the candidate service set of CPS manufacturing components for each upper-level subtask; based on the self-similar fractal characteristics of the CPS manufacturing component self-organizing aggregation model, and so on, until a hierarchical CPS manufacturing component combination that can complete the order production task is obtained.

[0143] 6) CPS manufacturing component autonomous aggregation example

[0144] Since the self-organizing aggregation process of each layer of CPS manufacturing components in this method has self-similarity, a practical application case of a scale of 50_4 (where “50” is the number of subtasks and “4” means that each subtask has 4 candidate services) is used to demonstrate the effectiveness of the proposed DPD-DQN.

[0145] The case data is as follows:

[0146] T＝[18,23,21,30,30,18,23,21,30,30,18,23,21,30,30,18,23,21,30,30,18,23,21,30,30,18,23,21,30,30,18,23,21,30,30,18,23,21,30,30,18,23,21,30,30,18,23,21,30,30；24,12,18,19,19,24,12,18,19,19,24,12,18,19,19,24,12,18,19,19,24,12,18,19,19,24,12,18,19,19,24,12,18,19,19,24,12,18,19,19,24,12,18,19,19,24,12,18,19,19；36,20,24,25,25,36,20,24,25,25,36,20,24,25,25,36,20,24,25,25,36,20,24,25,25,36,20,24,25,25,36,20,24,25,25,36,20,24,25,25,36,20,24,25,25,36,20,24,25,25；21,26,27,29,29,21,26,27,29,29,21,26,27,29,29,21,26,27,29,29,21,26,27,29,29,21,26,27,29,29,21,26,27,29,29,21,26,27,29,29,21,26,27,29,29,21,26,27,29,29；23,26,27,29,29,23,26,27,29,29,23,26,27,29,29,23,26,27,29,29,23,26,27,29,29,23,26,27,29,29,23,26,27,29,29,23,26,27,29,29,23,26,27,29,29,23,26,27,29,29]；

[0147] Q＝[8.1,8.4,8.5,9.1,9.1,8.1,8.4,8.5,9.1,9.1,8.1,8.4,8.5,9.1,9.1,8.1,8.4,8.5,9.1,9.1,8.1,8.4,8.5,9.1,9.1,8.1,8.4,8.5,9.1,9.1,8.1,8.4,8.5,9.1,9.1,8.1,8.4,8.5,9.1,9.1,8.1,8.4,8.5,9.1,9.1,8.1,8.4,8.5,9.1,9.1；8.6,8.7,7.9,7.8,7.8,8.6,8.7,7.9,7.8,7.8,8.6,8.7,7.9,7.8,7.8,8.6,8.7,7.9,7.8,7.8,8.6,8.7,7.9,7.8,7.8,8.6,8.7,7.9,7.8,7.8,8.6,8.7,7.9,7.8,7.8,8.6,8.7,7.9,7.8,7.8,8.6,8.7,7.9,7.8,7.8,8.6,8.7,7.9,7.8,7.8；9.2,9.4,8.2,8.1,7.8,9.2,9.4,8.2,8.1,7.8,9.2,9.4,8.2,8.1,7.8,9.2,9.4,8.2,8.1,7.8,9.2,9.4,8.2,8.1,7.8,9.2,9.4,8.2,8.1,7.8,9.2,9.4,8.2,8.1,7.8,9.2,9.4,8.2,8.1,7.8,9.2,9.4,8.2,8.1,7.8,9.2,9.4,8.2,8.1,7.8；8.3,8.3,9.0,7.8,7.8,8.3,8.3,9.0,7.8,7.8,8.3,8.3,9.0,7.8,7.8,8.3,8.3,9.0,7.8,7.8,8.3,8.3,9.0,7.8,7.8,8.3,8.3,9.0,7.8,7.8,8.3,8.3,9.0,7.8,7.8,8.3,8.3,9.0,7.8,7.8,8.3,8.3,9.0,7.8,7.8,8.3,8.3,9.0,7.8,7.8；8.0,8.3,8.0,7.8,7.8,8.0,8.3,8.0,7.8,7.8,8.0,8.3,8.0,7.8,7.8,8.0,8.3,8.0,7.8,7.8,8.0,8.3,8.0,7.8,7.8,8.0,8.3,8.0,7.8,7.8,8.0,8.3,8.0,7.8,7.8,8.0,8.3,8.0,7.8,7.8,8.0,8.3,8.0,7.8,7.8,8.0,8.3,8.0,7.8,7.8]；.

[0148] C＝[50,57,80,90,90,50,57,80,90,90,50,57,80,90,90,50,57,80,90,90,50,57,80,90,90,50,57,80,90,90,50,57,80,90,90,50,57,80,90,90,50,57,80,90,90,50,57,80,90,90；27,86,45,86,86,27,86,45,86,86,27,86,45,86,86,27,86,45,86,86,27,86,45,86,86,27,86,45,86,86,27,86,45,86,86,27,86,45,86,86,27,86,45,86,86,27,86,45,86,86；46,21,32,58,58,46,21,32,58,58,46,21,32,58,58,46,21,32,58,58,46,21,32,58,58,46,21,32,58,58,46,21,32,58,58,46,21,32,58,58,46,21,32,58,58,46,21,32,58,58；70,36,70,72,72,70,36,70,72,72,70,36,70,72,72,70,36,70,72,72,70,36,70,72,72,70,36,70,72,72,70,36,70,72,72,70,36,70,72,72,70,36,70,72,72,70,36,70,72,72；30,36,50,72,72,30,36,50,72,72,30,36,50,72,72,30,36,50,72,72,30,36,50,72,72,30,36,50,72,72,30,36,50,72,72,30,36,50,72,72,30,36,50,72,72,30,36,50,72,72]；

[0149] S＝[0.8,0.9,0.7,0.8,0.8,0.8,0.9,0.7,0.8,0.8,0.8,0.9,0.7,0.8,0.8,0.8,0.9,0.7,0.8,0.8,0.8,0.9,0.7,0.8,0.8,0.8,0.9,0.7,0.8,0.8,0.8,0.9,0.7,0.8,0.8,0.8,0.9,0.7,0.8,0.8,0.8,0.9,0.7,0.8,0.8,0.8,0.9,0.7,0.8,0.8；0.6,0.7,0.8,0.9,0.9,0.6,0.7,0.8,0.9,0.9,0.6,0.7,0.8,0.9,0.9,0.6,0.7,0.8,0.9,0.9,0.6,0.7,0.8,0.9,0.9,0.6,0.7,0.8,0.9,0.9,0.6,0.7,0.8,0.9,0.9,0.6,0.7,0.8,0.9,0.9,0.6,0.7,0.8,0.9,0.9,0.6,0.7,0.8,0.9,0.9；0.5,0.8,0.9,0.7,0.7,0.5,0.8,0.9,0.7,0.7,0.5,0.8,0.9,0.7,0.7,0.5,0.8,0.9,0.7,0.7,0.5,0.8,0.9,0.7,0.7,0.5,0.8,0.9,0.7,0.7,0.5,0.8,0.9,0.7,0.7,0.5,0.8,0.9,0.7,0.7,0.5,0.8,0.9,0.7,0.7,0.5,0.8,0.9,0.7,0.7；0.6,0.8,0.7,0.6,0.6,0.6,0.8,0.7,0.6,0.6,0.6,0.8,0.7,0.6,0.6,0.6,0.8,0.7,0.6,0.6,0.6,0.8,0.7,0.6,0.6,0.6,0.8,0.7,0.6,0.6,0.6,0.8,0.7,0.6,0.6,0.6,0.8,0.7,0.6,0.6,0.6,0.8,0.7,0.6,0.6,0.6,0.8,0.7,0.6,0.6；0.7,0.8,0.8,0.6,0.6,0.7,0.8,0.8,0.6,0.6,0.7,0.8,0.8,0.6,0.6,0.7,0.8,0.8,0.6,0.6,0.7,0.8,0.8,0.6,0.6,0.7,0.8,0.8,0.6,0.6,0.7,0.8,0.8,0.6,0.6,0.7,0.8,0.8,0.6,0.6,0.7,0.8,0.8,0.6,0.6,0.7,0.8,0.8,0.6,0.6];.

[0150] Location=[4,2,1,1,1,4,2,1,1,1,4,2,1,1,1,4,2,1,1,1,4,2,1,1,1,4,2 ,1,1,1,4,2,1,1,1,4,2,1,1,1,4,2,1,1,1,4,2,1,1,1;2,4,2,2,2,2,4,2,2 ,2,2,4,2,2,2,2,4,2,2,2,2,4,2,2,2,4,2,1,1,1,4,2,1,1,1,4,2,1,1,1,4,2,1,1,1,4,2,1,1,1;2,3,4,3,3,2,3,4,3,3,2,3,4,3,3,2,3,4 ,3,3,4,2,1,1,1,4,2,1,1,1,4,2,1,1,1,4,2,1,1,1,4,2,1,1,1; 1,2,1,4,4,1,2,1,4,4,1,2,1,4,4,1,2,1,4,4,1,2,1,4,4,4,2,1,1,1,4,2,1,1,1,4,2 ,1,1,1,4,2,1,1,1,4,2,1,1,1;1,2,1,4,4,1,2,1,4,4,1,2,1,4,4,1,2,1,4,4,1,2,1,1,1,4,2,1,1,1,4,2,1,1,1,4,2,1,1,1,1];

[0151] LC=[10,30,20,23,19,19; 30,10,26,29,20,20; 20,26,10,36,32,32; 23,29,26,10,41,41; 19,20,32,41,10,10; 19,20,32,41,10,10];

[0152] LT=[1,10,8,6,6,6;10,1,5,4,6,6;8,5,1,7,3,3;6,4,7,1,2,2;6,6,3,2,1,1;6,6,3,2,1,1];

[0153] w1=0.2;w2=0.4;w3=0.3;w4=0.1.batch size batch_size=16;training times 200;learning rate lr=0.0005;epsilion=0.95;γ=0.9;target network (Q_Target) update step length target_replace_iter=50, experience pool capacity memory_capacity=100.

[0154] The structural parameters of the DPD-DQN three-layer convolutional neural network are as follows:

[0155] (n_states×300,300×400,400×500,500×n_actions).

[0156] Figure 3 is a random DPD-DQN solution process. Figure 3 As shown in the figure, after 100 times of training, the network structure parameters of the DPD-DQN model were obviously optimized. The optimal result of this solution is the 171th result [2 3 3 1 1 5 52 4 3; 2 3 3 1 1 5 3 3 4 1; 5 3 4 3 3 2 3 3 1 1; 2 3 3 2 1 2 3 3 1 1; 2 3 3 3 1 23 4 3 1], and the target value is 1544.56.

[0157] Then, according to the Furutaguchi experimental method, the target weight value is changed according to the four levels of multiple factors, and retraining is performed to obtain the corresponding optimal CPS manufacturing component combination. By analogy, the candidate set of CPS manufacturing components for a subtask of the upper level is obtained. By analogy, a customized order service chain is obtained.

[0158] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A self-organizing hierarchical aggregation method based on CPS manufacturing components, characterized by: The self-organizing hierarchical aggregation method of CPS manufacturing components is carried out by using a self-similar modeling method of CPS manufacturing components. In a digital twin model, the unmanned production line is deconstructed according to the fractal principle and self-similarity to form a basic fractal component set, and the basic fractal components are autonomously hierarchically aggregated to obtain CPS manufacturing components. The formation method of the CPS manufacturing components and their self-organizing aggregation method are both self-similar. The method comprises the following steps: Get order task information; When the current production line status cannot complete the order task, the processing tasks at each level are decomposed into the next level of sub-processing task sets based on the order information; Decouple the current unmanned production line to obtain the lowest granularity CPS manufacturing components; Based on the sub-processing task sets of each layer, a candidate service set of CPS manufacturing components for specific processing tasks is formed; Each CPS manufacturing component aggregation unit corresponds to a processing task of the previous layer; according to the processing task of the previous layer, the current layer CPS manufacturing component self-organizes multiple times based on DPD-DQN to form a current layer CPS manufacturing component service combination that completes the processing task of the previous layer, that is, the CPS manufacturing component of the previous layer; after multiple aggregations of the current layer CPS manufacturing components self-organize and aggregate, a CPS manufacturing component candidate service set of a processing task of the previous layer is formed; the self-organization aggregation methods of the CPS manufacturing components of each layer are self-similar; The DPD-DQN represents a multi-agent reinforcement learning model; The CPS represents a cyber-physical system; Match the processing tasks at each level and their required CPS manufacturing components to obtain a set of CPS manufacturing components that meet the requirements in the current state, forming an order-customized reconstruction production line; The CPS manufacturing component layer-by-layer self-organization aggregation process is carried out according to the following steps: A low-level self-organizing aggregation model of CPS manufacturing components is constructed. Optimization strategies are introduced to address the overfitting, simple random sampling, and poor learning effects of DQN. DPD-DQN is proposed to solve the single-layer self-organizing aggregation model of CPS manufacturing components. The self-organizing aggregation model of CPS manufacturing components is solved layer by layer using DPD-DQN until a hierarchical CPS manufacturing component that can complete the order subtask is obtained. A multi-level autonomous aggregation model of CPS manufacturing components is formed. The traditional DQN uses a deep network to approximate the current value function and applies another deep network with the same structure but different parameters to generate the target Q value, i.e., Q 现实 , the Q value is calculated and updated according to the following formula: The DPD-DQN adopts multiple optimization strategies to prevent overfitting. 估计 The neural network calculates action a; The optimization strategy includes any one or a combination of any two or more of the following strategies: Optimization strategy 1: Use Q to prevent overfitting 估计 The neural network calculates action a, and the Q value calculation formula is revised as follows. Optimization strategy 2: Sampling strategy considering the importance of experience data Extract experience data according to the importance of DQN experience data, as follows: Apply the SumTree method to form a tree structure of experience data, each leaf stores the priority of each sample, each branch node has only two branches, and the node value is the sum of the two branch values; when sampling, divide the sum of p into batch_size intervals, n = sum(p) / batch_size, and then randomly select a number in each interval; Optimization strategy 3: During the model optimization process, the Q value update method of each action of DQN is revised to the following formula: Q(s,a;θ,α,β)=V(s;θ,β)+A(s,a;θ,α); The parameters in the DPD-DQN are defined as follows: In the traditional DQN, there are two neural networks with the same structure and parameter updates with time difference, namely: Q 估计 and Q 现实 ; Neural network hyperparameters: Learning rate α, minimum batch size b, sampling weight β, discount factor γ, ε; Indicates the Q in the original DQN 现实 Neural network update methods; represents Q using optimization strategy 1 现实 Neural network update method; S represents the state set of the environment; s represents a specific state; a represents a specific action in the action set taken by the agent; r represents the immediate return reward; γ represents the discount factor; R t+1 represents the reword value of the t+1th generation; θ represents the neural network parameter; Indicates the selection of the next state S t+1 The maximum Q value of an action; Q(s,a;θ,α,β) represents the Q value of state s when taking action a and the neural network parameters are (θ,α,β); V(s;θ,β) represents the state value of state s with (θ,β) neural network parameters; A(s,a,θ,α) represents the additional reward for state s to perform a specific action a.

2. The method for self-organizing hierarchical aggregation of CPS manufacturing components as claimed in claim 1, characterized in that: The DPD-DQN solves the self-organizing aggregation model of each layer of CPS manufacturing components according to the following steps: Initialize the DPD-DQN neural network parameters, including the learning rate α, the minimum batch size b, the sampling weight β, the discount coefficient γ, and the greedy strategy (ε-greedy) probability ε; input the TQCS data of the bottom-level CPS manufacturing component set; Enter the training process: In each training round, each agent selects a behavior; multiple agents form a joint action a; the environment executes the behaviors in the joint action set and updates the state for each action; after all actions are executed, the total state s of the system is updated; Calculate the immediate return r; Update storage s, a, r; wait until the experience pool overflows and start the learning mechanism; And update Q at a fixed step period 现实 Neural network parameters; After a fixed number of training times, the optimal CPS manufacturing component combination of a subtask in this layer is obtained. Then, according to the Taguchi experimental method, the target weight value is changed according to the four levels of multiple factors, and retraining is performed to obtain the corresponding optimal CPS manufacturing component combination; and so on, the candidate set of CPS manufacturing components for a subtask in the upper layer is obtained; DPD-DQN is applied to obtain the candidate service set of CPS manufacturing components for each subtask of the upper level, and so on, until a hierarchical combination of CPS manufacturing components that can complete the order production task is obtained.

3. The self-similar fractal reconstruction system of unmanned production line is characterized by: The self-similar formation mode and self-organization aggregation mode of CPS manufacturing components are used to construct a self-similar fractal reconstruction system for unmanned production lines, including: An order management unit, used to obtain order task information; When the current production line status cannot complete the order task, the order processing task is decomposed into units to decompose each layer of processing tasks into the next level of sub-processing task sets; The CPS manufacturing component generation unit is used to decouple the current unmanned production line to obtain low-granularity CPS manufacturing components; The CPS manufacturing component classification unit is used to form a candidate service set of CPS manufacturing components for a specific processing task based on the sub-processing task sets of each layer; CPS manufacturing component aggregation unit, each CPS manufacturing component aggregation unit corresponds to a processing task of the previous layer; according to the processing task of the previous layer, the current layer CPS manufacturing component self-organizes multiple times based on DPD-DQN to form a current layer CPS manufacturing component service combination that completes the processing task of the previous layer, that is, the CPS manufacturing component of the previous layer; after multiple aggregations of the current layer CPS manufacturing components self-organize and aggregate, a CPS manufacturing component candidate service set of a processing task of the previous layer is formed; the self-organization aggregation methods of CPS manufacturing components at each layer are self-similar; The order-customized reconfiguration production line generation unit is used to match the processing tasks at each level with the CPS manufacturing components they need, so as to obtain a hierarchical and multi-granular CPS manufacturing component set that meets the requirements in the current state, thus forming an order-customized reconfiguration production line; The CPS manufacturing component aggregation unit includes a CPS manufacturing component self-organization aggregation fractal model based on DPD-DQN and a weight parameter adjustment method based on Taguchi experimental method; The DPD-DQN-based CPS manufacturing component self-organizing aggregation fractal model is used to construct the CPS manufacturing component self-organizing aggregation model of each layer. The DPD-DQN-based CPS manufacturing component self-organizing aggregation fractal model introduces optimization strategies for the overfitting, simple random sampling, and poor learning effect of DQN. The traditional DQN uses a deep network to approximate the current value function and applies another deep network with the same structure but different parameters to generate the target Q value, i.e., Q 现实 , the Q value is calculated and updated according to the following formula: The DPD-DQN adopts multiple optimization strategies to prevent overfitting. 估计 The neural network calculates action a; The optimization strategy includes any one or a combination of any two or more of the following strategies: Optimization strategy 1: Using Q 估计 The neural network calculates action a, and the Q value calculation formula is revised as follows. Optimization strategy 2: Sampling strategy based on the importance of empirical data Extract empirical data according to the importance of DQN empirical data, as follows: Apply the SumTree method to form a tree structure of empirical data, each leaf stores the priority of each sample, each branch node has only two branches, and the node value is the sum of the two branch values; when sampling, divide the sum of p into batch_size intervals, that is, n = sum(p) / batch_size, and then randomly select a number in each interval; Optimization Strategy 3: During the model optimization process, the Q value update method for each action of DQN is revised to the following formula: Q(s,a;θ,α,β)=V(s;θ,β)+A(s,a;θ,α); The parameters in the DPD-DQN are defined as follows: In traditional DQN, there are two neural networks with the same structure and parameter updates with time difference: Q 估计 and Q 现实 ; Neural network hyperparameters: learning rate α, minimum batch size b, sampling weight β, discount factor γ, ε; Indicates the Q in the original DQN 现实 Neural network update methods; represents Q using optimization strategy 1 现实 Neural network update method; S represents the state set of the environment; s represents a specific state; a represents a specific action in the action set taken by the agent; r represents the immediate return reward; γ represents the discount factor; R t+1 represents the reword value of the t+1th generation; θ represents the neural network parameter; Indicates the selection of the next state S t+1 The maximum Q value of an action; Q(s,a;θ,α,β) represents the Q value of state s when taking action a and the neural network parameters are (θ,α,β); V(s;θ,β) represents the state value of state s with (θ,β) neural network parameters; A(s,a,θ,α) represents the additional reward for state s to perform a specific action a.

4. The self-similar fractal reconstruction system for unmanned production line according to claim 3, characterized in that: The DPD-DQN solves the self-organizing aggregation model of each layer of CPS manufacturing components according to the following steps: Initialize the DPD-DQN neural network parameters, including the learning rate α, the minimum batch size b, the sampling weight β, the discount factor γ, the greedy strategy (ε-greedy) probability ε, etc.; input the TQCS data of the bottom-level CPS manufacturing component set; Enter the training process: In each training round, each agent selects a behavior; multiple agents form a joint action a; the environment executes the behaviors in the joint action set and updates the state for each action; after all actions are executed, the total state s of the system is updated; Calculate the immediate return r; Update storage s, a, r; wait until the experience pool overflows and start the learning mechanism; And update Q at a fixed step period 现实 , i.e., the neural network parameters of Q; After a fixed number of training times, the optimal CPS manufacturing component combination of a subtask in this layer is obtained. Then, according to the Taguchi experimental method, the target weight value is changed according to the four levels of multiple factors, and retraining is performed to obtain the corresponding optimal CPS manufacturing component combination; and so on, the candidate set of CPS manufacturing components for a subtask in the upper layer is obtained; DPD-DQN is applied to obtain the candidate service set of CPS manufacturing components for each subtask of the upper level; based on the self-similar fractal characteristics of the self-organizing aggregation model of CPS manufacturing components, and so on, until a hierarchical CPS manufacturing component combination that can complete the order production task is obtained.

Images