Selective disassembly optimization method and device based on PPO algorithm and medium

Through PPO algorithm and Petri network modeling, the structural changes adaptability and high cost problems when disassembling complex EOL products are solved, and an efficient and stable disassembly strategy is realized, reducing environmental impact.

CN119005499AActive Publication Date: 2024-11-22SHANGHAI SHANYANZHE TECHNOLOGY CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411021248.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-28
Publication Date
2024-11-22
Estimated Expiration
2044-07-28

AI Technical Summary

Technical Problem

In the prior art, when disassembling complex and dynamically changing EOL products, heuristic algorithms are difficult to adapt to structural changes, resulting in high disassembly cost and unstable disassembly. Traditional DQN algorithms have problems with over-learning and reuse during training, resulting in unstable model training. .

Method used

The selective disassembly optimization method based on PPO algorithm is adopted, and the product topology structure and connection relationship are described through Petri network modeling, combined with the strategy optimization and cutting mechanism of reinforcement learning, the optimal disassembly strategy is independently learned to reduce the disassembly cost and environmental impact.

Benefits of technology

It improves the efficiency and stability of the disassembly process, adapts to complex dynamic structures, avoids the instability of policy updates, and has higher stability and convergence speed compared to the DQN algorithm, achieving better disassembly planning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005499A_ABST
    Figure CN119005499A_ABST
Patent Text Reader

Abstract

The invention provides a selective disassembly optimization method based on a PPO algorithm. The method comprises the steps that a DPN model for disassembling waste products is constructed according to scrapped (EOL) products needing to be disassembled; constructing a selective disassembly model based on reinforcement learning according to the DPN model, and training the selective disassembly model by using a PPO algorithm; and inputting the EOL product needing to be disassembled in real time into the trained selective disassembly model to obtain an optimal disassembly action set. According to the selective disassembly optimization method based on the PPO algorithm, the efficiency and effect of the disassembly process are improved, and higher adaptability and stability are shown in the face of complex dynamic disassembly tasks. Besides, in combination with other advanced reinforcement learning algorithms and exploration of a better modeling scheme, the performance and practicability of the selective disassembly planning method are expected to be further improved, and powerful technical support is provided for sustainable manufacturing and resource recovery.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of disassembly technology, and in particular relates to a selective disassembly optimization method, device and medium based on a PPO algorithm. Background Art

[0002] The manufacturing industry consumes a lot of energy and mineral resources, and end-of-life (EOL) products are piled up in landfills every year. EOL products such as electronic products, automobiles, shared bicycles and industrial equipment often contain valuable components and materials that can be reused or recycled. Recovering value from these products helps reduce raw material mining and energy consumption, which contributes to sustainable manufacturing. Disassembly is an indispensable process for recycling scrapped products, in which the optimal / near-optimal disassembly plan should be determined to maximize the value recovery of scrapped products. Such a decision-making process is called disassembly planning (DSP). Since many EOL products have been in service for a long time, their quality status will be more or less deteriorated from their original state. In view of this, disassembling EOL products is much more complicated than assembling new products, and although full DSP can disassemble all parts, not all parts are still valuable, and the cost of disassembly is too high. In response to this situation, domestic and foreign experts have also conducted in-depth research on selective disassembly and believe that selective disassembly is a better choice than complete disassembly.

[0003] As the EOL product structure becomes increasingly complex, the search space for disassembly sequences will become larger and larger. Therefore, heuristic algorithms are widely used to find the optimal disassembly sequence. The disassembly sequence is obtained before the offline product is disassembled. However, due to the uncertainty of the product structure, the predetermined disassembly sequence is often impractical. As for the heuristic algorithm, a slight change to the EOL product structure may restart the entire optimization process. Therefore, the decision of the heuristic algorithm cannot adapt to the changes in the EOL product structure. Some studies use deep Q networks (DQN) to speed up learning. DQN uses experience replay and target fixation to improve algorithm stability. However, due to the non-static distribution of samples and the accumulated errors during training, it sometimes leads to over-learning or reuse of certain experiences, making model training unstable. Summary of the invention

[0004] The purpose of the present invention is to propose a selective disassembly optimization method, device and medium based on the PPO algorithm, which reduces the disassembly cost and environmental impact by only disassembling valuable parts. The modeling method based on Petri net (DPN) can accurately describe the topological structure of the product and the connection relationship between parts, so that the constraint relationship of the disassembly process can be accurately expressed.

[0005] In order to achieve the above object, the present invention provides a selective disassembly optimization method based on the PPO algorithm, the method comprising:

[0006] Step 1: Construct a DPN model for disassembling waste products based on the EOL products that need to be disassembled;

[0007] Step 2: Build a selective disassembly model based on reinforcement learning according to the DPN model, and use the PPO algorithm to train the selective disassembly model;

[0008] Step 3: Input the EOL products that need to be disassembled in real time into the trained selective disassembly model to obtain the optimal set of disassembly actions.

[0009] Furthermore, the S1 specifically includes:

[0010] Step 1.1: Construct the five-tuple of the DPN model according to the topological structure of the EOL products, the connection relationship between the EOL products, and the constraint relationship between the disassembly operations;

[0011] Step 1.2: Calculate the transformation of the EOL product disassembly state based on the output of each transition of the quintuple of the DPN model.

[0012] Furthermore, the quintuple of the DPN model is expressed as follows:

[0013] DPN=(P, F, In×m, Om×n, m0)

[0014] Among them, P represents the set of places, the root node place represents the EOL product that has not been disassembled, the intermediate node represents the EOL product middleware that has not been disassembled to the end, and the leaf node represents the EOL product parts that have been completely disassembled; F represents the set of transitions, and the transition corresponds to the action in reinforcement learning, that is, the disassembly operation of the product; n represents the number of places; m represents the number of transitions; the input matrix In×m defines the directed arcs from the place to the transition, and its elements i, j, and k take values ​​0 or 1. When there is no directed arc from the j place to the k transition, it takes 0, otherwise it takes 1; the output matrix Om×n defines the directed arcs from the transition to the place, and its elements o, k, and j take values ​​0 or 1. When there is no directed arc from the k transition to the j place, it takes 0, otherwise it takes 1; m0 is an n-dimensional vector used to represent the initial disassembly state, and its elements take values ​​1 or 0. 1 represents that the disassembly operation has been performed, and 0 represents that the disassembly operation has not been performed.

[0015] Furthermore, each transition of the five-tuple of the DPN model includes at least one input and two outputs; each transition obtains tokens from at least one or more places and distributes the tokens to different places; when a place has a token, only one of the output transitions can be selected to be activated; when a transition points to different places from multiple directed arcs, if the transition is activated, all the places pointed to will obtain tokens;

[0016] Among them, the m-dimensional vector t fired Indicates the activation status of each transition. Its element value is 1 or 0, indicating that the corresponding transition has been activated or not, respectively. new Represents the new disassembly state, and the disassembly state transformation equation is as follows:

[0017] m new =m0+t fired ·(Om×nI T n×m)

[0018] Where T represents the transpose operation of the matrix.

[0019] Furthermore, the reward function of the selective disassembly model based on reinforcement learning includes: an environmental pollution index of the disassembly process, a recyclable and reusable quality index of the disassembly process, and a recycling and reusability value index of the disassembly process.

[0020] Furthermore, the S2 specifically includes:

[0021] Assume that there are α places and β transitions in the DPN model, and the state s is represented by the α-element vector s=(s1, s2, ..., s i ,s α ) indicates that its element s i The value of is 0 or 1, indicating the state of having or not having tokens in the corresponding library respectively;

[0022] The activity is represented by an integer a, the value range of a is 1≤a≤α, which represents the sequence number of the corresponding transition;

[0023] The set of all selectable activities in state s is represented by A(s), which is determined according to the DPN model; where the β-dimensional vector X = (x1, x2, ..., x α ) indicates whether the transition can be activated in state s, and the value of its element x i The value is 1 or 0, indicating whether the corresponding transition can be activated. The calculation formulas of x and A(s) are as follows:

[0024] x=s·Iα×β

[0025] A(s)={a∈{1, 2,...,β}|x i =1};

[0026] The reward function Reward for constructing the selective disassembly model is expressed as follows:

[0027] Reward=w1·h(x i )+w2·v(x i )+w3·w(x i )

[0028] Among them, w1, w2 and w3 represent the weights of each indicator, h(x i ) represents the disassembly operation x i The sum of the environmental pollution indexes of the selected parts removed, v(x i ) represents the disassembly operation x i The sum of the recyclable mass of the selected dismantled parts, w(x i ) indicates disassembly operation x i The sum of the recoverable and recyclable values ​​of the selected dismantled parts.

[0029] Furthermore, the environmental pollution index of the disassembly process is:

[0030] The selected parts are disassembled with the least disassembly operations, and the environmental pollution index fx(x) of the disassembly process is maximized, which is expressed as follows:

[0031]

[0032] Among them, h(x i ) represents the disassembly operation x i The sum of the environmental pollution indexes of the selected parts removed is calculated as follows:

[0033]

[0034] Among them, C s represents the set of parts that are selectively disassembled, C i represents the set of parts removed during the disassembly process except for the selective disassembly, har(c) represents the environmental pollution index level of part c, and N represents the number of parts;

[0035] The recyclable quality indicators of the disassembly process are:

[0036] Disassemble the selected parts with the least disassembly operation and make the disassembly process recyclable and reusable. v (x) is the largest, expressed as follows:

[0037]

[0038] Among them, v(x i ) represents the disassembly operation x i The sum of the recyclable mass of the selected dismantled parts is calculated as follows:

[0039]

[0040] Among them, val(c) represents the recyclable quality of part c;

[0041] The recyclable reuse value indicators of the disassembly process:

[0042] Disassemble the selected parts with the least disassembly operation and make the recycling value index of the disassembly process w (x) is the largest, expressed as follows:

[0043]

[0044] Where w(xi) represents the disassembly operation x i The sum of the recoverable and reusable values ​​of the selected dismantled parts is calculated as follows:

[0045]

[0046] Among them, wei(c) represents the recyclable reuse value of part c.

[0047] Furthermore, the PPO algorithm is an actor-critic network architecture, where the actor network update is expressed as follows:

[0048]

[0049] The critic network update is expressed as follows:

[0050] Among them, rt(θ) represents the probability ratio of the new and old strategies, which is used to measure the change range of the new and old strategies. The advantage function is used to measure the goodness of taking an action relative to the average action in the current state. t (θ), 1-∈, 1+∈) represents the value used to limit r t (θ) changes in the range, when r t When (θ) is too large, the Clip function will truncate it to 1+∈; when r t If (θ) is too small, the Clip function will truncate it to 1-∈, represents the expected average value, θ represents the parameters of the policy network, ∈ represents the hyperparameter used to control the clipping range of the importance sampling ratio, ranging from (0,1); V θ (s t ) represents the state value output by the critic network, Indicates actual return.

[0051] In a second aspect, an embodiment of the present invention further provides an electronic device, including:

[0052] one or more processors;

[0053] a memory for storing one or more programs,

[0054] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned selective disassembly optimization methods based on the PPO algorithm.

[0055] In a second aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned selective disassembly optimization methods based on the PPO algorithm.

[0056] The beneficial technical effects of the present invention are at least as follows:

[0057] The present invention is based on the selective disassembly planning problem of the PPO algorithm. Selective disassembly reduces disassembly costs and environmental impacts by only disassembling valuable parts. The modeling method based on Petri nets (DPN) can accurately describe the topological structure of the product and the connection relationship between parts, so that the constraint relationship of the disassembly process can be accurately expressed. By introducing the PPO algorithm in reinforcement learning, it is possible to autonomously learn and adapt to the optimal disassembly strategy when facing complex and dynamically changing product structures. Compared with the traditional DQN algorithm, the PPO algorithm has higher stability and convergence speed, mainly due to its policy optimization and clipping mechanism, which can effectively control the amplitude of policy updates, thereby avoiding instability during the policy update process.

[0058] The present invention not only improves the efficiency and effectiveness of the disassembly process, but also demonstrates stronger adaptability and stability when faced with complex and dynamic disassembly tasks. In addition, by combining other advanced reinforcement learning algorithms and exploring better modeling schemes, it is expected to further improve the performance and practicality of the selective disassembly planning method, providing strong technical support for sustainable manufacturing and resource recycling. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0060] Figure 1 This is a flow chart of the selective disassembly optimization method based on the PPO algorithm of the present invention.

[0061] Figure 2 This is an example diagram of DPN according to an embodiment of the present invention.

[0062] Figure 3 The figure is a solution flow chart of a selective disassembly planning method based on a PPO algorithm according to an embodiment of the present invention.

[0063] Figure 4 Schematic diagram of an experimental use case of an embodiment of the present invention.

[0064] Figure 5 Schematic diagram of the Petri net model of the experimental use case of the embodiment of the present invention.

[0065] Figure 6 Schematic diagram of reward curve for 500 iterations of the PPO algorithm according to an embodiment of the present invention.

[0066] Figure 7 Schematic diagram of reward curve for 500 iterations of the DQN algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0068] like Figure 1 As shown, the present invention discloses a selective disassembly optimization method based on the PPO algorithm, the method comprising:

[0069] Step 1: Construct a DPN model for disassembling end-of-life products based on the EOL products that need to be disassembled.

[0070] Furthermore, the S1 specifically includes:

[0071] Step 1.1: Construct the five-tuple of the DPN model according to the topological structure of the EOL products, the connection relationship between the EOL products, and the constraint relationship between the disassembly operations;

[0072] Step 1.2: Calculate the transformation of the EOL product disassembly state based on the output of each transition of the quintuple of the DPN model.

[0073] The five-tuple of the DPN model is expressed as follows:

[0074] DPN=(P, F, In×m, Om×n, m0)

[0075] Among them, P represents the set of places, the root node place represents the EOL product that has not been disassembled, the intermediate node represents the EOL product middleware that has not been disassembled to the end, and the leaf node represents the EOL product parts that have been completely disassembled; F represents the set of transitions, and the transition corresponds to the action in reinforcement learning, that is, the disassembly operation of the product; n represents the number of places; m represents the number of transitions; the input matrix In×m defines the directed arcs from the place to the transition, and its elements i, j, and k take values ​​0 or 1. When there is no directed arc from the j place to the k transition, it takes 0, otherwise it takes 1; the output matrix Om×n defines the directed arcs from the transition to the place, and its elements o, k, and j take values ​​0 or 1. When there is no directed arc from the k transition to the j place, it takes 0, otherwise it takes 1; m0 is an n-dimensional vector used to represent the initial disassembly state, and its elements take values ​​1 or 0. 1 represents that the disassembly operation has been performed, and 0 represents that the disassembly operation has not been performed.

[0076] Furthermore, in DPN, each transition has at least one input and two outputs, which means that a transition can obtain tokens from one or more places and distribute tokens to different places. In this case, an "or" logical relationship is formed: when a place has a token, only one of the output transitions can be selected to activate. On the other hand, a transition may have multiple directed arcs pointing to different places, which means that when the transition is activated, all the places it points to will obtain tokens. In this case, an "and" logical relationship is formed. Use the m-dimensional vector t fired Indicates the activation status of each transition. Its element value is 1 or 0, indicating that the corresponding transition has been activated or not, respectively. new Represents the new disassembly state, and the disassembly state transformation equation is as follows:

[0077] m new =m0+t fired ·(Om×nI T n×m)

[0078] Where T represents the transpose operation of the matrix.

[0079] Among them, Figure 2 As mentioned above, a simple example based on Petri net modeling is given. In the example, the set of places and transitions and the input matrix and output matrix are shown below. The initial state is m0. The root place has a token that can select transitions, but only one transition can be selected at a time, that is, t1 or t2. When t1 transition is selected, t fired =(1, 0, 0, 0), then the disassembly state will become m = (0, 1, 0, 0, 0, 1).

[0080] Step 2: Build a selective disassembly model based on reinforcement learning according to the DPN model, and use the PPO algorithm to train the selective disassembly model.

[0081] Specifically, state and action are very important concepts in reinforcement learning. The essence of reinforcement learning is to learn control strategies through trial and error through the interaction between the agent and the environment. In this process, the agent performs actions and observes changes in the state of the environment, and then continuously improves its own strategy based on the rewards (feedback) obtained to adapt to future tasks. In most reinforcement learning environments, the choice of action is fixed, but in the disassembly problem, the choice of action is related to the state, and the number of states corresponding to different states is different. Therefore, different action sets need to be selected according to different states.

[0082] Furthermore, the S2 specifically includes:

[0083] Assume that there are α places and β transitions in the DPN model, and the state s is represented by the α-element vector s=(s1, s2, ..., s i ,s α ) indicates that its element s i The value of is 0 or 1, indicating the state of having or not having tokens in the corresponding library respectively;

[0084] The activity is represented by an integer a, the value range of a is 1≤a≤α, which represents the sequence number of the corresponding transition;

[0085] The set of all selectable activities in state s is represented by A(s), which is determined according to the DPN model; where the β-dimensional vector X = (x1, x2, ..., x α ) indicates whether the transition can be activated in state s, and the value of its element x i The value is 1 or 0, indicating whether the corresponding transition can be activated. The calculation formulas of x and A(s) are as follows:

[0086] x=s·Iα×β

[0087] A(s)={a∈{1, 2,...,β}|x i =1};

[0088] The reward function Reward for constructing the selective disassembly model is expressed as follows:

[0089] Reward=w1·h(X i )+w2·v(x i )+w3·w(x i )

[0090] Among them, w1, w2 and w3 represent the weights of each indicator, h(x i) represents the disassembly operation x i The sum of the environmental pollution indexes of the selected parts removed, v(x i ) represents the disassembly operation x i The sum of the recyclable mass of the selected dismantled parts, w(x i ) indicates disassembly operation x i The sum of the recoverable and recyclable values ​​of the selected dismantled parts.

[0091] In reinforcement learning, reward is a very important concept that directly guides the learning process and behavior strategy of the agent. Reward is the feedback signal received by the agent when interacting with the environment. It can tell the agent how effective its behavior is, thereby guiding the agent to learn the best behavior strategy. The design of reward has a crucial impact on the performance of the reinforcement learning algorithm. A reasonable reward setting can help the agent learn the correct behavior strategy faster. The reward setting in this article mainly involves three goals:

[0092] Remove the selected parts with the least disassembly operation and make the environmental pollution index of the disassembly process h (x) is the largest, expressed as follows:

[0093]

[0094] Among them, h(x i ) represents the disassembly operation x i The sum of the environmental pollution indexes of the selected parts removed is calculated as follows:

[0095]

[0096] Among them, C s represents the set of parts that are selectively disassembled, C i It represents the set of parts removed during the disassembly process except for selective disassembly, har(c) represents the environmental pollution level of part c, and N represents the number of parts. The environmental pollution level of a part is represented by six integers from 0 to 5. The larger the value, the higher the degree of environmental pollution of the part. Parts with higher environmental pollution levels should be removed and then processed to avoid damage to the environment.

[0097] The recyclable quality indicators of the disassembly process are:

[0098] Disassemble the selected parts with the least disassembly operation and make the disassembly process recyclable and reusable. v (x) is the largest, expressed as follows:

[0099]

[0100] Among them, v(x i ) represents the disassembly operation x i The sum of the recyclable mass of the selected dismantled parts is calculated as follows:

[0101]

[0102] Among them, val(c) represents the recyclable quality of part c; similarly, six integers from 0 to 5 are used to represent the recyclable quality of the part, and the larger the value, the higher the quality;

[0103] The recyclable reuse value indicators of the disassembly process:

[0104] Disassemble the selected parts with the least disassembly operation and make the recycling value index of the disassembly process w (x) is the largest, expressed as follows:

[0105]

[0106] Where w(xi) represents the disassembly operation x i The sum of the recoverable and reusable values ​​of the selected dismantled parts is calculated as follows:

[0107]

[0108] Among them, wei(c) represents the recyclable and reusable value of part c, and six integers from 0 to 5 are used to represent the quality of the part. The higher the value, the higher the value and the more necessary it is to recycle.

[0109] Furthermore, the PPO algorithm is a near-line policy optimization algorithm that uses the actor-critic architecture. The PPO algorithm uses a clipping mechanism to limit the update range of the policy, which helps to improve the stability of training and reduce instability caused by excessive update ranges. PPO is insensitive to the selection of initial policy parameters and hyperparameters, and can adjust parameters more simply. In addition to the above points, PPO is also an on-policy algorithm. During the training process, the agent uses the current policy to interact directly with the environment and uses the data of interaction with the environment to directly update the policy. This is particularly important in large-scale disassembly operations. There are two neural networks, an actor network and a critic network. The actor network is used to select actions, and the critic network is used to calculate the advantage function. First, the actor network is used to interact with the environment to obtain a series of states, actions, rewards, and next state data, and then the critic network is used to calculate the advantage function and the actor network is used to calculate the ratio between the new and old policies based on the collected data. Finally, the policy update range is limited based on the ratio and clipping technology, and then the network parameters are updated. The following is the neural network parameter update method:

[0110]

[0111] The critic network update is expressed as follows:

[0112]

[0113] where r t (θ) is the probability ratio of the new and old strategies, which is used to measure the magnitude of change between the new and old strategies. is the advantage function used to measure the goodness of taking an action relative to the average action in the current state, clip(r t (θ), 1-∈, 1+∈) is used to limit r t (θ) changes in the range, when r t When (θ) is too large, the Clip function will truncate it to 1+∈; when r t If (θ) is too small, the Clip function will truncate it to 1-∈, which helps prevent the policy update from being too large or too small, thereby improving the stability of training and the effectiveness of the update; represents the expected average value, θ represents the parameters of the policy network, ∈ represents the hyperparameter used to control the clipping range of the importance sampling ratio, ranging from (0, 1), V θ (s t ) represents the state value output by the critic network, Indicates actual return.

[0114] like Figure 3As shown, the solution steps of the selective disassembly planning method based on the PPO algorithm are summarized as follows:

[0115] Step 1: Establish a DPN model for disassembling waste products, select the parts to be disassembled, and set the weights of each indicator;

[0116] Step 2: Set parameters such as δ, ∈, γ, the learning rate of the Actor and Critic networks, and the maximum number of iterations episodes;

[0117] Step 3: Reset the DPN initial state;

[0118] Step 4: Calculate the activity set A(s) under the current state s according to DPN and the activity set selection formula;

[0119] Step 5: The Actor network randomly selects an action from the current state's activity set using normal distribution;

[0120] Step 6: Interact with the environment and obtain the next state, reward, and data on whether to end the while loop;

[0121] Step 7: The Critic network uses the acquired data to calculate the error and advantage function;

[0122] Step 7: The Actor network uses the state to obtain the activity selection probability ratio of the new and old strategies and perform pruning;

[0123] Step 8: The Actor network and the Critic network update network parameters simultaneously;

[0124] Step 9: Repeat steps 4 to 6 until all parts that need to be disassembled are disassembled;

[0125] Step 10: Repeat steps 4 to 8 until the maximum number of iterations is reached;

[0126] Step 11: Continuously learn and update the neural network parameters through continuous interaction with the environment to obtain the optimal disassembly steps.

[0127] Step 3: Input the EOL products that need to be disassembled in real time into the trained selective disassembly model to obtain the optimal set of disassembly actions.

[0128] Furthermore, the EOL products that need to be disassembled in real time are first described and modeled, and then the experimental results are given. All algorithms are coded and experimented in pycharm2022.1.3 under the windows 10 operating system. The CPU model is Intel(R) Core(TM) i5-10500CPU@3.10GHz 3.10GHz, the GPU model is NVIDIA GeForce RTX2060SUPER, and the running memory is 16GB.

[0129] For example, Figure 4 As shown in the figure, the waste product has 8 parts, and the constraint relationship between the parts is represented by dotted lines. First, Petri net modeling is performed based on the constraint relationship between the parts for this test case. The established Petri net model is as follows: Figure 4 As shown, it includes 21 places and 19 transitions. The parts disassembled in each transition are different, but some places are the same. For example, there are three places in place No. 21 and three places in place No. 20. This is because if the disassembly routes are different, the disassembled parts will be different, which will result in multiple identical places. However, in the experiment, since a place can only choose one transition, that is, there can only be one disassembly route, the same parts will not be disassembled multiple times.

[0130] Select parts B, D, and F as the parts that need to be disassembled. Parts B, D, and F correspond to places P15, P17, and P19, respectively. The environmental pollution levels, standardized recycling quality, and value of all parts corresponding to known waste products are shown in Table 1. The weights of each indicator are set to w1=0.5, w2=0.3, and w3=0.2. In order to determine the ideal parameter combination, the present invention sets several important parameters, the learning rate of the Actor network, and the _ lr = {0.01, 0.001}, the learning rate of the Critic network _ lr = {0.01, 0.001}, cropping rate ∈ = {0.1, 0.2, 0.3}, so there are a total of 12 (i.e., 2×2×3) parameter combinations. Each parameter combination is used to implement selective disassembly based on the PPO algorithm. The experimental results are shown in Table 2.

[0131] Table 1 Properties of all disassembled parts

[0132] Part Number Environmental pollution level Standardizing recycling quality Standardized recycling value A 2 2 1 B 5 3 1 C 1 2 4 D 1 4 3 E 1 1 1 F 2 1 4 G 2 1 1 H 1 2 1

[0133] Table 2 Operation results of various parameter combinations

[0134]

[0135]

[0136] From Table 2, we can observe that the reward is the largest when the learning rate of the Actor network is 1e-3 and the learning rate of the Critic network is 1e-2. However, when the cropping rate is 0.1 and 0.3, the reward convergence state is not very good. Therefore, the cropping rate is 0.2, the learning rate of the Actor network is 1e-3, and the learning rate of the Critic network is 1e-2 as the optimal parameter combination. The specific image convergence curve is as follows: Figure 6 shown.

[0137] In order to further test the performance of the proposed method, the present invention selects the DQN algorithm for comparison. The DQN algorithm is widely used in the current reinforcement learning to solve the disassembly sequence planning problem. The DQN algorithm continuously learns and updates the q value by using a neural network to fit the q value to find the optimal solution. In order to accurately compare the advantages and disadvantages of the two methods, the environment used by the DQN algorithm is the same as that used by the PPO algorithm. The reward convergence of the DQN algorithm after 500 iterations is shown in the figure below. Figure 7 As shown in the figure, it can be clearly seen that the convergence speed of using the DQN algorithm to solve the selective disassembly planning problem is relatively slow and difficult to converge. The final reward will fluctuate between several values. This may be because the DQN algorithm uses the experience replay mechanism, which causes a lot of data to be used multiple times. However, the PPO algorithm is an on-policy algorithm, which directly uses the collected data instead of storing it first. In addition, the PPO algorithm directly updates the strategy instead of updating and iterating the value first. Compared with the value-based strategy, the policy-based method is more flexible and effective in dealing with complex and highly uncertain tasks.

[0138] In summary, the present invention is based on the selective disassembly planning problem of the PPO algorithm. Selective disassembly reduces disassembly costs and environmental impacts by only disassembling valuable parts. The modeling method based on Petri nets (DPN) can accurately describe the topological structure of the product and the connection relationship between parts, so that the constraint relationship of the disassembly process can be accurately expressed. By introducing the PPO algorithm in reinforcement learning, the present invention can autonomously learn and adapt to the optimal disassembly strategy when facing complex and dynamically changing product structures. Compared with the traditional DQN algorithm, the PPO algorithm has higher stability and convergence speed, mainly due to its policy optimization and clipping mechanism, which can effectively control the amplitude of policy updates, thereby avoiding instability during the policy update process.

[0139] The selective disassembly planning method based on the PPO algorithm performs well in dealing with complex disassembly tasks. By setting different learning rates of the Actor and Critic networks, as well as different combinations of clipping rate parameters, the present invention finds that when the Actor learning rate is 1e-3, the Critic learning rate is 1e-2, and the clipping rate is 0.2, the PPO algorithm has the best performance and can converge to the optimal solution in the shortest time. In contrast, the DQN algorithm converges slowly after 500 iterations under the same environment, and there are significant fluctuations in the convergence process. This shows that the DQN algorithm has certain limitations when dealing with large-scale state spaces and complex decision-making problems. It may be due to the data reuse problem caused by the experience replay mechanism, which makes it difficult for the model to stabilize during training.

[0140] The selective disassembly planning method based on the PPO algorithm not only improves the efficiency and effectiveness of the disassembly process, but also shows stronger adaptability and stability when facing complex and dynamic disassembly tasks. In addition, by combining other advanced reinforcement learning algorithms and exploring better modeling schemes, it is expected to further improve the performance and practicality of the selective disassembly planning method, providing strong technical support for sustainable manufacturing and resource recycling.

[0141] Although embodiments of the present invention have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A selective disassembly optimization method based on the PPO algorithm, characterized in that: The method comprises: Step 1: Construct a DPN model for disassembling waste products based on the EOL products that need to be disassembled; Step 2: Build a selective disassembly model based on reinforcement learning according to the DPN model, and use the PPO algorithm to train the selective disassembly model; Step 3: Input the EOL products that need to be disassembled in real time into the trained selective disassembly model to obtain the optimal set of disassembly actions.

2. The selective disassembly optimization method based on the PPO algorithm according to claim 1 is characterized in that: The S1 specifically includes: Step 1.1: Construct the five-tuple of the DPN model according to the topological structure of the EOL products, the connection relationship between the EOL products, and the constraint relationship between the disassembly operations; Step 1.2: Calculate the transformation of the EOL product disassembly state based on the output of each transition of the quintuple of the DPN model.

3. The selective disassembly optimization method based on the PPO algorithm according to claim 2 is characterized in that: The five-tuple of the DPN model is expressed as follows: DPN=(P,EIn×m,Om×n,m0) Among them, P represents the set of places, the root node place represents the EOL product that has not been disassembled, the intermediate node represents the EOL product middleware that has not been disassembled to the end, and the leaf node represents the EOL product parts that have been completely disassembled; F represents the set of transitions, and the transition corresponds to the action in reinforcement learning, that is, the disassembly operation of the product; n represents the number of places; m represents the number of transitions; the input matrix In×m defines the directed arcs from the place to the transition, and its elements i, j, and k take values ​​0 or 1. When there is no directed arc from the j place to the k transition, it takes 0, otherwise it takes 1; the output matrix Om×n defines the directed arcs from the transition to the place, and its elements o, k, and j take values ​​0 or 1. When there is no directed arc from the k transition to the j place, it takes 0, otherwise it takes 1; m0 is an n-dimensional vector used to represent the initial disassembly state, and its elements take values ​​1 or 0. 1 represents that the disassembly operation has been performed, and 0 represents that the disassembly operation has not been performed.

4. The selective disassembly optimization method based on the PPO algorithm according to claim 3 is characterized in that: Each transition of the five-tuple of the DPN model includes at least one input and two outputs; each transition obtains tokens from at least one or more places and distributes the tokens to different places; when a place has a token, only one of the output transitions can be activated; when a transition points to different places from multiple directed arcs, if the transition is activated, all the places pointed to will obtain tokens; Among them, the m-dimensional vector t fired Indicates the activation status of each transition. Its element value is 1 or 0, indicating that the corresponding transition has been activated or not, respectively. new Represents the new disassembly state, and the disassembly state transformation equation is as follows: m new =m0+t fired ·(Om×n-I T n×m) Where T represents the transpose operation of the matrix.

5. The selective disassembly optimization method based on the PPO algorithm according to claim 1 is characterized in that: The reward function of the selective disassembly model based on reinforcement learning includes: an environmental pollution index of the disassembly process, a recycling and reuse value index of the disassembly process, and a recycling and reuse value index of the disassembly process.

6. The selective disassembly optimization method based on the PPO algorithm according to claim 1 is characterized in that: The S2 specifically includes: Assume that there are α places and β transitions in the DPN model, and the state s is represented by the α-element vector s=(s1, s2, ..., s i ,s α ) indicates that its element s i The value of is 0 or 1, indicating the state of having or not having tokens in the corresponding library respectively; The activity is represented by an integer a, the value range of a is 1≤a≤α, which represents the sequence number of the corresponding transition; The set of all selectable activities in state s is represented by A(s), which is determined according to the DPN model; where the β-dimensional vector X = (x1, x2, ..., x α ) indicates whether the transition can be activated in state s, and the value of its element x i The value is 1 or 0, indicating whether the corresponding transition can be activated. The calculation formulas of x and A(s) are as follows: x=s·Iα×β A(s)={a∈{1,2,...,β}|x i =1}; The reward function Reward for constructing the selective disassembly model is expressed as follows: Reward=w1·h(x i )+w2·v(x i )+w3·w(x i ) Among them, w1, w2 and w3 represent the weights of each indicator, h(x i ) indicates the disassembly operation x i The sum of the environmental pollution indexes of the selected parts removed, v(x i ) indicates the disassembly operation x i The sum of the recyclable mass of the selected dismantled parts, w(x i ) refers to the sum of the recoverable and recyclable values ​​of the selected parts removed by the disassembly operation xi.

7. The selective disassembly optimization method based on the PPO algorithm according to any one of claims 5-6, characterized in that: Environmental pollution index of the disassembly process: Remove the selected parts with the least disassembly operation and make the environmental pollution index of the disassembly process h (x) is the largest, expressed as follows: Among them, h(x i ) indicates the disassembly operation x i The sum of the environmental pollution indexes of the selected parts removed is calculated as follows: Among them, C s represents the set of parts that are selectively disassembled, C i represents the set of parts removed during the disassembly process except for the selective disassembly, har(c) represents the environmental pollution index level of part c, and N represents the number of parts; The recyclable quality indicators of the disassembly process are: Disassemble the selected parts with the least disassembly operation and make the disassembly process recyclable and reusable. v (x) is the largest, expressed as follows: Among them, v(x i ) indicates the disassembly operation x i The sum of the recyclable mass of the selected dismantled parts is calculated as follows: Among them, val(c) represents the recyclable quality of part c; Recyclable reuse value indicators of the disassembly process: Disassemble the selected parts with the least disassembly operation and make the recycling value index of the disassembly process w (x) is the largest, expressed as follows: Among them, w(x i ) indicates the disassembly operation x i The sum of the recoverable and reusable values ​​of the selected dismantled parts is calculated as follows: Among them, wei(c) represents the recyclable reuse value of part c.

8. The selective disassembly optimization method based on the PPO algorithm according to claim 7 is characterized in that: The PPO algorithm is an actor-critic network architecture, where the actor network update is expressed as follows: The critic network update is expressed as follows: Among them, r t (θ) represents the probability ratio of the new and old strategies, which is used to measure the magnitude of change between the new and old strategies. The advantage function is used to measure the goodness of taking an action relative to the average action in the current state. t (θ), 1-∈, 1+∈) represents the value used to limit r t (θ) changes in the range, when r t When (θ) is too large, the Clip function will truncate it to 1+∈; when r t If (θ) is too small, the Clip function will truncate it to 1-∈. represents the expected average value, θ represents the parameters of the policy network, ∈ represents the hyperparameter used to control the clipping range of the importance sampling ratio, ranging from (0,1); V θ (s t ) represents the state value output by the critic network, Indicates actual return.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the selective disassembly optimization method based on the PPO algorithm as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a selective disassembly optimization method based on the PPO algorithm as described in any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Virtual prototype disassembly sequence planning method

    CN103164550A

  • Method for building application sequence of software protection technologies based on Petri network

    CN107545156A

  • Power distribution network voltage reactive power control method based on near-end strategy optimization algorithm

    CN115021276A

  • Selective disassembly planning method and system based on RQL algorithm

    CN117151425A

  • Airsoldier hierarchical autonomous behavior decision modeling method based on PPO and HIRO algorithms

    CN118276443A