Selective disassembly optimization method, equipment and medium based on PPO algorithm

Abstract

The present invention proposes a selective disassembly optimization method based on the PPO algorithm, which includes: constructing a DPN model for disassembling end-of-life (EOL) products that need to be disassembled; constructing a selective disassembly model based on reinforcement learning based on the DPN model, and training the selective disassembly model using the PPO algorithm; inputting the EOL products that need to be disassembled in real time into the trained selective disassembly model to obtain the optimal set of disassembly actions. The selective disassembly optimization method based on the PPO algorithm of the present invention not only improves the efficiency and effectiveness of the disassembly process, but also demonstrates greater adaptability and stability when facing complex and dynamic disassembly tasks. In addition, by combining other advanced reinforcement learning algorithms and exploring better modeling schemes, it is expected to further enhance the performance and practicality of the selective disassembly planning method, providing strong technical support for sustainable manufacturing and resource recycling.

Description

Technical Field

[0001] The present invention belongs to the technical field of disassembly technology, and in particular relates to a selective disassembly optimization method, device and medium based on a PPO algorithm. Background Art

[0002] The manufacturing industry consumes vast amounts of energy and mineral resources, and end-of-life (EOL) products accumulate in landfills each year. EOL products, such as electronics, automobiles, shared bicycles, and industrial equipment, often contain valuable components and materials that can be reused or recycled. Recovering value from these products helps reduce raw material extraction and energy consumption, contributing to sustainable manufacturing. Disassembly is an essential process for recycling EOL products, and optimal / near-optimal disassembly plans should be determined to maximize EOL value recovery. This decision-making process is known as disassembly planning (DSP). Because many EOL products have been in service for a long time, their quality condition may have deteriorated to some extent compared to their original state. Therefore, disassembling EOL products is significantly more complex than assembling new ones. While complete DSP can disassemble all parts, not all parts are valuable, and disassembly costs are prohibitive. In response to this situation, domestic and international experts have conducted in-depth research on selective disassembly and believe it is a better option than complete disassembly.

[0003] As the structure of EOL products continues to become more complex, the search space for disassembly sequences will become increasingly larger. Therefore, heuristic algorithms are widely used to find the optimal disassembly sequence. The disassembly sequence is obtained before the end-of-line product is disassembled. However, due to the uncertainty of the product structure, a predetermined disassembly sequence is often impractical. As for heuristic algorithms, slight changes to the EOL product structure may restart the entire optimization process. Therefore, the decisions of heuristic algorithms cannot adapt to changes in the EOL product structure. Some studies use deep Q-networks (DQNs) to accelerate learning. DQNs use experience replay and target fixation to improve algorithm stability. However, due to the non-static distribution of samples and accumulated errors during training, it sometimes leads to overlearning or repeated use of certain experiences, making model training unstable. Summary of the Invention

[0004] The present invention aims to propose a method, device, and medium for selective disassembly optimization based on the PPO algorithm. Selective disassembly reduces disassembly costs and environmental impact by removing only valuable parts. A modeling approach based on Petri nets (DPNs) accurately describes the product's topology and the connections between its parts, allowing precise expression of the constraints governing the disassembly process.

[0005] In order to achieve the above object, the present invention provides a selective disassembly optimization method based on the PPO algorithm, the method comprising:

[0006] Step 1: Construct a DPN model for dismantling waste products based on the EOL products that need to be dismantled;

[0007] Step 2: Build a selective disassembly model based on reinforcement learning according to the DPN model, and use the PPO algorithm to train the selective disassembly model;

[0008] Step 3: Input the EOL products that need to be disassembled in real time into the trained selective disassembly model to obtain the optimal set of disassembly actions.

[0009] Furthermore, the S1 specifically includes:

[0010] Step 1.1: Construct the five-tuple of the DPN model based on the topological structure of the EOL products, the connection relationship between the EOL products, and the constraint relationship between the disassembly operations;

[0011] Step 1.2: Calculate the transformation of the EOL product disassembly state based on the output of each transition of the DPN model's quintuple.

[0012] Furthermore, the quintuple of the DPN model is expressed as follows:

[0013] DPN=(P, F, In×m, Om×n, m0)

[0014] Where P represents the set of places, the root node places represent EOL products that have not been disassembled, the intermediate nodes represent EOL product middleware that has not been completely disassembled, and the leaf nodes represent EOL product parts that have been completely disassembled; F represents the set of transitions, which correspond to actions in reinforcement learning, that is, product disassembly operations; n represents the number of places; m represents the number of transitions; the input matrix In×m defines directed arcs from places to transitions, and its elements i, j, and k take values ​​0 or 1. When there is no directed arc from place j to place k, it takes 0, otherwise it takes 1; the output matrix Om×n defines directed arcs from transitions to places, and its elements o, k, and j take values ​​0 or 1. When there is no directed arc from place k to place j, it takes 0, otherwise it takes 1; m0 is an n-dimensional vector used to represent the initial disassembly state, and its elements take values ​​1 or 0. 1 represents that the disassembly operation has been performed, and 0 represents that the disassembly operation has not been performed.

[0015] Furthermore, each transition of the DPN model's quintuple includes at least one input and two outputs. Each transition obtains tokens from at least one place and distributes the tokens to different places. When a place has a token, only one of the output transitions can be activated. When a transition points to different places from multiple directed arcs, all the places pointed to will obtain tokens after the transition is activated.

[0016] Among them, the m-dimensional vector t fired Indicates the activation status of each transition. Its element value is 1 or 0, indicating that the corresponding transition has been activated or not. new Represents the new disassembly state, and the disassembly state transformation equation is as follows:

[0017] m new =m0+t fired ·(Om×nI T n×m)

[0018] Where T represents the transpose operation of the matrix.

[0019] Furthermore, the reward function of the selective disassembly model based on reinforcement learning includes: an environmental pollution index of the disassembly process, a recyclable and reuse quality index of the disassembly process, and a recycling and reuse value index of the disassembly process.

[0020] Furthermore, the S2 specifically includes:

[0021] Assume that there are α places and β transitions in the DPN model, and the state s is represented by the α element vector s = (s1, s2, ..., s i , s α ) indicates that its element s i The value of is 0 or 1, indicating that the corresponding library has or does not have a token.

[0022] The activity is represented by an integer a, the value range of a is 1≤a≤α, which represents the sequence number of the corresponding transition;

[0023] The set of all selectable activities in state s is represented by A(s), which is determined according to the DPN model; where the β-dimensional vector X = (x1, x2, ..., x α ) indicates whether the transition can be activated in state s, and the value of its element is x i The value is 1 or 0, indicating whether the corresponding transition can be activated. The calculation formulas of x and A(s) are as follows:

[0024] x=s·Iα×β

[0025] A(s)={a∈{1, 2,...,β}|x i =1};

[0026] The reward function Reward for constructing the selective disassembly model is expressed as follows:

[0027] Reward=w1·h(x i )+w2·v(x i )+w3·w(x i )

[0028] Among them, w1, w2 and w3 represent the weights of each indicator, h(x i ) represents the disassembly operation x i The sum of the environmental pollution indexes of the selected parts removed, v(x i ) represents the disassembly operation x i The sum of the recyclable mass of the selected dismantled parts, w(x i ) indicates disassembly operation x i The sum of the recoverable and reusable values ​​of the selected dismantled parts.

[0029] Furthermore, the environmental pollution index of the disassembly process is:

[0030] The selected parts are disassembled with the least disassembly operations, and the environmental pollution index fx(x) of the disassembly process is maximized, which is expressed as follows:

[0031]

[0032] Among them, h(x i ) represents the disassembly operation x i The sum of the environmental pollution indexes of the selected parts removed is calculated as follows:

[0033]

[0034] Among them, C s represents the set of parts that are selectively disassembled, C i represents the set of parts removed during the disassembly process except for selective disassembly, har(c) represents the environmental pollution index level of part c, and N represents the number of parts;

[0035] Recyclable and reusable quality indicators of the disassembly process:

[0036] Disassemble the selected parts with the least disassembly operation and make the disassembly process recyclable and reusable. v (x) is the largest, which is expressed as follows:

[0037]

[0038] Among them, v(x i ) represents the disassembly operation x i The sum of the recyclable mass of the selected dismantled parts is calculated as follows:

[0039]

[0040] Among them, val(c) represents the recyclable mass of part c;

[0041] Recyclable reuse value indicators of the disassembly process:

[0042] Disassemble the selected parts with the least disassembly operation and make the recycling value index of the disassembly process w (x) is the largest, which is expressed as follows:

[0043]

[0044] Where w(xi) represents the disassembly operation x i The sum of the recyclable and reusable values ​​of the selected dismantled parts is calculated as follows:

[0045]

[0046] Among them, wei(c) represents the recycling value of part c.

[0047] Furthermore, the PPO algorithm is an actor-critic network architecture, where the actor network update is expressed as follows:

[0048]

[0049] The critic network update is expressed as follows:

[0050] Among them, rt(θ) represents the probability ratio of the new and old strategies, which is used to measure the change range of the new and old strategies. The advantage function is used to measure the goodness of taking an action relative to the average action in the current state, clip(r t (θ), 1-∈, 1+∈) represents the function used to limit r t (θ) changes in the range, when r t When (θ) is too large, the Clip function will truncate it to 1+∈; when r t If (θ) is too small, the Clip function will truncate it to 1-∈, represents the expected average value, θ represents the parameters of the policy network, ∈ represents the hyperparameter used to control the clipping range of the importance sampling ratio, ranging from (0,1); V θ (s t ) represents the state value output by the critic network, Indicates actual return.

[0051] In a second aspect, an embodiment of the present invention further provides an electronic device, including:

[0052] one or more processors;

[0053] a memory for storing one or more programs,

[0054] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the aforementioned selective disassembly optimization methods based on the PPO algorithm.

[0055] In a second aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned selective disassembly optimization methods based on the PPO algorithm.

[0056] The beneficial technical effects of the present invention are at least as follows:

[0057] The present invention is based on the selective disassembly planning problem of the PPO algorithm. Selective disassembly reduces disassembly costs and environmental impacts by only disassembling valuable parts. The modeling method based on Petri nets (DPN) can accurately describe the topological structure of the product and the connection relationship between parts, so that the constraint relationship of the disassembly process can be accurately expressed. By introducing the PPO algorithm in reinforcement learning, it is possible to autonomously learn and adapt to the optimal disassembly strategy when faced with complex and dynamically changing product structures. Compared with the traditional DQN algorithm, the PPO algorithm has higher stability and convergence speed, mainly due to its policy optimization and clipping mechanism, which can effectively control the amplitude of policy updates, thereby avoiding instability during the policy update process.

[0058] This invention not only improves the efficiency and effectiveness of the disassembly process, but also demonstrates greater adaptability and stability when faced with complex and dynamic disassembly tasks. Furthermore, by combining it with other advanced reinforcement learning algorithms and exploring more optimized modeling schemes, it is expected to further enhance the performance and practicality of selective disassembly planning methods, providing strong technical support for sustainable manufacturing and resource recycling. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The present invention is further described with reference to the accompanying drawings. However, the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative effort.

[0060] Figure 1 This is a flow chart of the selective disassembly optimization method based on the PPO algorithm of the present invention.

[0061] Figure 2 This is an example diagram of DPN according to an embodiment of the present invention.

[0062] Figure 3 This is a flowchart of a selective disassembly planning method based on the PPO algorithm according to an embodiment of the present invention.

[0063] Figure 4 Schematic diagram of an experimental use case of an embodiment of the present invention.

[0064] Figure 5 Schematic diagram of the Petri net model of the experimental use case of the embodiment of the present invention.

[0065] Figure 6 Schematic diagram of the reward curve for 500 iterations of the PPO algorithm according to an embodiment of the present invention.

[0066] Figure 7 This is a schematic diagram of the reward curve for 500 iterations of the DQN algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0068] like Figure 1 As shown, the present invention discloses a selective disassembly optimization method based on the PPO algorithm, the method comprising:

[0069] Step 1: Construct a DPN model for dismantling end-of-life products based on the EOL products that need to be dismantled.

[0070] Furthermore, the S1 specifically includes:

[0071] Step 1.1: Construct the five-tuple of the DPN model based on the topological structure of the EOL products, the connection relationship between the EOL products, and the constraint relationship between the disassembly operations;

[0072] Step 1.2: Calculate the transformation of the EOL product disassembly state based on the output of each transition of the DPN model's quintuple.

[0073] The five-tuple of the DPN model is expressed as follows:

[0074] DPN=(P, F, In×m, Om×n, m0)

[0075] Where P represents the set of places, the root node places represent EOL products that have not been disassembled, the intermediate nodes represent EOL product middleware that has not been completely disassembled, and the leaf nodes represent EOL product parts that have been completely disassembled; F represents the set of transitions, which correspond to actions in reinforcement learning, that is, product disassembly operations; n represents the number of places; m represents the number of transitions; the input matrix In×m defines directed arcs from places to transitions, and its elements i, j, and k take values ​​0 or 1. When there is no directed arc from place j to place k, it takes 0, otherwise it takes 1; the output matrix Om×n defines directed arcs from transitions to places, and its elements o, k, and j take values ​​0 or 1. When there is no directed arc from place k to place j, it takes 0, otherwise it takes 1; m0 is an n-dimensional vector used to represent the initial disassembly state, and its elements take values ​​1 or 0. 1 represents that the disassembly operation has been performed, and 0 represents that the disassembly operation has not been performed.

[0076] Furthermore, in DPN, each transition has at least one input and two outputs, which means that a transition can obtain tokens from one or more places and distribute tokens to different places. In this case, an "or" logical relationship is formed: when a place has a token, only one of the output transitions can be activated. On the other hand, a transition may have multiple directed arcs pointing to different places, which means that when the transition is activated, all the places it points to will obtain tokens. In this case, an "and" logical relationship is formed. Use the m-dimensional vector t fired Indicates the activation status of each transition. Its element value is 1 or 0, indicating that the corresponding transition has been activated or not. new Represents the new disassembly state, and the disassembly state transformation equation is as follows:

[0077] m new =m0+t fired ·(Om×nI T n×m)

[0078] Where T represents the transpose operation of the matrix.

[0079] Among them, such as Figure 2 As mentioned above, a simple example based on Petri net modeling is given. In the example, the set of places and transitions and the input matrix and output matrix are shown below. The initial state is m0. The root place has a token that can select transitions, but only one transition can be selected at a time, that is, t1 or t2. When t1 transition is selected, t fired =(1,0,0,0), then the disassembly state will become m=(0,1,0,0,0,1).

[0080] Step 2: Build a selective disassembly model based on reinforcement learning according to the DPN model, and use the PPO algorithm to train the selective disassembly model.

[0081] Specifically, states and actions are crucial concepts in reinforcement learning. The essence of reinforcement learning is to learn control strategies through trial and error, through the interaction between an agent and its environment. In this process, the agent performs actions and observes changes in the environment's state. Based on the rewards (feedback) received, the agent continuously refines its strategy to adapt to future tasks. In most reinforcement learning environments, the choice of action is fixed. However, in the disassembly problem, the choice of action is state-dependent, and different states correspond to different numbers of states. Therefore, different action sets need to be selected based on different states.

[0082] Furthermore, the S2 specifically includes:

[0083] Assume that there are α places and β transitions in the DPN model, and the state s is represented by the α element vector s = (s1, s2, ..., s i , s α ) indicates that its element s i The value of is 0 or 1, indicating that the corresponding library has or does not have a token.

[0084] The activity is represented by an integer a, the value range of a is 1≤a≤α, which represents the sequence number of the corresponding transition;

[0085] The set of all selectable activities in state s is represented by A(s), which is determined according to the DPN model; where the β-dimensional vector X = (x1, x2, ..., x α ) indicates whether the transition can be activated in state s, and the value of its element is x i The value is 1 or 0, indicating whether the corresponding transition can be activated. The calculation formulas of x and A(s) are as follows:

[0086] x=s·Iα×β

[0087] A(s)={a∈{1, 2,...,β}|x i =1};

[0088] The reward function Reward for constructing the selective disassembly model is expressed as follows:

[0089] Reward=w1·h(X i )+w2·v(x i )+w3·w(x i )

[0090] Among them, w1, w2 and w3 represent the weights of each indicator, h(x i) represents the disassembly operation x i The sum of the environmental pollution indexes of the selected parts removed, v(x i ) represents the disassembly operation x i The sum of the recyclable mass of the selected dismantled parts, w(x i ) indicates disassembly operation x i The sum of the recoverable and reusable values ​​of the selected dismantled parts.

[0091] In reinforcement learning, reward is a crucial concept that directly guides the agent's learning process and behavioral strategy. Reward is the feedback signal an agent receives when interacting with the environment. It tells the agent how effective its behavior is, thereby guiding it to learn the optimal behavioral strategy. The design of reward has a crucial impact on the performance of reinforcement learning algorithms. A reasonable reward setting can help the agent learn the correct behavioral strategy more quickly. The reward setting in this article mainly involves three objectives:

[0092] Remove the selected parts with the least disassembly operation and make the environmental pollution index of the disassembly process f h (x) is the largest, which is expressed as follows:

[0093]

[0094] Among them, h(x i ) represents the disassembly operation x i The sum of the environmental pollution indexes of the selected parts removed is calculated as follows:

[0095]

[0096] Among them, C s represents the set of parts that are selectively disassembled, C i represents the set of parts removed during the disassembly process, excluding selective disassembly. har(c) represents the environmental pollution level of part c, and N represents the number of parts. The environmental pollution level of a part is represented by six integers from 0 to 5, with larger values ​​representing higher levels of environmental pollution. Parts with higher levels of environmental pollution should be removed and processed to avoid environmental damage.

[0097] Recyclable and reusable quality indicators of the disassembly process:

[0098] Disassemble the selected parts with the least disassembly operation and make the disassembly process recyclable and reusable. v (x) is the largest, which is expressed as follows:

[0099]

[0100] Among them, v(x i ) represents the disassembly operation x i The sum of the recyclable mass of the selected dismantled parts is calculated as follows:

[0101]

[0102] Among them, val(c) represents the recyclable quality of part c; similarly, six integers from 0 to 5 are used to represent the recyclable quality of the part, and the larger the value, the higher the quality;

[0103] Recyclable reuse value indicators of the disassembly process:

[0104] Disassemble the selected parts with the least disassembly operation and make the recycling value index of the disassembly process w (x) is the largest, which is expressed as follows:

[0105]

[0106] Where w(xi) represents the disassembly operation x i The sum of the recyclable and reusable values ​​of the selected dismantled parts is calculated as follows:

[0107]

[0108] Among them, wei(c) represents the recycling and reuse value of part c, and six integers from 0 to 5 are used to represent the quality of the part. The higher the value, the higher the value and the more necessary it is to recycle.

[0109] Furthermore, the PPO algorithm is a near-line policy optimization algorithm that utilizes an actor-critic architecture. PPO uses a clipping mechanism to limit the policy update amplitude, which helps improve training stability and reduce instability caused by excessive update amplitudes. PPO is insensitive to the choice of initial policy parameters and hyperparameters, making parameter tuning simpler. In addition to the above advantages, PPO is an on-policy algorithm. During training, the agent uses its current policy to directly interact with the environment and uses the data from this interaction to directly update the policy. This is particularly important in large-scale disassembly operations. It consists of two neural networks: an actor network and a critic network. The actor network selects actions, while the critic network calculates the advantage function. The actor network first interacts with the environment to obtain a series of state, action, reward, and next state data. The critic network then calculates the advantage function and the actor network calculates the ratio between the old and new policies based on the collected data. Finally, based on this ratio and clipping, the policy update amplitude is limited, and the network parameters are updated. The following is the neural network parameter update method:

[0110]

[0111] The critic network update is expressed as follows:

[0112]

[0113] where r t (θ) is the probability ratio of the new and old strategies, which is used to measure the change range of the new and old strategies. The advantage function is used to measure the goodness of taking an action relative to the average action in the current state, clip(r t (θ), 1-∈, 1+∈) is used to limit r t (θ) changes in the range, when r t When (θ) is too large, the Clip function will truncate it to 1+∈; when r t If (θ) is too small, the Clip function will truncate it to 1-∈, which helps prevent the policy update from being too large or too small, thereby improving the stability of training and the effectiveness of the update; represents the expected mean value, θ represents the parameters of the policy network, ∈ represents the hyperparameter used to control the clipping range of the importance sampling ratio, ranging from (0, 1), V θ (s t ) represents the state value output by the critic network, Indicates actual return.

[0114] like Figure 3As shown in Figure 2, the solution steps of the selective disassembly planning method based on the PPO algorithm are summarized as follows:

[0115] Step 1: Establish a DPN model for disassembling scrap products, select the parts to be disassembled, and set the weights of each indicator;

[0116] Step 2: Set parameters such as δ, ∈, γ, the learning rate of the Actor and Critic networks, and the maximum number of iterations episodes;

[0117] Step 3: Reset the DPN initial state;

[0118] Step 4: Calculate the activity set A(s) under the current state s according to the DPN and activity set selection formula;

[0119] Step 5: The Actor network randomly selects an action from the current state's activity set using normal distribution;

[0120] Step 6: Interact with the environment and obtain data on the next state, reward, and whether to end the while loop;

[0121] Step 7: The critic network uses the acquired data to calculate the error and advantage function;

[0122] Step 7: The Actor network uses the state to obtain the activity selection probability ratio of the new and old strategies and perform pruning;

[0123] Step 8: The Actor network and the Critic network update network parameters simultaneously;

[0124] Step 9: Repeat steps 4 to 6 until all parts that need to be disassembled are removed;

[0125] Step 10: Repeat steps 4 to 8 until the maximum number of iterations is reached;

[0126] Step 11: Continuously learn and update the neural network parameters through continuous interaction with the environment to obtain the optimal disassembly steps.

[0127] Step 3: Input the EOL products that need to be disassembled in real time into the trained selective disassembly model to obtain the optimal set of disassembly actions.

[0128] Furthermore, we first describe and model the EOL products that require real-time disassembly, and then present the experimental results. All algorithms were coded and tested in PyCharm 2022.1.3 under the Windows 10 operating system. The CPU model is Intel(R) Core(TM) i5-10500CPU @ 3.10GHz, the GPU model is NVIDIA GeForce RTX2060SUPER, and the running memory is 16GB.

[0129] For example, the experiment Figure 4 As shown, the discarded product has 8 parts, and the constraint relationship between the parts is represented by dot-dashed lines. First, Petri net modeling is performed based on the constraint relationship between the parts for this test case. The established Petri net model is as follows Figure 4 As shown, it contains 21 places and 19 transitions. The parts disassembled in each transition are different, but some places are the same. For example, there are three places in place No. 21 and three places in place No. 20. This is because if the disassembly routes are different, the disassembled parts will be different, and there will be multiple places with the same place. However, in the experiment, since a place can only choose one transition, that is, there can only be one disassembly route, the same part will not be disassembled multiple times.

[0130] Select parts B, D, and F as the parts that need to be disassembled. Parts B, D, and F correspond to places P15, P17, and P19, respectively. The environmental pollution levels, standardized recycling quality, and value of all parts corresponding to the known waste products are shown in Table 1. The weights of each indicator are set to w1 = 0.5, w2 = 0.3, and w3 = 0.2. In order to determine the ideal parameter combination, the present invention sets several important parameters, the learning rate of the Actor network, and the _ lr = {0.01, 0.001}, the learning rate of the Critic network _ lr = {0.01, 0.001}, and cropping ratio ∈ = {0.1, 0.2, 0.3}, resulting in a total of 12 (i.e., 2 × 2 × 3) parameter combinations. Each parameter combination was used to implement selective disassembly based on the PPO algorithm. The experimental results are shown in Table 2.

[0131] Table 1 Properties of all disassembled parts

[0132] Part Number Environmental pollution level Standardizing recycling quality Standardized recycling value A 2 2 1 B 5 3 1 C 1 2 4 D 1 4 3 E 1 1 1 F 2 1 4 G 2 1 1 H 1 2 1

[0133] Table 2 Running results of various parameter combinations

[0134]

[0135]

[0136] From Table 2, we can observe that the reward obtained when the learning rate of the Actor network is 1e-3 and the learning rate of the Critic network is 1e-2 is the largest. However, when the cropping rate is 0.1 and 0.3, the reward convergence state is not very good. Therefore, the cropping rate is 0.2, the learning rate of the Actor network is 1e-3, and the learning rate of the Critic network is 1e-2 as the optimal parameter combination. The specific image convergence curve is shown in the figure. Figure 6 shown.

[0137] In order to further test the performance of the proposed method, the present invention selected the DQN algorithm for comparison. The DQN algorithm is currently widely used in reinforcement learning to solve the disassembly sequence planning problem. The DQN algorithm continuously learns and updates the q value by using a neural network to fit the q value to find the optimal solution. In order to accurately compare the advantages and disadvantages of the two methods, the environment used by the DQN algorithm is the same as that used by the PPO algorithm. The reward convergence situation of 500 iterations of the DQN algorithm is as follows: Figure 7 As shown in the figure, it is clear that the DQN algorithm for solving the selective disassembly planning problem converges slowly and has difficulty converging. The final reward fluctuates between several values. This may be due to the DQN algorithm's use of experience replay, which results in repeated use of a lot of data. However, the PPO algorithm is an on-policy algorithm that directly uses collected data rather than storing it first. Furthermore, the PPO algorithm directly updates the policy rather than iterating on the value first. Compared to value-based strategies, policy-based approaches are more flexible and effective in handling complex and highly uncertain tasks.

[0138] In summary, the present invention is based on the selective disassembly planning problem of the PPO algorithm. Selective disassembly reduces disassembly costs and environmental impacts by only disassembling valuable parts. The modeling method based on Petri nets (DPN) can accurately describe the topological structure of the product and the connection relationship between parts, so that the constraint relationship of the disassembly process can be accurately expressed. By introducing the PPO algorithm in reinforcement learning, the present invention can autonomously learn and adapt to the optimal disassembly strategy when faced with complex and dynamically changing product structures. Compared with the traditional DQN algorithm, the PPO algorithm has higher stability and convergence speed, mainly due to its policy optimization and clipping mechanism, which can effectively control the amplitude of policy updates, thereby avoiding instability during the policy update process.

[0139] The selective disassembly planning method based on the PPO algorithm performs well in dealing with complex disassembly tasks. By setting different learning rates of the Actor and Critic networks, as well as different combinations of clipping rate parameters, the present invention found that when the Actor learning rate is 1e-3, the Critic learning rate is 1e-2, and the clipping rate is 0.2, the PPO algorithm performs best and can converge to the optimal solution in the shortest time. In contrast, the DQN algorithm converges slowly after 500 iterations under the same environment, and there are significant fluctuations in the convergence process. This shows that the DQN algorithm has certain limitations when dealing with large-scale state spaces and complex decision-making problems. It may be due to the data reuse problem caused by the experience replay mechanism, which makes it difficult for the model to stabilize during training.

[0140] The selective disassembly planning method based on the PPO algorithm not only improves the efficiency and effectiveness of the disassembly process, but also demonstrates greater adaptability and stability when faced with complex and dynamic disassembly tasks. Furthermore, by combining it with other advanced reinforcement learning algorithms and exploring more optimal modeling schemes, it is expected to further enhance the performance and practicality of the selective disassembly planning method, providing strong technical support for sustainable manufacturing and resource recycling.

[0141] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A selective disassembly optimization method based on the PPO algorithm is characterized by: The method comprises: Step 1: Construct a DPN model for dismantling waste products based on the EOL products that need to be dismantled; Step 2: Build a selective disassembly model based on reinforcement learning according to the DPN model, and use the PPO algorithm to train the selective disassembly model; Step 3: Input the EOL products that need to be disassembled in real time into the trained selective disassembly model to obtain the optimal set of disassembly actions; The reward function of the selective disassembly model based on reinforcement learning includes: an environmental pollution index of the disassembly process, a recycling and reuse value index of the disassembly process, and a recycling and reuse value index of the disassembly process; The step 2 specifically includes: Assume that there is Repositories and transition, state s is used Metavector Indicates that its elements The value of is 0 or 1, indicating that the corresponding library has or does not have a token. The activity is represented by an integer a, and the value range of a is 1≤a≤ , indicating the sequence number of the corresponding transition; The set of all selectable activities in state s is represented by A(s), which is determined according to the DPN model; dimensional vector Indicates whether the transition can be activated in state s, and the value of its elements The value is 1 or 0, indicating whether the corresponding transition can be activated. The calculation formulas of x and A(s) are as follows: ； ； Constructing a reward function for the selective disassembly model , which is expressed as follows: ； in, 、 and Indicates the weight of each indicator, Indicates disassembly operation The sum of the environmental pollution indexes of the selected parts removed, Indicates disassembly operation The sum of the recyclable mass of the selected dismantled parts, Indicates disassembly operation The sum of the recoverable and reusable values ​​of the selected dismantled parts; Environmental pollution index of the disassembly process: Remove the selected parts with the least disassembly operation and minimize the environmental pollution index during the disassembly process. Maximum, expressed as follows: ； in, Indicates disassembly operation The sum of the environmental pollution indexes of the selected parts removed is calculated as follows: ； in, Represents a set of parts that are selectively disassembled. Represents the collection of parts removed during the disassembly process except for selective disassembly. represents the environmental pollution index level of part c, and N represents the number of parts; Recyclable and reusable quality indicators of the disassembly process: Disassemble the selected parts with the least disassembly operation and make the disassembly process recyclable and reusable. Maximum, expressed as follows: ； in, Indicates disassembly operation The sum of the recyclable mass of the selected dismantled parts is calculated as follows: ； Among them, val(c) represents the recyclable mass of part c; Recyclable reuse value indicators of the disassembly process: Remove selected parts with minimal disassembly operations and maximize the recycling value of the disassembly process Maximum, expressed as follows: ； in, Indicates disassembly operation The sum of the recyclable and reusable values ​​of the selected dismantled parts is calculated as follows: ； Among them, wei(c) represents the recycling value of part c.

2. The selective disassembly optimization method based on the PPO algorithm according to claim 1 is characterized in that: The step 1 specifically includes: Step 1.1: Construct the five-tuple of the DPN model based on the topological structure of the EOL products, the connection relationship between the EOL products, and the constraint relationship between the disassembly operations; Step 1.2: Calculate the transformation of the EOL product disassembly state based on the output of each transition of the DPN model's quintuple.

3. The selective disassembly optimization method based on the PPO algorithm according to claim 2 is characterized in that: The five-tuple of the DPN model is expressed as follows: ； Where P represents the set of places, the root node places represent EOL products that have not been disassembled, the intermediate nodes represent EOL product middleware that has not been disassembled to the end, and the leaf nodes represent EOL product parts that have been completely disassembled; F represents the set of transitions, which correspond to actions in reinforcement learning, namely, product disassembly operations; n represents the number of places; m represents the number of transitions; the input matrix I Defines a directed arc from the library to the transition, whose elements i, j, k are 0 or 1. When there is no directed arc from the j library to the k library, it is 0, otherwise it is 1. Output matrix O Defines a directed arc from transition to place, whose elements o, k, j take values ​​0 or 1. When there is no directed arc from k to j place, the value is 0, otherwise it is 1. It is an n-dimensional vector used to represent the initial disassembly state. Its element value is 1 or 0. 1 represents that the disassembly operation has been performed, and 0 represents that the disassembly operation has not been performed.

4. The selective disassembly optimization method based on the PPO algorithm according to claim 3 is characterized in that: Each transition in the DPN model's quintuple consists of at least one input and two outputs. Each transition obtains tokens from at least one place and distributes them to different places. When a place has a token, only one of the output transitions can be activated. When a transition points to different places from multiple directed arcs, all the places it points to will obtain tokens if the transition is activated. Among them, the m-dimensional vector t fired Indicates the activation status of each transition. Its element value is 1 or 0, indicating that the corresponding transition has been activated or not. Represents the new disassembly state, and the disassembly state transformation equation is as follows: ； Where T represents the transpose operation of the matrix.

5. The selective disassembly optimization method based on the PPO algorithm according to claim 1 is characterized in that: The PPO algorithm is an actor-critic network architecture, where the actor network update is expressed as follows: ； The critic network update is expressed as follows: ； in, The probability ratio of the new and old strategies is used to measure the magnitude of the change between the new and old strategies. The advantage function is used to measure the goodness of taking an action relative to the average action in the current state. Indicates that it is used to limit The range of change, when If it is too large, the Clip function will truncate it to 1+ ;when If it is too small, the Clip function will truncate it to 1- , represents the expected mean value, represents the parameters of the policy network, Represents the hyperparameter used to control the clipping range of the importance sampling ratio, ranging from (0,1); Represents the state value output by the critic network, Indicates actual return.

6. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the selective disassembly optimization method based on the PPO algorithm as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the selective disassembly optimization method based on the PPO algorithm as described in any one of claims 1 to 5 is implemented.

Images