A Collaborative Optimization Method for Additive Manufacturing Paths Based on Deep Reinforcement Learning

Abstract

This invention discloses a collaborative optimization method for additive manufacturing paths and processes based on deep reinforcement learning. It involves training a path planning reinforcement learning model and a process optimization reinforcement learning model separately, then coupling them. By inputting several optimal actions from the path planning model into the process optimization model in real time, the optimal path and process combination is obtained. This optimal combination is then further input into the path planning model for deposition state updates, iterating continuously to obtain the optimal process path combination. This method exhibits good generalization ability and is not limited to single materials, single heat sources, or single-shaped components. It not only reduces the cost of manual path design but also reduces material and time costs during process exploration and improves the performance of additively manufactured components.

Description

Technical Field

[0001] This invention relates to a collaborative optimization method for additive manufacturing path processes based on deep reinforcement learning, belonging to the field of gradient materials additive manufacturing technology. Background Technology

[0002] Additive manufacturing is an emerging materials processing technology that creates three-dimensional objects by adding materials layer by layer. It offers advantages such as high material utilization and high design freedom. The quality and performance of additively manufactured components are directly related to the additive deposition path and deposition process parameters. However, current additive manufacturing deposition path planning mainly consists of simple paths such as linear, polygonal, and contour paths. Path planning methods that comprehensively consider the impact of the path on the performance of the additively manufactured component are even rarer, and there is no comprehensive method suitable for optimizing both the path and process simultaneously at the part level.

[0003] In recent years, deep reinforcement learning has become an important branch of machine learning. It combines deep learning and reinforcement learning, employing a method of learning data representations by constructing and training multi-layered (deep) neural networks. Reinforcement learning, on the other hand, allows agents to learn optimal behavioral strategies through interaction with their environment. It holds promise for solving the problem of co-optimization of additive manufacturing paths and processes at the part level. Patent CN116629128A – A method for controlling arc additive manufacturing based on deep reinforcement learning – presents a process optimization method based on existing paths, achieving the effect of controlling additive manufacturing. Building upon this, it is hoped that deep reinforcement learning can be further utilized to solve the aforementioned problems. Summary of the Invention

[0004] To address the technical problem of low performance of additive components in existing technologies and the inability to solve the collaborative optimization of additive paths and processes at the part level, this invention proposes a collaborative optimization method for additive paths and processes based on deep reinforcement learning.

[0005] The technical solution adopted by this invention to solve the above problems is as follows: This invention proposes a collaborative optimization method for additive manufacturing path processes based on deep reinforcement learning, comprising:

[0006] Step 1: Construct an additive path planning reinforcement learning environment;

[0007] Step 2: Set up a reward mechanism for the path planning agent to evaluate the deposition path selected by the agent;

[0008] Step 3: Train the path planning reinforcement learning model;

[0009] Step 4: Develop and train the additive manufacturing process optimization model environment;

[0010] Step 5: Couple the path planning reinforcement learning model and the process optimization model to obtain the path-process collaborative optimization reinforcement learning model;

[0011] Step 6: Further train the reinforcement learning model for collaborative optimization of the path process;

[0012] Step 7: Input the additive component into the trained path process collaborative mutual optimization reinforcement learning model to obtain the optimal deposition path and optimal process parameters, and perform actual additive manufacturing based on the optimal deposition path and optimal process parameters.

[0013] Optionally, the additive path planning reinforcement learning environment in step 1 includes the environment boundary of path planning, the initial path planning state, the path planning termination state, the agent's motion space, and path planning visualization.

[0014] Optionally, in step 2, the agent's reward mechanism is a reward function, which is a function of the number of deposition channels, deposition area, intersecting deposition nodes, deposition channel curvature and angle. The quality of the deposition path taken by the agent is comprehensively evaluated based on the reward function.

[0015] Optionally, step 3, training the path planning reinforcement learning model, includes the following steps:

[0016] Step 3.1: Use the real-time motion state of the agent within the constructed path planning environment as the model input;

[0017] Step 3.2: Obtain the agent's evaluation of the current path planning state and environment state, calculate the reward obtained by the agent, and use reinforcement learning algorithm to optimize and update the model parameters;

[0018] Step 3.3: Repeat steps 3.1-3.2 to continuously increase the rewards obtained by the agent during iterative training, thus obtaining an optimized path planning model.

[0019] Optionally, the steps in step 4 for optimizing the additive manufacturing process include:

[0020] Step 4.1: Develop a path-based mesh generation algorithm and further build an automated solution model for the entire process of thermo-mechanical coupling in additive manufacturing;

[0021] Step 4.2: Train the additive manufacturing process optimization model using a multi-machine parallel approach;

[0022] Step 4.3: Use the real-time temperature field, stress field and deformation field data of additive manufacturing as input to the trained additive manufacturing path process optimization model, and use a 3D convolutional network to extract the spatiotemporal information of the environmental state;

[0023] Step 4.4: Assess the current environmental state of additive manufacturing based on the extracted spatiotemporal information;

[0024] Step 4.5: Define the reward obtained by the agent as a function of melt depth, melt width, stress, deformation and peak temperature, and optimize the model parameters through reinforcement learning algorithm to obtain the optimized process planning model;

[0025] Step 4.6: Output optimized process parameters based on the trained additive manufacturing process optimization model to maintain stable deposition channel formation and minimize stress deformation.

[0026] Optionally, step 5, which involves obtaining the path process collaborative mutual optimization reinforcement learning model, includes:

[0027] Step 5.1: Simultaneously initialize the path planning reinforcement learning model and the additive manufacturing process optimization reinforcement learning model;

[0028] Step 5.2: Input the deposition path information contained in the real-time motion of the agent in the path planning reinforcement learning model into the additive path process optimization model, and perform additive thermo-mechanical coupling simulation of the corresponding deposition path;

[0029] Step 5.3: Obtain the optimal deposition path and optimal process parameters through the additive manufacturing process optimization model;

[0030] Step 5.4: The optimal deposition path is further input into the path planning reinforcement learning model, and the path planning model agent adopts the deposition path at the next time step;

[0031] Step 5.5: Repeat steps 5.2-5.4 to proceed with the next path planning and iterate continuously until the path and process are mutually optimized, resulting in a deposition scheme that achieves synergistic optimization of the additive manufacturing path and process.

[0032] The beneficial effects of this invention are as follows: This invention optimizes the path of diverse additive component slices (i.e., the path optimization environment boundary) using a reinforcement learning algorithm, and further optimizes the process using a numerical simulation-based process optimization model. By coupling the two and iterating continuously, mutual optimization of the path and process is achieved, ultimately obtaining the optimal deposition path and matching process parameters. This method has good generalization ability and is not limited to a single material, a single heat source, or a single shape component. This method not only reduces the cost of manual path design but also reduces the material and time costs during process exploration, and it also helps to improve the performance of additive components. Attached Figure Description

[0033] Figure 1 This is a flowchart of an additive manufacturing path process collaborative optimization method based on deep reinforcement learning provided in an embodiment of the present invention;

[0034] Figure 2 This is a framework diagram of a path planning reinforcement learning model provided in an embodiment of the present invention;

[0035] Figure 3 This is a framework diagram of the additive manufacturing path process optimization model provided in the embodiments of the present invention;

[0036] Figure 4 This is a framework diagram of the path process collaborative mutual optimization reinforcement learning model provided in the embodiments of the present invention. Example

[0037] Combination Figure 1-4 This embodiment will be described as follows: Figure 1 As shown, the steps of the additive manufacturing path collaborative optimization method based on deep reinforcement learning provided in this embodiment include:

[0038] Step S1: Construct an additive path planning reinforcement learning environment;

[0039] Furthermore, the additive path planning reinforcement learning environment in step S1 includes the path planning environment boundary, the initial path planning state, the path planning termination state, the agent's motion space, and path planning visualization.

[0040] Diverse closed curves are generated based on polylines, Bézier curves, spline curves, etc., and plotted on a small black image. The image is then binarized, with the pixel value inside the closed curve set to 255 and the pixel value of other parts of the image set to 0. This image is used as the initial state of the path planning environment, and the closed curves serve as the boundary of the environment.

[0041] Based on the aforementioned path environment, a line segment is initialized inside the closed curve. The direction of the line segment represents the deposition direction, and the width of the line segment represents the width of the deposition channel. This line segment serves as the agent in the path planning model. The agent's action is defined as moving at a certain speed along a certain direction within the environmental boundary. The direction of movement can change at any time, and the agent can choose a new starting point within the environment to begin a new movement, representing a new deposition channel. The area traversed by the agent is assigned a pixel value of 128, representing the deposited portion.

[0042] Step S2: Set up a reward mechanism for the path planning agent;

[0043] Furthermore, in step S2, in order to train the agent to learn path planning based on the path planning environment in step S1 using reinforcement learning, a suitable reward function needs to be set. In this embodiment, the reward function is defined as a function of the number of deposition channels, deposition area, intersecting deposition nodes, deposition channel curvature, and angle. The fewer the number of deposition channels, the larger the deposition area, the fewer the deposition channel intersections, the smaller the deposition channel curvature, and the smaller the angle, the higher the reward obtained by the agent.

[0044] Step S3: Train the path planning reinforcement learning model;

[0045] Furthermore, during specific training, the size of the input data and the complexity of the features should be considered to design a suitable reinforcement learning architecture, such as... Figure 2 As shown, the real-time motion state graph of the agent within the environment is used as the network input, and a reinforcement learning algorithm is used to optimize and update the network parameters, enabling the agent to have a more accurate evaluation of the current path planning environment state and maximize the rewards obtained during continuous training, thus continuously optimizing the path taken by the agent.

[0046] In this invention, the network can be selected from frameworks such as AlexNet, VGG16, and VGG19. Alternatively, a network framework of appropriate size can be built based on PyTorch according to the size of the input data. Reinforcement learning algorithms such as A2C, A3C, PPO, and SAC are used to train the model. The network frameworks and optimization algorithms applicable to this method include, but are not limited to, the above. Different network architectures and different reinforcement learning algorithms may result in different path planning strategies adopted by the agent. In practical applications, it is necessary to try different approaches to determine the optimal reinforcement learning model framework.

[0047] Step S4: Develop and train the additive manufacturing process optimization model environment;

[0048] Furthermore, such as Figure 3 As shown, to adapt the path optimization model trained in S3, a path-based mesh generation algorithm was developed. An automated additive manufacturing numerical simulation thermo-coupling model was developed as the process optimization environment. A large amount of data was acquired through this environment to develop the process optimization model. Real-time temperature, stress, and deformation field data from additive manufacturing were used as the environment state. A 3D convolutional network was employed to extract the spatiotemporal information of the environment state. Appropriate process parameters were selected based on the current state to maintain stable melt depth and width, and minimize stress and deformation. The reward obtained by the process optimization model agent was defined as a function of melt depth, melt width, stress, deformation, and peak temperature. Reinforcement learning algorithms were used to continuously optimize the network parameters, increasing the reward obtained by the agent and ultimately realizing a path-based process optimization model. To accelerate the training process and improve the model's generalization ability, a multi-machine parallel training method was adopted for the process model.

[0049] Step S5: Couple the path planning reinforcement learning model and the process optimization model to obtain the path-process collaborative optimization reinforcement learning model;

[0050] Furthermore, the steps for obtaining the path-process collaborative mutual optimization reinforcement learning model include:

[0051] like Figure 4As shown, a path planning reinforcement learning model and an additive manufacturing path process optimization model are initialized simultaneously. The deposition path information contained in several better actions taken by the agent in the path planning reinforcement learning model is input into the additive manufacturing path process optimization model. Additive thermo-mechanical coupling simulations are performed for several deposition paths. The optimal deposition path and its optimal process parameters are obtained from several deposition paths through the additive manufacturing path process optimization model. The optimal deposition path is then input into the path planning reinforcement learning model to determine the optimal action of the agent and proceed to the next path planning step. This process is iterated until the mutual optimization of path and process is completed, resulting in a path-process collaborative mutual optimization reinforcement learning model.

[0052] Step S6: Train the reinforcement learning model for collaborative optimization of the path process;

[0053] Because the process optimization and path optimization models are coupled together, the decision-making methods of the two model agents change after coupling. It is necessary to further optimize both models using reinforcement learning algorithms to improve the decision-making ability of the agents and obtain the optimal deposition path and the optimal process parameters that match it.

[0054] Step S7: Input the additive component into the trained path process collaborative mutual optimization reinforcement learning model to obtain the optimal deposition path and optimal process parameters, and perform actual additive manufacturing based on the optimal deposition path and optimal process parameters.

[0055] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the scope of the present invention, based on the technical essence of the present invention and within the spirit and principles of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A deep reinforcement learning based additive path process co-optimization method, characterized in that, The steps of the additive path process collaborative optimization method based on deep reinforcement learning include: Step 1: Constructing an additive path planning reinforcement learning environment; Step 2: Setting up a path planning agent reward mechanism for evaluating the deposition path selected by the agent; Step 3: Training the path planning reinforcement learning model; Step 4: Developing an additive process optimization model environment and training it; Step 5: Coupling the path planning reinforcement learning model and the process optimization model to obtain a path-process collaborative mutual optimization reinforcement learning model; The steps of obtaining the path-process collaborative mutual optimization reinforcement learning model in step 5 include: Step 5.1: Synchronously initializing the path planning reinforcement learning model and the additive process optimization reinforcement learning model; Step 5.2: Inputting the deposition path information contained in the real-time motion of the agent in the path planning reinforcement learning model into the additive path process optimization model to perform additive thermal coupling simulation of the corresponding deposition path; Step 5.3: Obtaining the optimal deposition path and optimal process parameters through the additive process optimization model; Step 5.4: Further inputting the optimal deposition path into the path planning reinforcement learning model, and the path planning model agent takes the deposition path at the next time; Step 5.5: Repeating steps 5.2-5.4 to plan the next path and continuously iterate until the mutual optimization of the path and the process is completed, obtaining the additive path-process collaborative mutual optimization deposition scheme; Step 6: Training the path-process collaborative mutual optimization reinforcement learning model; Step 7: Inputting the additive component into the trained path-process collaborative mutual optimization reinforcement learning model to obtain the optimal deposition path and optimal process parameters, and performing actual additive based on the optimal deposition path and optimal process parameters.

2. The method of claim 1, wherein, The additive path planning reinforcement learning environment in step 1 includes the environment boundary of path planning, the initial path planning state, the path planning termination state, the motion space of the agent, and the path planning visualization.

3. The method of claim 1, wherein, In step 2, the agent reward mechanism is a reward function, which is a function of the number of deposition paths, deposition area, cross-deposition nodes, deposition path curvature, and angle, and the agent's deposition path is evaluated based on the reward function.

4. The method of claim 1, wherein, The steps of training the path planning reinforcement learning model in step 3 include: Step 3.1: Taking the state of the agent's real-time motion in the constructed path planning environment as the model input; Step 3.2: Obtaining the agent's evaluation of the current path planning state environment state and calculating the reward obtained by the agent, and updating the model parameters using a reinforcement learning algorithm; Step 3.3: Repeating steps 3.1-3.2 to increase the reward of the agent in iterative training, and obtaining the optimized path planning reinforcement learning model.

5. The method of claim 1, wherein, The steps of developing an additive process optimization model environment and training it in step 4 include: Step 4.1: Developing a path-based grid generation algorithm and further building an additive manufacturing thermal coupling full-process automatic solution model; Step 4.2: Training the additive path process optimization model in a multi-machine parallel manner; Step 4.3: Input the real-time temperature field, stress field and deformation field data of the additive material into the trained additive path process optimization model as the environmental state, and use a 3D convolution network to extract the spatio-temporal information of the environmental state; Step 4.4: Evaluate the current additive environmental state based on the extracted spatio-temporal information; Step 4.5: Define the reward obtained by the intelligent agent as a function of the penetration, width, stress, deformation and peak temperature, and optimize the model parameters through a reinforcement learning algorithm to obtain an optimized process planning model; Step 4.6: Output the optimized process parameters based on the trained additive process optimization model to maintain stable deposition path formation and minimize stress and deformation.

Images