Deep q-network based directed energy deposition laser power off-line planning method

Abstract

The application discloses a kind of based on deep Q network's directional energy deposition laser power offline planning method, belong to the field of additive manufacturing.The numerical model of macroscopic temperature field based on directional energy deposition process is constructed as interactive environment, the volume of molten pool is extracted as the state feature representing thermal process, and the target molten pool volume is determined by experiment calibration.The power planning problem is constructed as Markov decision process, the state vector consisting of molten pool volume sequence, discrete power adjustment action set and Gaussian reward function centered on target volume are defined to form a deep reinforcement learning framework.Under this framework, the deep Q network agent is trained to learn the optimal power adjustment strategy.Finally, the trained agent is used to plan the laser power sequence of the deposition process offline.The application can plan the optimal laser power sequence offline, stabilize the heat input of directional energy deposition process, effectively suppress defects such as incomplete fusion and cracks, and improve the quality of additive manufacturing.

Description

Technical Field

[0001] This invention belongs to the field of additive manufacturing, and particularly relates to an offline planning method for directional energy deposition laser power based on deep Q-networks. Background Technology

[0002] Directed energy deposition (DED) is a technique in additive manufacturing that uses a high-energy laser beam as a heat source to melt and stack material in a specific area layer by layer. It is commonly used to manufacture and repair complex metal components. The high energy density heat input and rapid cooling solidification cause continuous changes in the molten pool state and solidification conditions during DED, affecting the dimensional accuracy and mechanical properties of the deposited part. Therefore, maintaining stable thermal conditions during deposition is crucial for improving process quality. Deposition path planning and laser parameter adjustment have proven to be effective solutions. Deposition path planning focuses on optimizing the internal heat distribution of the deposited part, while adjusting laser power parameters directly controls the heat input, making its effect more significant. Currently, laser power adjustment for DED is mainly divided into two methods: online monitoring and control, and offline parameter planning. Online monitoring and control typically establishes the molten pool profile or temperature as the monitoring object, processes signals in real time, and achieves rapid parameter control during the deposition process. Offline parameter planning has lower real-time requirements; based on experimental data or simulation scenarios, it optimizes the deposition laser power sequence through a full-process study.

[0003] The continuous nature of directional energy deposition (DOD) processes makes designing a universal and effective real-time parameter controller challenging, requiring a balance between signal volume and processing speed. Furthermore, online monitoring and control methods often utilize visible or infrared spectral sensors (such as high-speed cameras / melt pool cameras) to capture signals, with the equipment mounted coaxially or laterally relative to the nozzle. The acquired raw observation data contains detection noise such as powder splashes and metallic light reflections, necessitating filtering of complex image signals to extract crucial melt pool information, significantly impacting the real-time performance of feedback control. Offline parameter planning, on the other hand, has lower real-time requirements, extracting information from experimental data or simulation models and combining machine learning methods to process large volumes of signals. However, experimental research in DOD is costly, and simulation models require simplification for machine learning to improve computational efficiency. Therefore, designing efficient and reasonable state extraction methods and model training frameworks is crucial for integrating machine learning methods with offline parameter planning in DOD processes. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes an offline laser power planning method for directional energy deposition (DOD) based on a deep Q-network. By fusing deep reinforcement learning with a high-fidelity process model, a deep Q-network agent is trained to achieve optimized control of the molten pool volume. This method can plan the optimal laser power sequence offline, stabilizing the thermal input of the DOD process.

[0005] The technical solution adopted in this invention is as follows:

[0006] In a first aspect, this invention proposes an offline planning method for directional energy deposition laser power based on a deep Q-network, comprising the following steps:

[0007] S1. Construct a numerical model of the macroscopic temperature field based on the directional energy deposition process;

[0008] S2. In the finite element simulation calculation of the macroscopic temperature field numerical model, the volume of the molten pool is obtained by accumulating the volumes of all mesh elements with temperatures higher than the liquidus temperature of the material. The volume of the molten pool is used as a characteristic quantity to characterize the thermal state of the process. Based on the deposition experiment under ideal process parameters, the target molten pool volume is determined by geometric calibration and semi-ellipsoidal model calculation.

[0009] S3. The offline laser power planning problem is transformed into a Markov decision process, and a deep reinforcement learning framework is constructed. The state space is defined as a one-dimensional state vector, which is formed by the sequential combination of the melt pool volume values ​​at multiple equally divided positions of discrete deposition channel units. The action space is defined as a discrete set of instructions to increase the laser power by one unit, decrease it by one unit, or keep it unchanged. The unit corresponds to the minimum resolution of the laser output power. The reward function is defined as a Gaussian distribution function calculated for each component of the one-dimensional state vector with the target melt pool volume as the expected value. The calculation results of the Gaussian distribution function for each component constitute the branch reward.

[0010] S4. Using the macroscopic temperature field numerical model as the interactive environment, a deep Q-network agent is trained under the deep reinforcement learning framework, so that the agent learns to output the optimal laser power adjustment action based on the state vector.

[0011] S5. Using the trained deep Q-network agent, perform offline planning of the laser power sequence for the directional energy deposition process.

[0012] Furthermore, the macroscopic temperature field numerical model defines the motion mode of the laser heat source and the heat transfer mode:

[0013] The laser heat source moves as follows: a set of attitude coordinate points is introduced into the nozzle of the robotic arm. The deposition path and laser action direction are reconstructed, thereby discretizing the continuous deposition path into multiple unit deposition channels for time-series calculations; where N is the total time step of the deposition process. , and This indicates the position of the nozzle in the world coordinate system. , and This indicates the orientation of the nozzle in the cylindrical coordinate system of the robotic arm;

[0014] The heat transfer mode is defined by a Gaussian heat source model for characterizing the laser heat source, a function defining the spatial distribution of laser energy, a control equation for controlling the temperature change in the simulation region, and a master equation for heat transfer boundary conditions that simultaneously considers thermal convection and thermal radiation phenomena.

[0015] Furthermore, the formula for expressing the heat transfer mode is as follows:

[0016] ;

[0017] ;

[0018] ;

[0019] ;

[0020] in, It is the standard direction vector. It is heat flux. P is the absorption rate of the material to laser energy, and P is the laser power. It is a function that defines the spatial distribution of laser energy. It is the direction of the laser beam. It is to find the absolute value of a scalar. It is to find the magnitude of the vector. It is the standard deviation describing the degree of power density distribution of the laser heat source, and d is the distance of the heated surface point from the origin of the laser beam. These are the position coordinates of the points on the heated surface. These are the coordinates of the origin of the laser beam's action. Here, T is the material density, T is the temperature field, and t is time. It is the specific heat capacity of the material. It is a velocity field. Q is the change in temperature, and Q is the change in heat in the simulation domain. is the Hamiltonian operator, used to calculate the gradient of a scalar field and the divergence of a vector field; k is the thermal conductivity coefficient of the material. It is convective heat loss. It is the convective heat transfer coefficient. It is the ambient temperature. It is radiative heat loss. It's the emissivity. It is the Stefan-Boltzmann constant.

[0021] Furthermore, the determination of the target molten pool volume through geometric calibration and semi-ellipsoidal model calculation includes:

[0022] Planar single-pass deposition was performed under ideal process parameters;

[0023] The length of the molten pool is obtained by a laterally installed observation device, and the width and depth of the molten pool on the cross-section of the deposition channel are obtained by metallographic observation.

[0024] The molten pool is approximated as a semi-ellipsoid, with the length, width, and depth respectively serving as the three semi-axes of the semi-ellipsoid, and the target molten pool volume is calculated based on the semi-ellipsoid volume formula.

[0025] Furthermore, in S3, the reward function also includes an additional reward term and a failure penalty term;

[0026] The trigger condition for the additional reward item is that the melt pool volume values ​​in the one-dimensional state vector are all within the range of the Gaussian distribution function centered on the target melt pool volume;

[0027] The trigger condition for the failure penalty term is that the sum of the branch rewards corresponding to each component of the one-dimensional state vector is less than zero.

[0028] Furthermore, the formula for calculating the branch reward is as follows:

[0029] ;

[0030] in, It is the branch reward corresponding to the i-th component in the state vector at time t. It is the molten pool volume represented by the i-th component in the state vector at time t. It is the target molten pool volume. It is the standard deviation of the preset Gaussian distribution. This is the scaling factor, and m is the displacement parameter. It is a natural exponential function.

[0031] Furthermore, the accumulation of the volumes of all grid cells with temperatures higher than the material's liquidus temperature to obtain the molten pool volume includes:

[0032] Traverse all volume mesh elements in the finite element calculation of the macroscopic temperature field numerical model;

[0033] For each volume mesh element, determine whether its transient temperature obtained through node interpolation is higher than the liquidus temperature of the metallic material; if it is higher, the volume of the volume mesh element is included in the summation; if it is equal to or lower than the summation, it is not included.

[0034] The volumes of all volume mesh elements with temperatures above the liquidus temperature are summed, and the sum is used as the extracted molten pool volume.

[0035] Secondly, this invention proposes an offline planning system for directional energy deposition laser power based on a deep Q-network, which is used to implement the aforementioned offline planning method for directional energy deposition laser power based on a deep Q-network.

[0036] Thirdly, the present invention proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned method for offline planning of directional energy deposition laser power based on deep Q-network.

[0037] Fourthly, the present invention proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method for offline planning of directional energy deposition laser power based on a deep Q-network.

[0038] The beneficial effects of this invention are:

[0039] This invention integrates directional energy deposition (OED) technology with deep reinforcement learning methods. By constructing a deep Q-network decision agent, it achieves efficient processing of sample data. Through the establishment of a high-fidelity numerical model, it balances the accuracy and efficiency of sample data processing, reducing model training costs. Furthermore, this invention proposes a novel state extraction method, control objective, and model training framework, meeting the solution requirements of Markov decision problems under specific process conditions. This invention can efficiently solve the offline laser power planning problem in OED processes, which is of great significance for improving the thermal distribution of OED components and enhancing forming accuracy and quality. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the offline planning method for directional energy deposition laser power based on deep Q-networks.

[0041] Figure 2 This is a schematic diagram of a semi-ellipsoidal molten pool model, in which, Figure 2 (a) in the figure is a simulation diagram of the transient temperature field. Figure 2 (b) in the figure is a cross-sectional view of the transient temperature field simulation diagram. Figure 2 (c) in the figure is a schematic diagram of approximating the molten pool in the simulation as a semi-ellipsoidal geometry;

[0042] Figure 3 This is a schematic diagram of the geometric dimensions of the molten pool under geometric calibration experiments, in which... Figure 3 (a) in the diagram is a schematic diagram of the molten pool length. Figure 3 (b) in the diagram is a schematic diagram of the width and depth of the molten pool;

[0043] Figure 4 This is a schematic diagram of a deep reinforcement learning framework;

[0044] Figure 5 This is a schematic diagram comparing the laser power sequence under the offline planning strategy of the present invention and the traditional constant power strategy;

[0045] Figure 6 This is a schematic diagram comparing the average volume of the molten pool under the offline planning strategy of this invention with that under the traditional constant power strategy. Detailed Implementation

[0046] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0047] The accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0048] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0049] The flowchart of the offline planning method for directional energy deposition laser power based on deep Q-network is as follows: Figure 1 As shown, a reasonable control objective and a "state-action-reward" cyclic decision-making framework are designed, and a deep reinforcement learning method is introduced to realize the offline planning of the power parameter policy.

[0050] This invention mainly includes the establishment of a macroscopic temperature field numerical model based on directional energy deposition (OED) technology, the determination of the target control state of the molten pool, and the design of decision logic and training architecture. Due to the high cost of experimental research and multi-scale simulation of OED technology, and the difficulty in matching the data generation efficiency with the requirements of deep reinforcement learning, this invention constructs a macroscopic temperature field model for robot-assisted OED technology. This model simplifies the simulation of phenomena such as molten pool dynamics and microstructure evolution at a single scale to achieve efficient and accurate calculations. Based on this model, the molten pool volume, which can comprehensively assess heat input and dissipation, is extracted as a state feature, and the target control state for the molten pool volume is determined through experimental calculation. The offline laser power planning of OED technology is transformed into a Markov decision process, and a deep Q-network method is introduced to construct a decision agent. Through training, this agent acquires the ability to make optimal decisions.

[0051] The complete implementation process is as follows:

[0052] S1. A macroscopic temperature field numerical model is constructed based on the directed energy deposition (TED) process. This model is designed for robot-assisted metal TED manufacturing processes, simplifying the simulation of molten pool dynamics and microstructure evolution at a single scale, and discretizing the continuous deposition process. Based on the energy distribution characteristics of the fiber-optic laser source, the laser heat source is characterized using a Gaussian heat source model, and the heat flux is... As shown in formula (1). Wherein, It is the standard direction vector. It is the laser power. This refers to the direction of the laser beam. Considering that some energy is lost due to reflection when the laser strikes a material, the material's absorptivity is introduced. It reflects the ratio of energy absorbed. The Gaussian distribution characterizing laser energy is shown in Equation (2). It is the origin of the laser beam's action. It refers to the location of the surface point heated by the laser beam. It is the standard deviation describing the degree of distribution of the heat source power density. The temperature change in the simulation area is controlled by formula (3). (kg / m 3 ) is the material density. (J / (kg·K)) is the specific heat capacity of the material. (W / (m·K)) is the thermal conductivity coefficient of the material. (K) and (m / s) represent the temperature field and the velocity field, respectively. (W / m) represents the heat change in the simulation domain. The main governing equation of the heat transfer boundary of the simulation domain mainly considers the heat loss caused by heat convection and heat radiation, as shown in equation (4). This refers to convective heat loss, with the thermal convection boundary set on the surface of the target component in contact with the external environment. It is the convective heat transfer coefficient. (K) is the ambient temperature. It is radiative heat loss, and the thermal radiation boundary is set in the sedimentary layer and its surrounding area. It's the emissivity. It is the Stefan-Boltzmann constant, with a value of .

[0053]

[0054]

[0055]

[0056]

[0057] Furthermore, the numerical model decomposes the continuous deposition process into discrete deposition channel units for time-series analysis, and introduces the attitude coordinate point set P{X} of the robotic arm nozzle. n Y n Z n A n B n C n The deposition path and laser direction are reconstructed using the formulas {x, n ∈ N}. Here, N is the total time step of the deposition process, and X... n Y n and Z n Indicates the position of the nozzle in the world coordinate system, A n B n and C n Control the attitude of the nozzle in the cylindrical coordinate system of the robotic arm.

[0058] The proposed macroscopic temperature field numerical model defines the motion mode of the laser heat source and the heat transfer mode. The motion mode of the laser heat source is realized by introducing the attitude coordinate point set of the robotic arm nozzle, restoring the deposition path and the laser action direction, thereby discretizing the continuous deposition path into multiple unit deposition channels for time-series calculation; the heat transfer mode is defined by the above equations.

[0059] S2, determine the target control state of the molten pool.

[0060] To integrate the above numerical model into a deep reinforcement learning environment, quantifiable state indicators need to be extracted for the agent to make sequential decisions. This invention uses the melt pool volume to comprehensively evaluate the heat input and dissipation during the directed energy deposition process. As low-dimensional scalar data, it can effectively avoid the curse of dimensionality and is more suitable for the interaction requirements of the deep reinforcement learning environment. The melt pool volume is difficult to observe in the actual manufacturing process. In the finite element simulation calculation of the above macroscopic temperature field numerical model, the volume of all grid elements with a temperature higher than the liquidus temperature of the material is accumulated to obtain the melt pool volume. The melt pool volume is used as a characteristic quantity to characterize the thermal state of the process. Specifically, it is calculated by formula (5) in the finite element discretization state.

[0061]

[0062] in, Let N be the volume of the molten pool, and let N represent the total number of volume mesh elements in the solution domain. This is the liquidus temperature of the metallic material. This represents the transient temperature of each volume grid element obtained through grid node interpolation. This represents the volume of the corresponding mesh cell. Indicator function. Determine whether the transient temperature of the volume mesh element exceeds the liquidus temperature of the metallic material, and use this to define the range of the molten pool calculated by the model.

[0063] A stable melt pool volume indicates a dynamic balance between heat input and loss during the directional energy deposition process, creating favorable conditions for material deposition. Calculating the ideal melt pool volume under specific process conditions is a crucial step in determining the control target. This invention employs a geometric calibration calculation method to determine the target melt pool volume under heat-free conditions. Specifically, a side-mounted high-speed camera is used to photograph the single-pass planar deposition process under ideal process parameters, and the length information of the melt pool is obtained through image calibration. A metallographic sample is prepared by wire cutting of the deposition pass, and the width and depth information of the melt pool are obtained through microscopic observation of the cross-section. Figure 2 Figure (a) shows the simulation diagram of the transient temperature field. Figure 2 (b) is a cross-sectional view of the transient temperature field simulation. The actual molten pool volume is approximated using a semi-ellipsoidal model. The transient molten pool is abstracted as follows: Figure 2 The semi-ellipsoidal geometry shown in (c) is given. The semi-axis a, b, and c of the ellipsoid correspond to the length, width, and depth of the molten pool, respectively. The volume of the ideal molten pool can be calculated using formula (6).

[0064]

[0065] like Figure 3As shown in (a), the geometric center of the molten pool moves synchronously with the laser's point of action. Using the distance between the geometric centers of the molten pool in the two extracted images as a reference, the length of the molten pool after stabilization in this state is measured to be 2.5666 mm. Therefore, the semi-axis length... The value is 1.2833 mm. Microscopic measurements of the sedimentary channel cross-section are as follows: Figure 3 As shown in (b), the length of the fusion interface between the deposition channel and the substrate is 1.7774 mm, therefore the semi-axial length is... It is 0.8887mm; the maximum penetration depth is the length of the half-shaft. It is 0.6601 mm. This is calculated according to formula (6). =1.58mm 3 .

[0066] S3, building a deep reinforcement learning framework.

[0067] This invention constructs a deep reinforcement learning framework to efficiently solve the Markov decision process of offline laser planning in directional energy deposition processes. The framework is as follows: Figure 4 As shown. At time t, the state vector output by the agent perceiving the environment. Actions are selected from action set A using a deep Q-network. ( ) Execute in the simulator environment and provide feedback on the reward for the action. The state at the next moment During the training phase, the above time-series closed-loop process is repeated at each time step until convergence to the optimal policy. The detailed definitions of the elements of the deep reinforcement learning framework are as follows:

[0068] State Space Definition: The state space S is defined as a 32×1 dimension vector, formed by sequentially combining the melt pool volume values ​​at multiple equally divided locations of discrete depositional channel units. The melt pool volume values ​​are calculated based on a numerical model. The continuous deposition process is discretized into unit depositional channels in the numerical model for characterization; therefore, the state extracted at each time step must cover the entire process information of the corresponding depositional channel. In this embodiment, the unit depositional channel is divided into 33 equal parts. After removing endpoints that may have abrupt changes in melt pool volume, the melt pool volume scalars of the remaining 32 equally divided points are sequentially extracted and stored in the state vector.

[0069] Define the action space: The action space A is a 3×1 vector, including the laser power at time t. The three actions are: increasing the laser power by one unit, decreasing it by one unit, or keeping it constant. The laser energy is a proportional output; to ensure flexibility in power adjustment, the minimum resolvable value of the laser output power is selected as the change range. Actions The definition is shown in formula (7).

[0070]

[0071] Defining the reward function: The melt pool volume is allowed to fluctuate within a certain range; therefore, the core part of the reward function is designed as a Gaussian distribution function centered on the ideal melt pool volume. , For each component of the one-dimensional state vector, a Gaussian distribution function is calculated with the target melt pool volume as the expected value, and the calculation results for each component constitute a branch reward. Furthermore, this embodiment also ensures that the melt pool volume at all collection points on a single deposition channel is within the target melt pool volume. A fixed additional reward is set for each time step within the confidence interval to encourage approaching the control target at all times. Conversely, when the total reward at a certain time step is less than 0, it is considered a control failure, and a fixed negative reward is set. The reward function design is shown in formulas (8-11).

[0072]

[0073]

[0074]

[0075]

[0076] in, Let t be the state vector. The melt pool volume data at the i-th extraction point. The corresponding branch reward. The scaling factor is used to compress the reward to a suitable range, avoiding the gradient explosion problem. State vector The dimension, by rewarding each branch Additional rewards With negative rewards The summation yields the final comprehensive reward corresponding to a specific time step decision. .

[0077] S4. Train a deep Q-network agent.

[0078] In this step, a deep Q-network agent is trained under a deep reinforcement learning framework using a macroscopic temperature field numerical model as the interactive environment. This allows the agent to learn to output the optimal laser power adjustment action based on the state vector.

[0079] During training, when the sequential decision-making process fails consecutively, the state in subsequent time steps will deviate from the normal process range, and the state transition will no longer have process reference value. To prevent the agent from learning erroneous strategies under abnormal states, this invention sets a termination decision variable D. After receiving negative rewards for five consecutive time steps, variable D is set to 1, prematurely terminating the current training round.

[0080] This invention uses a deep Q-network to construct a decision agent to plan the power strategy for directional energy deposition laser. Its core idea is to use a deep neural network to approximate the action-value function (Q function) and perform optimal iteration through the Bellman equation, as shown in formula (12). and These are the current state and the current action, respectively. and It refers to future states and future actions. It is the intelligent agent that performs actions. Immediate rewards obtained from the environment, discount factor This represents the decay factor for future rewards. The Q-value of a state-action pair equals the current reward plus the future state. The maximum expected reward obtained by taking the optimal action is determined by recursively updating the Q value in the above manner, gradually bringing it closer to the optimal value. Thus, the optimal strategy is obtained. As shown in formula (13). The mean square error, as defined in formula (14), is used to measure the loss between the current Q value and the target Q value. The network parameters are updated via backpropagation using the Adam optimizer. During training, a separate target network is constructed to calculate the target Q-value. Parameters are updated from the current network every 5 episodes, effectively reducing fluctuations in the target Q-value and improving training stability. - A greedy strategy is used to select actions in order to achieve a balance between exploration and exploitation. As shown in formula (15), the probability is used to determine the actions. Randomly select actions, The probability of choosing the action with the highest current Q value. Exploration rate. With a fixed attenuation rate The value gradually decreases from 1 to 0.005 to ensure that space for random exploration is still retained after the training matures. Experiences generated by the agent's interaction with the environment are represented by quadruples. The data is stored in a fixed-size playback memory in a specific format. During each network update, a small batch of empirical calculations is randomly sampled from the playback memory, effectively avoiding correlations between data generated in adjacent time steps.

[0081]

[0082]

[0083]

[0084]

[0085] The hyperparameters affecting model training performance are shown in Table 1. The parameters used in this invention are as follows:

[0086] Table 1: Hyperparameters

[0087]

[0088] S5. Using the trained deep Q-network agent, perform offline planning of the laser power sequence for the directional energy deposition process.

[0089] For example, firstly, in an offline environment, the trained and converged deep Q-network agent model and its parameters are imported. Simultaneously, the process parameters for the planned deposition task are loaded, including the deposition geometry defined by the robotic arm's motion path point set, and the initial process parameters required for the macroscopic temperature field numerical model.

[0090] Secondly, using a numerical model as the simulation environment, the agent starts from the deposition initiation point and makes decisions based on the one-dimensional state vector calculated by the model. At each decision step corresponding to a deposition channel, the agent selects the optimal action from the action space based on the network's forward propagation results. After executing this action, the laser power value is updated, and the numerical model simulates and calculates the next one-dimensional state vector based on the new power. This closed-loop process progresses step-by-step along the entire deposition path until deposition is complete. During this process, the agent outputs a specific power adjustment command for each decision step on the path, ultimately forming a complete laser power sequence.

[0091] Finally, the system outputs the power sequence file, which can be directly loaded into the control system of the directional energy deposition equipment to guide the actual manufacturing process, thereby achieving active and optimized control of laser energy to stabilize the molten pool and improve the forming quality.

[0092] Figure 5 The laser power sequence under the offline planning strategy of this invention and the traditional constant power strategy was compared. The horizontal axis represents the deposition passes (0 to 100), and the vertical axis represents the laser power (W). In the figure, the constant power value is always maintained at around 1020W, forming a stable straight line; while the offline planning power dynamically changes between approximately 700W and 1020W. The results show that the method of this invention does not output a fixed power, but a dynamic power curve that is optimized and adjusted in real time according to the process conditions.

[0093] Figure 6The effects of different power strategies on melt pool volume are revealed. The horizontal axis represents the deposition pass, and the vertical axis represents the melt pool volume (mm). 3 In the figure, the target molten pool volume is stable at 1.6 mm. 3 The average volume fluctuates drastically under constant power, ranging from 2.0 to 4.0 mm. 3 The values ​​varied significantly, deviating severely from the target value. In contrast, the average volume under the offline planning power proposed in this invention almost perfectly matches the target value, remaining stable at 1.6 mm throughout. 3 The dynamic power sequence planned by the method of this invention can stably control the molten pool volume near the ideal target value, which is significantly better than the traditional constant power strategy, and is expected to fundamentally improve the stability and consistency of the forming quality of the directional energy deposition process.

[0094] This embodiment also provides an offline planning system for directional energy deposition laser power based on a deep Q-network for implementing the above method, comprising:

[0095] The model building module is used to build a numerical model of the macroscopic temperature field based on the directional energy deposition process;

[0096] The feature and target determination module is used to accumulate the volumes of all mesh elements with temperatures higher than the liquidus temperature of the material in the finite element simulation calculation of the macroscopic temperature field numerical model to obtain the melt pool volume, and use the melt pool volume as a characteristic quantity to characterize the thermal state of the process; and based on the deposition experiment under ideal process parameters, the target melt pool volume is determined by geometric calibration and semi-ellipsoidal model calculation.

[0097] The decision framework construction module is used to build a deep reinforcement learning framework and define the state space, action space and reward function.

[0098] The agent training module is used to train a deep Q-network agent within a deep reinforcement learning framework, using a macroscopic temperature field numerical model as the interactive environment.

[0099] The planning and execution module is used to perform offline planning of the laser power sequence for the directional energy deposition process using a trained deep Q-network agent. For the system embodiment, since it basically corresponds to the method embodiment, relevant details can be found in the description of the method embodiment; the implementation method of the module will not be repeated here. The system embodiment described above is merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0100] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0101] It should also be noted that the offline planning method for directional energy deposition laser power based on deep Q-networks in the above embodiments can essentially be executed by a computer program. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the method provided in the above embodiments, which includes a memory and a processor;

[0102] The memory is used to store computer programs;

[0103] The processor is configured to, when executing the computer program, implement the offline planning of directional energy deposition laser power based on deep Q-networks in the above embodiments.

[0104] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium.

[0105] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the method provided in the above embodiments. The storage medium stores a computer program that, when executed by a processor, can realize the offline planning of directional energy deposition laser power based on deep Q-networks in the above embodiments.

[0106] It is understood that the computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0107] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for offline planning of directional energy deposition laser power based on deep Q-networks, characterized in that, Includes the following steps: S1. Construct a numerical model of the macroscopic temperature field based on the directional energy deposition process; S2. In the finite element simulation calculation of the macroscopic temperature field numerical model, the volume of the molten pool is obtained by accumulating the volumes of all mesh elements with temperatures higher than the liquidus temperature of the material, and the volume of the molten pool is used as a characteristic quantity to characterize the thermal state of the process. Based on deposition experiments under ideal process parameters, the target melt pool volume was determined through geometric calibration and semi-ellipsoidal model calculation. S3. Construct a deep reinforcement learning framework, defining the state space as a one-dimensional state vector, which is formed by sequentially combining the melt pool volume values ​​at multiple equally divided positions of discrete deposition channel units; defining the action space as a discrete set of instructions to increase, decrease, or keep the laser power unchanged by one unit, with the unit corresponding to the minimum resolution of the laser output power; defining the reward function as a Gaussian distribution function calculated for each component of the one-dimensional state vector with the target melt pool volume as the expected value, wherein the calculation results of the Gaussian distribution function for each component constitute branch rewards; S4. Using the macroscopic temperature field numerical model as the interactive environment, a deep Q-network agent is trained under the deep reinforcement learning framework, so that the agent learns to output the optimal laser power adjustment action based on the state vector. S5. Using the trained deep Q-network agent, perform offline planning of the laser power sequence for the directional energy deposition process.

2. The method for offline planning of directional energy deposition laser power based on deep Q-networks according to claim 1, characterized in that, The macroscopic temperature field numerical model defines the movement mode of the laser heat source and the heat transfer mode: The laser heat source moves as follows: a set of attitude coordinate points is introduced into the nozzle of the robotic arm. The deposition path and laser action direction are reconstructed, thereby discretizing the continuous deposition path into multiple unit deposition channels for time-series calculations; where N is the total time step of the deposition process. , and This indicates the position of the nozzle in the world coordinate system. , and This indicates the orientation of the nozzle in the cylindrical coordinate system of the robotic arm; The heat transfer mode is defined by a Gaussian heat source model for characterizing the laser heat source, a function defining the spatial distribution of laser energy, a control equation for controlling the temperature change in the simulation region, and a master equation for heat transfer boundary conditions that simultaneously considers thermal convection and thermal radiation phenomena.

3. The method for offline planning of directional energy deposition laser power based on deep Q-networks according to claim 2, characterized in that, The formula for expressing the heat transfer mode is as follows: ； ； ； ； in, It is the standard direction vector. It is heat flux. P is the absorption rate of the material to laser energy, and P is the laser power. It is a function that defines the spatial distribution of laser energy. It is the direction of the laser beam. It is to find the absolute value of a scalar. It is to find the magnitude of the vector. It is the standard deviation describing the degree of power density distribution of the laser heat source, and d is the distance of the heated surface point from the origin of the laser beam. These are the position coordinates of the points on the heated surface. These are the coordinates of the origin of the laser beam's action. Here, T is the material density, T is the temperature field, and t is time. It is the specific heat capacity of the material. It is a velocity field. Q is the change in temperature, and Q is the change in heat in the simulation domain. It is the Hamiltonian operator; k is the thermal conductivity coefficient of the material. It is convective heat loss. It is the convective heat transfer coefficient. It is the ambient temperature. It is radiative heat loss. It's the emissivity. It is the Stefan-Boltzmann constant.

4. The method for offline planning of directional energy deposition laser power based on deep Q-networks according to claim 1, characterized in that, The method of determining the target molten pool volume through geometric calibration and semi-ellipsoidal model calculation includes: Planar single-pass deposition was performed under ideal process parameters; The length of the molten pool is obtained by a laterally installed observation device, and the width and depth of the molten pool on the cross-section of the deposition channel are obtained by metallographic observation. The molten pool is approximated as a semi-ellipsoid, with the length, width, and depth respectively serving as the three semi-axes of the semi-ellipsoid, and the target molten pool volume is calculated based on the semi-ellipsoid volume formula.

5. The method for offline planning of directional energy deposition laser power based on deep Q-networks according to claim 1, characterized in that, In S3, the reward function also includes an additional reward item and a failure penalty item; The trigger condition for the additional reward is that the melt pool volume values ​​in the one-dimensional state vector are all within one standard deviation of the Gaussian distribution function centered on the target melt pool volume. The trigger condition for the failure penalty term is that the sum of the branch rewards corresponding to each component of the one-dimensional state vector is less than zero.

6. The method for offline planning of directional energy deposition laser power based on deep Q-networks according to claim 1 or 5, characterized in that, The formula for calculating the branch reward is as follows: ； in, It is the branch reward corresponding to the i-th component in the state vector at time t. It is the molten pool volume represented by the i-th component in the state vector at time t. It is the target molten pool volume. It is the standard deviation of the preset Gaussian distribution. This is the scaling factor, and m is the displacement parameter. It is a natural exponential function.

7. The method for offline planning of directional energy deposition laser power based on deep Q-networks according to claim 1, characterized in that, The volume of the molten pool is obtained by summing the volumes of all mesh elements whose temperatures are higher than the liquidus temperature of the material, including: Traverse all volume mesh elements in the finite element calculation of the macroscopic temperature field numerical model; For each volume mesh element, determine whether its transient temperature obtained through node interpolation is higher than the liquidus temperature of the metallic material; if it is higher, the volume of the volume mesh element is included in the summation; if it is equal to or lower than the summation, it is not included. The volumes of all volume mesh elements with temperatures above the liquidus temperature are summed, and the sum is used as the extracted molten pool volume.

8. A system for offline planning of directional energy deposition laser power based on a deep Q-network, used to implement the method for offline planning of directional energy deposition laser power based on a deep Q-network as described in claim 1, characterized in that, The system includes: The model building module is used to build a numerical model of the macroscopic temperature field based on the directional energy deposition process; The feature and target determination module is used to accumulate the volumes of all mesh elements with temperatures higher than the liquidus temperature of the material in the finite element simulation calculation of the macroscopic temperature field numerical model to obtain the melt pool volume, and use the melt pool volume as a characteristic quantity to characterize the thermal state of the process; and based on the deposition experiment under ideal process parameters, the target melt pool volume is determined by geometric calibration and semi-ellipsoidal model calculation. The decision framework construction module is used to build a deep reinforcement learning framework and define the state space, action space and reward function. The agent training module is used to train a deep Q-network agent within a deep reinforcement learning framework, using a macroscopic temperature field numerical model as the interactive environment. The planning and execution module is used to perform offline planning of the laser power sequence for the directional energy deposition process using a trained deep Q-network agent.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the offline planning method for directional energy deposition laser power based on a deep Q-network, as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the offline planning method for directional energy deposition laser power based on a deep Q-network, as described in any one of claims 1 to 7.

Images