Powder bed fusion additive manufacturing defect detection and mitigation method based on reinforcement learning

Abstract

The present application relates to a kind of powder bed fusion additive manufacturing defect detection and inhibition method based on reinforcement learning, comprising the following steps S1 real-time acquisition in powder bed fusion additive manufacturing process Image and state data, and the image and the state data are preprocessed;S2 build reinforcement learning model based on D3QN, including defining environment, reward mechanism definition, build neural network and experience playback process;S3 the image information and state data in powder bed fusion additive manufacturing process obtained after pre-processing in step S1 are input into the reinforcement learning model based on D3QN in training, obtain the best decision information in powder bed fusion additive manufacturing process, and obtain corresponding training model;S4 the training model obtained using the reinforcement learning model based on D3QN in powder bed fusion additive manufacturing process defect detection and inhibition is realized, to improve the quality of the part of powder bed fusion additive manufacturing.

Description

Technical Field

[0001] This invention relates to the field of powder bed fused additive manufacturing technology, and in particular to a method for defect detection and suppression in powder bed fused additive manufacturing based on reinforcement learning. Background Technology

[0002] Powder bed fusion additive manufacturing (PBD) is a manufacturing technology that uses lasers or electron beams to thermally melt powder layers, thereby stacking metal materials layer by layer to form solid metal parts. It can form complex components that are difficult or impossible to form using traditional manufacturing techniques and has been applied in aerospace, biomedicine, automotive, and mold making fields. However, the development of PBD equipment requires extensive process testing to avoid defects in printed parts and to verify the printing effects of various process parameter combinations. This process is time-consuming, labor-intensive, and costly. Furthermore, the structural design, equipment parameters, and other unknown factors of the PBD equipment can also affect the quality of the printed parts. For example, in the development of large-size equipment based on powder bed fusion, large-format solid parts often exhibit defects such as bulging, blackening, warping, and cracks. Possible causes include inappropriate process parameters, wind field structure, and wind speed, requiring continuous adjustments. As additive manufacturing technology advances and its market share increases year by year, additive manufacturing companies are constantly iterating and updating their products to seize industry dominance. Therefore, reducing testing costs and accelerating the development of additive manufacturing equipment are key issues that researchers need to consider.

[0003] Reinforcement learning differs from supervised and unsupervised learning in that each piece of training data is not independent; there is a correlation between the data, and the current state will influence the agent's next decision. This "real-time" characteristic is well-suited for defect detection in additive manufacturing. Furthermore, D3QN, based on the Q-learning algorithm, not only shares the advantages of Q-learning such as easy convergence and wide applicability, but also addresses the problem of excessive error in Q-learning when dealing with large-scale state spaces through a dual-network architecture, making it suitable for the large-scale state space characteristics of additive manufacturing processes.

[0004] Therefore, it is necessary to design a defect detection and suppression method for powder bed fused additive manufacturing based on reinforcement learning. Summary of the Invention

[0005] Based on this, a method for defect detection and suppression in powder bed fused additive manufacturing based on reinforcement learning is provided. The learning process of reinforcement learning controls the forming quality in powder bed fused additive manufacturing, thereby achieving the purpose of suppressing defects.

[0006] A method for defect detection and suppression in powder bed fused additive manufacturing based on reinforcement learning includes the following steps: S1: Real-time acquisition of images and state data during the powder bed fused additive manufacturing process, and preprocessing of the images and state data; S2: Construction of a reinforcement learning model based on D3QN, including defining the environment, defining the reward and punishment mechanism, building the neural network, and the experience playback process; S3: Inputting the image information and state data obtained in step S1 after preprocessing into the reinforcement learning model based on D3QN for training, obtaining the optimal decision information in the powder bed fused additive manufacturing process, and obtaining the corresponding training model; S4: Using the training model obtained by the reinforcement learning model based on D3QN to realize defect detection and suppression in the powder bed fused additive manufacturing process.

[0007] In one embodiment, the specific steps for image preprocessing include: S11: converting the image to grayscale using a weighted average method to convert the acquired color image of the laser selective melting process into a grayscale image; S12: denoising the grayscale image using a Gaussian filtering method, and adjusting the Gaussian kernel and standard deviation according to the characteristics of powder bed fusion additive manufacturing images; S13: normalizing the pixel values ​​of the grayscale image. The images during the powder bed fusion additive manufacturing process are acquired in real-time using a high-precision digital camera; the pixel values ​​are normalized to facilitate neural network fitting.

[0008] In one embodiment, in step S12, a 3×3 square Gaussian kernel is selected based on common defects in the appearance of the laser selective melting formed part.

[0009] In one embodiment, in step S12, a standard deviation of 5 is selected based on common defects in the appearance of the laser selective melting formed part.

[0010] Based on the common defects in the appearance of laser selective melting formed parts, such as warping, bulging, blackening, and cracks, a smaller Gaussian kernel and a larger standard deviation are selected, namely a 3×3 square kernel and a standard deviation of 5, to preserve rich details in the image.

[0011] In one embodiment, in step S11, the grayscale value of the grayscale image is calculated as follows:

[0012] gray=R×0.299+G×0.587+B×0.114

[0013] Where R, G, and B represent the red, green, and blue component values ​​of a pixel, respectively.

[0014] In one embodiment, in step S13, the equation for normalizing the pixel values ​​of the image is:

[0015]

[0016] Where, x norm This represents the normalized pixel value, where x represents the original pixel value. σ represents the mean of all pixels, and σ represents the standard deviation of all pixels.

[0017] In one embodiment, the method further includes normalizing the state data, wherein the normalization equation is:

[0018]

[0019] Where, x min and x max These represent the minimum and maximum values ​​of the state variables.

[0020] In one embodiment, the specific steps for building the reinforcement learning model based on D3QN include: S21: Constructing the agent's state space S{s1,s2,…,s t Action space A{a1,a2,…,a…} t} and a reward or penalty function r; S22: Build a neural network, which includes an input layer, hidden layers, and an output layer; S23: Set up an evaluation network and a target network, wherein the evaluation network is used to estimate the current state (s) t Each action (a) t The target network is used to calculate the target Q value; S24: Set up an experience replay pool, which is used to hold experience samples during the interaction process of the agent. The network structures of the evaluation network and the target network are consistent with the neural network structure built in S22.

[0021] In one embodiment, the state space is:

[0022] S={Y,P a ,F,Q,T,C,H Z ,P,V,H,μ,G,L}

[0023] Where Y represents the oxygen content, Pa represents the pressure inside the forming cylinder, F represents the powder quantity in the powder storage cylinder, Q represents the protective gas flow rate, T represents the substrate temperature, and H represents the oxygen content. Z P represents the frequency of the circulating fan, V represents the laser power, H represents the scanning speed, μ represents the scanning spacing, G represents the layer thickness, and L represents the powder supply rate.

[0024] In one embodiment, the action space is:

[0025] A = {a1, a2, a3, a4, a5, a6, a7, a...}t}

[0026] Where a1 represents the action of adjusting laser power (W), a2 represents the action of adjusting scanning speed (mm / s), a3 represents the action of adjusting layer thickness (μm), a4 represents the action of adjusting scanning spacing (mm), and a5 represents the action of adjusting powder supply (mm). 3 The actions are as follows: a6 represents the action of adjusting the frequency (Hz) of the circulating fan, a7 represents the action of adjusting the substrate temperature, and a8 represents the action of adjusting the filling strategy.

[0027] In one embodiment, the reward or punishment function represents the agent in a certain state s. k Below, after a certain action a is completed k Rules that result in rewards or punishments, where the reward or punishment is denoted by r: r t+1 =r(s t ,a t ), where r t+1 This indicates that at time t, the agent is at time s. t Execute a in state t The reward or punishment received for an action.

[0028] In one embodiment, the input layer includes a first branch and a second branch, whereby the first branch is used to input an image and the second branch is used to input numerical features. One branch is a convolutional branch used to extract image features, and the other branch is a fully connected layer, where each neuron corresponds to a numerical feature.

[0029] In one embodiment, the hidden layer includes at least one convolutional layer, one pooling layer, and one fully connected layer. The fully connected layer connected to the output layer includes two branch networks: a value network and an advantage network.

[0030] In one embodiment, the output layer is a fully connected layer, where each neuron corresponds to an action.

[0031] In one embodiment, the Q-value output of the evaluation network is:

[0032]

[0033] Where |A| represents the number of actions, V(s) represents the state-value function, and A(s,a) represents the dominance function. i ) indicates that the i-th action a is performed in state s. i The dominant function;

[0034] The function values ​​of V(s) and A(s,a) are both approximated using neural networks:

[0035] V(s)=f θ (s)

[0036]

[0037] Where fθ represents the neural network, θ represents the network parameters, and f θ ′ represents a neural network based on another set of network parameters θ.

[0038] In one embodiment, the target Q-value output formula of the target network is:

[0039] y t =r t +γQ t (s t+1 argmax a Q e (s t+1 ,a;ω e );ω t )

[0040] Among them, y t For the target Q value, r t argmax is the agent's immediate reward in the current state, γ∈(0,1) is the discount factor, and argmax is the quotient. a Q e (s t+1 ,a;ω e To obtain s using the evaluation network t+1 The action corresponding to the optimal action value in a given state;

[0041] Q e Medium loss function (L e The mean squared error (MSE) function is defined as follows:

[0042] L e =E[(r t +γmax a 'Q t (s t+1 ,a t+1 ;ω t )-Q e (s t ,a t ;ω e )) 2 ]

[0043] Where, max a 'Q t (s t+1 ,a t+1 ;ω tQ is the maximum action value estimate among all possible actions the target network can perform in the next state. e (s t ,a t ;ω e ) represents the action value estimate of the evaluation network performing an action in the current state. The network parameters of the evaluation network are denoted by ω. e This indicates that stochastic gradient descent (SGD) is used to update ω. e The network parameters of the target network are determined by ω. t It means that ω t The update rule is to copy ω in the evaluation network every T time step. e .

[0044] In one embodiment, the empirical sample is:

[0045] es = {s t ,a t ,r t ,s t+1 ,end}

[0046] Among them, s t Given the current state, a t s represents the action taken by the agent at the current moment. t+1 The state for the next moment is defined by `end`, which is a flag indicating when the agent reaches the termination state. The data structure of the experience samples uses a circular queue to achieve the iteration of old and new data within a fixed experience replay pool size.

[0047] In one embodiment, data from the experience playback pool is collected using a priority sampling method. The priority of each piece of experience data is calculated using a temporal difference algorithm, determined by δ. t The δ t The expression is:

[0048] δ t =r t +γmax a 'Q t (s t+1 ,a t+1 ;ω t )-Q e (s t ,a t ;ω e )

[0049] The term derived from δ t The larger the value, the greater the probability that the empirical sample was collected.

[0050] In one embodiment, the specific steps of S3 include: S31: Initializing the network parameters of the evaluation network and the target network, and creating an empty experience replay pool; S32: The agent collects experience samples, selects actions using an ε-greedy algorithm, and generates the next state and reward value after selecting an action; S33: Storing the experience samples collected in step S32 into the empty experience replay pool; S34: When the number of experience samples in the empty experience replay pool reaches a set number, training the evaluation network begins, and the parameters of the evaluation network are updated using the loss function and stochastic gradient descent method in the evaluation network; S35: Every T time interval, the parameters of the evaluation network are copied to the target network, and the target network is updated; S36: Repeating steps S32 to S35 until a preset training time is reached or the average cumulative reward on the validation set does not improve after a set number of training sessions, training is stopped.

[0051] In one embodiment, the specific steps of using the ε-greedy algorithm to select actions are as follows: randomly select actions in the action space with a probability of ε, and select the action that the evaluation network estimates to have the maximum value with a probability of 1-ε.

[0052] In one embodiment, the relationship between the network parameters of the evaluation network and the network parameters of the target network in step S35 is as follows:

[0053]

[0054] Where 0 < τ << 1, i represents the current iteration number, and ω t The ω represents the network parameters of the target network. e This represents the network parameters of the evaluation network. This indicates that the network parameters are evaluated during the i-th iteration. This represents the network parameters of the target network at the i-th iteration. This represents the network parameters of the target network at the (i+1)th iteration.

[0055] In general, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0056] 1. This invention employs a reinforcement learning model based on the D3QN algorithm, which possesses self-learning and adaptive capabilities. For the model itself, it can continuously optimize and improve its parameters based on actual conditions to maximize cumulative rewards, thereby quickly and accurately adjusting process or equipment parameters in the powder bed fusion additive manufacturing process, thus improving the quality of parts manufactured using this method. For the powder bed fusion additive manufacturing process, the reinforcement learning model based on the D3QN algorithm effectively reduces manual intervention and labor costs by continuously learning and adapting to new materials, processes, and equipment, accelerating the R&D speed for researchers.

[0057] 2. This invention can be used for defect detection in powder bed fusion additive manufacturing. Compared with traditional machine learning algorithms, the reinforcement learning model based on D3QN has the ability to handle complex image scenes and learn autonomously in image recognition. It can process the entire image scene in an end-to-end manner.

[0058] 3. This invention can be used to suppress defects in powder bed fused additive manufacturing. The model can automatically monitor and control the powder bed fused additive manufacturing process, and adjust parameters through decision-making before defects are detected or occur, thereby reducing labor costs and improving the robustness of the powder bed fused additive manufacturing process.

[0059] 4. This invention uses D3QN as the core of the defect detection and suppression strategy in powder bed fused additive manufacturing. By building a dual Q network and an experience replay pool, it has a faster convergence speed, can handle higher-dimensional state spaces, and has stronger generalization ability compared with traditional reinforcement learning models such as Q-learning. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of a method for defect detection and suppression in powder bed fused additive manufacturing based on reinforcement learning, according to an embodiment of the present invention.

[0061] Figure 2 This is a schematic diagram of the specific process of S1 in a powder bed fused additive manufacturing defect detection and suppression method based on reinforcement learning according to an embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram of the specific process of S2 in a powder bed fused additive manufacturing defect detection and suppression method based on reinforcement learning according to an embodiment of the present invention;

[0063] Figure 4 This is a schematic diagram of the specific process of S3 in a powder bed fused additive manufacturing defect detection and suppression method based on reinforcement learning according to an embodiment of the present invention;

[0064] Figure 5This is an overall framework diagram of a powder bed fused additive manufacturing defect detection and suppression method based on reinforcement learning, according to an embodiment of the present invention.

[0065] Figure 6 This is a D3QN model diagram of a powder bed fused additive manufacturing defect detection and suppression method based on reinforcement learning, according to an embodiment of the present invention.

[0066] Figure 7 This is a flowchart illustrating a method for detecting and suppressing defects in powder bed fused additive manufacturing based on reinforcement learning, according to an embodiment of the present invention. Detailed Implementation

[0067] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0068] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0069] The following describes, with reference to the accompanying drawings, a method for detecting and suppressing defects in powder bed fused additive manufacturing according to some embodiments of the present invention.

[0070] like Figures 1 to 7 As shown, this embodiment discloses a method for defect detection and suppression in powder bed fused additive manufacturing, including the following steps: S1: Real-time acquisition of images and state data during the powder bed fused additive manufacturing process, and preprocessing of the images and state data; S2: Construction of a reinforcement learning model based on D3QN, including defining the environment, defining the reward and punishment mechanism, building the neural network, and the experience playback process; S3: Inputting the image information and state data obtained in step S1 after preprocessing into the reinforcement learning model based on D3QN for training, obtaining the optimal decision information in the powder bed fused additive manufacturing process, and obtaining the corresponding training model; S4: Using the training model obtained by the reinforcement learning model based on D3QN to realize defect detection and suppression in the powder bed fused additive manufacturing process.

[0071] like Figure 5As shown, the overall framework comprises two parts: an agent and the powder bed fusion additive manufacturing process. The agent performs actions on the powder bed fusion additive manufacturing process to control its forming quality. Simultaneously, the powder bed fusion additive manufacturing process feeds back to the agent with state information after the actions and the corresponding reward value. This process can also be viewed as an "action-feedback-action" strategy, through which the agent aims to suppress defects.

[0072] like Figure 7 As shown, a method for defect detection and suppression in powder bed fused additive manufacturing based on reinforcement learning includes the following steps:

[0073] Step 1: Use a high-precision digital camera to acquire images in real time during the laser selective melting printing process, and perform grayscale conversion and noise reduction preprocessing. At the same time, use oxygen content sensors, pressure sensors, level gauges, air flow meters, temperature sensors, and printing software to acquire status parameters such as oxygen content, forming cylinder pressure, powder quantity in the powder storage tank, protective air flow, substrate temperature, powder spreading cart speed, circulating fan frequency, laser power, scanning speed, scanning spacing, layer thickness, powder supply, and filling strategy during the laser selective melting printing process.

[0074] Step 2: Build a reinforcement learning model based on D3QN, including defining the environment, defining the reward and punishment mechanism, building the neural network, and the experience replay process;

[0075] Step 3: Input the preprocessed image data described in Step 1 and the state data during the laser selective melting process into a reinforcement learning model based on D3QN for training, so as to obtain the optimal decision information for the powder bed melting additive manufacturing process and obtain the corresponding model.

[0076] Step 4: Use a reinforcement learning model based on D3QN to realize defect detection and suppression in the laser selective melting process.

[0077] The present invention provides further detailed description of the above technical solutions:

[0078] like Figure 2 As shown, the image preprocessing steps in step 1 further include:

[0079] 1.1) Grayscale

[0080] The weighted average method is used to convert the color images of the laser selective melting process captured by the high-precision digital camera into grayscale images. The grayscale value is calculated using the following formula:

[0081] gray=R×0.299+G×0.587+B×0.114

[0082] Where R, G, and B represent the red, green, and blue component values ​​of a pixel, respectively.

[0083] 1.2) Noise Reduction

[0084] Gaussian filtering was used to denoise the image. Based on the common defects in the appearance of laser selective melting formed parts, such as warping, bulging, blackening, and cracks, a smaller Gaussian kernel and a larger standard deviation were selected, namely a 3×3 square kernel and a standard deviation of 5, to preserve the rich details of the image.

[0085] 1.3) Normalization

[0086] To facilitate neural network fitting, the pixel values ​​of the grayscale image are normalized:

[0087]

[0088] Where, x norm This represents the normalized pixel value, where x represents the original pixel value. σ represents the mean of all pixels, and σ represents the standard deviation of all pixels.

[0089] Similarly, the state variables in step 1 are preprocessed by normalization:

[0090]

[0091] Where, x min and x max Represents the minimum and maximum values ​​of state variables;

[0092] like Figure 3 As shown, further, step 2, building a reinforcement learning model based on D3QN, includes:

[0093] 2.1) Define the environment

[0094] The defined environment defines the state space:

[0095] S={Y,P a ,F,Q,T,C,H Z ,P,V,H,μ,G,L}

[0096] Where Y represents the oxygen content value, P a The pressure inside the forming cylinder is represented by F, the powder quantity in the powder storage cylinder is represented by Q, the protective gas flow rate is represented by T, and the substrate temperature is represented by H. Z The values ​​represent the circulating fan frequency, P represents the laser power, V represents the scanning speed, H represents the scanning interval, μ represents the layer thickness, G represents the powder supply, and L represents the filling strategy.

[0097] Define the action space:

[0098] A = {a1, a2, a3, a4, a5, a6, a7, a...} t}

[0099] Where a1∈(1,2,…,500) represents the action of adjusting laser power (W), a2∈(5,10,15,…,2000) represents the action of adjusting scanning speed (mm / s), a3∈(10,20,…,100) represents the action of adjusting layer thickness (μm), a4∈(0.01,0.02,…,0.2) represents the action of adjusting scanning spacing (mm), and a5∈(5,10,…,1000) represents the action of adjusting powder supply (mm). 3 The actions of a6∈(1,2,…,50) represent the action of adjusting the frequency (Hz) of the circulating fan, a6∈(1,2,…,80) represent the action of adjusting the substrate temperature, and a7∈(segmented, striped, whole) represent the action of adjusting the filling strategy.

[0100] Intelligent agents change the defined state of the environment through actions and receive rewards;

[0101] 2.2) Establish a reward and punishment mechanism

[0102] After each layer of metal powder is laid, if the powder supply is adequate, the agent is given a positive reward of 5; if the powder supply is insufficient or excessive, a penalty of -1 is imposed. After each layer of solid and support is printed, if the part has a metallic luster and is free from warping, cracks, bulges, or blackening, the agent is given a positive reward of 5; if the part has warping, cracks, bulges, or blackening, a penalty of -10 is imposed.

[0103] The reward or punishment formula is as follows:

[0104]

[0105] 2.3) Building a neural network

[0106] like Figure 6 As shown, the neural network includes an input layer, a hidden layer, and an output layer;

[0107] The input layer consists of two branches: one for inputting the image and the other for inputting numerical features. The hidden layers include one convolutional layer, one pooling layer, and one fully connected layer. The convolutional layer has 5×5 kernels, 512 kernels, and a stride of 1. The pooling layer uses max pooling with a 3×3 window and a stride of 1. The fully connected layer, connected to the output layer, consists of two branch networks: a state-value network (Value) and an action-advantage network (Advantage). The Value network outputs the state value, while the Advantage network outputs the advantage value for each action. The output layer is also fully connected, with each neuron corresponding to one action.

[0108] 2.4) Setting up the evaluation network (Q e ) and target network (Q t )

[0109] The network structures of the evaluation network and the target network are consistent with the neural network structure built in 2.3); the evaluation network is used to estimate the current state (s). t Each action (a) t The Q value of the network is determined by ω. e This indicates that stochastic gradient descent (SGD) is used to update ω. e The target network is used to calculate the target Q value, and its network parameters are determined by ω. t It means, and ω t The update rule is to copy ω in the evaluation network every T time step. e ;

[0110] Q value calculation formula:

[0111]

[0112] Where |A| is the number of actions, V(s) is the state value function, A(s,a) is the advantage function, and A(s,a) is the value function. i ) indicates that the i-th action a is performed in state s. i The advantage function of both is approximated using neural networks:

[0113] V(s)=f θ (s)

[0114]

[0115] Where fθ represents a neural network, θ is the network parameter, and f θ ′ represents a neural network based on another set of network parameters θ.

[0116] Formula for calculating the target Q value:

[0117] y t =r t +γQ t (s t+1 argmax a Q e (s t+1 ,a;ω e );ω t )

[0118] Among them, y t For the target Q value, r t argmax is the agent's immediate reward in the current state, γ∈(0,1) is the discount factor, and argmax is the quotient. a Q e (st+1 ,a;ω e To obtain s using the evaluation network t+1 The action corresponding to the optimal action value in a given state.

[0119] Q e Medium loss function (L e The mean squared error (MSE) function is defined as follows:

[0120] L e =E[(r t +γmax a 'Q t (s t+1 ,a t+1 ;ω t )-Q e (s t ,a t ;ω e )) 2 ]

[0121] Where, max a 'Q t (s t+1 ,a t+1 ;ω t Let Q be the maximum action value estimate among all possible actions the target network can take in the next state. e (s t ,a t ;ω e This is an estimate of the action value of the network performing actions in its current state.

[0122] 2.5) Set up the experience replay pool (D)

[0123] The size of the experience playback pool should accommodate a sufficient number of experience samples:

[0124] es = {s t ,a t ,r t ,s t+1 ,end}

[0125] Among them, s t Given the current state, a t s represents the action taken by the agent at the current moment. t+1 The next state is defined by 'end', which is the marker indicating when the agent reaches the termination state. Its data structure uses a circular queue to achieve the replacement of old and new data with a fixed experience replay pool size.

[0126] Data from the experience replay pool is collected using a priority sampling method. The priority of each piece of experience data is calculated using the temporal difference algorithm (TD), and is determined by δ. tmeasure:

[0127] δ t =r t +γmax a 'Q t (s t+1 ,a t+1 ;ω t )-Q e (s t ,a t ;ω e )

[0128] The larger the value, the greater the probability that the empirical sample will be collected.

[0129] like Figure 4 As shown, step 3 is further based on the D3QN reinforcement learning model training process:

[0130] 3.1) Initialize Q e Q t and experience replay pool

[0131] Initialize Q e and Q t Based on the network parameters, create an empty experience replay pool.

[0132] 3.2) Sampling

[0133] The agent collects action data (es). An ε-greedy algorithm is used to select actions, where ε is a small value. Actions are randomly selected from the action space with a probability of ε, and the action that evaluates the network to the maximum value is selected with a probability of 1-ε. After selection, the next state and reward value are generated.

[0134] 3.3) Store experience samples

[0135] Store es in the experience replay pool.

[0136] 3.4) Update Q e

[0137] When the number of experience replay pools reaches a certain amount, training Q begins. e Through L e Update Q with SGD e parameter.

[0138] 3.5) Update Q t

[0139] Every T time interval, Q e Copy the parameters to Q t In, that is:

[0140]

[0141] Where 0 < τ < < 1, and i represents the current iteration number.

[0142] 3.6) Iteration

[0143] Repeat steps 3.2)-3.5) until the maximum training time is reached or the average cumulative reward on the validation set shows no improvement after several rounds of training.

[0144] The specific steps of the above algorithm are as follows:

[0145] Input: Current state S t ={Y,P a ,F,Q,T,C,H Z ,P,V,H,μ,G,L},image

[0146] Evaluation network model Q e

[0147] Target network model Q t

[0148] Experience Replay Pool D

[0149] Output: D3QN reinforcement learning model

[0150] process:

[0151] Initialize Q e Q t D

[0152] Establish reward and punishment mechanisms

[0153] for t=1,2,…,T do

[0154] Get the current state s t Image

[0155] Choose action a in the action space with probability ε. t

[0156] The maximum action feedback reward r estimated by the evaluation network is selected with a probability of 1-ε. t and the next state s t+1

[0157] Store the experience samples es in D.

[0158] Samples from D in TD format

[0159] Calculate the target value: y t =r t +γQ t (s t+1 argmaxa Q e (s t+1 ,a;ω e );ω t )

[0160] Network update: L e =E[(r t +γmax a 'Q t (s t+1 ,a t+1 ;ω t )-Q e (s t ,a t ;ω e )) 2 ].

[0161] As can be seen from the above description, the embodiments of the present invention have the following beneficial effects:

[0162] 1. This invention employs a reinforcement learning model based on the D3QN algorithm, which possesses self-learning and adaptive capabilities. For the model itself, it can continuously optimize and improve its parameters based on actual conditions to maximize cumulative rewards, thereby quickly and accurately adjusting process or equipment parameters in the powder bed fusion additive manufacturing process, thus improving the quality of parts manufactured using this method. For the powder bed fusion additive manufacturing process, the reinforcement learning model based on the D3QN algorithm effectively reduces manual intervention and labor costs by continuously learning and adapting to new materials, processes, and equipment, accelerating the R&D speed for researchers.

[0163] 2. This invention can be used for defect detection in powder bed fusion additive manufacturing. Compared with traditional machine learning algorithms, the reinforcement learning model based on D3QN has the ability to handle complex image scenes and learn autonomously in image recognition. It can process the entire image scene in an end-to-end manner.

[0164] 3. This invention can be used to suppress defects in powder bed fused additive manufacturing. The model can automatically monitor and control the powder bed fused additive manufacturing process, and adjust parameters through decision-making before defects are detected or occur, thereby reducing labor costs and improving the robustness of the powder bed fused additive manufacturing process.

[0165] 4. This invention uses D3QN as the core of the defect detection and suppression strategy in powder bed fused additive manufacturing. By building a dual Q network and an experience replay pool, it has a faster convergence speed, can handle higher-dimensional state spaces, and has stronger generalization ability compared with traditional reinforcement learning models such as Q-learning.

[0166] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0167] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for defect detection and suppression in powder bed fused additive manufacturing based on reinforcement learning, characterized in that, Includes the following steps: S1: Real-time acquisition of images and status data during the powder bed fusion additive manufacturing process, and preprocessing of the images and status data; S2: Build a reinforcement learning model based on D3QN, including defining the environment, defining the reward and punishment mechanism, building the neural network, and the experience replay process; S3: Input the image information and state data of the powder bed fusion additive manufacturing process obtained after preprocessing in step S1 into the reinforcement learning model based on D3QN for training, obtain the best decision information in the powder bed fusion additive manufacturing process, and obtain the corresponding training model. S4: The trained model obtained by the reinforcement learning model based on D3QN is used to realize defect detection and suppression in the powder bed fused additive manufacturing process; The specific steps for building a reinforcement learning model based on D3QN include: S21: Construct the state space S{s1,s2,…,s} of the intelligent agent. t Action space A{a1,a2,…,a…} t } and the reward or penalty function r; S22: Construct a neural network, which includes an input layer, a hidden layer, and an output layer; S23: Set up an evaluation network and a target network, wherein the evaluation network is used to estimate the current state s. t Next action a t The target network is used to calculate the target Q value; S24: Set up an experience replay pool, which is used to hold experience samples during the interaction process of the intelligent agent; The state space is as follows: S={Y,P a ,F,Q,T,C,H Z ,P,V,H,μ,G,L} Where Y represents the oxygen content value, P a The pressure inside the forming cylinder is represented by F, the powder quantity in the powder storage cylinder is represented by Q, the protective gas flow rate is represented by T, and the substrate temperature is represented by H. Z The values ​​represent the circulating fan frequency, P represents the laser power, V represents the scanning speed, H represents the scanning interval, μ represents the layer thickness, G represents the powder supply, and L represents the filling strategy. The action space is: A = {a1, a2, a3, a4, a5, a6, a7, a8} Where a1 represents the action of adjusting laser power, a2 represents the action of adjusting scanning speed, a3 represents the action of adjusting layer thickness, a4 represents the action of adjusting scanning spacing, a5 represents the action of adjusting powder supply, a6 represents the action of adjusting circulating fan frequency, a7 represents the action of adjusting substrate temperature, and a8 represents the action of adjusting filling strategy.

2. The method for defect detection and suppression in powder bed fused additive manufacturing based on reinforcement learning according to claim 1, characterized in that, The specific steps for image preprocessing include: S11: The image is converted to grayscale using a weighted average method to convert the acquired color image of the laser selective melting process into a grayscale image. S12: The grayscale image is denoised using a Gaussian filtering method, and the Gaussian kernel and standard deviation are adjusted according to the characteristics of powder bed fused additive manufacturing images. S13: Normalize the pixel values ​​of the grayscale image.

3. The method for defect detection and suppression in powder bed fused additive manufacturing based on reinforcement learning according to claim 2, characterized in that, In step S12, a 3×3 square Gaussian kernel is selected based on common defects in the appearance of the laser selective melting formed part; and / or In step S12, a standard deviation of 5 is selected based on common defects in the appearance of the laser selective melting formed part; and / or In step S11, the grayscale value of the grayscale image is calculated using the following formula: gray=R×0.299+G×0.587+B×0.114 Where R, G, and B represent the red, green, and blue component values ​​of a pixel, respectively.

4. The method for defect detection and suppression in powder bed fused additive manufacturing based on reinforcement learning according to claim 1, characterized in that, The reward or punishment function represents the agent's state s in a certain state. k Below, after a certain action a is completed k Rules that result in rewards or punishments, where the reward or punishment is denoted by r: r t+1 =r(s t ,a t ) Where, r t+1 This indicates that at time t, the agent is at time s. t Execute a in state t The reward or punishment received for an action.

5. The method for defect detection and suppression in powder bed fused additive manufacturing based on reinforcement learning according to claim 1, characterized in that, The Q-value output formula of the evaluation network is: Where |A| represents the number of actions, V(s) represents the state-value function, and A(s,a) represents the dominance function. i ) indicates that the i-th action a is performed in state s. i The dominant function; The function values ​​of V(s) and A(s,a) are both approximated using neural networks: V(s)=f θ (s) Where fθ represents the neural network, θ represents the network parameters, and f θ′ This represents a neural network based on another set of network parameters θ'; The target Q-value output formula of the target network is: y t =r t +γQ t (s t+1 ,argmax a Q e (s t+1 ,a;ω e );oh t ) Among them, y t For the target Q value, r t argmax is the agent's immediate reward in the current state, γ∈(0,1) is the discount factor, and argmax is the quotient. a Q e (s t+1 ,a;ω e To obtain s using the evaluation network t+1 The action corresponding to the optimal action value in a given state; Q e Loss function L e Defined as the mean square error function: L e =E[(r t +γmax a′ Q t (s t+1 ,a t+1 Oh t )-Q e (s t ,a t Oh e )) 2 ] Where, max a ′ Q t (s t+1 ,a t+1 ;ω t Q is the maximum action value estimate among all possible actions the target network can perform in the next state. e (s t ,a t ;ω e ) represents the action value estimate of the evaluation network performing an action in the current state. The network parameters of the evaluation network are denoted by ω. e This indicates that stochastic gradient descent is used to update ω. e The network parameters of the target network are determined by ω. t It means that ω t The update rule is to copy ω in the evaluation network every T time step. e ; The empirical sample is: es={s t ,a t ,r t ,s t+1 ,end} Among them, s t Given the current state, a t s represents the action taken by the agent at the current moment. t+1 The state is the next time step, and end is the flag indicating when the agent reaches the termination state. The data structure of the experience samples adopts a circular queue.

6. The method for defect detection and suppression in powder bed fused additive manufacturing based on reinforcement learning according to claim 5, characterized in that, The input layer includes a first branch and a second branch, wherein the first branch is used to input images and the second branch is used to input numerical features; and / or The hidden layer comprises at least one convolutional layer, one pooling layer, and one fully connected layer. The fully connected layer connected to the output layer comprises two branch networks: a state value network and an action advantage network; and / or The output layer is a fully connected layer; and / or Data in the experience replay pool is collected using a priority sampling method. The priority of each piece of experience data is calculated using a time-series difference algorithm, denoted by δ. t The δ t The expression is: d t =r t +γmax a ′Q t (s t+1 ,a t+1 Oh t )-Q e (s t ,a t Oh e ) Where, r t The immediate reward of the agent in the current state, γ∈(0,1) is the discount factor, and max a 'Q t (s t+1 ,a t+1 ;ω t Q is the maximum action value estimate among all possible actions the target network can perform in the next state. e (s t ,a t ;ω e ) represents the action value estimate of the evaluation network performing actions in its current state.

7. The method for defect detection and suppression in powder bed fused additive manufacturing based on reinforcement learning according to claim 1, characterized in that, The specific steps of S3 include: S31: Initialize the network parameters of the evaluation network and the target network, and create an empty experience replay pool; S32: The agent collects experience samples, uses the ε-greedy algorithm to select actions, and after selecting an action, generates the next state and reward value; S33: Store the experience samples collected in step S32 into the empty experience playback pool; S34: When the number of experience samples in the empty experience replay pool reaches a set number, start training the evaluation network and update the parameters of the evaluation network through the loss function and stochastic gradient descent method in the evaluation network; S35: Every T time interval, the parameters of the evaluation network are copied to the target network to update the target network; S36: Repeat steps S32 to S35 until the preset training time is reached or the average cumulative reward on the validation set does not improve after a set number of training sessions, then stop training.

8. The method for defect detection and suppression in powder bed fused additive manufacturing based on reinforcement learning according to claim 7, characterized in that, The specific steps for selecting actions using the ε-greedy algorithm are as follows: randomly select an action from the action space with probability ε, and select the action that yields the maximum value estimated by the evaluation network with probability 1-ε; and / or The relationship between the network parameters of the evaluation network and the network parameters of the target network in step S35 is as follows: Where 0 < τ << 1, i represents the current iteration number, and ω t The ω represents the network parameters of the target network. e This represents the network parameters of the evaluation network. This indicates that the network parameters are evaluated during the i-th iteration. This represents the network parameters of the target network at the i-th iteration. This represents the network parameters of the target network at the (i+1)th iteration.

Images