A method for adjusting process parameters of annular polishing based on reinforcement learning

Abstract

This invention discloses a method for adjusting process parameters in ring polishing based on reinforcement learning, comprising the following steps: First, acquiring process parameters during ring polishing, constructing a historical processing database, and linking historical records according to process control procedures to form an ordered chain; Second, pre-training a neural network using historical processing data to make the probability distribution of the neural network's output data close to that of historical data; Third, combining expert experience with reinforcement learning to train the neural network online, evaluating the neural network's output strategy online, and continuously iteratively optimizing the neural network; Fourth, constructing an experience pool, placing the interaction results between the neural network and the environment into the experience pool, and using the records in the experience pool as a training set to continuously expand the dataset iteratively; Fifth, solidifying the trained neural network and deploying it in the production environment to improve the processing efficiency of ring polishing and reduce the number of processing iterations for optical components.

Description

Technical Field

[0001] The invention belongs to the technical field of intelligent manufacturing, and in particular relates to a method for adjusting process parameters of annular polishing based on reinforcement learning. Background Art

[0002] Annular polishing is a polishing process for large-diameter flat optical glass. Annular polishing has the advantages of full-band surface error control and easy large-scale production. It is widely used in the fine polishing stage of the production process of optical flat components, and determines the final surface accuracy of optical components. Especially in the processing of high-power laser optical components, annular polishing has significant advantages in suppressing medium-frequency ripple errors, shortening processing cycles, and reducing processing costs. The annular polishing system is mainly composed of polishing discs, correction discs, component discs, beam brackets, etc. The polishing disc has a two-layer structure. The lower layer is a turntable made of natural granite, and the upper layer is a table top cast with materials used for polishing, mainly asphalt. The correction disc is used to correct the plastic deformation of the polishing disc surface, and its diameter is slightly larger than the radius of the polishing disc. The component disc is used to fix the optical element. When the annular polishing system is working, the polishing disc rotates at a faster speed; the correction disc and the component disc are both fixed on the beam bracket and driven to rotate by the motor. The speed of both is lower than that of the polishing disc, and the speed of the component disc is slightly lower than that of the polishing disc. The correction disk is in close contact with the polishing disk, the component disk is suspended above the polishing disk, and the optical element is in close contact with the polishing disk.

[0003] Annular polishing can make the surface error PV value of optical components converge to 0.3λ (λ=632.8mm), but optical components usually cannot reach 0.3λ through a single annular polishing. It needs to go through multiple processing iterations, sometimes even up to 10 times. Before each polishing, the process parameters are adjusted according to the current surface error of the optical component, including polishing time, polishing disk speed, correction disk speed, correction disk lifting position, correction disk radial position, correction disk unloading force, workpiece disk speed, polishing liquid outlet temperature, polishing liquid flow, ambient temperature, ambient humidity and other parameters. Therefore, the control of optical component surface error mainly depends on process experience and lacks effective parameter control methods, making annular polishing a bottleneck in the field of advanced optical manufacturing for a long time.

[0004] Due to the iterative nature of annular polishing, the process of adjusting the process parameters of annular polishing is a time-related sequential decision-making process, that is, a Markov decision process. Therefore, the use of reinforcement learning methods to intelligently adjust the parameters of annular polishing, improve the effectiveness of parameter selection, and reduce processing time are issues that the industry urgently needs to solve. Summary of the invention

[0005] The purpose of the present invention is to solve the above problems and provide a method for adjusting the process parameters of annular polishing based on reinforcement learning, which can perform intelligent parameter adjustment, improve the effectiveness of parameter selection, and reduce processing time.

[0006] In order to solve the above technical problems, the technical solution of the present invention is: a method for adjusting process parameters of annular polishing based on reinforcement learning, comprising the following steps:

[0007] S1. Obtain the process parameters during the annular polishing process, build a historical processing database, and link the historical records according to the process control process to form an orderly chain;

[0008] S2. Use historical processing data to pre-train the neural network so that the probability distribution of the output data of the neural network is close to that of the historical data;

[0009] S3. Combine expert experience and reinforcement learning to train the neural network online, evaluate the output strategy of the neural network online, and continuously iterate and optimize the neural network;

[0010] S4. Build an experience pool, put the interaction results between the neural network and the environment into the experience pool, use the records in the experience pool as a training set, and continuously iterate to expand the data set;

[0011] S5. Solidify the trained neural network and deploy it in the production environment to improve the processing efficiency of annular polishing and reduce the number of processing iterations of optical components.

[0012] Furthermore, the process parameters of the annular polishing in step S1 can basically be obtained by a CNC machine tool, including the polishing disc speed, the correction disc speed, the correction disc lifting position, the correction disc radial position, the correction disc unloading force, the workpiece disc speed, etc., and the process parameters can be adjusted by a CNC machine tool; in addition, some process parameters are located in the peripheral auxiliary equipment of the main machine tool, including the polishing liquid outlet temperature, the polishing liquid flow rate, etc., which also play a key role in the surface error of the optical element; finally, there are environmental parameters, including ambient temperature, ambient humidity, etc. A stable workshop environment is also conducive to the processing of optical elements; because of the iterative processing nature of optical elements, a processing chain can be formed on the same optical element: {(S o , A o ),......,(S N , A N )},S k (0≤k≤N) is the surface shape detection result of the optical element, A k (0≤k≤N) is the process parameter for processing optical elements, therefore, {(S k , A k )} N It is a time-dependent sequential decision process.

[0013] Furthermore, in step S2, the historical processing database constructed in S1 is used to pre-train the neural network, and the input of the neural network is S k , the output is A k ; The neural network selects a deep convolutional neural network, the input is a two-dimensional matrix, and the output is the value of the process parameter, so it is a single-input and multiple-output network model.

[0014] Furthermore, the surface error S of the given optical element in step S3 is k , the neural network gives a corresponding processing parameter A to the component in this state k ; A k First, the process experts will evaluate it. If it meets the expected range of parameter control, the machine tool will use parameter A. k Processing is carried out; if it does not meet expectations, the process expert can ask the neural network to re-evaluate the surface error of the optical element and give a new set of process parameters; the cycle is repeated multiple times until the parameters meet the expected requirements, otherwise the process expert will specify the parameters for this processing and the neural network will continue to train.

[0015] Furthermore, the step S4 specifically includes: constructing two experience pools, one is the expert experience pool, which stores the evaluation results of the process parameters given by the process experts to the neural network; the other is the processing experience pool, which stores the results of the interaction between the neural network and the machine tool. When the neural network interacts with the environment continuously, a large amount of data is generated, which can be used to expand the training set; at the same time, the experience pool is persisted and data is continuously accumulated in actual production; the data in the experience pool is used to iteratively optimize the neural network to improve the effectiveness of the process parameters.

[0016] Furthermore, the step S5 is specifically as follows: when the loss value of the neural network is less than a given loss value, it can be solidified and applied to the production environment.

[0017] The beneficial effect of the present invention is that the annular polishing process parameter adjustment method based on reinforcement learning provided by the present invention can intelligently adjust the parameters of the annular polishing, improve the effectiveness of parameter selection, and reduce the processing time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a method for adjusting process parameters of annular polishing based on reinforcement learning of the present invention;

[0019] Figure 2 It is a diagram of the establishment of the historical database and the neural network pre-training method of the present invention;

[0020] Figure 3 It is a flowchart of neural network training combining expert knowledge and reinforcement learning in the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments:

[0022] like Figures 1 to 3 As shown, the present invention provides a method for adjusting process parameters of annular polishing based on reinforcement learning, comprising the following steps:

[0023] S1. Obtain the process parameters during the annular polishing process, build a historical processing database, and link the historical records according to the process control process to form an orderly chain.

[0024] The process parameters of the annular polishing in step S1 can basically be obtained through the CNC machine tool, including the polishing disk speed, the correction disk speed, the correction disk lifting position, the correction disk radial position, the correction disk unloading force, the workpiece disk speed, etc., and the process parameters can be adjusted by the CNC machine tool; in addition, some process parameters are located in the peripheral auxiliary equipment of the main machine tool, including the polishing liquid outlet temperature, the polishing liquid flow rate, etc., which also play a key role in the surface error of the optical element; finally, there are environmental parameters, including ambient temperature, ambient humidity, etc. A stable workshop environment is also conducive to the processing of optical elements; because of the iterative processing nature of optical elements, a processing chain can be formed on the same optical element: {(S o , A o ),......,(S N , A N )},S k (0≤k≤N) is the surface shape detection result of the optical element, A k (0≤k≤N) is the process parameter for processing optical elements, therefore, {(S k , A k )} N It is a time-dependent sequential decision process.

[0025] Taking the existing Siemens CNC machine tools as an example, Siemens CNC machine tools can be connected through the OPCUA protocol and directly access the registers on the Sinumerik to obtain values. For example, the polishing disc speed, correction disc speed, correction disc lifting position, correction disc radial position, correction disc unloading force, workpiece disc speed, etc. can all be obtained by accessing the CNC machine tool. The peripheral auxiliary equipment is connected to the acquisition card, and the acquisition card communicates data using the ModBus TCP protocol. By reading the values ​​on the acquisition card, the corresponding process parameters can be obtained, such as the polishing liquid outlet temperature, polishing liquid flow, ambient temperature, ambient humidity, etc.

[0026] S2. Use historical processing data to pre-train the neural network so that the probability distribution of the output data of the neural network is close to the historical data.

[0027] In step S2, the historical processing database constructed in S1 is used to pre-train the neural network. The input of the neural network is S k , the output is A k The neural network is a deep convolutional neural network, the input is a two-dimensional matrix, and the output is the value of the process parameter, so it is a single-input and multi-output network model. Pre-training makes the output distribution of the neural network close to the distribution of historical data, which is conducive to the adjustment of reinforcement learning parameters.

[0028] like Figure 2 As shown, the historical processing is sorted out and a processing chain is formed based on the component batch. The surface error is a binary matrix, which is the interferometer detection result of the optical component, from which the PV value for judging whether the component meets the standard can be calculated. k It is the process parameter value given by the process expert. (S k , A k ) form a pair of state and process parameter values, and the state value of the next group is the result of processing the process parameters of the previous state.

[0029] S3. Combine expert experience with reinforcement learning to conduct online training of the neural network, conduct online evaluation of the output strategy of the neural network, and continuously iterate and optimize the neural network.

[0030] like Figure 2 As shown, the neural network is pre-trained. The input data is the surface error S k The output result is process parameter A k . Because there are multiple process parameters, this is a multi-objective optimization problem. The loss function is selected as a variant of KL divergence, as shown in the following formula:

[0031]

[0032] P(x) is the probability distribution of the output of the neural network, and Q(x) is the probability distribution of the actual process parameters. The output of the neural network is made to have the same distribution as the process parameters given by the expert by minimizing the loss function. The integrity of the process chain is maintained during the training process, that is, data is taken in the order of the process chain, and component batches are randomly selected.

[0033] The surface error S of the given optical element in step S3 is k , the neural network gives a corresponding processing parameter A to the component in this state k ; A k First, the process experts will evaluate it. If it meets the expected range of parameter control, the machine tool will use parameter A. kProcessing is carried out; if it does not meet expectations, the process expert can ask the neural network to re-evaluate the surface error of the optical component and give a new set of process parameters; the cycle is repeated multiple times until the parameters meet the expected requirements, otherwise the process expert will specify the parameters for this processing and the neural network will continue to train.

[0034] S4. Build an experience pool, put the interaction results between the neural network and the environment into the experience pool, use the records in the experience pool as a training set, and continuously iterate to expand the data set.

[0035] Step S4 specifically includes: constructing two experience pools, one is the expert experience pool, which stores the evaluation results of the process parameters given by the process experts to the neural network; the other is the processing experience pool, which stores the results of the interaction between the neural network and the machine tool. When the neural network interacts with the environment continuously, a large amount of data is generated, which can be used to expand the training set; at the same time, the experience pool is persisted and data is continuously accumulated in actual production; the data in the experience pool is used to iteratively optimize the neural network to improve the effectiveness of the process parameters.

[0036] like Figure 3 As shown in FIG. 1 , the present invention mainly implements reinforcement learning in the form of DQN (Deep Q-network). The neural network interacts with the environment as an intelligent agent, that is, Figure 3 PolicyNet and TargetNet in.

[0037] First, for the surface error S of the optical element at a certain moment t , PolicyNet will give a process parameter adjustment strategy a t , process experts on strategy a t Conduct an assessment and the assessment results are * t After a certain number of interactions, a set of data {(S t , a * t )} batchsize PolicyNet is optimized, and the loss function is selected as (1). This part integrates expert experience knowledge as a preliminary assessment before the intelligent agent interacts with the processing environment, which is helpful to avoid extreme strategies from entering the processing environment.

[0038] Secondly, once the PolicyNet strategy is evaluated by the process experts, it interacts with the workshop environment. This is a Markov decision process that satisfies the following conditions:

[0039] p(s t+1 |s t ,a t )=p(s t+1|h t ,a t ) (2)

[0040] h t ={s1,s2,s3,……,s t} (3)

[0041] The state history is h t , including all previous states. The reward function is shown in (4). The current state (i.e., the surface error of the optical element) and the action taken will affect the reward obtained by the agent.

[0042] R(s t =s,a t =a) = E[r t |s t =s,a t =a] (4)

[0043] The Q function is shown in (5), which describes the expected reward obtained by taking a certain action in a certain state.

[0044] Q π (s,a)=E π [G t |s t =s,a t =a] (5)

[0045] G t =r t+1 +γr t+2 +γ 2 r t+3 +……+γ T-t-1 r T (6)

[0046] G t is the return, which is the gradual superposition of rewards, γ is the discount factor, and π is the strategy selected at this moment. The goal of reinforcement learning is to maximize the return, that is, to maximize the Q function.

[0047] Q(s,a)=argmax a Q π (s,a) (7)

[0048] For the optimization of the agent, the loss function is selected as:

[0049] L = ∑(y t -Q(s t ,a t )) 2 (8)

[0050]

[0051] Q stands for PolicyNet. That is TargetNet. Figure 3 PolicyNet describes the current state of the optical component. t Propose a strategy t , then the machine tool equipment uses process parameter strategy a t After the processing is completed, the optical element will continue to undergo interferometer surface detection to obtain a new state s t+1 By comparing t With s t+1 , given the policy a given by PolicyNet t A reward for t+1 The surface shape error is based on the PV value and defines the following reward function, θ t With θ t+1 They are the PV values ​​of optical components before and after processing:

[0052]

[0053] The result of PolicyNet interacting with the environment is (s t , a t ,s t+1 , r t+1 ) is stored in the processing experience pool. After PolicyNet interacts with the environment a certain number of times, a batch of data will be taken out of the processing experience pool for optimization training. The Q value of the training data is calculated by PolicyNet, and the expected Q value is calculated by TargetNet. Then, the loss value is obtained by combining the Q value and the expected Q value through loss loss, and finally, PolicyNet is optimized by gradient descent. After a certain number of training optimizations, PolicyNet assigns its own neural network parameters to TargetNet, so that the two networks can achieve the effect of updating and optimization.

[0054] The entire method of combining expert experience and reinforcement learning is in an online form, acquiring and processing data in real time. While acquiring data online, it also stores the data well to facilitate training of new networks.

[0055] S5. Solidify the trained neural network and deploy it in the production environment to improve the processing efficiency of annular polishing and reduce the number of processing iterations of optical components.

[0056] Step S5 is as follows: when the loss value of the neural network is less than the given loss value, it can be solidified and applied to the production environment. When the loss value of PolicyNet is less than the given value, the entire training module can be stopped, and PolicyNet can be solidified and put into the production environment to assist workers in adjusting process parameters when processing optical components.

[0057] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for adjusting process parameters of annular polishing based on reinforcement learning, characterized in that: The following steps are involved: S1. Obtain the process parameters during the annular polishing process, build a historical processing database, and link the historical records according to the process control process to form an orderly chain; S2. Use historical processing data to pre-train the neural network so that the probability distribution of the output data of the neural network is close to that of the historical data; S3. Combine expert experience and reinforcement learning to train the neural network online, evaluate the output strategy of the neural network online, and continuously iterate and optimize the neural network; S4. Build an experience pool, put the interaction results between the neural network and the environment into the experience pool, use the records in the experience pool as a training set, and continuously iterate to expand the data set; S5. Solidify the trained neural network and deploy it in the production environment to improve the processing efficiency of annular polishing and reduce the number of processing iterations of optical components; The process parameters of the annular polishing in step S1 can basically be obtained by a CNC machine tool, including the polishing disc speed, the correction disc speed, the correction disc lifting position, the correction disc radial position, the correction disc unloading force, and the workpiece disc speed. The process parameters can be adjusted by a CNC machine tool; In addition, the process parameters in the peripheral auxiliary equipment of the main machine tool include the polishing liquid outlet temperature and polishing liquid flow rate, which also play a key role in the surface error of the optical element; finally, the environmental parameters include ambient temperature and ambient humidity. A stable workshop environment is also conducive to the processing of optical elements. Due to the iterative processing nature of optical elements, a processing chain can be formed on the same optical element: {(S o , A o ),.......,(S N , A N )},S k (0≤k≤N) is the surface shape detection result of the optical element, A k (0≤k≤N) is the process parameter for processing optical elements, therefore, {(Sk, Ak)}N is a time-dependent sequential decision process; In the step S2, the historical processing database constructed in S1 is used to pre-train the neural network, the input of the neural network is Sk, and the output is Ak; the neural network selects a deep convolutional neural network, the input is a two-dimensional matrix, and the output is the value of the process parameter, so it is a single-input and multi-output network model; In step S3, the surface error Sk of the optical element is given, and the neural network gives a corresponding processing parameter Ak for the element in this state; Ak is first evaluated by the process expert, and if it meets the expected range of parameter regulation, the machine tool will perform processing with the parameter Ak; if it does not meet the expectations, the process expert can ask the neural network to re-evaluate the surface error of the optical element and give a new set of process parameters; the cycle is repeated many times until the parameters meet the expected requirements, otherwise the process expert specifies the parameters for this processing, and the neural network continues to train; The step S4 specifically includes: constructing two experience pools, one is an expert experience pool, which stores the evaluation results of the process parameters given by the process experts to the neural network; the other is a processing experience pool, which stores the results of the interaction between the neural network and the machine tool; when the neural network and the environment continuously interact, a large amount of data is generated, which can be used to expand the training set; at the same time, the experience pool is persistently saved, and data is continuously accumulated in actual production; the data in the experience pool is used to iteratively optimize the neural network to improve the effectiveness of the process parameters; The step S5 is specifically as follows: when the loss value of the neural network is less than a given loss value, the neural network can be solidified and applied to the production environment; For the surface error S of optical components t , PolicyNet will give a process parameter adjustment strategy a t , process experts on strategy a t Conduct an assessment and the assessment results are * t Stored in the expert experience pool; after interaction, a set of data {(S t , a * t )} batchsize PolicyNet is optimized, and the loss function is selected as formula (1); this part integrates expert experience knowledge and serves as a preliminary assessment before the intelligent agent interacts with the processing environment, which is helpful to avoid extreme strategies from entering the processing environment; Secondly, once the PolicyNet strategy is evaluated by the process experts, it interacts with the workshop environment. This is a Markov decision process that satisfies the following conditions: p(s t+1 |s t ,a t )=p(s t+1 |h t ,a t ) (2) <h2 style=";text-align:left;direction:ltr">h<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> (s1,s2,s3,……,s)<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr">} (3) The state history is h t , including all previous states; the reward function is shown in (4), the current state, the state is the surface error of the optical element, the current state and the action taken will affect the reward obtained by the agent; R(s t =s,a t =a)=E[r t |s t =s,a t =a] (4) The Q function is shown in (5), which describes the expected reward obtained by taking a certain action in a certain state; Qπ(s,a)=Eπ[G t |with t =s,a t =a] (5) G t =r t+1 +γr t+2 +g 2 r t+3 +……+c T-t-1 r T (6) G t is the return, which is the gradual superposition of rewards, γ is the discount factor, π is the strategy selected at this moment; the goal of reinforcement learning is to maximize the return, that is, to maximize the Q function; Q(s,a)=argmax a Qπ(s,a) (7) For the optimization of the agent, the loss function is selected as: L=∑(y t -Q(s t ,a t )) 2 (8) Q stands for PolicyNet. That is TargetNet; PolicyNet is the current state of the optical element s t Propose a strategy t , then the machine tool equipment uses process parameter strategy a t Process the optical element; after the processing is completed, the optical element will continue to perform interferometer surface detection to obtain a new state s t+1 ; By comparing t With s t+1 , given the policy a given by PolicyNet t A reward for t+1 ; The surface shape error takes the PV value as the standard and defines the following reward function, θ t With θ t+1 They are the PV values ​​of optical components before and after processing: The result of PolicyNet interacting with the environment is (s t , a t ,s t+1 , r t+1 ) is stored in the processing experience pool in the form of a structure; after PolicyNet interacts with the environment a certain number of times, a batch of data will be taken out from the processing experience pool for optimization training; The Q value of the training data is calculated by PolicyNet, and the expected Q value is calculated by TargetNet. Then, the loss value is obtained by combining the Q value and the expected Q value through loss loss, and finally PolicyNet is optimized by the gradient descent method. After a certain number of training optimizations, PolicyNet assigns its own neural network parameters to TargetNet, so that the two networks can achieve the effect of updating and optimization.

Images