PI coating equipment motion control method and system based on reinforcement learning

Abstract

The invention provides a PI coating equipment motion control method and system based on reinforcement learning, and relates to the technical field of coating equipment control. Firstly, a virtual simulation platform for simulating the actual operation environment of PI coating equipment is built, and a training environment is provided for reinforcement learning. And creating and initializing a Q matrix, and designing a comprehensive reward function considering the process performance parameter weight of the PI coating equipment. Then, obtaining operation data through interaction with the virtual simulation platform, and training a Q matrix based on reinforcement learning by using the data; and in the training process, dynamically adjusting the weight of the reward function by using a feedback mechanism so as to optimize the learning process until the training is completed. And finally, inputting the current state of the PI coating equipment into the trained Q matrix to obtain an optimal action, and guiding the PI coating equipment to operate. Motion control over the PI coating equipment is achieved, and the coating quality and efficiency are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of coating equipment control, and more particularly relates to a motion control method and system for a PI coating equipment based on reinforcement learning. Background Art

[0002] PI (polyimide) coating equipment is mainly used for coating high-performance polymer materials and is widely used in the electronics and semiconductor industries. The PI (polyimide) coating equipment can uniformly coat a PI film on a glass substrate. The PI (polyimide) film is crucial in the manufacture of flexible electronic products and applications in high-temperature environments due to its excellent heat resistance, electrical insulation, and mechanical strength. When preparing polyimide films in the laboratory, a film coater is often used to coat polyamic acid slurry on a glass plate, and then polyimide films are formed through curing and high-temperature imidization.

[0003] Currently, the control method of PI coating equipment mainly adopts the "control system design" method. Through precise adjustment of control parameters, the control system design can improve the dynamic response of the system, reduce overshoot and steady-state error. Through precise control and automated monitoring, the need for manual intervention and maintenance is reduced, and labor costs are lowered. However, the control system design requires a large amount of time and professional knowledge, with high complexity, and is highly challenging to debug in dynamic and nonlinear systems. Moreover, its performance depends on the accuracy and stability of sensors and actuating devices, making it difficult to ensure the coating quality and efficiency.

[0004] In practical applications, with the increase in coating speed, the average coating thickness generally decreases; the increase in acceleration limits the fluidity of the coating material, resulting in fluctuations in coating thickness. There is an interaction between coating speed and coating acceleration, which has a significant impact on the uniformity of the coating thickness. Existing motion control methods for coating equipment based on reinforcement learning usually use dual models or even multiple models to separately train key parameters of the coating equipment such as coating speed and coating acceleration, as well as the quality of the glass substrate such as thickness uniformity. However, problems such as excessive dependence on data, high time costs, and overfitting occur, making it difficult to ensure coating quality and efficiency. Summary of the Invention

[0005] To solve the problem that the current control method of PI coating equipment is difficult to ensure coating quality and efficiency, the present invention provides a motion control method for PI coating equipment based on reinforcement learning to improve coating quality and efficiency.

[0006] To achieve the above technical effects, the technical solution of the present invention is as follows:

[0007] S1: Build a virtual simulation platform that simulates the actual operating environment of the PI coating equipment to provide a training environment for reinforcement learning;

[0008] S2: Create a Q matrix and initialize it, where the rows of the Q matrix represent the states of the PI coating equipment, and the columns of the Q matrix represent the actions taken by the PI coating equipment;

[0009] S3: Design a comprehensive reward function considering the weights of the process performance parameters of the PI coating equipment;

[0010] S4: Obtain data related to the operation of the PI coating equipment by interacting with the virtual simulation platform. Based on reinforcement learning, use the data to train the Q matrix. During the training process, adopt a feedback mechanism to dynamically adjust the weights of the comprehensive reward function until the training ends;

[0011] S5: Input the current state of the PI coating equipment into the trained Q matrix, output the corresponding action of the current state, and the PI coating equipment operates according to the action.

[0012] Further, the state space of the reinforcement learning includes: the speed adjustment of the PI coating equipment, the acceleration adjustment of the PI coating equipment, and the coating thickness adjustment. The action space includes: the speed of the PI coating equipment, the acceleration of the PI coating equipment, the coating thickness, the environmental temperature, and the environmental humidity.

[0013] Further, the comprehensive reward function includes: the PI coating equipment acceleration stability reward function, the coating quality reward function, and the PI coating equipment speed stability reward function;

[0014] The expression of the PI coating equipment acceleration stability reward function is:

[0015] R a =-|a t |

[0016] In the formula, R a represents the PI coating equipment acceleration stability reward function, and a t represents the acceleration of the PI coating equipment at time t;

[0017] The expression of the coating quality reward function is:

[0018] R h =-|δh|

[0019] In the formula, R h represents the coating quality reward function, and δh represents the error between the actual coating thickness and the ideal thickness;

[0020] The expression of the PI coating equipment speed stability reward is:

[0021] R v =-|v t -v target |

[0022] In the formula, R v represents the speed stability reward of the PI coating equipment, v t represents the actual speed of the PI coating equipment, v tatget represents the target speed of the PI coating equipment;

[0023] The acceleration stability reward function, coating quality reward function, and speed stability reward function of the PI coating equipment are weighted and calculated to obtain a comprehensive reward function. The expression is:

[0024] R total = α a R a + α h R h + α v R v

[0025] In the formula, R total represents the comprehensive reward function, α a represents the weight coefficient of the acceleration stability reward function of the PI coating equipment, α h represents the weight coefficient of the coating quality reward function, α v represents the weight coefficient of the speed stability reward of the PI coating equipment.

[0026] Furthermore, sensors are configured in the virtual simulation platform. Through interaction with the virtual simulation platform, data related to the operation of the PI coating equipment is obtained. The data is key parameters during the movement of the coating equipment collected by the sensors, including: the speed, acceleration, temperature, humidity, pressure, coating quality index thickness and uniformity of the coating equipment, feedback data on the instability or coating failure of the PI coating equipment, and historical operation data of the PI coating equipment.

[0027] Furthermore, during the testing process, a feedback mechanism is adopted to dynamically adjust the weights of the comprehensive reward function. When the actual values of the acceleration, coating quality, and speed of the PI coating equipment are close to or reach the required target values, the weights of their corresponding reward functions are reduced. When the actual values deviate from the target values, the weights of their corresponding reward functions are increased. The expression satisfied by the dynamic adjustment is:

[0028]

[0029] In the formula, α i (t + 1) represents the adjusted parameter weight, α i (t) is the weight in the current iteration. i represents any one of the acceleration stability reward function, coating quality reward function, and speed stability reward function of the PI coating equipment. σ represents the scaling factor of the weight ratio, P targetRepresents the target value of the PI coating equipment, P i (t) represents the actual value of the PI coating equipment in the current iteration.

[0030] According to the above technical means, the comprehensive reward function combines the coating quality, equipment running speed and acceleration, and at the same time dynamically adjusts the weights through the adoption of a feedback mechanism to achieve the global optimization of the coating quality and adapt to the coating production with different process requirements faster.

[0031] Furthermore, the steps of using data to train the Q matrix in S4 are as follows:

[0032] S401: Set the current state of the PI coating equipment to the initial state of the PI coating equipment, and set the learning rate, discount factor, number of training times, maximum number of training steps and ε-greedy strategy during the training process;

[0033] S402: According to the initial state and the Q matrix, select a decision-making action using the ε-greedy strategy;

[0034] S403: Execute the decision-making action to obtain a new state;

[0035] S404: Observe the new state and the immediate reward obtained according to the comprehensive reward function;

[0036] S405: Update the Q matrix based on the Q-learning update rule;

[0037] S406: Set the new state as the current state;

[0038] Repeat steps S401 - S406 until the termination state is reached or the preset number of training steps at the beginning is reached, and end the training.

[0039] Furthermore, the expression of the Q-learning update rule is:

[0040] Q(S t ,a t )←Q(S t ,a t )+α·[R t +γ×maxQ(S t+1 ,a)-Q(S t ,a t )]

[0041] In the formula, Q(S t ,a t ) represents the Q value of taking action a in state S t , α represents the learning rate, R t represents the immediate reward, γ represents the discount factor, maxQ(S t represents the maximum Q value of taking action a in state S t+1, a) represents the state S t+1 The maximum Q value for taking action a.

[0042] According to the above technical means, the Q matrix autonomously learns the optimal motion control strategy actions of the PI coating equipment through interaction with the environment, and can dynamically adjust the control parameters according to the real-time state of the PI coating equipment, thereby achieving more precise motion control of the PI coating equipment and improving the quality and efficiency of coating.

[0043] Furthermore, the motion control method of the PI coating equipment based on reinforcement learning further includes: using external sensors of the PI coating equipment to measure the environmental temperature, environmental humidity, pressure, coating speed, and coating acceleration parameters of the coating equipment, and using measurement tools to measure the quality thickness uniformity of the glass substrate; inputting the measured data into the trained Q matrix to test whether the trained Q matrix can output the optimal action according to the current state of the PI coating equipment during actual operation; at the same time, evaluating the stability of the PI coating equipment operating according to the action, and when the stability evaluation result is lower than the preset target, retraining the Q matrix until the stability evaluation result reaches the preset target.

[0044] Furthermore, use SimuLink software to build a virtual simulation platform that simulates the actual operating environment of the PI coating equipment.

[0045] The present invention also provides a motion control system for a PI coating equipment based on reinforcement learning, including:

[0046] A virtual simulation platform building module for building a virtual simulation platform that simulates the actual operating environment of the PI coating equipment and providing a training environment for reinforcement learning;

[0047] A Q matrix creation module for creating and initializing the Q matrix, where the rows of the Q matrix represent the states of the PI coating equipment, and the columns of the Q matrix represent the actions taken by the PI coating equipment;

[0048] A comprehensive reward function setting module for designing a comprehensive reward function that considers the weight of the process performance parameters of the PI coating equipment;

[0049] A training module for obtaining data related to the operation of the PI coating equipment by interacting with the virtual simulation platform, training the Q matrix based on reinforcement learning using the data, and dynamically adjusting the weight of the comprehensive reward function using a feedback mechanism during the training process until the training ends;

[0050] An operation module for inputting the current state of the PI coating equipment into the trained Q matrix and outputting the action corresponding to the current state, and the PI coating equipment operates according to the action.

[0051] Compared with the prior art, the beneficial effects of this method are:

[0052] The present invention provides a motion control method and system for a PI coating device based on reinforcement learning. First, a virtual simulation platform that mimics the actual operating environment of the PI coating device is built to provide a low-cost and high-efficiency training environment for reinforcement learning. By interacting with the virtual simulation platform, data related to the operation of the PI coating device is obtained. Based on reinforcement learning, the Q matrix is trained using the data. Meanwhile, during the training process, a feedback mechanism is adopted to dynamically adjust the weights of the comprehensive reward function to achieve global optimization of the coating quality and more quickly adapt to the coating production requirements of different processes. Finally, the current state of the PI coating device is input into the trained Q matrix, and the optimal action is output to operate the PI coating device. The present invention realizes the motion control of the PI coating device, improves the coating quality and efficiency, by building a virtual simulation platform, designing a comprehensive reward function, and creating a training Q matrix. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It shows the flowchart of the motion control method for the PI coating device based on reinforcement learning proposed in the embodiment of the present invention;

[0054] Figure 2 It shows the flowchart of training the Q matrix using data proposed in the embodiment of the present invention;

[0055] Figure 3 It shows the structural diagram of the motion control system for the PI coating device based on reinforcement learning proposed in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] The drawings are only for illustrative purposes and should not be construed as a limitation of this patent;

[0057] To better illustrate this embodiment, some parts of the drawings are omitted, enlarged, or reduced, and do not represent the actual size;

[0058] For those skilled in the art, it is understandable that some well-known content descriptions in the drawings may be omitted.

[0059] The technical solutions of the present invention will be further described below with reference to the drawings and embodiments.

[0060] The description of the positional relationship in the drawings is only for illustrative purposes and should not be construed as a limitation of this patent;

[0061] Embodiment 1

[0062] This embodiment provides a motion control method for a PI coating device based on reinforcement learning. As shown in the flowchart of this method, the method proposed in this embodiment generally includes the following steps: Figure 1 shown, the method proposed in this embodiment generally includes the following steps:

[0063] S1: Build a virtual simulation platform to simulate the actual operating environment of the PI coating equipment, providing a training environment for reinforcement learning;

[0064] S2: Create and initialize the Q matrix. Here, the rows of the Q matrix represent the states of the PI coating equipment, and the columns of the Q matrix represent the actions taken by the PI coating equipment;

[0065] S3: Design a comprehensive reward function considering the weight of the process performance parameters of the PI coating equipment;

[0066] S4: By interacting with the virtual simulation platform, obtain data related to the operation of the PI coating equipment. Based on reinforcement learning, use the data to train the Q matrix. During the training process, adopt a feedback mechanism to dynamically adjust the weight of the comprehensive reward function until the training ends;

[0067] S5: Input the current state of the PI coating equipment into the trained Q matrix, output the corresponding action for the current state, and the PI coating equipment operates according to the action.

[0068] In this embodiment, first, the relationship between the coating speed, coating acceleration, and coating thickness uniformity is studied. Part of the data is shown in Table 1.

[0069] Table 1

[0070] Coating speed (m / min) <![CDATA[Acceleration (m / s 2 )]]> Average thickness (μm) Standard deviation (μm) 1 0.1 50 5 1 0.5 52 7 1 1 54 8 2 0.1 45 4 2 0.5 46 5 2 1 50 6 3 0.1 40 6 3 0.5 39 9 3 1 41 11 4 0.1 38 8 4 0.5 36 10 4 1 35 12

[0071] According to the partial experimental data shown in the above figure, through data analysis, it is obtained that when the coating speed increases, the average thickness generally decreases. Since the flow time of the coating on the substrate surface is shortened, the deposition of the coating becomes uneven. At different coating speeds, as the acceleration increases, the thickness uniformity decreases. The faster the coating speed, the increase in acceleration will limit the fluidity of the coating, resulting in thickness fluctuations. The experimental data shows that there is an interaction between the coating speed and coating acceleration, which has a significant impact on the coating thickness uniformity. In the present invention, by using a single method to train multiple parameters affecting coating, the training efficiency and generalization ability of the motion control method of the PI coating equipment are improved, and the phenomenon of overfitting is avoided.

[0072] In this embodiment, the SimuLink software is used to build a virtual simulation platform to simulate the actual operating environment of the PI coating equipment, and the initial parameters of the virtual simulation platform are set at the same time.

[0073] In this embodiment, the external data acquisition module of the PI coating equipment is used to collect relevant data of the glass substrate during the process flow. For example, the attitude angle of the glass substrate in the equipment at that time is collected through a gyroscope, the temperature and humidity data of the glass substrate during movement are collected through a temperature and humidity measurement module, and the pressure of the environment where the glass substrate is located at that time is obtained through a pressure gauge. According to the data collected by the external acquisition module and the operation mode of the glass substrate in the process flow, the initial parameters are determined by using the parameter empirical tuning method. At the same time, parameter constraints are imposed on the initial parameters, and a suitable parameter range is set to avoid overfitting during the training process. In a specific environment, ensure that the strategy meets physical or environmental requirements, prevent excessive exploration, and avoid taking overly risky behaviors during training exploration.

[0074] As a machine learning method, reinforcement learning learns how to make decisions by interacting with the environment. Its core consists of "agent", "environment", "state", "action", and "reward". The agent selects actions in each state and adjusts the strategy according to the rewards feedback by the environment to maximize the long-term return.

[0075] In this embodiment, the state space of reinforcement learning is set to include: the speed adjustment of the PI coating equipment, the acceleration adjustment of the PI coating equipment, and the coating thickness adjustment. The action space includes: the speed of the PI coating equipment, the acceleration of the PI coating equipment, the coating thickness, the environmental temperature, and the environmental humidity.

[0076] Configure sensors in the virtual simulation platform. Through interaction with the virtual simulation platform, obtain data related to the operation of the PI coating equipment. The data are the key parameters during the movement of the coating equipment collected by the sensors, including: the speed, acceleration, temperature, humidity, pressure, coating quality index thickness and uniformity of the coating equipment; the feedback data when the PI coating equipment is unstable or the coating fails; the historical operation data of the PI coating equipment.

[0077] The obtained equipment data shows the behavior and performance of the PI coating equipment under different conditions, which can help optimize parameters such as coating speed and acceleration, thereby improving coating quality and production efficiency. The feedback data when the PI coating equipment is unstable or the coating fails is used to help learn the optimal control method during the training process. At the same time, according to different feedback data, the control strategy is dynamically adjusted to optimize the control method to improve coating quality. The historical operation data of the PI coating equipment can be used to analyze the effects of different control methods, optimize the control methods, and ensure the stability and reliability of the control methods.

[0078] The Q-Learning algorithm is a value-function-based reinforcement learning algorithm. It learns a Q matrix through interaction with the environment of the PI coating equipment, thereby guiding the agent to select the optimal action in each state. It does not require prior knowledge of the dynamic model of the environment, but learns the optimal strategy through trial and error.

[0079] like Figure 2 As shown in the flowchart, the steps of using data to train the Q matrix in the embodiment are:

[0080] S401: setting the current PI coating device state to the initial state of the PI coating device, setting the learning rate, discount factor, number of trainings, maximum number of training steps and ε-greedy strategy during the training process;

[0081] The learning rate controls the step size of the Q-value update, the discount factor controls the importance of measuring future rewards, and the ε-greedy strategy is used to balance exploration and exploitation by setting the exploration rate ∈.

[0082] S402: Select a decision action using the ε-greedy strategy according to the initial state and the Q matrix;

[0083] Use the ε-greedy strategy to select actions, select the action with the highest Q value with probability 1-∈ (greedy selection), and randomly select an action with probability ∈ (exploration), which ensures that both the known optimal strategy and the exploration of new strategies can be used during training.

[0084] S403: Execute the decision action to obtain a new state;

[0085] S404: Observe the new state and the immediate reward obtained according to the comprehensive reward function;

[0086] The immediate reward obtained through the comprehensive reward function determines the quality of the decision-making action;

[0087] S405: Update the Q matrix based on the Q-learning update rule;

[0088] The expression of Q-learning update rule is:

[0089] Q(S t ,a t )←Q(S t ,a t )+α·[R t +γ×maxQ(S t+1 ,a)-Q(S t ,a t )]

[0090] In the formula, Q(S t ,a t ) indicates state S t Take action a t Q value, α represents the learning rate, R t represents the immediate reward, γ represents the discount factor, maxQ(S t+1 ,a) means in state St+1 The maximum Q-value for taking action a.

[0091] S406: Set the new state as the current state;

[0092] Repeat steps S401 - S406 until the termination state is reached or the preset number of training steps at the start is reached, and end the training.

[0093] The trained Q matrix stores the optimal Q-value for each state-action pair, that is, the expected cumulative reward for taking a certain action in a given state. Input the current state of the PI coating equipment, output the action corresponding to the current state, and the PI coating equipment operates according to the action.

[0094] Exemplarily, set all initial Q-values to 0, set the learning rate α to 0.6, the discount factor γ to 0.8, the number of training times to 200, the maximum number of training steps to 20, and ε in the ε-greedy policy to 0.2. Train the Q matrix according to the above-set parameters, then input the current state of the PI coating equipment into the trained Q matrix, output the action corresponding to the current state, and the PI coating equipment operates according to the action. The specific values of the above parameters can be adjusted accordingly according to the specific process design scenario.

[0095] The motion control method of the PI coating equipment based on reinforcement learning further includes: using external sensors of the coating equipment to measure parameters such as the environmental temperature, environmental humidity, pressure, coating speed, and coating acceleration of the coating equipment, and using measuring tools to measure the quality thickness uniformity of the glass substrate; inputting the measured data into the trained Q matrix to test whether the trained Q matrix can output the optimal action according to the current state of the PI coating equipment during actual operation; at the same time, evaluating the stability of the PI coating equipment operating according to the action. When the stability evaluation result is lower than the preset target, retrain the Q matrix until the stability evaluation result reaches the preset target.

[0096] In this embodiment, when the stability evaluation result is lower than the preset target, adjust parameters such as the learning rate α, discount factor γ, and number of training times in the Q matrix, retrain the Q matrix, and re-input the current state of the PI coating equipment until the stability performance evaluation result of the glass substrate movement is higher than or equal to the preset expectation.

[0097] Embodiment 2

[0098] In this embodiment, the design process of the comprehensive reward function is described in detail. The comprehensive reward function includes: the PI coating equipment acceleration stability reward function, the coating quality reward function, and the PI coating equipment speed stability reward function.

[0099] The acceleration stability reward function of the PI coating equipment is used to encourage the PI coating equipment to keep the coating acceleration within the set range. The smaller the absolute value of the acceleration, the more stable the acceleration, and the higher the reward value. The expression is:

[0100] R a =-|a t |

[0101] In the formula, R a represents the acceleration stability reward function of the PI coating equipment, and a t represents the acceleration of the PI coating equipment at time t;

[0102] The coating quality reward function is used to encourage the actual thickness of the coating to be as close as possible to the ideal thickness. The smaller the error, the better the coating quality, and the higher the reward value. The expression is:

[0103] R h =-|δh|

[0104] In the formula, R h represents the coating quality reward function, and δh represents the error between the actual thickness and the ideal thickness of the coating;

[0105] The speed stability reward function of the PI coating equipment is used to encourage the PI coating equipment to keep the speed stable. The smaller the deviation between the actual speed and the target speed, the more stable the speed, and the higher the reward value. The expression:

[0106] R v =-|v t -v target |

[0107] In the formula, R v represents the speed stability reward of the PI coating equipment, v t represents the actual speed of the PI coating equipment, and v target represents the target speed of the PI coating equipment;

[0108] The acceleration stability reward function, coating quality reward function, and speed stability reward function of the PI coating equipment are weighted and calculated to obtain the comprehensive reward function. The expression is:

[0109] R total =α a R a +α h R h +α v R v

[0110] In the formula, R total represents the comprehensive reward function, α a represents the weight coefficient of the acceleration stability reward function of the PI coating equipment, αh represents the weight coefficient of the coating quality reward function, α v represents the weight coefficient of the PI coating equipment speed stability reward.

[0111] Design an integrated reward function that can simultaneously consider three key performance indicators: acceleration stability, coating quality, and speed stability, so as to optimize the operation strategy of the PI coating equipment. The selection of the weight coefficient depends on the relative importance of each performance indicator in the specific application scenario and can be adjusted according to actual needs.

[0112] In this embodiment, during the test process, a feedback mechanism is used to dynamically adjust the weights of the integrated reward function. When the actual values of the PI coating equipment acceleration, coating quality, and PI coating equipment speed are close to or reach the required target values, the weights of their corresponding reward functions are reduced. When the actual values deviate from the target values, the weights of their corresponding reward functions are increased. The expression satisfied by the dynamic adjustment is:

[0113]

[0114] In the formula, α i (t + 1) represents the adjusted parameter weight, α i (t) is the weight in the current iteration. i represents any one of the PI coating equipment acceleration stability reward function, coating quality reward function, and PI coating equipment speed stability reward function. σ represents the scaling factor of the weight ratio, P target represents the target value of the PI coating equipment, P i (t) represents the actual value of the PI coating equipment in the current iteration.

[0115] Specifically, when the fluctuation range of the coating thickness is large or the coating acceleration suddenly changes violently, the corresponding weight is increased, so that it receives more attention in subsequent training.

[0116] Exemplarily, in the initial training stage, first focus on the uniformity of the coating thickness and the stability of the movement. Therefore, the weights of the PI coating equipment acceleration stability reward function, coating quality reward function, and PI coating equipment speed stability reward function are set as follows:

[0117] α a = 0.4, α h = 0.5, α v = 0.1

[0118] When reaching the mid - stage of training, after several rounds of training, when the coating thickness has stabilized near the target value, the weight of the coating thickness is reduced, and more attention is paid to the speed stability of the PI coating equipment. Therefore, the weights of the acceleration stability reward function of the PI coating equipment, the coating quality reward function, and the speed stability reward function of the PI coating equipment are set as follows:

[0119] α a = 0.4, α h = 0.2, α v = 0.4

[0120] By the late stage of training, the coating thickness, the motion stability of the coating equipment, and the coating speed have all reached the ideal state. Therefore, the weights of the three should be averaged. So, the weights of the acceleration stability reward function of the PI coating equipment, the coating quality reward function, and the speed stability reward function of the PI coating equipment are set as follows:

[0121] α a = 0.33, α h = 0.33, α v = 0.33

[0122] In the actual application process, the specific weight values should be calculated through the weight adjustment formula according to the actual performance of the PI coating equipment.

[0123] In this embodiment, in the initial stage of training or in the stage where the feedback changes rapidly, a larger learning rate is adopted to accelerate the adjustment of the weights of the reward function, so as to have a more rapid response to the target with poor performance; as the training progresses, in the middle and late stages or when it has entered the stable stage, a smaller learning rate is adopted, which can accurately approach the target value.

[0124] Exemplarily, in the initial stage of training or in the stage where the feedback changes rapidly, the learning rate is set to 0.05. In the middle and late stages or when it has entered the stable stage, the learning rate is set to 0.01.

[0125] Embodiment 3

[0126] As Figure 3 shown in the structural diagram, in this embodiment, a motion control system of a PI coating equipment based on reinforcement learning is also provided. The system includes:

[0127] A virtual simulation platform building module, which is used to build a virtual simulation platform that simulates the actual operating environment of the PI coating equipment and provides a training environment for reinforcement learning;

[0128] A Q - matrix creation module, which creates and initializes the Q - matrix. Among them, the rows of the Q - matrix represent the states of the PI coating equipment, and the columns of the Q - matrix represent the actions taken by the PI coating equipment;

[0129] A comprehensive reward function setting module for designing a comprehensive reward function that takes into account the weights of the process performance parameters of the PI coating equipment;

[0130] A training module for obtaining data related to the operation of the PI coating equipment by interacting with a virtual simulation platform, training a Q matrix using the data based on reinforcement learning, and dynamically adjusting the weights of the comprehensive reward function during the training process until the training is completed;

[0131] An operation module for inputting the current state of the PI coating equipment into the trained Q matrix and outputting the corresponding action for the current state, and the PI coating equipment operates according to the action.

[0132] The embodiments are only examples given to clearly illustrate the present invention and are not limitations on the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to enumerate all the implementation manners here. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the claims of the present invention.

Claims

1. A motion control method for a PI coating device based on reinforcement learning, characterized in that, It includes the following steps: S1: Build a virtual simulation platform that simulates the actual operating environment of the PI coating equipment to provide a training environment for reinforcement learning; S2: Create and initialize a Q matrix, where the rows of the Q matrix represent the states of the PI coating equipment, and the columns of the Q matrix represent the actions taken by the PI coating equipment; S3: Design a comprehensive reward function that considers the weight of the process performance parameters of the PI coating equipment; S4: Through interaction with the virtual simulation platform, obtain data related to the operation of the PI coating equipment. Based on reinforcement learning, use the data to train the Q matrix. During the training process, adopt a feedback mechanism to dynamically adjust the weight of the comprehensive reward function until the training ends; S5: Input the current state of the PI coating equipment into the trained Q matrix, output the action corresponding to the current state, and the PI coating equipment operates according to the action.

2. The motion control method of a PI coating device based on reinforcement learning according to claim 1, wherein The state space of reinforcement learning includes: speed adjustment of the PI coating equipment, acceleration adjustment of the PI coating equipment, and coating thickness adjustment. The action space includes: speed of the PI coating equipment, acceleration of the PI coating equipment, coating thickness, environmental temperature, and environmental humidity.

3. A motion control method for a PI coating device based on reinforcement learning according to claim 1, characterized in that The comprehensive reward function includes: PI coating equipment acceleration stability reward function, coating quality reward function, and PI coating equipment speed stability reward function; The expression of the PI coating equipment acceleration stability reward function is: R a = -|a t | where, R a represents the acceleration stability reward function of the PI coating equipment, and a t represents the acceleration of the PI coating equipment at time t; The expression of the coating quality reward function is: R h = -|δh| where R h represents the coating quality reward function, and δh represents the error between the actual thickness and the ideal thickness of the coating; The expression of the PI coating equipment speed stability reward is: R v = -|v t - v target | Wherein, R v represents the PI coating equipment speed stability reward, v t represents the actual speed of the PI coating equipment, v target represents the target speed of the PI coating equipment; Perform weighted calculation on the PI coating equipment acceleration stability reward function, coating quality reward function, and PI coating equipment speed stability reward function to obtain the comprehensive reward function, and the expression is: R total = α a R a + α h R h + α v R v where R total represents the comprehensive reward function, α a represents the weight coefficient of the PI coating equipment acceleration stability reward function, α h represents the weight coefficient of the coating quality reward function, α v represents the weight coefficient of the PI coating equipment speed stability reward.

4. The method for controlling the movement of the PI coating equipment based on reinforcement learning according to claim 1, wherein, Configure sensors in the virtual simulation platform. Through interaction with the virtual simulation platform, obtain data related to the operation of the PI coating equipment. The data is the key parameters during the movement of the coating equipment collected by the sensors, including: speed, acceleration, temperature, humidity, pressure, coating quality index thickness and uniformity of the coating equipment, feedback data when the PI coating equipment is unstable or coating fails, and historical operation data of the PI coating equipment.

5. A motion control method for a PI coating device based on reinforcement learning according to claim 4, characterized in that During the testing process, adopt a feedback mechanism to dynamically adjust the weight of the comprehensive reward function. When the actual values of the PI coating equipment acceleration, coating quality, and PI coating equipment speed are close to or reach the required target values, reduce the weight of the corresponding reward function. When the actual value deviates from the target value, increase the weight of the corresponding reward function. The expression satisfied by the dynamic adjustment is: Where, α i (t + 1) represents the adjusted parameter weight, and α i (t) is the weight in the current iteration. i represents any one of the PI coater acceleration stability reward function, the coating quality reward function, and the PI coater speed stability reward function. σ represents the scaling factor of the weight ratio, and P target represents the target value of the PI coater, and P i (t) represents the actual value of the PI coater in the current iteration.

6. The motion control method of a PI coating device based on reinforcement learning according to claim 5, characterized in that The steps of using data to train the Q matrix in S4 are: S401: Set the current state of the PI coating equipment as the initial state of the PI coating equipment, and set the learning rate, discount factor, number of training times, maximum number of training steps, and ε-greedy strategy during the training process; S402: According to the initial state and the Q matrix, select a decision-making action using the ε-greedy strategy; S403: Execute the decision-making action to obtain a new state; S404: Observe the new state and the immediate reward obtained according to the comprehensive reward function; S405: Update the Q matrix based on the Q-learning update rule; S406: Set the new state as the current state; Repeat steps S401 - S406 until reaching the termination state or the preset number of training steps at the start, and end the training.

7. A motion control method for a PI coating device based on reinforcement learning according to claim 6, characterized in that, The expression of the Q - learning update rule is: Q(S t , a t ) ← Q(S t , a t ) + α · [R t + γ × maxQ(S t+1 , a) - Q(S t , a t )] Wherein, Q(S t , a t ) represents the Q-value of taking action a in state S t , α represents the learning rate, R t represents the immediate reward, γ represents the discount factor, and maxQ(S t , a) represents the maximum Q-value of taking action a in state S t+1 . t+1 ​ 8. A motion control method for a PI coating device based on reinforcement learning according to claim 1, characterized in that It also includes: Use the external sensors of the PI coating equipment to measure the environmental temperature, environmental humidity, pressure, coating speed, and coating acceleration parameters of the coating equipment, and use measuring tools to measure the quality thickness uniformity of the glass substrate; input the measured data into the trained Q - matrix, and test whether the trained Q - matrix can output the optimal action according to the current state of the PI coating equipment during actual operation; at the same time, evaluate the stability of the PI coating equipment when operating according to the action. When the stability evaluation result is lower than the preset target, retrain the Q - matrix until the stability evaluation result reaches the preset target.

9. A motion control method for a PI coating device based on reinforcement learning according to claim 1, characterized in that, Use SimuLink software to build a virtual simulation platform that simulates the actual operating environment of the PI coating equipment.

10. A motion control system for a PI coating equipment based on reinforcement learning, which is used to implement a motion control method for a PI coating equipment based on reinforcement learning described in any one of claims 1-9, characterized in that, It includes: A virtual simulation platform building module, used to build a virtual simulation platform that simulates the actual operating environment of the PI coating equipment, providing a training environment for reinforcement learning; A Q - matrix creation module, which creates and initializes the Q - matrix. Among them, the rows of the Q - matrix represent the states of the PI coating equipment, and the columns of the Q - matrix represent the actions taken by the PI coating equipment; A comprehensive reward function setting module, used to design a comprehensive reward function considering the weight of the process performance parameters of the PI coating equipment; A training module, used to obtain data related to the operation of the PI coating equipment by interacting with the virtual simulation platform, and based on reinforcement learning, use the data to train the Q - matrix. During the training process, adopt a feedback mechanism to dynamically adjust the weight of the comprehensive reward function until the training ends; An operation module, used to input the current state of the PI coating equipment into the trained Q - matrix, output the action corresponding to the current state, and the PI coating equipment operates according to the action.

Images