AOI optical scheme automatic optimization method based on reinforcement learning

Abstract

The invention discloses an AOI optical scheme automatic optimization method based on reinforcement learning, and belongs to the technical field of automatic optical detection. According to the method, a reinforcement learning technology is applied to automatic adjustment of optical schemes in a machine vision system, wafer images under different optical schemes are collected by using an AOI system, and scoring is performed by using a plurality of semantic segmentation models; constructing and training a non-reference image quality evaluation network, evaluating the optical imaging quality in real time, and designing a reward function on the basis; by using the Actor-Critic algorithm, the intelligent agent can autonomously learn an optimal parameter adjustment strategy, quickly adapt to different scenes, realize dynamic optimization of light source parameters and remarkably improve image quality, and an efficient and intelligent solution is provided for efficient application of a machine vision system in a complex industrial environment.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic optical inspection (AOI), and in particular to an automatic optimization method for AOI optical schemes based on reinforcement learning. Background Art

[0002] Automated Optical Inspection (AOI) is a non-contact, vision-based, automated process for quality inspection of products exiting a production line. AOI uses optical imaging, automatically scanning the workpiece with a camera, capturing an image of the part at extremely high resolution. It then employs a variety of algorithms to analyze and identify the captured image, replacing manual inspection. AOI offers advantages such as high reliability, ease of use, and low cost. The high-definition imaging in AOI helps identify subtle defects and avoid missed and false detections.

[0003] To obtain high-definition workpiece images, in addition to extremely high-resolution lenses, a good optical solution plays a vital role. Different optical solutions, i.e., different light source parameters (such as light source angle and intensity), can significantly affect image quality and, in turn, defect detection accuracy.

[0004] Currently, there is limited research on automated adjustment methods for light source parameters. Most AOI systems rely on manual adjustments or simple automatic adjustment algorithms, lacking efficient automated adjustment strategies. Manual adjustment methods are difficult to adapt to the needs of different inspection tasks, especially when inspecting multiple types of defects. Frequent parameter adjustments are required to achieve the best inspection results, increasing operational complexity and time costs. Summary of the Invention

[0005] The problem to be solved by the present invention is to provide an automatic optimization method for AOI optical solutions based on reinforcement learning, which applies reinforcement learning technology to the automatic adjustment of optical solutions in machine vision systems. It can quickly adjust parameters according to image quality, adapt to changing detection needs, improve detection efficiency and defect recognition accuracy, and enhance system stability and reliability.

[0006] The present invention adopts the following technical solution: a method for automatic optimization of AOI optical scheme based on reinforcement learning, comprising the following steps: Step 1: Use the AOI system to collect wafer images under different optical schemes, establish an initial wafer image dataset, and perform preprocessing; Step 2: Use multiple semantic segmentation models to score the preprocessed wafer image dataset, train a no-reference image quality assessment network, fit the scores, and obtain an immediate reward signal based on the trained no-reference image quality assessment network; Step 3: Build a reinforcement learning model: Step 3.1, construct the initial state space and obtain the light source parameters and environmental state; Step 3.2: Define the action space, control the robot arm to move to the surrounding grid, and adjust the coarse-to-fine image acquisition strategy; Step 3.3: Establish a reward function based on the score of each wafer image. Step 3.4: Build the policy network and value network, and use the environment state as multimodal input to enter the policy network and value network; Step 4: Training using the Advantage Actor-Critic algorithm: The reinforcement learning model outputs DEI and feeds it back to the value network as the environment’s reward for the agent’s behavior; The policy network outputs the action probability distribution, selects the most likely action, and executes the parameter action; The value network is trained using images and rewards, outputting value estimates for input states to guide the update of the strategy network. Each network is trained synchronously to achieve automatic optimization of the AOI optical solution.

[0007] Preferably, step 2 includes the following sub-steps: Step 2.1: Use multiple semantic segmentation models to score the pre-processed wafer image dataset, and use MIoU to reflect the detection ability of the semantic segmentation model to show the wafer image quality under the current optical scheme, and obtain the image quality of each wafer. A score to indicate the degree of defect exposure; Step 2.2: Train the no-reference image quality assessment network using the existing dataset and fit the score of the wafer image dataset. , obtain the trained no-reference image quality assessment network; Step 2.3: Use the trained no-reference image quality assessment network as part of the subsequent environment to obtain an immediate reward signal.

[0008] Preferably, in step 3.1, the light source parameters include light intensity and lighting angle, and changes in the light source parameters lead to changes in the environmental state. The light source parameters are represented as a two-dimensional vector; The environmental state consists of three parts, specifically: Simulate the acquired high-resolution wafer images: Build an image acquisition simulation model to simulate the imaging process of industrial cameras and light sources in the AOI system, help establish the state space, and obtain observation images under different optical parameter combinations; Current location information: One-hot encoding is used to indicate the current location of the agent; Access trajectory matrix record: records the defect exposure index value of the currently visited grid point to form environmental memory.

[0009] Preferably, in step 3.2, light intensity and lighting angle are used as core adjustment parameters to construct a two-dimensional discrete action space, and the light intensity and lighting angle are adjusted by adjusting the action; the action space is used to control the movement of the robotic arm to the surrounding grid to achieve coarse-to-fine image acquisition strategy adjustment.

[0010] Preferably, in step 3.3, a reward function is established based on the score of each wafer image. The higher the DEI score, the higher the image quality. The reward function includes: instant change reward, progress reward, repeated visit penalty, and close target reward. The final reward function is obtained. ; Further set round reward smoothing to reduce reward fluctuations during training.

[0011] Preferably, in step 3.4, the environment state is used as a multimodal input into the policy network and the value network, including the following sub-steps: Step 3.4.1: Input the current location information and access trajectory matrix records into the Vision Transformer model, and input the simulated high-resolution wafer image into the EfficientViT model. After average pooling, perform feature fusion and project it to 128 dimensions through a fully connected layer to obtain the fused state value feature vector. Step 3.4.2: The fused 128-dimensional state-value feature vector is input into the policy network after passing through the ReLu activation function. The fully connected layer outputs a 4-dimensional vector. The softmax activation function is used to generate the action distribution, and the four actions of up, down, left, and right are obtained. After the policy network is trained, the output is the action with the highest probability. Step 3.4.3: The fused 128-dimensional state-value feature vector is input into the value network after passing through the ReLu activation function, and then outputted as a 64-dimensional vector through a fully connected layer, and then outputted as a scalar state value estimate through another fully connected layer.

[0012] Preferably, in step 4, the Advantage Actor-Critic algorithm is used for training, which includes the following sub-steps: Step 4.1. Initialize the policy network, value network parameters and experience pool, and input the current state , actions taken , rewards received and the next state , stored through the experience pool; Step 4.2: In each round of training, the estimated value network and the target value network are used to evaluate the value of the current state and the next state respectively, and the loss function is calculated to measure the difference between the estimated value and the target value. Step 4.3: After each round of training, perform a gradient update on the policy network and value network based on the state sequence, action sequence, and reward sequence collected in this round; The value network parameters are estimated by gradient descent, and the policy network outputs the action probability distribution. The target policy network provides stable policy evaluation and assists in updating the policy network. The policy network parameters are adjusted using the advantage function through policy gradient ascent. Step 4.4: Output the updated policy network, value network parameters, and loss value to guide the agent's behavior in the environment to maximize long-term rewards.

[0013] Preferably, in step 4.1, initialize the policy network and value network Parameters and , set the initial policy network learning rate , value network learning rate , discount factor , Greedy Probability , the total number of steps is T, initialize the experience pool to store experience.

[0014] At time step When the agent follows the greedy probability Select Action , is the status Next action The state action value function represents the execution of an action in the state The expected cumulative reward obtained by the agent; When the sampling probability is greater than hour, Indicates that the status Next, select the action that maximizes the Q value; When the sampling probability is less than or equal to When , the agent randomly selects an action to explore the new strategy space and avoid falling into the local optimum.

[0015] At each time step t, the policy network outputs the action probability distribution based on the current state and samples the action ; The value network estimates the value function of the current state; The agent performs actions After that, the robot arm makes a move to change the optical parameters, and the environment returns to the next state, obtaining a new wafer image and current position information, and accessing the trajectory matrix record; The new wafer image is input into the no-reference image quality assessment network to obtain DEI, and the environment returns an immediate reward based on the DEI score. , used to measure the degree of improvement in collection effect; Observe the immediate reward and next state , the quadruple Deposit into experience pool.

[0016] Preferably, in step 4.3, after each round of training is completed, a gradient update is performed on the policy network and the value network based on the state sequence, action sequence, and reward sequence collected in this round, as follows: Step 4.3.1. Randomly and uniformly sample a small batch of experiences from the experience pool as samples, calculate the policy gradient, and combine it with the advantage function to maximize the expected return by directly optimizing the policy network parameters; Step 4.3.2: Update the value network parameters using the gradient descent algorithm; update the policy network parameters using the advantage function using the policy gradient ascent algorithm. Step 4.3.3: Minimize the loss function of the policy network and update the policy network; minimize the loss function of the value network and update the value network. Synchronously update the parameters of the two networks and calculate the cumulative discounted reward.

[0017] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects: 1. The reinforcement learning model built in this invention has strong adaptability and can automatically adjust the light source parameters according to the current inspection task and workpiece type. This adaptability enables the AOI system to quickly respond to different inspection requirements, improving its flexibility and versatility.

[0018] 2. The AOI optical scheme automatic optimization method of the present invention combines the advantages of deep learning and reinforcement learning, and can quickly learn and optimize light source parameters. The reference-free image quality assessment network can objectively evaluate image quality and provide a reliable reward signal for reinforcement learning.

[0019] 3. The Actor-Critic architecture of the present invention further accelerates the learning speed. Compared with the traditional DQN (Deep Q-Network)-based method, it significantly improves the automation level and efficiency of optical solution optimization, and can provide innovative solutions for the intelligent upgrade of industrial quality inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Advantage Actor-Critic algorithm training flow chart of the present invention; Figure 2 This is the overall framework diagram of the reinforcement learning model of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the application are further elaborated in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in the present invention. All non-innovative embodiments of other researchers in this field on this embodiment fall within the scope of protection of the present invention. At the same time, the step numbers in the embodiments of the present invention are only set for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0022] In one embodiment of the present invention, a method for automatic optimization of an AOI optical solution based on reinforcement learning includes the following steps:

[0023] 1. Use the AOI system to collect images.

[0024] In this embodiment, 10 different light source intensities and 10 different illumination angles are set, and the incident light source and the chip are evenly sampled at 10 different angles ranging from 22.5° to 67.5°, forming a total of 100 different optical solutions.

[0025] Images were taken under different optical schemes to establish an initial dataset with a total of 7,800 images with a resolution of 2048×2048. The collected images were further preprocessed and resized to 224×224 for subsequent input into the model for processing.

[0026] 2. Use multiple semantic segmentation models to score the established wafer image dataset.

[0027] MIoU reflects the detection capability of the model and can indicate the quality of wafer images. Therefore, in this embodiment, the score of each wafer image is defined as the defect exposure capability, that is, the Defect Exposure Index (DEI), and is scored using the following formula: ; in, Indicates the i-th Model, Indicates the number of semantic segmentation models used, A score for each wafer image, indicating the degree of defect exposure.

[0028] In this embodiment, preferably, .

[0029] 3. The reference-free image quality assessment network TOPIQ is trained using the existing dataset to enable it to effectively fit DEI.

[0030] The trained no-reference image quality assessment network model is used as part of the subsequent environment to obtain immediate reward signals. The score predicted by the no-reference image quality assessment network reflects the quality of the image under the current optical scheme, that is, the degree of defect exposure.

[0031] 4. Construct the initial state space.

[0032] In this embodiment, the light source parameters include light intensity (Intensity) and lighting angle (Angel), which are expressed as a two-dimensional vector, as follows: , ; ; in, Indicates different optical schemes, Represents different light intensities, Indicates different lighting angles.

[0033] Changes in light source parameters will lead to changes in the environmental state. The environmental state consists of the following three parts, which enter the policy network and value network as multimodal inputs.

[0034] (1) The high-resolution wafer image collected by the current simulation: Represents the image captured under the current optical solution configuration. Due to the high overhead of real-time image acquisition and complex control parameters, this embodiment constructs an image acquisition simulation model to simulate the imaging process of the industrial camera and light source in the AOI system, helping to establish the state space, that is, to obtain observation images under different optical parameter combinations.

[0035] (2) Current location information: One-hot encoding is used to represent the current location of the agent.

[0036] (3) Access trajectory matrix records: The defect exposure index value of the currently visited grid point constitutes the environmental memory.

[0037] Furthermore, the current position information tensor and access trajectory matrix records are input into the Vision Transformer model, and the high-resolution wafer image collected by simulation is input into the EfficientViT model. The three are respectively average-pooled and then feature-fused. They are projected to 128 dimensions through the fully connected layer. The fused state-value feature vector is activated by the ReLu (Rectified Linear Unit) function and sent to the policy network and value network respectively to define the action space.

[0038] 5. Use light intensity and lighting angle as core adjustment parameters to construct a two-dimensional discrete action space.

[0039] In this embodiment, the light intensity and lighting angle are adjusted by adjusting the action to define the action space. , used to control the robot arm to move toward the surrounding grid and implement coarse-to-fine image acquisition strategy adjustment: ; in, Respectively represent the four actions of up, down, left and right.

[0040] 6. Design the reward function.

[0041] The reward function is designed based on DEI. The higher the DEI score, the higher the image quality. In this embodiment, the reward function includes: instant change reward, progress reward, repeated visit penalty, and close-to-target reward.

[0042] (1) Instantaneous change rewards, used to encourage the agent to take actions that improve DEI: ; in, Indicates the new DEI score of the image after the agent performs the current action; Represents the current DEI score of the image before the agent performs the action (i.e., the DEI value of the previous time step).

[0043] (2) Progress Rewards: ; in, To prevent division by zero errors, DEI represents the initial state, The DEI represents the maximum possibility, and the molecule calculates the increment from the initial state to the current state, which is and use To ensure non-negative, divide by , normalized to scope.

[0044] The progress reward measures the long-term progress of the agent from the initial state to the current state, rather than just a single step change. Lower than ,but , to prevent negative rewards from interfering with learning.

[0045] (3) Access Penalty: ; If the agent visits the same location repeatedly, it is affected by The penalty can prevent the agent from spinning in place or falling into a loop and encourage the agent to explore new areas, avoiding overly conservative strategies.

[0046] (4) Rewards for approaching the target: ; When the agent's state is very close to the target DEI (the error in this example is <0.01), an additional reward of 0.2 is given. This reward helps the agent converge to the optimal solution accurately and ensures that the agent eventually completes the task.

[0047] In this embodiment, the final reward function is set as: .

[0048] Furthermore, this embodiment also sets round reward smoothing: ; in, Represents the total reward of the current episode, Represents the exponential moving average (EMA) of historical round rewards, which can reduce reward fluctuations during training and make policy optimization more stable.

[0049] 7. Build a strategy network (Actor) and a value network (Critic).

[0050] Specifically, the Advantage Actor-Critic algorithm is used for training, the policy network parameters are updated through policy gradient ascent, and the value network parameters are updated through the TD algorithm.

[0051] The fused state-value feature vector is input into the policy network after passing through the ReLu activation function. The 128-dimensional vector is output as a 4-dimensional vector through the fully connected layer. The action distribution is then generated through the softmax activation function, namely the four actions of up, down, left, and right. After training, the policy network outputs the action with the highest probability.

[0052] The fused state value feature vector is input into the value network after passing through the ReLu activation function. The 128-dimensional vector outputs a 64-dimensional vector through a fully connected layer, and then passes through another fully connected layer to output a scalar state value estimate.

[0053] 8. The reinforcement learning model is built through steps 4 to 7 above.

[0054] The reinforcement learning model outputs DEI and feeds it back to the value network as the environment's reward for the agent's behavior, while the policy network outputs the action probability distribution, selects the most likely action, and executes the parameter action. The value network is trained using images and rewards, outputs value estimates for the input state, and guides the update of the policy network. Each network is trained synchronously to achieve decision optimization.

[0055] 9. Start training.

[0056] At the beginning of each episode, initialize the policy network and value network Parameters and ,in, are the weight and bias parameters of the policy network, are the weight and bias parameters of the value network.

[0057] Set the initial policy network learning rate , value network learning rate , discount factor , Greedy Probability , the total number of steps is T, and the experience pool Experience-Pool is initialized to store experience.

[0058] In this embodiment, the greedy strategy is expressed as: ; At time step When the agent follows the greedy probability Select Action , is the status Next action The state action value function. It represents the execution of the action in the state After that, the expected cumulative reward that the agent can obtain.

[0059] When the sampling probability is greater than hour, Indicates that the status Next, select the action that maximizes the Q value.

[0060] When the sampling probability is less than or equal to When , the agent randomly selects an action to explore the new strategy space and avoid falling into the local optimum.

[0061] At each time step t, the policy network outputs the action probability distribution based on the current state and samples the action , while the value network estimates the value function of the current state .

[0062] Agent performs actions After that, the action of the robot arm can change the optical parameters, the environment returns to the next state, and a new wafer image and current position information are obtained. The trajectory matrix record is accessed, and the new wafer image is input into the reference-free image quality assessment model TOPIQ to obtain DEI.

[0063] The environment returns instant rewards based on DEI scores , used to measure the degree of improvement in collection effect. Observe the reward and the next state , the quadruple Deposit into experience pool.

[0064] 10. After each episode is completed, the system will perform a gradient update on the policy network (Actor) and the value network (Critic) based on the state sequence, action sequence, and reward sequence collected in this episode.

[0065] Specifically, a small batch of experience is randomly and uniformly sampled from the experience pool. Based on this batch of samples, the policy gradient is calculated and combined with the advantage function. The expected return is maximized by directly optimizing the policy parameters, the loss function of the policy network is minimized, and the Actor network is updated.

[0066] At the same time, minimize the loss function of the value network, update the Critic network, and update the parameters of the two networks.

[0067] 11. Calculate cumulative discount rewards (Return): ; in, Indicates the immediate reward for the current step, is the discount factor. To improve training stability, the return value is standardized. Then: ; in, , Respectively represent all the The mean and standard deviation of A small constant to prevent division by zero.

[0068] 12. Calculate the advantage function.

[0069] In this embodiment, the Actor network is updated through the advantage function: ; in, is the current state value predicted by the Critic network, Represents how good the current action is compared to the average policy.

[0070] The policy network is updated to: ; in, is the loss function of the policy network, is the strategy output by the Actor network, where The larger it is, the more it strengthens the probability of the current action, and the policy network updates parameter, .

[0071] Value Network Update: ; in, is the loss function of the value network, and the value network updates parameter, .

[0072] Construct the total loss: ;

[0073] This embodiment optimizes the two losses uniformly, so that the policy network continuously adjusts its decision direction according to the feedback from the value network, thereby achieving better policy learning.

[0074] In this embodiment, the model training process is as follows Figure 1 As shown, the overall framework of the model is as follows Figure 2 As shown, the total training process is as follows:

[0075] During training, the input includes the current state , actions taken , rewards received and the next state ,The experience pool stores this information for training.

[0076] The estimated value network and the target value network evaluate the value of the current state and the next state respectively, calculate the loss function to measure the difference between the estimated value and the target value, and update the estimated value network parameters through gradient descent.

[0077] The policy network outputs a probability distribution of actions, while the target policy network provides stable policy evaluation and assists in updating the policy network. Policy gradient ascent uses the advantage function to adjust the policy network parameters.

[0078] Finally, the updated policy and value network parameters, as well as the loss value, are output to guide the agent's behavior in the environment to maximize long-term rewards.

[0079] In summary, the reinforcement learning-based automatic optical solution optimization method proposed in this paper innovatively applies reinforcement learning technology to the automated adjustment of optical solutions in machine vision systems. By constructing a reference-free image quality evaluation network, it evaluates optical imaging quality in real time and uses this as the basis for designing a reward function. Furthermore, by utilizing the actor-critic algorithm, the intelligent agent can autonomously learn the optimal parameter adjustment strategy, quickly adapt to different scenarios, and dynamically optimize light source parameters, significantly improving image quality. This provides an efficient and intelligent solution for the effective application of machine vision systems in complex industrial environments.

[0080] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A reinforcement learning-based automatic optimization method for AOI optical solutions, characterized in that: The steps include: Step 1: Use the AOI system to collect wafer images under different optical schemes, establish an initial wafer image dataset, and perform preprocessing; Step 2: Use multiple semantic segmentation models to score the preprocessed wafer image dataset, train a no-reference image quality assessment network, fit the scores, and obtain an immediate reward signal based on the trained no-reference image quality assessment network; Step 3: Build a reinforcement learning model: Step 3.1, construct the initial state space and obtain the light source parameters and environmental state; Step 3.2: Define the action space, control the robot arm to move to the surrounding grid, and adjust the coarse-to-fine image acquisition strategy; Step 3.3: Establish a reward function based on the score of each wafer image. Step 3.4: Build the policy network and value network, and use the environment state as multimodal input to enter the policy network and value network; Step 4: Training using the Advantage Actor-Critic algorithm: The reinforcement learning model outputs DEI and feeds it back to the value network as the environment’s reward for the agent’s behavior; The policy network outputs the action probability distribution, selects the most likely action, and executes the parameter action; The value network is trained using images and rewards, outputting value estimates for input states to guide the update of the strategy network. Each network is trained synchronously to achieve automatic optimization of the AOI optical solution.

2. The AOI optical scheme automatic optimization method based on reinforcement learning according to claim 1 is characterized in that: Step 2 includes the following sub-steps: Step 2.1: Use multiple semantic segmentation models to score the preprocessed wafer image dataset. MIoU is used to reflect the detection capability of the semantic segmentation model and to show the wafer image quality under the current optical solution. The scoring formula is as follows: ； in, Indicates the i-th Model, Indicates the number of semantic segmentation models used, The score of each wafer image is used to indicate the degree of defect exposure; Step 2.2: Train the no-reference image quality assessment network using the existing dataset and fit the score of the wafer image dataset. , obtain the trained no-reference image quality assessment network; Step 2.3: Use the trained no-reference image quality assessment network as part of the subsequent environment to obtain an immediate reward signal.

3. The AOI optical scheme automatic optimization method based on reinforcement learning according to claim 1, characterized in that: In step 3.1, the light source parameters include light intensity and lighting angle. Changes in light source parameters lead to changes in environmental conditions. The light source parameters are represented as a two-dimensional vector: , ； ； in, Indicates different optical schemes, Represents different light intensities, Indicates different lighting angles; The environmental status consists of three parts, specifically: Simulate the acquired high-resolution wafer images: Build an image acquisition simulation model to simulate the imaging process of industrial cameras and light sources in the AOI system, help establish the state space, and obtain observation images under different optical parameter combinations; Current location information: One-hot encoding is used to indicate the current location of the agent; Access trajectory matrix record: records the defect exposure index value of the currently visited grid point to form environmental memory.

4. The AOI optical scheme automatic optimization method based on reinforcement learning according to claim 3 is characterized in that: In step 3.2, light intensity and lighting angle are used as core adjustment parameters to construct a two-dimensional discrete action space, and light intensity and lighting angle are adjusted by adjusting the action; The action space is used to control the robot arm to move toward the surrounding grid to achieve coarse-to-fine image acquisition strategy adjustment, which is expressed as: ; in, Respectively represent the four actions of up, down, left and right.

5. The AOI optical scheme automatic optimization method based on reinforcement learning according to claim 4 is characterized in that: In step 3.3, a reward function is established based on the score of each wafer image. The higher the DEI score, the higher the image quality. This includes: instant change reward, progress reward, repeated visit penalty, and close-to-target reward: The instant change reward is used to encourage the agent to take actions that improve DEI, expressed as : ； in, Indicates the new DEI score of the image after the agent performs the current action; Represents the current DEI score of the image before the agent performs the action; The progress reward is used to measure the long-term progress of the agent from the initial state to the current state and is expressed as : ； in, 、 Denote the DEI of the initial state and the current state respectively, DEI represents the greatest possible Used to prevent division by zero errors; the increment of the molecule from the initial state to the current state is expressed as and use Ensure it is non-negative by dividing , normalized to Range; if Lower than ,but ,prevent negative rewards from interfering with learning; The repeated visit penalty is used to prevent the agent from spinning in place or falling into a loop and encourages the agent to explore new areas, which is expressed as : ； If the agent visits the same location repeatedly, it is affected by punishment; The close-to-target reward is used to help the agent converge to the optimal solution and ensure that the agent eventually completes the task, which is expressed as : ； When the error between the agent's state and the target DEI is less than 0.01, an additional reward of 0.2 is given; The final reward function is obtained, expressed as : ； Round reward smoothing is also set to reduce reward fluctuations during training, expressed as : ； in, Represents the total reward of the current round, Represents the exponential moving average of historical round rewards.

6. The AOI optical scheme automatic optimization method based on reinforcement learning according to claim 5, characterized in that: In step 3.4, the environment state is used as a multimodal input to the policy network and value network, which includes the following sub-steps: Step 3.4.1: Input the current location information and access trajectory matrix records into the Vision Transformer model, and input the simulated high-resolution wafer image into the EfficientViT model. After average pooling, perform feature fusion and project it to 128 dimensions through a fully connected layer to obtain the fused state value feature vector. Step 3.4.2: The fused 128-dimensional state-value feature vector is input into the policy network after passing through the ReLu activation function. The fully connected layer outputs a 4-dimensional vector. The softmax activation function is used to generate the action distribution, and the four actions of up, down, left, and right are obtained. After the policy network is trained, the output is the action with the highest probability. Step 3.4.3: The fused 128-dimensional state-value feature vector is input into the value network after passing through the ReLu activation function, and then outputted as a 64-dimensional vector through a fully connected layer, and then outputted as a scalar state value estimate through another fully connected layer.

7. The AOI optical scheme automatic optimization method based on reinforcement learning according to claim 6, characterized in that: In step 4, the training using the Advantage Actor-Critic algorithm includes the following sub-steps: Step 4.

1. Initialize the policy network, value network parameters and experience pool, and input the current state , actions taken , rewards received and the next state , stored through the experience pool; Step 4.2: In each round of training, the estimated value network and the target value network are used to evaluate the value of the current state and the next state respectively, and the loss function is calculated to measure the difference between the estimated value and the target value. Step 4.3: After each round of training, perform a gradient update on the policy network and value network based on the state sequence, action sequence, and reward sequence collected in this round; The value network parameters are estimated by gradient descent, and the policy network outputs the action probability distribution. The target policy network provides stable policy evaluation and assists in updating the policy network. The policy network parameters are adjusted using the advantage function through policy gradient ascent. Step 4.4: Output the updated policy network, value network parameters, and loss value to guide the agent's behavior in the environment to maximize long-term rewards.

8. The AOI optical scheme automatic optimization method based on reinforcement learning according to claim 7, characterized in that: In step 4.1, initialize the policy network and value network Parameters and ,in, It's a state. It is the action taken, are the weight and bias parameters of the policy network, are the weight and bias parameters of the value network; Set the initial policy network learning rate , value network learning rate , discount factor , Greedy Probability , the total number of steps is T, initialize the experience pool to store experience; The greedy strategy is expressed as: ； At time step When the agent follows the greedy probability Select Action , is the status Next action The state action value function represents the execution of an action in the state The expected cumulative reward obtained by the agent; When the sampling probability is greater than hour, Indicates that the status Next, select The action with the largest value; When the sampling probability is less than or equal to When , the agent randomly selects an action to explore the new strategy space and avoid falling into the local optimum; At each time step t, the policy network outputs the action probability distribution based on the current state and samples the action ; The value network estimates the value function of the current state; The agent performs actions After that, the robot arm makes a move to change the optical parameters, and the environment returns to the next state, obtaining a new wafer image and current position information, and accessing the trajectory matrix record; The new wafer image is input into the no-reference image quality assessment network to obtain DEI, and the environment returns an immediate reward based on the DEI score. , used to measure the degree of improvement in collection effect; Observe the immediate reward and next state , the quadruple Deposit into experience pool.

9. The AOI optical scheme automatic optimization method based on reinforcement learning according to claim 7, characterized in that: In step 4.3, after each round of training, a gradient update is performed on the policy network and value network based on the state sequence, action sequence, and reward sequence collected in this round. The method is as follows: Step 4.3.

1. Randomly and uniformly sample a small batch of experiences from the experience pool as samples, calculate the policy gradient, and combine it with the advantage function to maximize the expected return by directly optimizing the policy network parameters; Step 4.3.2: Update the value network parameters using the gradient descent algorithm; update the policy network parameters using the advantage function using the policy gradient ascent algorithm. Step 4.3.3: Minimize the loss function of the policy network and update the policy network; minimize the loss function of the value network and update the value network. Synchronously update the parameters of the two networks and calculate the cumulative discounted reward.

10. The AOI optical scheme automatic optimization method based on reinforcement learning according to claim 9, characterized in that: The cumulative discount reward is calculated as follows: ； in, Indicates the immediate reward for the current step, is the discount factor; To improve training stability, the reward value is standardized: ； in, , Represents all the The mean and standard deviation of A small constant to prevent division by zero; Update the policy network through the advantage function: ； in, represents the current state value predicted by the value network, Indicates how good the current action is compared to the average strategy; The policy network is updated to: ； in, is the loss function of the policy network, is the policy output by the policy network, The larger it is, the more it strengthens the probability of the current action, and the policy network updates parameter, ; The value network is updated to: ； in, is the loss function of the value network, and the value network updates parameter, ; Construct the total loss: ； The total loss is used to enable the strategy network to continuously adjust its decision direction based on the feedback from the value network.

Images