Deep learning-based defect detection system, object aberration compensation method, and defect detection method

Abstract

This application discloses a defect detection system, an object aberration compensation method, and a defect detection method based on deep learning. When the aberration compensation mode is activated, a pre-trained neural network predicts the target surface profile of an optical modulation device based on an image of an aberration calibration plate on the objective lens surface. Since the target surface profile is predicted using the neural network, and the neural network is trained based on test images acquired from the aberration calibration plate in the neural network training mode and the expected surface profile of the optical modulation device optimized based on the test images, the neural network can predict a surface profile adjustment scheme suitable for optimizing the optical modulation device to compensate for aberrations based on the images acquired in the aberration compensation mode. This application uses a neural network to adjust the surface profile of the optical modulation device, thereby completing the compensation of object aberrations. This real-time compensation operation is intelligent and convenient, effectively preventing the continuous deterioration of object image quality.

Description

Technical Field

[0001] This application relates to the field of defect detection technology, and in particular to a deep learning-based defect detection system, an object image aberration compensation method, and a defect detection method. Background Technology

[0002] In semiconductor manufacturing, defect detection of prepared materials or samples is a crucial step in determining product yield. Bright-field defect inspection equipment can detect various defect types. The power of the light source directly affects the equipment's detection performance; for example, increasing the light source power can improve detection resolution, expand the field of view, and achieve higher throughput. However, increased light source power also makes the lens components in the equipment more prone to heat accumulation. Heat accumulation leads to aberrations, which in turn affects the imaging quality of the equipment or system. More profoundly, it can also interfere with the accuracy of defect detection, increasing the difficulty of inspection.

[0003] After the equipment is assembled and adjusted, aberration compensation is difficult to achieve and requires disassembly and reassembly of the equipment. On the one hand, this disassembly and reassembly interrupts the continuity of the inspection process; on the other hand, it incurs additional time and labor costs. Both of these issues reduce the efficiency of defect detection. Currently, there is a lack of defect detection equipment capable of real-time object aberration compensation during the inspection process. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a deep learning-based defect detection system, an object aberration compensation method, and a defect detection method, aiming to improve the real-time performance of aberration compensation and enhance the execution efficiency and smoothness of defect detection.

[0005] The embodiments of this application disclose the following technical solutions:

[0006] The first aspect of this application provides a defect detection system based on deep learning, which has three functional modes: neural network training mode, aberration compensation mode, and defect detection mode; the system includes: a light source module, a light modulation device, an objective lens, a defect detection module, and an aberration calibration plate;

[0007] When the system activates the aberration compensation mode, the aberration calibration plate is placed on the object plane of the objective lens. The light emitted by the light source module passes sequentially through the optical modulator and the objective lens to reach the aberration calibration plate, and is reflected on the aberration calibration plate. The reflected light passes sequentially through the objective lens and the optical modulator and is collected by the defect detection module. The defect detection module generates an image of the aberration calibration plate based on the collected light. The image is used as input to a neural network to predict the target surface shape that the optical modulator needs to be adjusted to. The neural network is trained based on a training set. The training set includes multiple sets of data pairs. Each set of data pairs includes a test image of the aberration calibration plate acquired by the system in the neural network training mode, and the expected surface shape of the optical modulator optimized based on the test image.

[0008] In an optional implementation, when the system starts the neural network training mode, the aberration calibration plate is positioned on the object plane of the objective lens; the test images in the multiple sets of data pairs are acquired at different times when the system runs the neural network training mode.

[0009] In an optional implementation, the expected surface shape of the optical modulation device in each data pair in the training set is the surface shape optimized for the optical modulation device using an optimization algorithm based on the test image of that data pair.

[0010] In an optional implementation, the optimization objective of the surface shape of the optical modulation device is that the value of the modulation transfer function of the system reaches the target convergence value.

[0011] In an optional implementation, the target in the aberration calibration plate comprises horizontal or vertical lines, and the value of the modulation transfer function is calculated using the knife-edge method; or,

[0012] The target in the aberration calibration plate includes horizontal or vertical lines tilted at 2° to 10°, and the value of the modulation transfer function is calculated using the tilted edge method; or...

[0013] The target in the aberration calibration plate includes black and white grating stripes with precision meeting preset conditions, and the value of the modulation transfer function is calculated by the sinusoidal target method.

[0014] In an optional implementation, the aberration calibration plate includes multiple rectangles of the same size and shape; two parallel sides of each rectangle form an angle of 2° to 10° with the horizontal reference direction of the aberration calibration plate.

[0015] In an optional implementation, the system further includes: a computation control module; both the defect detection module and the optical modulation device are electrically connected to the computation control module;

[0016] When the system activates the aberration compensation mode, the computation control module calls the neural network and uses the image of the aberration calibration plate as the input of the neural network to obtain the target surface shape output by the neural network. Based on the target surface shape, the optical modulation device is controlled to perform surface shape adjustment in order to compensate for the aberration of the objective lens by adjusting the optical modulation device.

[0017] In an optional implementation, the optical modulation device is a deformable mirror; a plurality of piezoelectric ceramics are disposed on the back side of the deformable mirror;

[0018] When the system activates the aberration compensation mode, the computation control module is specifically used to determine the amount of displacement that each piezoelectric ceramic needs to move based on the difference between the current surface shape of the optical modulation device and the target surface shape; and to send an electrical control command to the deformable mirror to drive each piezoelectric ceramic to move according to its respective required displacement, so as to adjust the current surface shape of the deformable mirror to the target surface shape.

[0019] In an optional implementation, the optical modulation device is a deformable mirror; a thermal resistor is disposed on the back side of the deformable mirror;

[0020] When the system activates the aberration compensation mode, the computation control module is specifically used to determine the current or voltage value to be applied to the thermal resistor based on the difference between the current surface shape of the optical modulation device and the target surface shape; send an electrical control command to the deformable mirror to energize the thermal resistor based on the current or voltage value, and use the heat generated after the thermal resistor is energized to adjust the current surface shape of the deformable mirror to the target surface shape.

[0021] In an optional implementation, the system further includes: a beam splitter; the light emitting end of the light source module and the light receiving end of the defect detection module are respectively located on both sides of the beam splitter surface of the beam splitter; the beam splitter is located between the light source module and the optical modulation device, and the beam splitter is located between the optical modulation device and the optical detection module.

[0022] In an optional implementation, after the optical modulator is adjusted to the target surface shape, the system switches to the defect detection mode, the aberration calibration plate moves out of the object surface, and the sample to be tested moves into the object surface; the light emitted by the light source module passes sequentially through the optical modulator after surface shape adjustment and the objective lens to reach the sample to be tested, and is reflected on the sample to be tested; the reflected light passes sequentially through the objective lens and the optical modulator after surface shape adjustment and is collected by the defect detection module; the defect detection module is used to generate the defect detection result of the sample to be tested based on the collected light.

[0023] The second aspect of this application provides a deep learning-based object aberration compensation method, which is applied to the deep learning-based defect detection system described in any implementation of the first aspect; the deep learning-based object aberration compensation method includes:

[0024] Activate the aberration compensation mode of the system and place the aberration calibration plate on the object plane of the objective lens;

[0025] When the light source module is turned on, the light emitted by the light source module passes sequentially through the light modulation device and the objective lens to reach the aberration calibration plate, and is reflected on the aberration calibration plate.

[0026] The defect detection module collects the reflected light that returns from the aberration calibration plate and passes sequentially through the objective lens and the optical modulation device before reaching the defect detection module.

[0027] The image of the aberration calibration plate is generated based on the reflected light collected by the defect detection module;

[0028] The image of the aberration calibration plate is used as the input of a neural network to predict the target surface shape that the current optical modulation device needs to be adjusted to; the surface shape of the current optical modulation device is adjusted to the target surface shape to compensate for the aberration of the objective lens.

[0029] In an optional implementation, the method provided by the second aspect further includes, before activating the aberration compensation mode of the system:

[0030] The neural network training mode of the system is activated, and the aberration calibration plate is placed on the object plane of the objective lens;

[0031] At different times when the system is running the neural network training mode, the aberration calibration plate is illuminated by the light source module, and the light returned by the aberration calibration plate is collected by the defect detection module to generate test images of the aberration calibration plate at different times.

[0032] For each test image, an optimization algorithm is used to optimize the expected surface shape of the optical modulation device, and each test image and the optimized expected surface shape are constructed as a set of data pairs to obtain a training set including multiple sets of data pairs.

[0033] The neural network is obtained by training the training set.

[0034] A third aspect of this application provides a deep learning-based defect detection method, which is applied to the deep learning-based defect detection system described in any implementation of the first aspect; the deep learning-based defect detection method includes:

[0035] Activate the defect detection mode of the system;

[0036] During the operation of the defect detection mode of the system, when it is determined that the objective lens of the system needs aberration compensation, the system is switched to the aberration compensation mode;

[0037] Activate the aberration compensation mode of the system and place the aberration calibration plate on the object plane of the objective lens;

[0038] When the light source module is turned on, the light emitted by the light source module passes sequentially through the light modulation device and the objective lens to reach the aberration calibration plate, and is reflected on the aberration calibration plate.

[0039] The defect detection module collects the reflected light that returns from the aberration calibration plate and passes sequentially through the objective lens and the optical modulation device before reaching the defect detection module.

[0040] The image of the aberration calibration plate is generated based on the reflected light collected by the defect detection module;

[0041] The image of the aberration calibration plate is used as the input of a neural network, which predicts the target surface shape that the current optical modulation device needs to be adjusted to; the surface shape of the current optical modulation device is adjusted to the target surface shape to compensate for the aberration of the objective lens;

[0042] Switch the system to the defect detection mode;

[0043] Control the aberration calibration plate to move out of the object plane of the objective lens and move the sample to be tested into the object plane;

[0044] When the light source module is turned on, the light emitted by the light source module passes sequentially through the light modulation device with the surface shape adjusted and the objective lens to reach the sample to be tested, and is reflected on the sample to be tested.

[0045] The defect detection module collects the reflected light from the sample under test, which then passes sequentially through the objective lens and the light modulation device after surface adjustment before reaching the defect detection module.

[0046] The defect detection module generates the defect detection result of the sample under test based on the collected light.

[0047] In an optional implementation, the method provided by the third aspect further includes, before activating the aberration compensation mode of the system:

[0048] The neural network training mode of the system is activated, and the aberration calibration plate is placed on the object plane of the objective lens;

[0049] At different times when the system is running the neural network training mode, the aberration calibration plate is illuminated by the light source module, and the light returned by the aberration calibration plate is collected by the defect detection module to generate test images of the aberration calibration plate at different times.

[0050] For each test image, an optimization algorithm is used to optimize the expected surface shape of the optical modulation device, and each test image and the optimized expected surface shape are constructed as a set of data pairs to obtain a training set including multiple sets of data pairs.

[0051] The neural network is obtained by training the training set.

[0052] Compared with the prior art, this application has the following beneficial effects:

[0053] This application discloses a deep learning-based defect detection system, an object aberration compensation method, and a defect detection method. When the aberration compensation mode is activated, a pre-trained neural network predicts the target surface shape of an optical modulation device based on an image of an aberration calibration plate on the objective lens surface. Since the target surface shape is predicted using a neural network trained on a training set with multiple data pairs, and each data pair in the training set includes a test image of the aberration calibration plate acquired by the system in the neural network training mode, and the expected surface shape of the optical modulation device optimized based on the test image, the neural network can predict a surface shape adjustment scheme suitable for optimizing the optical modulation device to compensate for aberrations based on the image acquired in the aberration compensation mode. This application uses a neural network to adjust the surface shape of the optical modulation device, thereby completing the compensation of object aberrations. This real-time compensation operation is intelligent and convenient, effectively preventing the continuous decline in object image quality. This application not only improves the real-time performance of aberration compensation and enhances the execution efficiency and smoothness of defect detection, but also ensures the accuracy of defect detection and reduces detection difficulty through timely aberration compensation. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 A schematic diagram of the structure of a deep learning-based defect detection system provided in an embodiment of this application;

[0056] Figure 2 A schematic diagram of an aberration calibration plate provided in an embodiment of this application;

[0057] Figure 3 A schematic diagram of a deep learning-based defect detection system in aberration compensation mode, provided for an embodiment of this application;

[0058] Figure 4 A schematic diagram of a deep learning-based defect detection system in defect detection mode, provided in an embodiment of this application;

[0059] Figure 5 A schematic diagram of another deep learning-based defect detection system provided in an embodiment of this application;

[0060] Figure 6 This is a schematic diagram of the structure of a deformable mirror provided in an embodiment of this application;

[0061] Figure 7 A schematic diagram of the structure of another deep learning-based defect detection system provided in the embodiments of this application;

[0062] Figure 8 A schematic diagram of another deep learning-based defect detection system provided in an embodiment of this application;

[0063] Figure 9 A flowchart illustrating a deep learning-based object image aberration compensation method provided in this application embodiment;

[0064] Figure 10 A flowchart of a deep learning-based defect detection method provided in an embodiment of this application. Detailed Implementation

[0065] Currently, compensation for image aberration in defect detection equipment can generally only be performed during the equipment assembly and adjustment phase. Once assembly and adjustment are complete, image aberration can only be compensated by disassembling and reassembling the equipment. In other words, there is currently a lack of technical solutions that can compensate for image aberration during defect detection after the equipment has been assembled and adjusted. If image aberration cannot be compensated in real time, it may affect the accuracy and smoothness of the equipment's defect detection of materials or samples.

[0066] In view of the above problems, the inventors have proposed a defect detection system, an objective aberration compensation method, and a defect detection method based on deep learning. The defect detection system has a neural network training mode, an aberration compensation mode, and a defect detection mode. These three modes can be freely switched. For example, after the system is in defect detection mode, if aberration compensation of the objective lens is required, simply switch the mode to aberration compensation mode. After the aberration compensation of the objective lens is completed, the mode can be switched back to defect detection mode. In this way, real-time compensation of objective aberration can be achieved during the detection process. The technical solution of this application does not require disassembling and reassembling the system structure; only the object placed on the objective lens surface needs to be moved. When the defect detection system is in neural network training mode or aberration compensation mode, the object placed on the objective lens surface is an aberration calibration plate; while when the defect detection system is in defect detection mode, the object placed on the objective lens surface is the sample to be tested. This system uses a neural network trained during the neural network training phase to combine real-time images from an aberration calibration plate to automatically predict the appropriate surface shape for the optical modulator. By adjusting the surface shape of the optical modulator, the system compensates for object aberrations throughout the entire system.

[0067] See Figure 1 The figure is a schematic diagram of the structure of a deep learning-based defect detection system provided in an embodiment of this application. Figure 1 As shown, the deep learning-based defect detection system includes: a light source module 01, a light modulation device 02, an objective lens 03, a defect detection module 04, and an aberration calibration plate 05. The placement of the aberration calibration plate 05 is related to the mode of the system. Figure 1 The red arrows show the optical path from the light source module 01 to the object on the objective lens 03 surface, and the blue arrows show the optical path from the object on the objective lens 03 surface to the defect detection module 04. It should be noted that in practical applications, the direction of the optical path is not limited to the specific elements with beam-reversing capabilities in the system, depending on their placement and number. Figure 1 The arrow direction is shown. Figure 1 The red and blue arrows in the text are for illustrative purposes only.

[0068] Specifically, the deep learning-based defect detection system has three functional modes: neural network training mode, aberration compensation mode, and defect detection mode. During the operation of neural network training mode and aberration compensation mode, it is necessary to photograph the aberration calibration plate 05. Therefore, when the system is in neural network training mode or aberration compensation mode, the aberration calibration plate 05 is placed on the object surface of the objective lens 03 during the period when it is necessary to photograph the aberration calibration plate 05.

[0069] In this embodiment, the optical modulation device 02 is disposed between the light source module 01 and the objective lens 03, and also between the objective lens 03 and the defect detection module 04. The optical modulation device 02 can compensate for aberrations caused by heat accumulation in the objective lens 03 by adjusting its surface shape. As an example, the optical modulation device 02 can be a deformable mirror, which reflects the light beam in the system's optical path. The back of the deformable mirror is provided with adjustment elements for adjusting its surface shape. These elements can be multiple piezoelectric ceramics or thermal resistors. Piezoelectric ceramics can change the surface shape of the entire deformable mirror by displacement; thermal resistors can change the surface shape of the entire deformable mirror by generating heat through electrical current.

[0070] In this embodiment, the deep learning-based defect detection system generally does not need to reactivate the neural network training mode for a short period after starting it once. Only when the aberration compensation of the objective lens is found to be unsatisfactory does it need to re-enter the neural network training mode. When the deep learning-based defect detection system is in neural network training mode, it mainly uses a training set to train a neural network capable of predicting the expected surface shape that the optical modulation device 02 should transform into based on the test image of the aberration calibration plate 05. In this application, the image acquired by the system in neural network training mode for the aberration calibration plate 05 on the object surface of the objective lens 03 is referred to as the test image. In neural network training mode, the defect detection module 04 in the system collects the light reflected from the aberration calibration plate 05 to form the test image of the aberration calibration plate 05.

[0071] The training set used to train the neural network in this application includes multiple data pairs. Each data pair includes a test image of the aberration calibration plate 05 acquired by a deep learning-based defect detection system in neural network training mode, and the expected surface shape of the optical modulation device 02 optimized based on the test image. For example, optimization algorithms such as genetic annealing are used to optimize the surface shape of the optical modulation device 02.

[0072] In practical applications, the test images in multiple data pairs are acquired at different times during the neural network training mode of the deep learning-based defect detection system. This is because the aberrations of the objective lens vary at different times of system operation due to factors such as thermal aberrations and airflow disturbances. Therefore, the test images of the optical modulation device 02 acquired at different times can exhibit differentiated characteristics, thus enriching the data diversity of test images and expected surface shapes in the multiple data pairs within the training set. For example, a test image acquired at 15:00 on a certain afternoon and a test image acquired at 15:20 on the same afternoon may differ due to the different acquisition times and differences in indoor airflow and heat, potentially resulting in differences in the test images of the same aberration calibration plate 05. Since the expected surface shape of the optical modulation device 02 in each data pair is optimized based on the test images in that data pair, the expected surface shape S1 optimized using the test image acquired at 15:00 on a certain afternoon and the expected surface shape S2 optimized using the test image acquired at 15:20 on the same afternoon will differ.

[0073] The aberration calibration plate 05 includes a target. The purpose of the target is to facilitate subsequent determination of how to optimize the system image quality. Image quality optimization aims to determine the ideal surface shape that the optical modulator 02 should occupy; this ideal surface shape is the expected surface shape mentioned above. The expected surface shape corresponds to the expected system image quality (ideal image quality). In this application, the expected surface shape of the optical modulator 02 is determined through convergent optimization using an optimization algorithm combined with system image quality evaluation metrics.

[0074] The target form of the optical modulator 02 is related to the optimization of its surface shape. In practical applications, the optimization goal of the surface shape of the optical modulator 02 is to achieve a target convergence value for the Modulation Transfer Function (MTF) of the deep learning-based defect detection system. MTF refers to the modulation density as a function of spatial frequency; it is a recognized standard for lens performance measurement, i.e., a quantitative method for comprehensively evaluating the resolution and contrast of a lens. Here, MTF serves as the objective function for optimizing the surface shape of the optical modulator 02, and the target convergence value can be set by the user. For example, setting the target convergence value to MTF=0.5 is considered to indicate that the expected surface shape has converged. The target convergence value can be set or determined according to actual needs (such as requirements for computation time, number of iterations, etc.), and is not limited here. The following describes several requirements for the target form in the aberration calibration plate 05.

[0075] (1) In one possible implementation, the target in the aberration calibration plate 05 comprises horizontal or vertical lines. For this type of target form, the value of the modulation transfer function can be calculated by the edge method when optimizing the surface shape of the optical modulator 02.

[0076] (2) In another possible implementation, the target in the aberration calibration plate 05 includes horizontal or vertical lines tilted at 2° to 10°. For this type of target form, the value of the modulation transfer function is calculated by the slanted edge method when optimizing the surface shape of the optical modulator 02. Figure 2 This is a schematic diagram of an aberration calibration plate provided in an embodiment of this application, as shown below. Figure 2 As shown, the aberration calibration plate consists of multiple rectangles of the same size and shape, i.e., 12 tilted squares serving as targets. Figure 2 In each small rectangle serving as the target, two parallel sides form a 7° angle with the horizontal reference direction of the aberration calibration plate. Of course, Figure 2 The target shape on the aberration calibration plate is for illustrative purposes only. In practical applications, the angle between the two parallel sides of the small rectangle and the horizontal reference direction of the aberration calibration plate should be within the range of 2° to 10°. Here, the horizontal reference direction can refer to... Figure 2 The direction pointing to the right from the middle horizontal. Utilizing Figure 2 The aberration calibration plate shown in the figure indicates that the system can test the MTF in four directions within the same field of view, and can also test the MTF in different fields of view.

[0077] Spatial frequency response (SFR) is an experimental method for calculating MTF (Mean Transmission Frequency) using tilted edges. In applications... Figure 2 When performing MTF testing on the aberration calibration plate shown, the MTF value is calculated using the tilted edge method. Optionally, oversampling technology is used to improve the sampling accuracy of the edge spread function (ESF), thereby calculating the MFT value more accurately.

[0078] (3) In another possible implementation, the target in the aberration calibration plate 05 includes black and white grating stripes with precision that meet preset conditions. For this type of target, the value of the modulation transfer function is calculated by the sinusoidal target method when optimizing the surface shape of the optical modulator 02.

[0079] In practical applications, if the Random Target Method is used to calculate the value of the modulation transfer function, an aberration calibration plate without the above-mentioned special requirements can also be used to capture test images.

[0080] In summary, after initiating neural network training, the deep learning-based defect detection system collects test images of the aberration calibration plate 05 at multiple different times and uses an optimization algorithm to converge the expected surface shape of the optical modulation device 02, thus constructing a training set containing multiple sets of data pairs. Using this training set, a neural network can be trained to predict how the optical modulation device 02 should perform surface shape adjustment based on the input images of the aberration calibration plate 05, i.e., a neural network that predicts the target surface shape that needs to be adjusted.

[0081] Having completed the neural network training, when the deep learning-based defect detection system needs to compensate for the aberrations of the objective lens 03, the system can activate the aberration compensation mode and call the neural network to analyze and process the image, thereby predicting the target surface shape of the optical modulation device 02.

[0082] When the deep learning-based defect detection system activates aberration compensation mode, the aberration calibration plate 05 is placed on the object plane of the objective lens 03. It should be noted that if the previous mode was neural network training mode before activating aberration compensation mode, the aberration calibration plate 05 was already on the object plane of the objective lens 03; therefore, after activating aberration compensation mode, simply keeping the aberration calibration plate 05 on the object plane of the objective lens 03 is sufficient. However, if the previous mode was defect detection mode, the aberration calibration plate 05 is located outside the system's optical path in defect detection mode (i.e., not on the object plane of the objective lens 03). Therefore, after activating aberration compensation mode, the sample originally placed on the object plane needs to be removed, and the aberration calibration plate 05 needs to be moved onto the object plane of the objective lens 03 to facilitate imaging of the aberration calibration plate 05 in aberration compensation mode.

[0083] Figure 3 This is a schematic diagram illustrating a deep learning-based defect detection system in aberration compensation mode, as provided in an embodiment of this application. Figure 3 The blue arrow in the middle indicates the optical path through which the light reflected from the aberration calibration plate 05 finally reaches the defect detection module 04. Figure 4 This is a schematic diagram illustrating a deep learning-based defect detection system in defect detection mode, as provided in an embodiment of this application. Figure 4 The green arrow indicates the optical path of light reflected from the sample to the defect detection module 04. This is compared with... Figure 3 and Figure 4 The objects below objective lens 03 also allow us to observe changes in the objects on the surface of objective lens 03.

[0084] When the system is in aberration compensation mode, the light emitted by the light source module 01 passes sequentially through the optical modulator 02 and the objective lens 03 to reach the aberration calibration plate 05, where it is reflected. The reflected light then passes sequentially through the objective lens 03 and the optical modulator 02 before being collected by the defect detection module 04. The defect detection module 04 generates an image of the aberration calibration plate 05 based on the collected light. This image serves as the input to a pre-trained neural network. By analyzing and processing the input image through this neural network, the target surface shape that the optical modulator 02 needs to be adjusted to can be predicted.

[0085] In one possible implementation scenario, the adjustment of the surface shape of the optical modulator 02 is achieved through the defect detection module 04. For example, the defect detection module 04 can be electrically connected to the optical modulator 02. After the defect detection module 04 calls the neural network to predict the target surface shape of the optical modulator 02, it controls the optical modulator 02 to adjust to the target surface shape.

[0086] Figure 5 This is a schematic diagram of another deep learning-based defect detection system provided in an embodiment of this application. It differs from... Figure 1 The structure shown is in Figure 5 The system structure shown further includes a computational control module 06. The defect detection module 04 and the optical modulation device 02 are both electrically connected to the computational control module 06. Figure 5 In the diagram, electrical connections are shown as dashed lines. (As shown in the image...) Figure 5 In another possible implementation scenario, the adjustment of the surface profile of the optical modulator 02 is achieved through the computational control module 06. When the deep learning-based defect detection system activates the aberration compensation mode, the computational control module 06 calls the neural network and uses the image of the aberration calibration plate 05 as the input to the neural network to obtain the target surface profile output by the neural network. Based on the target surface profile, the optical modulator 02 is controlled to adjust its surface profile in order to compensate for the aberration of the objective lens 03 by adjusting the optical modulator 02.

[0087] Based on different deformation mechanisms, the methods for controlling the deformation of the optical modulation device 02 also differ. In this embodiment, the optical modulation device 02 is a deformable mirror.

[0088] In one implementation, multiple piezoelectric ceramics are disposed on the back of the deformable mirror. Figure 6 This is a schematic diagram of the structure of a deformable mirror provided in an embodiment of this application, as shown below. Figure 6 The deformable mirror shown has nine piezoelectric ceramics evenly distributed on its back surface. Figure 6As indicated by the bidirectional arrows on the right side of the piezoelectric ceramic, the piezoelectric ceramic can move in the direction indicated by the arrows, thereby changing the surface shape of the deformable mirror. Each piezoelectric ceramic on the back of the deformable mirror can be moved independently. When the deep learning-based defect detection system activates the aberration compensation mode, the computational control module 06 is specifically used to determine the displacement required for each piezoelectric ceramic based on the difference between the current surface shape of the deformable mirror and its target surface shape; then, it sends electrical control commands to the deformable mirror to drive each piezoelectric ceramic to move according to its required displacement, thereby adjusting the current surface shape of the deformable mirror to the target surface shape. The electrical control commands sent to the deformable mirror can specifically include the displacement and direction of movement of the piezoelectric ceramic.

[0089] In another implementation, a thermal resistor is provided on the back of the deformable mirror. The thermal resistor can affect the surface shape of the deformable mirror through heat changes. In this embodiment, the number of thermal resistors provided on the back of the deformable mirror can be one or more. If multiple thermal resistors are provided, the adjustment of the surface shape is more flexible and controllable. When the deep learning-based defect detection system starts the aberration compensation mode, the computation control module 06 is specifically used to determine the current or voltage value to be applied to the thermal resistor based on the difference between the current surface shape of the deformable mirror and its target surface shape. An electrical control command is sent to the deformable mirror to energize the thermal resistor based on the current or voltage value. The heat generated after the thermal resistor is energized can adjust the current surface shape of the deformable mirror to the target surface shape. The electrical control command sent to the deformable mirror may specifically include electrical parameters such as the energizing current value or energizing voltage value corresponding to the thermal resistor.

[0090] After the optical modulator is adjusted to the target surface shape, the deep learning-based defect detection system switches to defect detection mode. In this mode, the aberration calibration plate 05 moves out of the object surface of the objective lens 03, and the sample under test moves into the object surface of the objective lens 03. The light emitted by the light source module 01 passes sequentially through the surface-adjusted optical modulator 02 and the objective lens 03 to reach the sample under test, and is reflected on the sample. The reflected light passes sequentially through the objective lens 03 and the surface-adjusted optical modulator 02 before being collected by the defect detection module 04. The defect detection module 04 finally generates the defect detection result of the sample under test on the object surface of the objective lens 03 based on the collected light. Since generating the defect detection result based on the reflected light of the sample under test is a relatively mature technology for defect detection in the semiconductor manufacturing field, it will not be elaborated here.

[0091] The neural network-based defect detection system provided in this application has three flexibly switchable functional modes. Using a neural network training mode, a neural network for object aberration compensation can be trained. In aberration compensation mode, the surface shape of the optical modulation device can be adjusted based on the target surface shape predicted by the neural network, thereby achieving aberration compensation. After aberration compensation, switching to defect detection mode allows for the acquisition of defect detection results based on low-aberration detection images. Since the target surface shape is predicted using a neural network, and the neural network is trained on a training set with multiple data pairs, where each data pair includes a test image acquired by the system from the aberration calibration plate in the neural network training mode, and the expected surface shape of the optical modulation device optimized based on the test image, the neural network can predict a surface shape adjustment scheme suitable for optimizing the optical modulation device to compensate for aberrations based on the image acquired in aberration compensation mode. This real-time compensation operation, which adjusts the surface shape of the optical modulation device based on the neural network to achieve object aberration compensation, is intelligent and convenient, effectively preventing a continuous decline in object image quality. This application not only improves the real-time performance of aberration compensation and enhances the efficiency and smoothness of defect detection, but also ensures the accuracy of defect detection and reduces the difficulty of detection through timely aberration compensation.

[0092] Figure 7 This is a schematic diagram of another deep learning-based defect detection system provided in an embodiment of this application. Figure 7 The system structure shown also includes a beam splitter 07. The light emitting end of the light source module 01 and the light receiving end of the defect detection module 04 are located on opposite sides of the beam splitting surface of the beam splitter 07. Figure 7 As shown, the beam splitter 07 is located between the light source module 01 and the optical modulation device 02, and also between the optical modulation device 02 and the defect detection module 04. Figure 7 In the example, the light emitted by the light source module 01 is transmitted to the light modulator 02 through the beam splitter 07; the light beam transmitted from the light modulator 02 is reflected to the defect detection module 04 when it passes through the beam splitter 07.

[0093] Figure 8 This is a schematic diagram of another deep learning-based defect detection system provided in an embodiment of this application. From... Figure 8 As can be seen, compared to Figure 7 A reflector 08 was added to the system. The reflector is placed between the light source module 01 and the beam splitter 07 to reflect and deflect the light beam emitted by the light source module 01.

[0094] In practical applications, the number and position of functional components such as beam splitters and reflectors in the system are not limited. Therefore, the attached figure is only a structural example of the system. The optical path configuration between the light source module 01 and the optical modulation device 02 is not limited, nor is the optical path configuration between the optical modulation device 02 and the defect detection module 04.

[0095] This application also proposes that if a deformable mirror (or deformable mirror) is used as the optical modulation device 02, and it fails to accurately achieve the target surface shape after adjustment, it may indicate a quality problem with the deformable mirror itself. In practical applications, the success of the deformable mirror in adjusting its surface shape to the target surface shape, i.e., whether the compensation has taken effect as expected, can be determined by analyzing the MTF data before and after compensation (i.e., before and after surface shape adjustment).

[0096] Based on the deep learning-based defect detection system described in the foregoing embodiments, this application also provides a deep learning-based object aberration compensation method. This method can be applied to the system described in the foregoing embodiments.

[0097] Figure 9 Here is a flowchart of the method, such as Figure 9 As shown, object image aberration compensation methods based on deep learning include:

[0098] S901. Activate the aberration compensation mode of the system and place the aberration calibration plate on the object plane of the objective lens.

[0099] S902. Turn on the light source module. The light emitted by the light source module passes through the light modulation device and the objective lens in sequence to reach the aberration calibration plate, and is reflected on the aberration calibration plate.

[0100] S903. The reflected light collected by the defect detection module from the aberration calibration plate and passing through the objective lens and the optical modulation device in sequence before reaching the defect detection module.

[0101] S904. Generate an image of the aberration calibration plate based on the reflected light collected by the defect detection module.

[0102] S905. The image of the aberration calibration plate is used as the input of the neural network, and the neural network predicts the target surface shape that the current optical modulation device needs to be adjusted to.

[0103] S906. Adjust the surface shape of the current optical modulation device to the target surface shape to compensate for the aberration of the objective lens.

[0104] Figure 9The illustrated process primarily describes how the input image for the neural network is obtained after the system activates the aberration compensation mode; that is, the image captured by the system on the aberration calibration board in aberration compensation mode. Using this image and the aforementioned neural network, the target surface shape is ultimately predicted. The system then adjusts the current surface shape of the optical modulation device based on this target surface shape. Since the target surface shape is predicted using a neural network, and the neural network is trained by the system in neural network training mode based on multiple sets of data pairs in the training set, and the expected surface shape in these multiple sets of data pairs is optimized from the test images within those data pairs, indicating that the system can obtain superior image quality with minimal aberration impact under the expected surface shape, adjusting the surface shape of the optical modulation device to the target surface shape during the aberration compensation stage can compensate for the influence of object image aberration. Therefore, after compensation, the accuracy of defect detection in the sample under test can be improved.

[0105] In practical applications, the above-mentioned deep learning-based object aberration compensation method may further include the following steps before activating the aberration compensation mode of the system, i.e., before S901:

[0106] The neural network training mode of the system is activated, and the aberration calibration plate is placed on the object plane of the objective lens;

[0107] At different times when the system is running the neural network training mode, the aberration calibration plate is illuminated by the light source module, and the light returned by the aberration calibration plate is collected by the defect detection module to generate test images of the aberration calibration plate at different times.

[0108] For each test image, an optimization algorithm is used to optimize the expected surface shape of the optical modulation device, and each test image and the optimized expected surface shape are constructed as a set of data pairs to obtain a training set including multiple sets of data pairs.

[0109] The neural network is obtained by training the training set.

[0110] The above steps describe the process of collecting the training set for the neural network and indicate that in the neural network training mode, the system uses the collected training set to train and obtain the neural network that is finally applied to the aberration compensation mode. In this application, in the technical implementation of constructing the training set and optimizing the surface shape based on the test image, the value of the system's modulation transfer function can reach the target convergence value, which can be used as the optimization target for the surface shape of the optical modulation device.

[0111] Based on the systems and methods described in the foregoing embodiments, this application also provides a defect detection method based on deep learning. Figure 10 Here is a flowchart of the method, such as Figure 10 As shown, the deep learning-based defect detection method includes:

[0112] S1001. Start the defect detection mode of the system.

[0113] S1002. During the operation of the defect detection mode of the system, when it is determined that the objective lens of the system needs aberration compensation, the system is switched to the aberration compensation mode.

[0114] S1003. Activate the aberration compensation mode of the system and place the aberration calibration plate on the object plane of the objective lens.

[0115] S1004. Turn on the light source module. The light emitted by the light source module passes through the light modulation device and the objective lens in sequence to reach the aberration calibration plate, and is reflected on the aberration calibration plate.

[0116] S1005. The reflected light collected by the defect detection module from the aberration calibration plate and passing through the objective lens and the optical modulation device in sequence before reaching the defect detection module.

[0117] S1006. Generate an image of the aberration calibration plate based on the reflected light collected by the defect detection module.

[0118] S1007. The image of the aberration calibration plate is used as the input of the neural network, and the neural network is used to predict the target surface shape that the current optical modulation device needs to be adjusted to.

[0119] S1008. Adjust the surface shape of the current optical modulation device to the target surface shape to compensate for the aberration of the objective lens.

[0120] S1009. Switch the system to the defect detection mode.

[0121] S1010: Control the aberration calibration plate to move out of the objective lens and move the sample to be tested into the objective lens.

[0122] S1011. Turn on the light source module. The light emitted by the light source module passes sequentially through the light modulation device after surface adjustment and the objective lens to reach the sample to be tested, and is reflected on the sample to be tested.

[0123] S1012. The reflected light collected by the defect detection module from the sample to be tested, which passes through the objective lens and the light modulation device after surface adjustment, and arrives at the defect detection module.

[0124] S1013. The defect detection module generates the defect detection result of the sample to be tested based on the collected light.

[0125] Figure 10The illustrated process demonstrates how the system switches from defect detection mode to aberration compensation mode and then back to defect detection mode. This process aligns with the typical requirements and mode switching methods for defect detection. Generally, when excessive aberrations are detected, affecting image quality and interfering with defect detection, there is a need to compensate for object aberrations. Once aberration compensation is complete, the system can switch back to defect detection mode to continue detecting defects in the sample. Because aberrations have been compensated, the system exhibits more stable and accurate defect detection performance when switching back to defect detection mode.

[0126] In practical applications, the above-mentioned deep learning-based defect detection method may further include the following steps before activating the aberration compensation mode of the system, i.e., before S1003:

[0127] The neural network training mode of the system is activated, and the aberration calibration plate is placed on the object plane of the objective lens;

[0128] At different times when the system is running the neural network training mode, the aberration calibration plate is illuminated by the light source module, and the light returned by the aberration calibration plate is collected by the defect detection module to generate test images of the aberration calibration plate at different times.

[0129] For each test image, an optimization algorithm is used to optimize the expected surface shape of the optical modulation device, and each test image and the optimized expected surface shape are constructed as a set of data pairs to obtain a training set including multiple sets of data pairs.

[0130] The neural network is obtained by training the training set.

[0131] The above steps describe the process of collecting the training set for the neural network and indicate that in the neural network training mode, the system uses the collected training set to train and obtain the neural network that is finally applied to the aberration compensation mode. In this application, in the technical implementation of constructing the training set and optimizing the surface shape based on the test image, the value of the system's modulation transfer function can reach the target convergence value, which can be used as the optimization target for the surface shape of the optical modulation device.

[0132] In the method embodiments of this application, a pre-trained neural network is used to predict the target surface shape of the optical modulator based on the image of the aberration calibration plate on the objective lens surface. After adjusting the optical modulator to the target surface shape, the defect detection mode is activated, the aberration calibration plate is removed, and the sample to be tested is moved onto the objective lens surface. Light emitted by the light source module gradually passes through the surface-adjusted optical modulator and the objective lens to reach the sample to be tested. The light reflected from the sample to be tested passes through the objective lens and the surface-adjusted optical modulator and is collected by the defect detection module. Finally, the defect detection module generates the defect detection result of the sample to be tested based on the collected light. This method uses a neural network to adjust the surface shape of the optical modulator, thereby completing the compensation of objective lens aberrations. This real-time compensation operation is intelligent and convenient, and can effectively prevent the continuous decline of objective lens quality. This application not only improves the real-time performance of aberration compensation and enhances the execution efficiency and smoothness of defect detection, but also ensures the accuracy of defect detection and reduces the detection difficulty through timely aberration compensation.

[0133] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A defect detection system based on deep learning, characterized in that, The system has three functional modes: neural network training mode, aberration compensation mode, and defect detection mode; the system includes: a light source module, a light modulation device, an objective lens, a defect detection module, and an aberration calibration plate; When the system activates the aberration compensation mode, the aberration calibration plate is placed on the object plane of the objective lens. The light emitted by the light source module passes sequentially through the optical modulator and the objective lens to reach the aberration calibration plate, and is reflected on the aberration calibration plate. The reflected light passes sequentially through the objective lens and the optical modulator and is collected by the defect detection module. The defect detection module generates an image of the aberration calibration plate based on the collected light. The image is used as input to a neural network to predict the target surface shape that the optical modulator needs to be adjusted to. The neural network is trained based on a training set. The training set includes multiple sets of data pairs. Each set of data pairs includes a test image of the aberration calibration plate acquired by the system in the neural network training mode, and the expected surface shape of the optical modulator optimized based on the test image.

2. The system according to claim 1, characterized in that, When the system starts the neural network training mode, the aberration calibration plate is positioned on the object plane of the objective lens; the test images in the multiple sets of data pairs are acquired at different times when the system is running the neural network training mode.

3. The system according to claim 1, characterized in that, The expected surface shape of the optical modulation device in each data pair in the training set is the surface shape optimized for the optical modulation device using an optimization algorithm based on the test image of that data pair.

4. The system according to claim 3, characterized in that, The optimization objective of the surface shape of the optical modulation device is that the value of the modulation transfer function of the system reaches the target convergence value.

5. The system according to claim 4, characterized in that, The target in the aberration calibration plate includes horizontal or vertical lines, and the value of the modulation transfer function is calculated using the knife-edge method; or... The target in the aberration calibration plate includes horizontal or vertical lines tilted at 2° to 10°, and the value of the modulation transfer function is calculated using the tilted edge method; or... The target in the aberration calibration plate includes black and white grating stripes with precision meeting preset conditions, and the value of the modulation transfer function is calculated by the sinusoidal target method.

6. The system according to claim 5, characterized in that, The aberration calibration plate comprises multiple rectangles of the same size and shape; two parallel sides of each rectangle form an angle of 2° to 10° with the horizontal reference direction of the aberration calibration plate.

7. The system according to claim 1, characterized in that, The system further includes: a computation control module; the defect detection module and the optical modulation device are both electrically connected to the computation control module; When the system activates the aberration compensation mode, the computation control module calls the neural network and uses the image of the aberration calibration plate as the input of the neural network to obtain the target surface shape output by the neural network. Based on the target surface shape, the optical modulation device is controlled to perform surface shape adjustment in order to compensate for the aberration of the objective lens by adjusting the optical modulation device.

8. The system according to claim 7, characterized in that, The optical modulation device is a deformable mirror; multiple piezoelectric ceramics are disposed on the back of the deformable mirror; When the system activates the aberration compensation mode, the computation control module is specifically used to determine the amount of displacement that each piezoelectric ceramic needs to move based on the difference between the current surface shape of the optical modulation device and the target surface shape; and to send an electrical control command to the deformable mirror to drive each piezoelectric ceramic to move according to its respective required displacement, so as to adjust the current surface shape of the deformable mirror to the target surface shape.

9. The system according to claim 7, characterized in that, The optical modulation device is a deformable mirror; a thermal resistor is provided on the back of the deformable mirror; When the system activates the aberration compensation mode, the computation control module is specifically used to determine the current or voltage value that needs to be applied to the thermal resistor based on the difference between the surface shape of the current optical modulation device and the target surface shape. An electrical control command is sent to the deformable mirror to energize the thermal resistor based on the current value or the voltage value, and the heat generated after the thermal resistor is energized is used to adjust the current surface shape of the deformable mirror to the target surface shape.

10. The system according to claim 1, characterized in that, The system further includes: a beam splitter; the light emitting end of the light source module and the light receiving end of the defect detection module are respectively located on both sides of the beam splitter surface of the beam splitter; the beam splitter is located between the light source module and the optical modulation device, and the beam splitter is located between the optical modulation device and the optical detection module.

11. The system according to any one of claims 1-10, characterized in that, After the optical modulator is adjusted to the target surface shape, the system switches to the defect detection mode. The aberration calibration plate is moved out of the object surface, and the sample to be tested is moved into the object surface. The light emitted by the light source module passes sequentially through the optical modulator after surface shape adjustment and the objective lens to reach the sample to be tested, and is reflected on the sample. The reflected light passes sequentially through the objective lens and the optical modulator after surface shape adjustment and is collected by the defect detection module. The defect detection module is used to generate defect detection results for the sample under test based on the collected light.

12. A deep learning-based object image aberration compensation method, characterized in that, The method is applied to the deep learning-based defect detection system according to any one of claims 1-11; the method includes: Activate the aberration compensation mode of the system and place the aberration calibration plate on the object plane of the objective lens; When the light source module is turned on, the light emitted by the light source module passes sequentially through the light modulation device and the objective lens to reach the aberration calibration plate, and is reflected on the aberration calibration plate. The defect detection module collects the reflected light that returns from the aberration calibration plate and passes sequentially through the objective lens and the optical modulation device before reaching the defect detection module. The image of the aberration calibration plate is generated based on the reflected light collected by the defect detection module; The image of the aberration calibration plate is used as the input of a neural network, which predicts the target surface shape that the current optical modulation device needs to be adjusted to. The surface profile of the current optical modulation device is adjusted to the target surface profile to compensate for the aberrations of the objective lens.

13. The method according to claim 12, characterized in that, Before activating the aberration compensation mode of the system, the method further includes: The neural network training mode of the system is activated, and the aberration calibration plate is placed on the object plane of the objective lens; At different times when the system is running the neural network training mode, the aberration calibration plate is illuminated by the light source module, and the light returned by the aberration calibration plate is collected by the defect detection module to generate test images of the aberration calibration plate at different times. For each test image, an optimization algorithm is used to optimize the expected surface shape of the optical modulation device, and each test image and the optimized expected surface shape are constructed as a set of data pairs to obtain a training set including multiple sets of data pairs. The neural network is obtained by training the training set.

14. A defect detection method based on deep learning, characterized in that, The method is applied to the deep learning-based defect detection system according to any one of claims 1-11; the method includes: Activate the defect detection mode of the system; During the operation of the defect detection mode of the system, when it is determined that the objective lens of the system needs aberration compensation, the system is switched to the aberration compensation mode; Activate the aberration compensation mode of the system and place the aberration calibration plate on the object plane of the objective lens; When the light source module is turned on, the light emitted by the light source module passes sequentially through the light modulation device and the objective lens to reach the aberration calibration plate, and is reflected on the aberration calibration plate. The defect detection module collects the reflected light that returns from the aberration calibration plate and passes sequentially through the objective lens and the optical modulation device before reaching the defect detection module. The image of the aberration calibration plate is generated based on the reflected light collected by the defect detection module; The image of the aberration calibration plate is used as the input of a neural network, which predicts the target surface shape that the current optical modulation device needs to be adjusted to. The surface shape of the current optical modulation device is adjusted to the target surface shape to compensate for the aberration of the objective lens; Switch the system to the defect detection mode; Control the aberration calibration plate to move out of the object plane of the objective lens and move the sample to be tested into the object plane; When the light source module is turned on, the light emitted by the light source module passes sequentially through the light modulation device with the surface shape adjusted and the objective lens to reach the sample to be tested, and is reflected on the sample to be tested. The defect detection module collects the reflected light from the sample under test, which then passes sequentially through the objective lens and the light modulation device after surface adjustment before reaching the defect detection module. The defect detection module generates the defect detection result of the sample under test based on the collected light.

15. The method according to claim 14, characterized in that, Before activating the aberration compensation mode of the system, the method further includes: The neural network training mode of the system is activated, and the aberration calibration plate is placed on the object plane of the objective lens; At different times when the system is running the neural network training mode, the aberration calibration plate is illuminated by the light source module, and the light returned by the aberration calibration plate is collected by the defect detection module to generate test images of the aberration calibration plate at different times. For each test image, an optimization algorithm is used to optimize the expected surface shape of the optical modulation device, and each test image and the optimized expected surface shape are constructed as a set of data pairs to obtain a training set including multiple sets of data pairs. The neural network is obtained by training the training set.

Images