Defect detection system based on deep learning, objective lens image difference compensation method and defect detection method

Abstract

The invention discloses a defect detection system based on deep learning, an objective lens image difference compensation method and a defect detection method. If the aberration compensation mode is started, a pre-trained neural network is used to predict the target surface type of the light modulator based on the image of the aberration calibration plate on the object plane of the objective lens. The target surface type is predicted by using the neural network, and the neural network is trained on the basis of the test image acquired by the aberration calibration plate and the expected surface type of the light modulator optimized on the basis of the test image in the neural network training mode. The neural network can predict a surface type adjustment scheme suitable for optimizing the light modulator to compensate for aberration based on the image acquired in the aberration compensation mode. According to the invention, the surface type adjustment of the light modulation device is realized based on the neural network, the compensation of the objective lens image difference is completed, the compensation operation which can be carried out in real time is intelligent and convenient, and the continuous reduction of the objective lens image quality can be effectively avoided.

Description

Technical Field

[0001] The present application relates to the field of defect detection technology, and in particular to a defect detection system based on deep learning, an objective aberration compensation method, and a defect detection method. Background Art

[0002] Defect detection of prepared materials or samples during the semiconductor manufacturing process is a critical step in determining product yield. Brightfield defect inspection equipment is capable of detecting a wide range of defect types. Light source power directly impacts the equipment's inspection performance. For example, increasing light source power can improve inspection resolution, expand the field of view, and achieve higher throughput. However, increasing light source power also makes the equipment's lens components more susceptible to heat accumulation. Heat accumulation can cause image aberrations, which in turn affect the imaging quality of the equipment or system. Furthermore, it can interfere with defect detection accuracy, making detection more difficult.

[0003] After equipment is assembled and adjusted, compensating for aberrations is difficult, requiring complete disassembly and reassembly. This interrupts the continuity of the inspection process; it also consumes additional time and labor. Both of these issues reduce defect detection efficiency. Currently, there is a lack of defect detection equipment capable of performing real-time objective aberration compensation during the inspection process. Summary of the Invention

[0004] Based on the above problems, the present application provides a defect detection system, an objective aberration compensation method and a defect detection method based on deep learning, with the aim of improving the real-time performance of aberration compensation and enhancing the execution efficiency and smoothness of defect detection.

[0005] The embodiments of this application disclose the following technical solutions: In a first aspect, the present application provides a deep learning-based defect detection system having three functional modes: a neural network training mode, an aberration compensation mode, and a defect detection mode; the system comprises: a light source module, a light modulation device, an objective lens, a defect detection module, and an aberration calibration plate; When the system starts the aberration compensation mode, the aberration calibration plate is placed on the object plane of the objective lens, and the light emitted by the light source module passes through the light modulator and the objective lens in sequence to reach the aberration calibration plate, and is reflected on the aberration calibration plate. The reflected light passes through the objective lens and the light modulator in sequence and is collected by the defect detection module; the defect detection module is used to generate an image of the aberration calibration plate based on the collected light; the image is used as input to the neural network to predict the target surface shape to which the current light modulator needs to be adjusted through the neural network; the neural network is trained based on a training set; the training set includes multiple groups of data pairs, each group of data pairs includes a test image of the aberration calibration plate collected by the system in the neural network training mode, and an expected surface shape of the light modulator optimized based on the test image.

[0006] In an optional implementation, when the system starts the neural network training mode, the aberration calibration plate is on the object plane of the objective lens; the test images in the multiple sets of data pairs are collected at different times when the system runs the neural network training mode.

[0007] In an optional implementation, the expected surface shape of the light modulator in each data pair in the training set is a surface shape optimized for the light modulator using an optimization algorithm based on the test image of the data pair.

[0008] In an optional implementation, the optimization goal of the surface shape of the optical modulation device is that the value of the modulation transfer function of the system reaches a target convergence value.

[0009] In an optional implementation, the target in the aberration calibration plate includes horizontal or vertical lines, and the value of the modulation transfer function is calculated by a knife-edge method; or, The target in the aberration calibration plate includes horizontal or vertical lines inclined at 2° to 10°, and the value of the modulation transfer function is calculated by the slanted edge method; or The target in the aberration calibration plate includes black and white grating stripes whose precision meets preset conditions, and the value of the modulation transfer function is calculated by a sine target method.

[0010] In an optional implementation, the aberration calibration plate includes a plurality of rectangles of the same size and shape; two parallel sides of the rectangles form an angle of 2° to 10° with the horizontal reference direction of the aberration calibration plate.

[0011] In an optional implementation, the system further includes: an operation control module; the defect detection module and the optical modulation device are both electrically connected to the operation control module; When the system starts the aberration compensation mode, the operation control module is used to call the neural network and use the image of the aberration calibration plate as the input of the neural network to obtain the target surface shape output by the neural network, and control the light modulator to adjust the surface shape based on the target surface shape, so as to compensate for the aberration of the objective lens by adjusting the light modulator.

[0012] In an optional implementation, the light modulation device is a deformable mirror; a plurality of piezoelectric ceramics are provided on the back of the deformable mirror; When the system starts the aberration compensation mode, the operation control module is specifically used to determine the displacement amount that each piezoelectric ceramic needs to move based on the difference between the current surface shape of the optical modulator and the target surface shape; and send an electrical control instruction to the deformable mirror to drive each piezoelectric ceramic to move according to the displacement amount that each needs to move, so as to adjust the current surface shape of the deformable mirror to the target surface shape.

[0013] In an optional implementation, the light modulation device is a deformable mirror; a thermal resistor is provided on the back of the deformable mirror; When the system starts the aberration compensation mode, the operation control module is specifically used to determine the current value or voltage value that needs to be applied to the thermistor based on the difference between the current surface shape of the optical modulator and the target surface shape; send an electrical control instruction to the deformable mirror to energize the thermistor based on the current value or the voltage value, and use the heat generated by the thermistor after energization to adjust the current surface shape of the deformable mirror to the target surface shape.

[0014] In an optional implementation, the system also includes: a spectroscopic element; the light emitting end of the light source module and the light receiving end of the defect detection module are respectively located on both sides of the spectroscopic surface of the spectroscopic element; the spectroscopic element is located between the light source module and the optical path of the optical modulator, and the spectroscopic element is located between the optical path of the optical modulator and the defect detection module.

[0015] In an optional implementation, after the optical modulator is adjusted to the target surface shape, the system switches to the defect detection mode, the aberration calibration plate is moved out of the object plane, and the sample to be tested is moved into the object plane; the light emitted by the light source module passes through the optical modulator with the surface shape adjusted and the objective lens in turn to reach the sample to be tested, and is reflected on the sample to be tested, and the reflected light passes through the objective lens and the optical modulator with the surface shape adjusted in turn and is collected by the defect detection module; the defect detection module is used to generate a defect detection result of the sample to be tested based on the collected light.

[0016] A second aspect of the present application provides a method for compensating an objective aberration based on deep learning, which is applied to the deep learning-based defect detection system described in any implementation of the first aspect; the method for compensating an objective aberration based on deep learning includes: Starting the aberration compensation mode of the system and placing the aberration calibration plate on the object plane of the objective lens; Turning on the light source module, so that the light emitted by the light source module passes through the light modulation device and the objective lens in sequence and reaches the aberration calibration plate, and is reflected on the aberration calibration plate; collecting, by the defect detection module, reflected light that returns from the aberration calibration plate and sequentially passes through the objective lens and the light modulation device before reaching the defect detection module; generating an image of the aberration calibration plate based on the reflected light collected by the defect detection module; The image of the aberration calibration plate is used as the input of a neural network, and the neural network predicts the target surface shape to which the current optical modulator needs to be adjusted; the surface shape of the current optical modulator is adjusted to the target surface shape to compensate for the aberration of the objective lens.

[0017] In an optional implementation, the method provided in the second aspect further includes, before starting the aberration compensation mode of the system: Starting a neural network training mode of the system, and placing the aberration calibration plate on the object plane of the objective lens; At different moments when the system runs the neural network training mode, the light source module illuminates the aberration calibration plate, and the defect detection module collects light returned by the aberration calibration plate to generate test images corresponding to the aberration calibration plate at the different moments; For each test image, an optimization algorithm is used to optimize an expected surface shape for the light modulator, and each test image and the optimized expected surface shape are constructed into a set of data pairs, thereby obtaining a training set including multiple sets of data pairs; The neural network is obtained by training using the training set.

[0018] A third aspect of the present application provides a deep learning-based defect detection method, which is applied to the deep learning-based defect detection system described in any implementation of the first aspect. The deep learning-based defect detection method includes: activating the defect detection mode of the system; During the period when the system is operating in the defect detection mode, when it is determined that the objective lens of the system needs to be compensated for aberration, switching the system to the aberration compensation mode; Starting the aberration compensation mode of the system and placing the aberration calibration plate on the object plane of the objective lens; Turning on the light source module, so that the light emitted by the light source module passes through the light modulation device and the objective lens in sequence and reaches the aberration calibration plate, and is reflected on the aberration calibration plate; collecting, by the defect detection module, reflected light that returns from the aberration calibration plate and sequentially passes through the objective lens and the light modulation device before reaching the defect detection module; generating an image of the aberration calibration plate based on the reflected light collected by the defect detection module; Using the image of the aberration calibration plate as an input to a neural network, the neural network predicts a target surface shape to which the current optical modulator needs to be adjusted; adjusting the current surface shape of the optical modulator to the target surface shape to compensate for the aberration of the objective lens; switching the system to the defect detection mode; Controlling the aberration calibration plate to move out from the object plane of the objective lens, and moving the sample to be measured into the object plane; Turning on the light source module, the light emitted by the light source module sequentially passes through the light modulation device with adjusted surface shape and the objective lens to reach the sample to be tested, and is reflected on the sample to be tested; The defect detection module collects reflected light that is returned from the sample to be tested and reaches the defect detection module after sequentially passing through the objective lens and the light modulation device after the surface shape is adjusted; The defect detection module generates a defect detection result of the sample to be tested based on the collected light.

[0019] In an optional implementation, the method provided in the third aspect further includes, before starting the aberration compensation mode of the system: Starting a neural network training mode of the system, and placing the aberration calibration plate on the object plane of the objective lens; At different moments when the system runs the neural network training mode, the light source module illuminates the aberration calibration plate, and the defect detection module collects light returned by the aberration calibration plate to generate test images corresponding to the aberration calibration plate at the different moments; For each test image, an optimization algorithm is used to optimize an expected surface shape for the light modulator, and each test image and the optimized expected surface shape are constructed into a set of data pairs, thereby obtaining a training set including multiple sets of data pairs; The neural network is obtained by training using the training set.

[0020] Compared with the existing technology, this application has the following beneficial effects: The present application discloses a defect detection system, an objective aberration compensation method and a defect detection method based on deep learning. If the aberration compensation mode is started, a pre-trained neural network is used to predict the target surface shape of the light modulator based on the image of the aberration calibration plate on the object surface of the objective lens. Since the target surface shape is predicted by using a neural network, and the neural network is trained based on a training set with multiple data pairs, and each data pair in the training set includes a test image of the aberration calibration plate collected by the system under the neural network training mode, and the expected surface shape of the light modulator optimized based on the test image, the neural network can predict a surface shape adjustment scheme suitable for optimizing the light modulator to compensate for the aberration based on the image collected in the aberration compensation mode. The present application realizes the surface shape adjustment of the light modulator based on the neural network, and then completes the compensation of the objective aberration. This real-time compensation operation is intelligent and convenient, and can effectively avoid the continuous decline of the objective image quality. The present application not only improves the real-time performance of aberration compensation and improves the execution efficiency and fluency of defect detection, but also ensures the accuracy of defect detection and reduces the difficulty of detection through timely compensation of aberrations. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1 A schematic diagram of the structure of a deep learning-based defect detection system provided in an embodiment of the present application; Figure 2 A schematic diagram of an aberration calibration plate provided in an embodiment of the present application; Figure 3 A schematic diagram of a deep learning-based defect detection system in an aberration compensation mode provided in an embodiment of the present application; Figure 4 A schematic diagram of a deep learning-based defect detection system in defect detection mode provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of another deep learning-based defect detection system provided in an embodiment of the present application; Figure 6 A schematic structural diagram of a deformable mirror provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of another deep learning-based defect detection system provided in an embodiment of the present application; Figure 8A schematic diagram of the structure of another deep learning-based defect detection system provided in an embodiment of the present application; Figure 9 A flowchart of a method for compensating objective aberrations based on deep learning provided in an embodiment of the present application; Figure 10 A flowchart of a deep learning-based defect detection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] Currently, compensation for objective aberrations in defect detection equipment can generally only be performed during the equipment's setup. Once setup is complete, compensation can only be performed by disassembling and reassembling the equipment. In other words, there is currently a lack of technical solutions that can compensate for objective aberrations after defect detection equipment is fully setup and during defect detection. Failure to compensate for objective aberrations in real time could affect the accuracy and smoothness of the equipment's defect detection of materials or samples.

[0024] In view of the above problems, the inventors have proposed a defect detection system, an objective aberration compensation method and a defect detection method based on deep learning after research. The defect detection system has a neural network training mode, an aberration compensation mode and a defect detection mode. The three modes can be switched freely. For example, after the system enables the defect detection mode, if it is necessary to compensate for the aberration of the objective lens in the system, it is only necessary to switch the mode to the aberration compensation mode. After the aberration compensation of the objective lens is completed, the mode can be switched to the defect detection mode again. In this way, real-time compensation for the objective aberration in the detection link can be achieved. The technical solution of the present application does not require the structure of the system to be disassembled and reassembled, and only the object placed on the object surface of the objective lens needs to be moved. When the defect detection system is in the neural network training mode or the aberration compensation mode, the object placed on the object surface of the objective lens is the aberration calibration plate; and when the defect detection system is in the defect detection mode, the object placed on the object surface of the objective lens is the sample to be tested. The system uses a neural network trained in the neural network training phase to combine with the real-time image of the aberration calibration plate to automatically predict the surface shape to which the light modulator should be adjusted. By adjusting the surface shape of the light modulator, the objective lens aberration in the entire system can be compensated.

[0025] See also Figure 1 , which is a structural diagram of a defect detection system based on deep learning provided by an embodiment of the present application. Figure 1 As shown, the deep learning-based defect detection system includes: a light source module 01, a light modulation device 02, an objective lens 03, a defect detection module 04 and an aberration calibration plate 05. The placement position of the aberration calibration plate 05 is related to the mode of the system. Figure 1The red arrow in the middle shows the optical path from the light source module 01 to the object on the object surface of the objective lens 03, and the blue arrow shows the optical path from the object on the object surface of the objective lens 03 to the defect detection module 04. It should be noted that in actual applications, the direction of the optical path is not limited to the following: Figure 1 Direction of arrow shown in . Figure 1 The red and blue arrows are for illustration purposes only.

[0026] Specifically, the deep learning-based defect detection system has three functional modes: neural network training mode, aberration compensation mode, and defect detection mode. During the operation of the neural network training mode and the aberration compensation mode, the aberration calibration plate 05 needs to be photographed. Therefore, when the system is in the neural network training mode or the aberration compensation mode, the aberration calibration plate 05 needs to be placed on the object plane of the objective lens 03 during the period when the aberration calibration plate 05 needs to be photographed.

[0027] In an embodiment of the present application, the light modulator 02 is arranged between the light path of the light source module 01 and the objective lens 03, and the light modulator 02 is arranged between the objective lens 03 and the optical path of the defect detection module 04. The light modulator 02 can compensate for the aberration of the objective lens 03 caused by heat accumulation by adjusting the surface shape. As an example, the light modulator 02 can be a deformable mirror, and the deformable mirror in the system optical path plays the role of reflecting the light beam. An adjustment element for adjusting the surface shape is provided on the back of the deformable mirror. The adjustment element can be a plurality of piezoelectric ceramics or a thermistor. Piezoelectric ceramics can change the surface shape of the entire deformable mirror by displacement; the thermistor can change the surface shape of the entire deformable mirror by generating heat through electricity.

[0028] In an embodiment of the present application, the defect detection system based on deep learning generally does not need to repeatedly enable the neural network training mode in a short period of time after starting the neural network training mode once. Only when it is found that the compensation effect of the objective lens is still not ideal after the aberration compensation, it is necessary to repeatedly enter the neural network training mode again. When the defect detection system based on deep learning is in the neural network training mode, the training set is mainly used to train a neural network that can predict the expected surface shape that the light modulator 02 should be transformed into based on the test image of the aberration calibration plate 05. In this application, the image collected from the aberration calibration plate 05 on the object plane of the objective lens 03 when the system is in the neural network training mode is referred to as a test image. Under the neural network training mode, the defect detection module 04 in the system forms a test image of the aberration calibration plate 05 by collecting light reflected from the aberration calibration plate 05.

[0029] The training set used to train the neural network in this application includes multiple data pairs, each of which includes a test image of the aberration calibration plate 05 captured by the deep learning-based defect detection system in neural network training mode, and the expected surface shape of the light modulator 02 optimized based on the test image. For example, an optimization algorithm such as genetic annealing is used to optimize the surface shape of the light modulator 02.

[0030] In actual applications, the test images in multiple data pairs are acquired at different times while the deep learning-based defect detection system is running in neural network training mode. This is because the aberrations of the objective lens vary at different times during system operation due to factors such as thermal aberration and airflow disturbances. Therefore, the test images of the light modulator 02 acquired at different times can exhibit different performances, thereby enriching the data diversity of the test images and expected surface shapes in the multiple data pairs in the training set. For example, a test image acquired at 3:00 PM on a certain day and a test image acquired at 3:20 PM on the same day may differ from the test images of the same aberration calibration plate 05 due to differences in acquisition time, indoor airflow, and heat. Since the expected surface shape of the light modulator 02 in each data pair is optimized based on the test images in that data pair, the expected surface shape S1 optimized using the test image acquired at 3:00 PM on a certain day and the expected surface shape S2 optimized using the test image acquired at 3:20 PM on the same day will differ.

[0031] The aberration calibration plate 05 contains a target. The target's purpose is to facilitate subsequent determination of how the system image quality should be optimized. This optimization determines the ideal surface shape of the light modulator 02. This ideal surface shape is the expected surface shape mentioned above. The expected surface shape corresponds to the expected system image quality (ideal image quality). In this application, the expected surface shape of the light modulator 02 is determined through an optimization algorithm combined with convergence optimization of system image quality evaluation indicators.

[0032] The target form of the optical modulator 02 is associated with the optimization of its surface shape. In practical applications, the surface shape optimization goal of the optical modulator 02 is to achieve a target convergence value for the modulation transfer function (MTF) of the deep learning-based defect detection system. MTF refers to the function of how modulation varies with spatial frequency. It is a recognized lens performance metric, serving as a quantitative method for comprehensively evaluating a lens's resolution and contrast. Here, MTF serves as the target function for optimizing the surface shape of the optical modulator 02, and the target convergence value can be customized. For example, when the target convergence value is set to MTF = 0.5, the expected surface shape is considered to have converged. The target convergence value can be set or determined based on actual requirements (such as computation time, number of iterations, and other requirements) and is not limited here. The following describes several target form requirements for the aberration calibration plate 05.

[0033] (1) In one possible implementation, the target in the aberration calibration plate 05 includes horizontal or vertical lines. For this type of target, when optimizing the surface shape of the optical modulator 02, the value of the modulation transfer function can be calculated using the edge method.

[0034] (2) In another possible implementation, the target in the aberration calibration plate 05 includes horizontal or vertical lines inclined at 2° to 10°. For this type of target, when optimizing the surface shape of the optical modulator 02, the modulation transfer function value is calculated using the slanted edge method. Figure 2 A schematic diagram of an aberration calibration plate provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the aberration calibration plate includes a plurality of rectangles of the same size and shape, that is, 12 inclined squares as targets. Figure 2 In each small rectangle used as a target, two parallel sides form an angle of 7° with the horizontal reference direction of the aberration calibration plate. Figure 2 The target shape on the aberration calibration plate in the figure is only used as an example. In actual application, the angle between the two parallel sides of the small rectangle and the horizontal reference direction of the aberration calibration plate can be within the range of 2°~10°. Here, the horizontal reference direction can refer to Figure 2 The direction of the horizontal line pointing to the right. Figure 2 The aberration calibration plate shown in the figure can test the MTF in four directions within the same field of view. In addition, it can also test the MTF of different fields of view.

[0035] Spatial frequency response (SFR) is an experimental method to calculate MTF by using slanted edges. Figure 2When performing MTF testing on the aberration calibration plate shown, the MTF value is calculated using the slanted edge method. Optionally, oversampling technology is used to improve the sampling accuracy of the edge spread function (ESF), thereby more accurately calculating the MFT value.

[0036] (3) In another possible implementation, the target in the aberration calibration plate 05 includes black and white grating stripes whose precision meets preset conditions. For this type of target form, when optimizing the surface shape of the optical modulator 02, the value of the modulation transfer function is calculated using the sinusoidal target method.

[0037] In practical applications, if the random target method is used to calculate the value of the modulation transfer function, an aberration calibration plate without the above-mentioned special target requirements can also be used to capture and form a test image.

[0038] In general, after initiating the neural network training mode, the deep learning-based defect detection system collects test images of the aberration calibration plate 05 at multiple different times and applies an optimization algorithm to converge on the expected surface shape of the light modulator 02, thereby constructing a training set containing multiple data pairs. Using this training set, a neural network can be trained to predict how the light modulator 02 should perform surface shape adjustments based on the input image of the aberration calibration plate 05, that is, to predict the target surface shape that needs to be adjusted.

[0039] On the basis of the completed neural network training, when the deep learning-based defect detection system needs to compensate for the aberration of the objective lens 03, the system can start the aberration compensation mode and then call the neural network to analyze and calculate the image, and then predict the target surface shape of the light modulator 02.

[0040] When the deep learning-based defect detection system starts the aberration compensation mode, the aberration calibration plate 05 is placed on the object plane of the objective lens 03. It should be noted that if, before the system starts the aberration compensation mode, the previous mode is the neural network training mode, then since the aberration calibration plate 05 is already on the object plane of the objective lens 03, after starting the aberration compensation mode, it is sufficient to keep the aberration calibration plate 05 on the object plane of the objective lens 03. If, before the system starts the aberration compensation mode, the previous mode is the defect detection mode, then since the aberration calibration plate 05 is located outside the system optical path in the defect detection mode, that is, the aberration calibration plate 05 is not on the object plane of the objective lens 03, then after starting the aberration compensation mode, it is necessary to remove the sample to be tested originally placed on the object plane and move the aberration calibration plate 05 into the object plane of the objective lens 03, so as to image the aberration calibration plate 05 in the aberration compensation mode.

[0041] Figure 3 This is a schematic diagram of a deep learning-based defect detection system in an aberration compensation mode provided by an embodiment of the present application. Figure 3 The blue arrow in the middle indicates the optical path of the light reflected from the aberration calibration plate 05 and finally reaching the defect detection module 04 . Figure 4 A schematic diagram of a deep learning-based defect detection system in defect detection mode provided in an embodiment of the present application is provided. Figure 4 The green arrow in the middle indicates the optical path of the light reflected from the sample to be tested and finally reaches the defect detection module 04. Figure 3 and Figure 4 The objects below the objective lens 03 can also be used to understand the changes in the objects on the object surface of the objective lens 03.

[0042] When the system is in aberration compensation mode, light emitted by light source module 01 passes through optical modulator 02 and objective lens 03, reaches aberration calibration plate 05, and is reflected from the aberration calibration plate 05. The reflected light then passes through objective lens 03 and optical modulator 02, and is collected by defect detection module 04. Defect detection module 04 generates an image of aberration calibration plate 05 based on the collected light. This image serves as input to a trained neural network. The neural network analyzes and processes the input image to predict the target surface shape to which optical modulator 02 needs to be adjusted.

[0043] In one possible implementation scenario, the surface shape of the optical modulator 02 is adjusted by the defect detection module 04. For example, the defect detection module 04 can be electrically connected to the optical modulator 02. After the defect detection module 04 uses a neural network to predict the target surface shape of the optical modulator 02, it controls the optical modulator 02 to adjust to the target surface shape.

[0044] Figure 5 This is a schematic diagram of the structure of another defect detection system based on deep learning provided in the embodiment of the present application. Figure 1 The structure shown in Figure 5 The system structure shown further includes an operation control module 06. The defect detection module 04 and the optical modulation device 02 are both electrically connected to the operation control module 06. Figure 5 In FIG, electrical connections are shown with dashed lines. Figure 5 In another possible implementation scenario, the adjustment of the surface shape of the light modulator 02 is achieved through the calculation control module 06. When the deep learning-based defect detection system starts the aberration compensation mode, the calculation control module 06 is used to call the neural network and use the image of the aberration calibration plate 05 as the input of the neural network to obtain the target surface shape output by the neural network. Based on the target surface shape, the light modulator 02 is controlled to perform surface shape adjustment to compensate for the aberration of the objective lens 03 by adjusting the light modulator 02.

[0045] Based on different deformation mechanisms, the ways of controlling the deformation of the light modulator 02 are also different. In the embodiment of the present application, the light modulator 02 is a deformable mirror.

[0046] In one implementation, a plurality of piezoelectric ceramics are provided on the back side of the deformable mirror. Figure 6 A structural diagram of a deformable mirror provided in an embodiment of the present application is shown in FIG. Figure 6 The back of the deformable mirror shown is provided with 9 piezoelectric ceramics, which are evenly distributed on the back of the deformable mirror. Figure 6 As shown by the double-headed arrow to the right of the piezoelectric ceramic, the piezoelectric ceramic can move in the direction indicated by the arrow, thereby changing the surface shape of the deformable mirror. Each piezoelectric ceramic on the back of the deformable mirror can be independently controlled to achieve movement. When the deep learning-based defect detection system starts the aberration compensation mode, the operation control module 06 is specifically used to determine the displacement required for each piezoelectric ceramic based on the difference between the current deformable mirror surface shape and its target surface shape; then, an electrical control instruction is sent to the deformable mirror to drive each piezoelectric ceramic to move according to the required displacement amount, thereby adjusting the current deformable mirror surface shape to the target surface shape. The electrical control instruction sent to the deformable mirror can specifically include the displacement amount and movement direction of the piezoelectric ceramic.

[0047] In another implementation, a thermistor is provided on the back of the deformable mirror. The thermistor can affect the surface shape of the deformable mirror through heat changes. In the embodiment of the present application, the number of thermistors provided on the back of the deformable mirror can be one or more. If multiple thermistors are provided, the adjustment of the surface shape is more flexible and controllable. When the deep learning-based defect detection system starts the aberration compensation mode, the operation control module 06 is specifically used to determine the current value or voltage value that needs to be applied to the thermistor based on the difference between the current surface shape of the deformable mirror and its target surface shape. An electrical control instruction is sent to the deformable mirror to energize the thermistor based on the current value or voltage value, and the heat generated after the thermistor is energized can adjust the surface shape of the current deformable mirror to the target surface shape. The electrical control instruction sent to the deformable mirror may specifically include electrical parameters such as the current value or voltage value corresponding to the thermistor.

[0048] After the optical modulator is adjusted to the target surface shape, the deep learning-based defect detection system switches to the defect detection mode. In this mode, the aberration calibration plate 05 is moved out of the object plane of the objective lens 03, and the sample to be tested is moved into the object plane of the objective lens 03. The light emitted by the light source module 01 passes through the optical modulator 02 and the objective lens 03 after the surface shape is adjusted in turn to reach the sample to be tested, and is reflected on the sample to be tested. The reflected light passes through the objective lens 03 and the optical modulator 02 after the surface shape is adjusted in turn and is collected by the defect detection module 04. The defect detection module 04 finally generates the defect detection result of the sample to be tested on the object plane of the objective lens 03 based on the collected light. Since the defect detection result is generated based on the collected reflected light of the sample to be tested, it is a relatively mature technology for defect detection in the field of semiconductor manufacturing, so it will not be described here.

[0049] The neural network-based defect detection system provided by the technical solution of this application has three flexibly switchable functional modes. Using the neural network training mode, a neural network for objective aberration compensation can be trained; in the aberration compensation mode, the surface shape of the light modulator can be adjusted based on the target surface shape predicted by the neural network, thereby achieving aberration compensation; after aberration compensation, switching to the defect detection mode can obtain defect detection results based on low-aberration detection images. Since the target surface shape is predicted by the neural network, and the neural network is trained based on a training set with multiple data pairs, and each data pair in the training set includes a test image of the aberration calibration plate captured by the system in the neural network training mode, and the expected surface shape of the light modulator optimized based on the test image, the neural network can predict a surface shape adjustment scheme suitable for optimizing the light modulator to compensate for aberrations based on the image captured in the aberration compensation mode. Based on the neural network, the surface shape of the light modulator is adjusted to compensate for the objective aberration. This real-time compensation operation is intelligent and convenient, and can effectively prevent the continuous decline in objective image quality. This application not only improves the real-time performance of aberration compensation and enhances the efficiency and fluency of defect detection, but also ensures the accuracy of defect detection and reduces the difficulty of detection through timely compensation of aberrations.

[0050] Figure 7 This is a structural diagram of another defect detection system based on deep learning provided in the embodiment of the present application. Figure 7 The system structure shown also includes a light splitting element 07. The light emitting end of the light source module 01 and the light receiving end of the defect detection module 04 are respectively located on both sides of the light splitting surface of the light splitting element 07. Figure 7 As shown, the beam splitter 07 is located between the light source module 01 and the optical modulator 02, and the beam splitter 07 is located between the optical modulator 02 and the defect detection module 04. Figure 7In this example, the light emitted by the light source module 01 is transmitted through the beam splitter 07 to the light modulator 02 ; the light beam transmitted from the light modulator 02 is reflected to the defect detection module 04 when passing through the beam splitter 07 .

[0051] Figure 8 A structural diagram of another defect detection system based on deep learning provided in an embodiment of the present application. Figure 8 As can be seen, compared to Figure 7 A reflector 08 is added to the system. The reflector is set between the light source module 01 and the optical path of the beam splitter 07 to reflect and deflect the light beam emitted by the light source module 01.

[0052] In actual applications, there is no limit on the number and position of functional elements such as spectrometers and reflectors in the system. Therefore, the attached drawings are only examples of the system structure. There is no limit on the optical path between the light source module 01 and the optical modulator 02, nor is there a limit on the optical path between the optical modulator 02 and the defect detection module 04.

[0053] This embodiment of the present application also proposes using a deformable mirror (also known as a deformable mirror) as the light modulator 02. If the deformable mirror fails to accurately achieve the target surface shape after adjustment, this may indicate a quality issue with the deformable mirror itself. In practical applications, analyzing the MTF data before and after compensation (i.e., before and after surface adjustment) can determine whether the deformable mirror has successfully adjusted to the target surface shape, and thus whether the compensation has been effective as expected.

[0054] Based on the deep learning-based defect detection system described in the aforementioned embodiment, this application also provides a deep learning-based objective aberration compensation method, which can be applied to the system described in the aforementioned embodiment.

[0055] Figure 9 The flowchart of this method is as follows: Figure 9 As shown in FIG, the objective aberration compensation method based on deep learning includes: S901, starting the aberration compensation mode of the system, and placing the aberration calibration plate on the object plane of the objective lens.

[0056] S902 , turning on the light source module. The light emitted by the light source module passes through the light modulation device and the objective lens in sequence to reach the aberration calibration plate, and is reflected on the aberration calibration plate.

[0057] S903. Collect, through the defect detection module, the reflected light that returns from the aberration calibration plate and sequentially passes through the objective lens and the light modulation device before reaching the defect detection module.

[0058] S904: Generate an image of the aberration calibration plate according to the reflected light collected by the defect detection module.

[0059] S905 , using the image of the aberration calibration plate as an input of a neural network, and predicting the target surface shape to which the current optical modulation device needs to be adjusted through the neural network.

[0060] S906: Adjust the current surface shape of the light modulation device to the target surface shape to compensate for the aberration of the objective lens.

[0061] Figure 9 The process shown mainly describes how to obtain the input image of the neural network after the system starts the aberration compensation mode, that is, the image of the aberration calibration plate captured by the system in the aberration compensation mode. Using this image and the above-mentioned neural network, the target surface shape is finally predicted. The system adjusts the current surface shape of the light modulator according to the target surface shape. Since the target surface shape is predicted by using a neural network, and the neural network is trained by the system in the neural network training mode based on multiple sets of data pairs in the training set, and the expected surface shape in the multiple sets of data pairs is optimized based on the test image in the data pairs, it indicates that the system can obtain better image quality under the expected surface shape and the aberration effect is small. Therefore, adjusting the surface shape of the light modulator to the target surface shape in the aberration compensation stage can compensate for the effect of the objective lens aberration. Therefore, after compensation, the accuracy of defect detection of the sample to be tested can be improved.

[0062] In practical applications, the above-mentioned objective aberration compensation method based on deep learning may further include the following steps before starting the aberration compensation mode of the system, that is, before S901: Starting a neural network training mode of the system, and placing the aberration calibration plate on the object plane of the objective lens; At different moments when the system runs the neural network training mode, the light source module illuminates the aberration calibration plate, and the defect detection module collects light returned by the aberration calibration plate to generate test images corresponding to the aberration calibration plate at the different moments; For each test image, an optimization algorithm is used to optimize an expected surface shape for the light modulator, and each test image and the optimized expected surface shape are constructed into a set of data pairs, thereby obtaining a training set including multiple sets of data pairs; The neural network is obtained by training using the training set.

[0063] The above steps describe the process of collecting a neural network training set and indicate that, in the neural network training mode, the system uses the collected training set to train a neural network ultimately used in the aberration compensation mode. In this application, constructing a training set and optimizing the surface shape based on test images can achieve a target convergence value for the system's modulation transfer function, which serves as the optimization target for the surface shape of the optical modulator.

[0064] Based on the systems and methods introduced in the aforementioned embodiments, the present application also provides a defect detection method based on deep learning. Figure 10 The flowchart of this method is as follows: Figure 10 As shown in Figure 2, the deep learning-based defect detection method includes: S1001. Start the defect detection mode of the system.

[0065] S1002 : While the system is running in the defect detection mode, when it is determined that the objective lens of the system needs to be compensated for aberration, switching the system to the aberration compensation mode.

[0066] S1003 , starting the aberration compensation mode of the system, and placing the aberration calibration plate on the object plane of the objective lens.

[0067] S1004 , turning on the light source module. The light emitted by the light source module passes through the light modulation device and the objective lens in sequence to reach the aberration calibration plate, and is reflected on the aberration calibration plate.

[0068] S1005. Collect, through the defect detection module, the reflected light that returns from the aberration calibration plate and passes through the objective lens and the light modulation device in sequence before reaching the defect detection module.

[0069] S1006. Generate an image of the aberration calibration plate according to the reflected light collected by the defect detection module.

[0070] S1007: Using the image of the aberration calibration plate as an input of a neural network, and predicting the target surface shape to which the current optical modulation device needs to be adjusted through the neural network.

[0071] S1008: Adjust the current surface shape of the light modulation device to the target surface shape to compensate for the aberration of the objective lens.

[0072] S1009: Switch the system to the defect detection mode.

[0073] S1010, controlling the aberration calibration plate to move out from the object plane of the objective lens, and moving the sample to be measured into the object plane.

[0074] S1011 , turning on the light source module, the light emitted by the light source module sequentially passes through the light modulation device with adjusted surface shape and the objective lens to reach the sample to be tested, and is reflected on the sample to be tested.

[0075] S1012. Reflected light that is returned from the sample to be tested and reaches the defect detection module after sequentially passing through the objective lens and the light modulation device after the surface shape is adjusted is collected by the defect detection module.

[0076] S1013: The defect detection module generates a defect detection result of the sample to be tested based on the collected light.

[0077] Figure 10 The process shown mainly shows the process of the system switching from defect detection mode to aberration compensation mode and then switching back to defect detection mode. This process also meets the general requirements and mode switching methods for defect detection of the system. Generally speaking, when it is found that the aberration accumulation of the system is too much, affecting the imaging quality of the system and thus interfering with defect detection, there is a need to compensate for the objective lens aberration. When the aberration compensation is completed, it is possible to switch back to defect detection mode and continue to perform defect detection on the sample to be tested. Since the aberration has been compensated, when switching back to defect detection mode, the system can exert more stable and accurate defect detection performance.

[0078] In practical applications, the above-mentioned deep learning-based defect detection method may further include the following steps before starting the aberration compensation mode of the system, that is, before S1003: Starting a neural network training mode of the system, and placing the aberration calibration plate on the object plane of the objective lens; At different moments when the system runs the neural network training mode, the light source module illuminates the aberration calibration plate, and the defect detection module collects light returned by the aberration calibration plate to generate test images corresponding to the aberration calibration plate at the different moments; For each test image, an optimization algorithm is used to optimize an expected surface shape for the light modulator, and each test image and the optimized expected surface shape are constructed into a set of data pairs, thereby obtaining a training set including multiple sets of data pairs; The neural network is obtained by training using the training set.

[0079] The above steps describe the process of collecting a neural network training set and indicate that, in the neural network training mode, the system uses the collected training set to train a neural network ultimately used in the aberration compensation mode. In this application, constructing a training set and optimizing the surface shape based on test images can achieve a target convergence value for the system's modulation transfer function, which serves as the optimization target for the surface shape of the optical modulator.

[0080] In the method embodiment of the present application, a pre-trained neural network is used to predict the target surface shape of the light modulator based on the image of the aberration calibration plate on the object surface of the objective lens. After the light modulator is adjusted to the target surface shape, the defect detection mode is started, the aberration calibration plate is moved out, and the sample to be tested is moved into the object surface of the objective lens. Light is emitted by the light source module, and gradually passes through the light modulator and the objective lens after the surface shape adjustment to reach the sample to be tested. The light reflected from the sample to be tested passes through the objective lens and the light modulator after the surface shape adjustment and is collected by the defect detection module. Finally, the defect detection module generates a defect detection result of the sample to be tested based on the collected light. This method is based on a neural network to adjust the surface shape of the light modulator, and then completes the compensation of the objective lens aberration. This compensation operation that can be performed in real time is intelligent and convenient, and can effectively avoid the continuous decline in the image quality of the objective lens. The present application not only improves the real-time performance of aberration compensation, improves the execution efficiency and fluency of defect detection, but also can ensure the accuracy of defect detection and reduce the difficulty of detection through timely compensation of aberrations.

[0081] The above is merely one specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A defect detection system based on deep learning, characterized in that: The system has three functional modes: neural network training mode, aberration compensation mode and defect detection mode; the system includes: a light source module, a light modulation device, an objective lens, a defect detection module and an aberration calibration plate; When the system starts the aberration compensation mode, the aberration calibration plate is placed on the object plane of the objective lens, and the light emitted by the light source module passes through the light modulator and the objective lens in sequence to reach the aberration calibration plate, and is reflected on the aberration calibration plate. The reflected light passes through the objective lens and the light modulator in sequence and is collected by the defect detection module; the defect detection module is used to generate an image of the aberration calibration plate based on the collected light; the image is used as input to the neural network to predict the target surface shape to which the current light modulator needs to be adjusted through the neural network; the neural network is trained based on a training set; the training set includes multiple groups of data pairs, each group of data pairs includes a test image of the aberration calibration plate collected by the system in the neural network training mode, and an expected surface shape of the light modulator optimized based on the test image.

2. The system according to claim 1, wherein: When the system starts the neural network training mode, the aberration calibration plate is located on the object plane of the objective lens; and the test images in the multiple sets of data pairs are collected at different times when the system runs the neural network training mode.

3. The system according to claim 1, wherein: The expected surface shape of the light modulator in each data pair in the training set is a surface shape optimized for the light modulator using an optimization algorithm based on the test image of the data pair.

4. The system according to claim 3, characterized in that The optimization goal of the surface shape of the optical modulation device is that the value of the modulation transfer function of the system reaches a target convergence value.

5. The system according to claim 4, characterized in that The target in the aberration calibration plate includes horizontal or vertical lines, and the value of the modulation transfer function is calculated by a knife-edge method; or The target in the aberration calibration plate includes horizontal or vertical lines inclined at 2° to 10°, and the value of the modulation transfer function is calculated by the slanted edge method; or The target in the aberration calibration plate includes black and white grating stripes whose precision meets preset conditions, and the value of the modulation transfer function is calculated by a sine target method.

6. The system according to claim 5, characterized in that The aberration calibration plate includes a plurality of rectangles of the same size and shape; two parallel sides of the rectangles form an angle of 2° to 10° with the horizontal reference direction of the aberration calibration plate.

7. The system according to claim 1, wherein: The system further comprises: an operation control module; the defect detection module and the optical modulation device are both electrically connected to the operation control module; When the system starts the aberration compensation mode, the operation control module is used to call the neural network and use the image of the aberration calibration plate as the input of the neural network to obtain the target surface shape output by the neural network, and control the light modulator to adjust the surface shape based on the target surface shape, so as to compensate for the aberration of the objective lens by adjusting the light modulator.

8. The system according to claim 7, characterized in that The light modulation device is a deformable mirror; a plurality of piezoelectric ceramics are provided on the back of the deformable mirror; When the system starts the aberration compensation mode, the operation control module is specifically used to determine the displacement amount that each piezoelectric ceramic needs to move based on the difference between the current surface shape of the optical modulator and the target surface shape; and send an electrical control instruction to the deformable mirror to drive each piezoelectric ceramic to move according to the displacement amount that each needs to move, so as to adjust the current surface shape of the deformable mirror to the target surface shape.

9. The system according to claim 7, wherein: The light modulation device is a deformable mirror; a thermal resistor is provided on the back of the deformable mirror; When the system starts the aberration compensation mode, the operation control module is specifically used to determine the current value or voltage value that needs to be applied to the thermal resistor according to the difference between the current surface shape of the optical modulation device and the target surface shape; An electrical control instruction is sent to the deformable mirror to energize the thermal resistor based on the current value or the voltage value, and the current surface shape of the deformable mirror is adjusted to the target surface shape by using the heat generated after the thermal resistor is energized.

10. The system according to claim 1, wherein: The system also includes: a spectroscopic element; the light emitting end of the light source module and the light receiving end of the defect detection module are respectively located on both sides of the spectroscopic surface of the spectroscopic element; the spectroscopic element is located between the light source module and the optical path of the optical modulator, and the spectroscopic element is located between the optical path of the optical modulator and the defect detection module.

11. The system according to any one of claims 1 to 10, characterized in that After the light modulator is adjusted to the target surface shape, the system switches to the defect detection mode, the aberration calibration plate is moved out of the object plane, and the sample to be tested is moved into the object plane; the light emitted by the light source module sequentially passes through the light modulator after the surface shape is adjusted and the objective lens to reach the sample to be tested, and is reflected on the sample to be tested, and the reflected light sequentially passes through the objective lens and the light modulator after the surface shape is adjusted and is collected by the defect detection module; The defect detection module is used to generate a defect detection result of the sample to be tested based on the collected light.

12. A method for compensating objective aberration based on deep learning, characterized in that: A deep learning-based defect detection system according to any one of claims 1 to 11; the method comprising: Starting the aberration compensation mode of the system and placing the aberration calibration plate on the object plane of the objective lens; Turning on the light source module, so that the light emitted by the light source module passes through the light modulation device and the objective lens in sequence and reaches the aberration calibration plate, and is reflected on the aberration calibration plate; collecting, by the defect detection module, reflected light that returns from the aberration calibration plate and sequentially passes through the objective lens and the light modulation device before reaching the defect detection module; generating an image of the aberration calibration plate based on the reflected light collected by the defect detection module; Using the image of the aberration calibration plate as an input to a neural network, and predicting the target surface shape to which the current optical modulator needs to be adjusted through the neural network; The current surface shape of the light modulation device is adjusted to the target surface shape to compensate for the aberration of the objective lens.

13. The method according to claim 12, characterized in that Before starting the aberration compensation mode of the system, the method further includes: Starting a neural network training mode of the system, and placing the aberration calibration plate on the object plane of the objective lens; At different moments when the system runs the neural network training mode, the light source module illuminates the aberration calibration plate, and the defect detection module collects light returned by the aberration calibration plate to generate test images corresponding to the aberration calibration plate at the different moments; For each test image, an optimization algorithm is used to optimize an expected surface shape for the light modulator, and each test image and the optimized expected surface shape are constructed into a set of data pairs, thereby obtaining a training set including multiple sets of data pairs; The neural network is obtained by training using the training set.

14. A defect detection method based on deep learning, characterized in that: A deep learning-based defect detection system according to any one of claims 1 to 11; the method comprising: activating the defect detection mode of the system; During the period when the system is operating in the defect detection mode, when it is determined that the objective lens of the system needs to be compensated for aberration, switching the system to the aberration compensation mode; Starting the aberration compensation mode of the system and placing the aberration calibration plate on the object plane of the objective lens; Turning on the light source module, so that the light emitted by the light source module passes through the light modulation device and the objective lens in sequence and reaches the aberration calibration plate, and is reflected on the aberration calibration plate; collecting, by the defect detection module, reflected light that returns from the aberration calibration plate and sequentially passes through the objective lens and the light modulation device before reaching the defect detection module; generating an image of the aberration calibration plate based on the reflected light collected by the defect detection module; Using the image of the aberration calibration plate as an input to a neural network, and predicting the target surface shape to which the current optical modulator needs to be adjusted through the neural network; Adjusting the current surface shape of the light modulation device to the target surface shape to compensate for the aberration of the objective lens; switching the system to the defect detection mode; Controlling the aberration calibration plate to move out from the object plane of the objective lens, and moving the sample to be measured into the object plane; Turning on the light source module, the light emitted by the light source module sequentially passes through the light modulation device with adjusted surface shape and the objective lens to reach the sample to be tested, and is reflected on the sample to be tested; The defect detection module collects reflected light that is returned from the sample to be tested and reaches the defect detection module after sequentially passing through the objective lens and the light modulation device after the surface shape is adjusted; The defect detection module generates a defect detection result of the sample to be tested based on the collected light.

15. The method according to claim 14, characterized in that Before starting the aberration compensation mode of the system, the method further includes: Starting a neural network training mode of the system, and placing the aberration calibration plate on the object plane of the objective lens; At different moments when the system runs the neural network training mode, the light source module illuminates the aberration calibration plate, and the defect detection module collects light returned by the aberration calibration plate to generate test images corresponding to the aberration calibration plate at the different moments; For each test image, an optimization algorithm is used to optimize an expected surface shape for the light modulator, and each test image and the optimized expected surface shape are constructed into a set of data pairs, thereby obtaining a training set including multiple sets of data pairs; The neural network is obtained by training using the training set.

Images