Optical element surface defect detection method and device and computer equipment
Through the improved U2-Net network model in the surface defect detection of optical components, combined with ECA module and bicubital interpolation upsampling, the problem of low detection accuracy in the prior art is solved, and efficient and accurate defect detection is achieved.
Patent Information
- Application Number
- CN202510464464.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-14
- Publication Date
- 2025-07-25
AI Technical Summary
The existing optical element surface defect detection methods have the problem of low detection accuracy, especially the visual method is limited by the human eye resolution ability and subjective judgment of the detector, and the filter imaging method is limited by the optical sensor resolution.
The improved U2-Net network model is used for surface defect detection, and image feature extraction and defect detection are performed by adding an effective channel attention (ECA) module between two adjacent encoding units of the encoder of the original U2-Net network model, and using bicubital interpolation upsampling in the decoder, combined with a deep learning algorithm.
It improves the accuracy and efficiency of surface defect detection of optical components, enhances the ability to identify key features, reduces the loss of details during the upsampling process, and improves the accuracy of the detection results.
Smart Images
Figure CN120374560A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of optical detection, and more particularly to a method, apparatus and computer device for detecting surface defects of optical elements. Background Art
[0002] With the continuous development of the optical industry, optical elements have become an indispensable part of optical systems and have been widely used in fields such as aerospace and high-precision equipment. This has led to an increasing demand for the surface quality of optical elements. In high-precision optical systems in fields such as aerospace and high-precision equipment, it is particularly important to quickly and accurately evaluate the surface defects of optical elements.
[0003] Currently, the methods for detecting surface defects of optical elements include visual inspection and filtering imaging. Among them, visual inspection is limited by the resolution of the human eye and the subjective judgment of the inspectors, resulting in uneven detection quality, low detection efficiency and low accuracy; although the filtering imaging method improves the detection speed through an optical sensor, the machine is limited by the resolution of the optical sensor when observing the size of the defects, and the detection accuracy is still limited. These methods generally have the problem of low detection accuracy. Summary of the Invention
[0004] Embodiments of the present invention provide a method, apparatus and computer device for detecting surface defects of optical elements, which are used to solve the problem of low detection accuracy in detecting surface defects of optical elements in the prior art.
[0005] Embodiments of the present invention provide a method for detecting surface defects of optical elements, including:
[0006] Collecting an initial surface image of the optical element to be measured based on the principle of dark-field microscopy imaging;
[0007] Inputting the initial surface image into a pre-trained improved U 2 -Net network model to obtain a surface defect detection result output by the pre-trained improved U 2 -Net network model;
[0008] Wherein, the pre-trained improved U 2 -Net network model is used to detect surface defect features of the initial surface image, and the improved U 2 -Net network model is obtained by adding an effective channel attention (ECA) module between two adjacent coding units of the encoder of the original U 2 -Net network model.
[0009] In one embodiment, the improved U 2 -Net network model is obtained by adding an effective channel attention (ECA) module between two adjacent coding units of the encoder of the original U 2The ECA module is added between two adjacent encoding units of the encoder of the U-Net network model, and the transposed convolutional upsampling between two adjacent decoding units of the decoder of the original U-Net network model is replaced with bicubic interpolation upsampling; 2 wherein, the encoding unit and the decoding unit are residual U-shaped blocks.
[0010] In one embodiment, inputting the initial surface image into a pre-trained improved U-Net network model to obtain the surface defect detection result output by the pre-trained improved U-Net network model, including:
[0011] filtering the initial surface image to obtain a filtered image; 2 inputting the filtered image into the first encoding unit of the pre-trained improved U-Net network model. After feature extraction by the first encoding unit, input the extracted feature map into the ECA module connected to the first encoding unit to obtain the encoded feature map output by the ECA module connected to the first encoding unit, and input the encoded feature map into the next encoding unit to continue feature extraction until all encoding units and ECA modules of the encoder are traversed. Input the feature map output by the last encoding unit into the first decoding unit in the decoder of the pre-trained improved U-Net network model for decoding, and upsample the decoded image obtained by decoding; 2 For each remaining decoding unit in the decoder except the first decoding unit, perform a skip connection between the previous decoded feature map and the encoded feature map output by the target ECA module, and then input it into the remaining decoding unit for decoding to obtain the decoded feature map output by each remaining decoding unit; wherein, the previous decoded feature map is the feature map obtained by upsampling the decoded image output by the previous decoding unit of the remaining decoding unit; the target ECA module is the ECA module connected to the output end of the target encoding unit in the encoder, and the target encoding unit is the encoding unit in the encoder corresponding to the remaining decoding unit;
[0012] Input the decoded feature map output by each decoding unit in the decoder into the fusion layer of the pre-trained improved U-Net network model to obtain the surface defect detection result output by the fusion layer; the fusion layer is used for image feature fusion of the decoded feature map output by each decoding unit.
[0013] 2 2
[0014]
[0015] 2
[0016] In one embodiment, the pre-trained improved U 2 -Net network model is trained based on the following steps:
[0017] Collect the surface image of the sample optical element based on the principle of dark-field microscopy to obtain an initial sample image;
[0018] Filter the initial sample image to obtain a sample filtered image;
[0019] Label the surface defects of the sample filtered image to obtain a label image;
[0020] Perform the same data augmentation on the sample filtered image and the label image corresponding to the sample filtered image to obtain a target sample image and a target label image;
[0021] Based on the target sample image, the target label image, and the target loss function, train the improved U 2 -Net network model to obtain the pre-trained improved U 2 -Net network model.
[0022] In one embodiment, the target loss function is:
[0023]
[0024] wherein, L is the target loss function, M is the number of encoders in the encoder-decoder stage of the improved U 2 -Net network model, m is a variable, and is the loss function of the prediction result output by the m-th decoding unit of the improved U 2 -Net network model, is the corresponding loss weight, and l fuse is the loss function of the prediction map after fusing the prediction results output by all decoding units of the improved U 2 -Net network model, and w fuse is the loss weight corresponding to l fuse
[0025] In one embodiment, the and l fuse are binary cross-entropy loss functions.
[0026] In one embodiment, based on the target sample image, the target label image, and the target loss function, train the improved U 2 -Net network model to obtain the pre-trained improved U2 -Net network model, including:
[0027] Obtain a set target learning rate;
[0028] Based on the target sample image, the target label image, and the target loss function, using the target learning rate as the learning rate, use the Adam optimizer to optimize the improved U 2 -Net network model for training to obtain the pre-trained improved U 2 -Net network model.
[0029] An embodiment of the present invention provides an optical element surface defect detection device, including:
[0030] An acquisition module for acquiring an initial surface image of the measured optical element based on the dark field microscopy imaging principle;
[0031] A detection module for inputting the initial surface image into the pre-trained improved U 2 -Net network model to obtain the surface defect detection result output by the pre-trained improved U 2 -Net network model;
[0032] Wherein, the pre-trained improved U 2 -Net network model is used to detect surface defect features of the initial surface image, and the improved U 2 -Net network model is obtained by adding an effective channel attention ECA module between two adjacent coding units of the encoder of the original U 2 -Net network model.
[0033] An embodiment of the present invention provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the optical element surface defect detection method described in any one of the above.
[0034] An embodiment of the present invention provides a computer-readable storage medium, storing a computer program, and when the computer program is executed by a processor, the processor executes the optical element surface defect detection method described in any one of the above.
[0035] The optical element surface defect detection method, device and computer device provided by the embodiments of the present invention first acquire an initial surface image of the measured optical element based on the dark field microscopy imaging principle, and then input the initial surface image into the pre-trained improved U 2 -Net network model, through the pre-trained improved U 2The -Net network model detects surface defect features of the initial surface image to obtain a pre-trained improved U 2 - The surface defect detection result output by the -Net network model realizes the rapid detection of surface defects of optical elements by means of a deep learning algorithm, improving the detection accuracy. Among them, the improved U 2 - The -Net network model is obtained by adding an Efficient Channel Attention (ECA) module between two adjacent encoding units of the encoder of the original U 2 - The -Net network model. This ECA module can enhance the feature expression of the image, effectively enhance the key features of the image, thereby improving the model's recognition ability for important information, enabling the model to better identify and utilize key features, and further improving the accuracy of surface defect detection of optical elements. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.
[0037] Figure 1 It is a schematic structural diagram of the dark-field microscopy measurement system provided by the embodiment of the present invention;
[0038] Figure 2 It is a schematic flow diagram of the method for detecting surface defects of optical elements provided by the embodiment of the present invention;
[0039] Figure 3 It is the structural schematic diagram of the original U 2 -Net network model in the prior art;
[0040] Figure 4 It is the structural schematic diagram of the improved U 2 -Net network model provided by the embodiment of the present invention;
[0041] Figure 5 It is the schematic flow diagram of the training method of the pre-trained improved U 2 -Net network model provided by the embodiment of the present invention;
[0042] Figure 6 It is the schematic principle diagram of data augmentation provided by the embodiment of the present invention;
[0043] Figure 7 It is the U 2Schematic diagram of the change comparison of the loss value before and after the improvement of the U-Net network model;
[0044] Figure 8 The following is a schematic structural diagram of the optical element surface defect detection device provided by the embodiment of the present invention. Detailed implementation manners
[0045] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
[0046] Optical elements are an indispensable part of optical systems, and the quality of their surfaces directly affects the quality of optical systems. Especially in high-precision optical systems, how to quickly and accurately evaluate the surface defects of optical elements is particularly important.
[0047] In related technologies, the surface defect detection methods of optical elements include visual inspection method, filtering imaging method, and contact measurement method, etc. Among them, the visual inspection method is the most primitive detection method, but its detection results are limited by the resolution of the human eye and the subjective judgment of the detection personnel, resulting in uneven detection quality, low efficiency and low accuracy; although the filtering imaging method can improve the detection speed through an optical sensor, the instrument is limited by the resolution of the optical sensor when observing the surface defects of optical elements, and the detection accuracy is still limited; the contact measurement method detects the surface of the element by performing contact scanning on the element surface with a contact probe. Although it has high measurement accuracy, the detection efficiency is low. Especially for elements with large sizes, the detection efficiency will be greatly reduced.
[0048] With the continuous development of deep learning technology, defect detection methods based on deep learning have been more and more widely used in industrial scenarios. Applying deep learning technology to the detection of surface defects of optical elements will greatly improve the accuracy and efficiency of surface defect detection of optical elements.
[0049] In view of this, the embodiment of the present invention provides a surface defect detection method based on an improved U 2 -Net convolutional neural network model, which is based on dark-field microscopic imaging measurement and uses a deep learning algorithm to quickly detect surface defects of optical elements.
[0050] The method for detecting surface defects of an optical element provided by an embodiment of the present invention can be applied to a computer device in a dark-field microscopy measurement system, or other computer devices communicatively connected to the computer device, where the computer device can include a terminal device or a server, etc. Among them, the terminal device can include a mobile phone, a computer, a tablet computer, etc.; the server can include an independent server, a cluster server, or a cloud server, etc. The camera scanning method for detecting surface defects of the optical element can also be applied to a camera scanning device for detecting surface defects of the optical element provided in the computer device, and the camera scanning device for detecting surface defects of the optical element can be implemented by software, hardware, or a combination of both.
[0051] Figure 1 The structural schematic diagram of the dark-field microscopy measurement system is shown. The dark-field microscopy measurement system can measure the surface of the optical element to be measured based on the dark-field microscopy principle. Among them, dark-field microscopy (or dark-field microscopy) is a special microscopy technique in optical microscopy and electron microscopy. Light or electrons other than the object to be measured enter the objective lens of the microscope, making the field of view background observed in the eyepiece black, and only the edge of the object to be measured is bright. The dark-field microscopy imaging principle is to block the direct light and only allow the oblique light to irradiate on the object to be measured, and use the scattered light of the object to be measured for imaging. Specifically, a dark-field microscope uses a special condenser lens, which produces an inverted hollow conical illumination area with the focus on the plane of the object to be measured. Where there is no object to be measured, the oblique light will cross each other and form a dark background, while the oblique light passing through the object to be measured will produce diffraction, reflection, or refraction to form a bright sample image.
[0052] Refer to Figure 1 As shown, the dark-field microscopy measurement system can include a microscope camera 11, an annular light source 12, an electric control translation stage 13, and a computer device 14. Specifically, when measuring the optical element to be measured, the optical element 100 to be measured can be placed on the stage of the electric control translation stage 13, and the annular light source 12 emits annular light and irradiates it on the optical element 100 to be measured. If there are defects on the surface of the optical element 100 to be measured, the backscattered light caused by the defects will enter the microscopic imaging system through the objective lens of the microscope camera 11 and converge on the target surface of the microscope camera 11, while the reflected light reflected by the optical element 100 to be measured does not enter the microscopic imaging system, forming a dark field, and making the defect image form a bright image against a dark background. The electric control translation stage 13 clamps the microscope camera 11, and the computer device 14 can drive the microscope camera 11 to move by controlling the movement of the translation mechanism of the electric control translation stage 13. At the same time, the computer device 14 can also acquire the image collected by the microscope camera 11 and process the image.
[0053] Based on Figure 1The dark-field microscopy measurement system will be described in detail below for the optical element surface defect detection method provided by the embodiments of the present invention.
[0054] Figure 2 FIG. shows a schematic flowchart of the optical element surface defect detection method provided by the embodiments of the present invention. Referring to Figure 2 as shown, the optical element surface defect detection method may include the following steps 210 to 220.
[0055] Step 210: Collect an initial surface image of the optical element to be measured based on the dark-field microscopy principle.
[0056] The computer device may collect the surface image of the optical element to be measured according to the working principle of the dark-field microscopy measurement system described above Figure 1 to obtain the initial surface image.
[0057] Step 220: Input the initial surface image into a pre-trained improved U 2 -Net network model to obtain the surface defect detection result output by the pre-trained improved U 2 -Net network model.
[0058] Among them, the pre-trained improved U 2 -Net network model is used to detect the surface defect features of the initial surface image. The improved U 2 -Net network model is obtained by adding an ECA module between two adjacent encoding units of the encoder of the original U 2 -Net network model.
[0059] U 2 -Net network model (i.e., the dual U-Net network model) has the core idea of using multiple U-Net sub-networks to capture features at different scales and improving the segmentation accuracy by fusing information at different scales. Each U-Net sub-network extracts and restores the multi-layer features of the image through two stages of downsampling and upsampling, and then these features are merged through a fusion layer to extract more spatial information.
[0060] Figure 3 FIG. shows a schematic structural diagram of the original U 2 -Net network model. Referring to Figure 3 as shown, the original U 2The encoder of the U-Net network model has a total of 6 stages, and each stage is composed of residual U-shaped (RSU) blocks with multi-scale features. The left side is the encoder structure, and the right side is the decoder structure. The encoder is a downsampling process used to extract image features; the decoder is an upsampling process used to obtain the position information of the image. Specifically, the first four encoding units use residual U-shaped blocks RSU-7, RSU-6, RSU-5, and RSU-4 respectively. Among them, "7", "6", "5", and "4" represent the height L of the RSU block, and this height L is usually configured according to the spatial resolution of the input feature map. For feature maps with a large height and width, a larger L value is used to capture more large-scale information. The resolutions of the feature maps in the fifth and sixth encoding units are relatively low. If these feature maps are further downsampled, it will cause the loss of useful context. Therefore, in the fifth and sixth encoding unit stages, RSU-4F is used. The "4" in it represents the height of the RSU block, and "F" means that the RSU block is an expanded version and the pooling and upsampling operations are replaced by dilated convolutions. All intermediate feature maps of RSU-4F have the same resolution as their input feature maps. Original U 2 The structure of the decoder of the U-Net network model is symmetrically similar to that of its encoder. Each decoding unit also adopts a residual U-shaped (RSU) block, except that transposed convolutional upsampling is performed between two adjacent decoding units. That is, each decoding unit in the decoder stage takes the concatenation of the upsampled feature map from its previous decoding unit and the feature map from the symmetric encoding unit in its encoder stage as input.
[0061] In an embodiment of the present invention, it can be in the original U 2 -ECA modules are added between two adjacent encoding units of the encoder of the U-Net network model to obtain an improved U 2 -Net network model. For example, Figure 4 shows the structural schematic diagram of the improved U 2 -Net network model. Referring to Figure 4 as shown, it can be in the original U 2 -ECA modules are added between two adjacent residual U-shaped blocks of the encoder of the U-Net network model to enhance the weight distribution of different channels in the feature map. In this way, by adding an ECA module at the end of each encoding unit, a channel descriptor can be obtained through global average pooling, and one-dimensional convolution is used to generate the corresponding channel weights. These channel weights are then applied to the input feature map of the next encoding unit to effectively enhance the key features, thereby improving the recognition ability of the entire network model for important information, enabling the entire network model to better recognize and utilize key features, and enhancing the expression ability of the features.
[0062] Specifically, the residual U-shaped blocks of the first four encoding units can use 3×3 convolutions to extract features, gradually expand the receptive field, and then reduce the resolution of the feature map through max pooling while increasing the number of channels. Adding an ECA module after convolution can enhance the weights of important features through the channel attention mechanism and output feature maps with gradually decreasing resolutions. The fifth and sixth encoding units use dilated convolutions instead of pooling operations, which can expand the receptive field while maintaining high resolution. Adding an ECA module after the dilated convolution can further enhance the feature expression ability and output high-resolution feature maps.
[0063] In another embodiment of the present invention, an ECA module can be added between two adjacent encoding units of the encoder of the original U 2 -Net network model, and the transposed convolution upsampling between two adjacent decoding units of the decoder of the original U 2 -Net network model is replaced with bicubic interpolation upsampling to obtain an improved U 2 -Net network model. For example, as Figure 4 shown, an ECA module can be added between two adjacent residual U-shaped blocks of the encoder of the original U 2 -Net network model. At the same time, before the skip connection between the upsampled feature map of each decoding unit (RSU block) of the decoder and the output feature map of the corresponding encoding unit, the upsampled feature map is upsampled using bicubic interpolation to reduce image distortion and improve the quality of the segmentation result. The input feature map of each decoding unit is first upsampled by bicubic interpolation and adjusted accordingly according to the target output size. The upsampled feature map will be skip-connected to the output feature map of the corresponding encoding unit in the encoder to form a fused feature map. In this way, higher-level feature information can be extracted and the segmentation effect can be further improved. Compared with the upsampling implemented by transposed convolution in the original U 2 -Net network model, the improved U 2 -Net network model provided by the embodiment of the present invention uses bicubic interpolation upsampling instead of transposed convolution upsampling, effectively controlling the resolution of the feature map, improving the quality of the upsampled feature map, and reducing the aliasing effect caused by upsampling. At the same time, combined with the high computational efficiency of the ECA module, the network model can maintain a high computational efficiency while improving performance.
[0064] For the above-mentioned improved U 2 -Net network model, the surface image of the sample optical element can be used as the sample image, and the surface defects of the sample image can be labeled to obtain the corresponding label image. Then, using a large number of sample images, the label image corresponding to the sample image, and the loss function, the improved U 2 -Net network model is trained to obtain a pre-trained improved U2 -Net network model.
[0065] After obtaining the initial surface image, the computer device can input the initial surface image into a pre-trained improved U 2 -Net network model. The improved U 2 -Net network model detects surface defect features from the initial surface image, and can obtain the surface defect detection result output by the pre-trained improved U 2 -Net network model.
[0066] In a specific embodiment, step 220 may include the following steps 221 to 224.
[0067] Step 221: Filter the initial surface image to obtain a filtered image.
[0068] After the computer device obtains the initial surface image, it can filter the initial surface image according to a preset filtering method to obtain a filtered image. Among them, the preset filtering method may include at least one of Gaussian filtering, bilateral filtering, mean filtering, and Gaussian bilateral filtering, etc., and the embodiments of the present application do not limit this.
[0069] Step 222: Input the filtered image into the first encoding unit of the pre-trained improved U 2 -Net network model. After feature extraction by the first encoding unit, input the extracted feature map into the ECA module connected to the first encoding unit to obtain the encoded feature map output by the ECA module connected to the first encoding unit, and input the encoded feature map into the next encoding unit for further feature extraction until all the encoding units and ECA modules of the encoder are traversed. Input the feature map output by the last encoding unit into the first decoding unit of the decoder of the pre-trained improved U 2 -Net network model for decoding, and upsample the decoded image obtained by decoding.
[0070] Exemplarily, after the computer device obtains the filtered image, it can input the filtered image into the first encoding unit of the pre-trained improved U 2 -Net network model, or, the filtered image can be further grayed to obtain a grayed filtered image, and then input the grayed filtered image into the first encoding unit of the pre-trained improved U 2 -Net network model.
[0071] Exemplarily, as Figure 4 shown, bicubic interpolation upsampling can be used for upsampling.
[0072] Step 223: For each remaining decoding unit in the decoder except the first decoding unit, the previous decoded feature map is skip-connected with the encoded feature map output by the target ECA module and then input into this remaining decoding unit for decoding, to obtain the decoded feature map output by each remaining decoding unit.
[0073] Among them, the previous decoded feature map is the feature map obtained by upsampling the decoded image output by the previous decoding unit of the remaining decoding unit; the target ECA module is the ECA module connected to the output end of the target encoding unit in the encoder, and the target encoding unit is the encoding unit in the encoder corresponding to the remaining decoding unit.
[0074] For example, as Figure 4 shown, after the feature map output by the last encoding unit RSU-4F of the encoder is input into the first decoding unit RSU-4F in the decoder for decoding, the decoded image obtained by decoding is upsampled by bicubic interpolation to obtain the corresponding decoded feature map. Then, this decoded feature map is skip-connected with the encoded feature map output by the last ECA module connected to the output end of the fifth encoding unit RSU-4F of the encoder and then input into the second decoding unit RSU-4F of the decoder. Next, after the decoded image output by the second decoding unit RSU-4F is upsampled by bicubic interpolation, the obtained decoded feature map is continue to be skip-connected with the encoded feature map output by the ECA module connected to the output end of the fourth encoding unit RSU-4 in the encoder and then input into the third decoding unit RSU-4 of the decoder. And so on, the decoded feature map output by each decoding unit can be obtained.
[0075] Step 224: Input the decoded feature map output by each decoding unit in the decoder into the fusion layer of the pre-trained improved U 2 -Net network model to obtain the surface defect detection result output by the fusion layer.
[0076] Among them, the fusion layer is used to perform image feature fusion on the decoded feature map output by each decoding unit.
[0077] The optical element surface defect detection method provided by the embodiments of the present invention first collects the initial surface image of the measured optical element based on the principle of dark field microscopy, and then inputs the initial surface image into the pre-trained improved U 2 -Net network model. The pre-trained improved U 2 -Net network model is used to detect the surface defect features of the initial surface image, to obtain the surface defect detection result output by the pre-trained improved U 2 -Net network model. The rapid detection of optical element surface defects is realized by means of deep learning algorithms, and the detection accuracy is improved. Among them, the improved U 2-Net network model is obtained by adding an ECA module between two adjacent encoding units of the encoder of the original U 2 -Net network model. This ECA module can enhance the feature expression of the image, effectively strengthen the key features of the image, thereby improving the model's recognition ability for important information, enabling the model to better recognize and utilize key features, and further improving the accuracy of optical element surface defect detection. Further, the improved U 2 -Net network model uses bicubic interpolation upsampling in the decoder, reducing the detail loss in the upsampling process and being able to generate a smoother high-resolution feature map. Moreover, by fusing feature maps of different scales output by the encoding units, the accuracy of salient object detection is improved.
[0078] Based on Figure 2 the optical element surface defect detection method of the corresponding embodiment, in an embodiment of the present invention, Figure 5 exemplarily shows the flow schematic diagram of the training method of the pre-trained improved U 2 -Net network model. Referring to Figure 5 as shown, the pre-trained improved U 2 -Net network model can be trained based on the following steps 510 to step 550.
[0079] Step 510: Collect the surface image of the sample optical element based on the dark field microscopy principle to obtain the initial sample image.
[0080] Step 520: Filter the initial sample image to obtain the sample filtered image.
[0081] Among them, the filtering method for filtering the initial sample image may include at least one of Gaussian filtering, bilateral filtering, mean filtering, and Gaussian bilateral filtering, etc. The embodiments of the present invention do not limit this. For example, through Gaussian bilateral filtering, the edge information and detail information of the image can be retained, and noise can be suppressed.
[0082] Step 530: Label the surface defects of the sample filtered image to obtain the label image.
[0083] After obtaining the sample filtered image, the surface defects of the sample filtered image can be labeled using a labeling tool. The defect area and non-defect area can be clearly distinguished through labeling. The main content of the labeling is to mark the position of the surface defect of the sample optical element with a rectangular box, and label the marked rectangular box with a label name, such as naming it in English, and then save the file. During the labeling process, try to mark the outline of the defect as much as possible to obtain the label image corresponding to this image.
[0084] For example, a graphic image annotation tool Labelme can be used to manually mark the defective areas on the surface of the sample optical element in the sample filtered image, convert it into a file in.json format, and then obtain the label image corresponding to the sample filtered image by parsing the.json file. The role of this label image is to enable the model to accurately identify and locate defects, and it can be used to calculate the required evaluation index parameters through operations with the result image of the deep learning neural network.
[0085] Exemplarily, after filtering the initial sample image to obtain the sample filtered image, the sample filtered image can be grayscaled to obtain the grayscaled sample filtered image, and then the surface defects of the grayscaled sample filtered image can be labeled to obtain the label image.
[0086] Step 540: Perform the same data augmentation on the sample filtered image and the corresponding label image to obtain the target sample image and the target label image.
[0087] For example, Figure 6 shows a schematic diagram of the principle of data augmentation. Referring to Figure 4 as shown, during the data augmentation process, it is necessary to correspond the sample filtered image and the label image one by one and perform the same data augmentation. That is, for each sample filtered image, the same data augmentation is performed on this sample filtered image and its corresponding unique label image. The sample size can be enlarged through data augmentation.
[0088] Specifically, the data augmentation can include at least one of image flipping, image translation, and image rotation, etc. Among them, image flipping can include horizontal flipping and / or vertical flipping. Horizontal flipping is performed with the central connection line in the horizontal direction of the image as the axis, and vertical flipping is performed with the central connection line in the vertical direction of the image as the axis. Image translation mainly adjusts the position of the image in the horizontal (left and right movement) or vertical (up and down movement) direction according to the set movement distance to generate new samples that originally do not exist in the sample space. Image rotation is to rotate the image around the center point without changing the original defect features of the image, and the rotation angle is, for example, at least one of 45°, 90°, and 135°, etc.
[0089] Step 550: Based on the target sample image, the target label image, and the target loss function, train the improved U 2 -Net network model to obtain the pre-trained improved U 2 -Net network model.
[0090] After obtaining the target sample images and target label images through data augmentation, these images can be divided into datasets. For example, the dataset composed of these images can be divided into a training set and a validation set according to a ratio of 4:1. Among them, the number of defective images in the target sample images can be, for example, 4380, including 2100 stippling defects and 2280 scratch defects.
[0091] Before training the improved U 2 -Net network model, set the relevant parameters for model training, including the learning rate, optimizer type, number of training epochs, etc., as well as the setting of the training environment to ensure that the model can run effectively.
[0092] Specifically, step 550 trains the improved U 2 -Net network model based on the target sample images, target label images, and target loss function to obtain a pre-trained improved U 2 -Net network model, which can be achieved through the following steps: obtain the set target learning rate; based on the target sample images, target label images, and target loss function, use the Adam optimizer to train the improved U 2 -Net network model with the target learning rate as the learning rate to obtain a pre-trained improved U 2 -Net network model.
[0093] For example, model training can be carried out on the established optical component surface defect dataset. The optimizer can adopt Adam, the learning rate can be set to 0.001, and the component surface defect images are iterated 200 times in total.
[0094] Exemplarily, the target loss function can be:
[0095] Among them, L is the target loss function, M is the number of encoding units in the encoding and decoding stage of the improved U 2 -Net network model, m is a variable, is the loss function of the prediction result output by the m-th decoding unit of the improved U 2 -Net network model, is the corresponding loss weight, l fuse is the loss function of the prediction map after fusing the prediction results output by all decoding units of the improved U 2 -Net network model, w fuse is the loss weight corresponding to l fuse
[0096] Exemplarily, and l fuse It can be the binary cross-entropy loss function, which calculates the difference between the model's predicted values and the actual labels.
[0097] During the training process, the network weights are continuously adjusted to minimize the loss value of the objective loss function. As the network parameters are optimized, the loss value of the objective loss function will gradually decrease. If the loss value continues to decrease, it indicates that the model is learning effective features. When the loss value tends to be stable and no longer decreases significantly, it means that the model has converged and the training can stop.
[0098] Figure 7 Shows U 2 - Schematic diagram comparing the changes in the loss values before and after improving the Net network model. Refer to Figure 7 As shown, the original U 2 - Net network model and the improved U 2 - Net network model are trained respectively with the same training parameters and training environment. Curve ① is the change curve of the loss value during the training of the original U 2 - Net network model, and curve ② is the change curve of the loss value during the training of the improved U 2 - Net network model. In the first 50 iterations, the loss value experienced a sharp decline. This cliff-like decline trend highlights the network model's ability to quickly learn in the initial stage, and the learning ability of the improved U 2 - Net network model is significantly higher than that of the original U 2 - Net network model. As the iteration progresses, when the number of iterations accumulates to 200 times, the decline rate of the loss value significantly slows down and finally stabilizes at about 0.18, indicating that the network model has reached convergence, that is, the training process of the network model tends to be completed and its performance tends to be optimal. Through comparison, it can be seen that the performance of the improved U 2 - Net network model is significantly higher than that of the original U 2 - Net network model.
[0099] After the model training is completed, it can be verified on the validation set to ensure the generalization ability of the model. For example, metrics such as intersection over union (IoU), Dice similarity, and accuracy can be used to evaluate the trained improved U 2 - Net network model. The value ranges of these metrics are all between 0 and 1. When the extraction result is optimal, the value of the metric is 1; when it is the worst, the value of the metric is 0. Therefore, the higher the metric value, the better the extraction effect of the network. In this way, the generalization ability of the trained improved U 2 - Net network model can be verified.
[0100] Through the verification of the validation set, U 2By comparing the evaluation metrics before and after the improvement of the U-Net network model, the comparison results shown in Table 1 can be obtained:
[0101] Table 1
[0102] Network model Accuracy IoU Dice similarity <![CDATA[Original U 2 -Net]]> 0.928 0.874 0.877 <![CDATA[Improved U 2 -Net]]> 0.957 0.955 0.959
[0103] As can be seen from Table 1, the improved U 2 -Net network model has good generalization ability.
[0104] The improved U 2 -Net network model provided by the embodiment of the present invention adds an ECA network learning unit in each encoding stage, which not only enhances the feature expression ability, but also helps to solve the problem of gradient disappearance in the training of deep networks, thereby accelerating the training speed of the model. Moreover, the decoder realizes effective control of the resolution of the feature map through bicubic interpolation upsampling. At the same time, the ECA module has high computational efficiency, enabling the entire network model to maintain high computational efficiency while improving performance.
[0105] Based on the same inventive concept, the embodiment of the present invention provides an optical element surface defect detection device. Since the principle of the device for solving technical problems is similar to the optical element surface defect detection method provided by the embodiment of the present invention, the implementation of the optical element surface defect detection device can refer to the implementation of the method, and the repeated parts will not be described again.
[0106] Figure 8 The structural schematic diagram of the optical element surface defect detection device provided by the embodiment of the present invention is shown. The optical element surface defect detection device may include:
[0107] An acquisition module 810, configured to acquire an initial surface image of the optical element to be measured based on the principle of dark field microscopy imaging;
[0108] A detection module 820, configured to input the initial surface image into a pre-trained improved U 2 -Net network model to obtain a surface defect detection result output by the pre-trained improved U 2 -Net network model;
[0109] Wherein, the pre-trained improved U 2 -Net network model is used to detect surface defect features of the initial surface image. The improved U 2 -Net network model is obtained by adding an ECA module between two adjacent encoding units of the encoder of the original U 2 -Net network model.
[0110] In an embodiment of the present invention, the improved U 2- The -Net network model is based on the original U 2 - An ECA module is added between two adjacent encoding units of the encoder of the -Net network model, and the original U 2 - The transposed convolutional upsampling between two adjacent decoding units of the decoder of the -Net network model is replaced with bicubic interpolation upsampling; wherein, the encoding unit and the decoding unit are residual U-shaped blocks.
[0111] In an embodiment of the present invention, the detection module 820 may include:
[0112] A filtering unit for filtering the initial surface image to obtain a filtered image;
[0113] A first feature extraction unit for inputting the filtered image into the first encoding unit of a pre-trained improved U 2 -Net network model. After feature extraction by the first encoding unit, the extracted feature map is input into the ECA module connected to the first encoding unit to obtain an encoded feature map output by the ECA module connected to the first encoding unit, and the encoded feature map is input into the next encoding unit for continued feature extraction until all encoding units and ECA modules of the encoder are traversed. The feature map output by the last encoding unit is input into the first decoding unit in the decoder of the pre-trained improved U 2 -Net network model for decoding, and the decoded image obtained by decoding is upsampled;
[0114] A second feature extraction unit for, for each remaining decoding unit in the decoder except the first decoding unit, performing a skip connection between the previous decoded feature map and the encoded feature map output by the target ECA module and inputting the result into the remaining decoding unit for decoding to obtain the decoded feature map output by each remaining decoding unit; wherein, the previous decoded feature map is a feature map obtained by upsampling the decoded image output by the previous decoding unit of the remaining decoding unit; the target ECA module is the ECA module connected to the output end of the target encoding unit in the encoder, and the target encoding unit is the encoding unit in the encoder corresponding to the remaining decoding unit;
[0115] A fusion unit for inputting the decoded feature map output by each decoding unit in the decoder into the fusion layer of the pre-trained improved U 2 -Net network model to obtain the surface defect detection result output by the fusion layer; wherein, the fusion layer is used for image feature fusion of the decoded feature map output by each decoding unit.
[0116] In an embodiment of the present invention, the optical element surface defect detection device may further include a training module, and the training module is used for: collecting a surface image of a sample optical element based on the principle of dark-field microscopy imaging to obtain an initial sample image; filtering the initial sample image to obtain a sample filtered image; annotating surface defects on the sample filtered image to obtain a label image; performing the same data augmentation on the sample filtered image and the label image corresponding to the sample filtered image to obtain a target sample image and a target label image; based on the target sample image, the target label image, and the target loss function, training the improved U 2 -Net network model to obtain a pre-trained improved U 2 -Net network model.
[0117] In an embodiment of the present invention, when the training module trains the improved U 2 -Net network model based on the target sample image, the target label image, and the target loss function to obtain a pre-trained improved U 2 -Net network model, it is specifically used for: obtaining a set target learning rate; based on the target sample image, the target label image, and the target loss function, using the Adam optimizer to train the improved U 2 -Net network model with the target learning rate as the learning rate to obtain a pre-trained improved U 2 -Net network model.
[0118] It should be understood that the units included in the above optical element surface defect detection device are only logical divisions according to the functions implemented by the device. In actual applications, the above units can be superimposed or split. And the functions implemented by the optical element surface defect detection device provided in this embodiment correspond one by one to the optical element surface defect detection method provided in the above embodiment. For the more detailed processing flow implemented by this device, it has been described in detail in the above method embodiment, and will not be described in detail here.
[0119] Another embodiment of the present invention further provides a computer device, which includes: a processor and a memory; the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the computer device executes each step of the optical element surface defect detection method shown in the above method embodiment.
[0120] Another embodiment of the present invention further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions run on a computer device, the computer device executes each step of the optical element surface defect detection method shown in the above method embodiment.
[0121] Although the preferred embodiments of the present invention have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention.
[0122] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.
Claims
1. An optical element surface defect detection method, characterized in that, Including: Collecting an initial surface image of the optical element to be measured based on the principle of dark-field microscopy imaging; Input the initial surface image into a pre-trained improved U 2 -Net network model to obtain the surface defect detection result output by the pre-trained improved U 2 -Net network model; Among them, the pre-trained improved U 2 -Net network model is used to detect surface defect features of the initial surface image, and the improved U 2 -Net network model is obtained by adding an effective channel attention ECA module between two adjacent encoding units of the encoder of the original U 2 -Net network model.
2. The method for detecting surface defects of an optical element according to claim 1, characterized in that, The improved U 2 -Net network model is obtained by adding the ECA module between two adjacent encoding units of the encoder of the original U 2 -Net network model, and replacing the transposed convolutional upsampling between two adjacent decoding units of the decoder of the original U 2 -Net network model with bicubic interpolation upsampling; Wherein, the encoding unit and the decoding unit are residual U-shaped blocks.
3. The method for detecting surface defects of an optical element according to claim 1 or 2, characterized in that, Inputting the initial surface image into a pre-trained improved U 2 -Net network model to obtain the surface defect detection result output by the pre-trained improved U 2 -Net network model, including: Filtering the initial surface image to obtain a filtered image; Input the filtered image into the first encoding unit of the pre-trained improved U 2 -Net network model. After feature extraction by the first encoding unit, input the extracted feature map into the ECA module connected to the first encoding unit to obtain the encoded feature map output by the ECA module connected to the first encoding unit, and input the encoded feature map into the next encoding unit to continue feature extraction until all encoding units and ECA modules of the encoder are traversed. Input the feature map output by the last encoding unit into the first decoding unit in the decoder of the pre-trained improved U 2 -Net network model for decoding, and upsample the decoded image obtained by decoding; For each remaining decoding unit in the decoder except the first decoding unit, performing a skip connection between the previous decoded feature map and the encoded feature map output by the target ECA module and inputting the result into the remaining decoding unit for decoding to obtain the decoded feature map output by each remaining decoding unit; wherein, the previous decoded feature map is a feature map obtained by upsampling the decoded image output by the previous decoding unit of the remaining decoding unit; the target ECA module is the ECA module connected to the output end of the target encoding unit in the encoder, and the target encoding unit is the encoding unit in the encoder corresponding to the remaining decoding unit; Input the decoded feature maps output by each decoding unit in the decoder into the fusion layer of the pre-trained improved U 2 -Net network model to obtain the surface defect detection result output by the fusion layer; the fusion layer is used to perform image feature fusion on the decoded feature maps output by each decoding unit.
4. The method for detecting surface defects of an optical element according to claim 1 or 2, characterized in that, The pre-trained improved U 2 -Net network model is obtained by training based on the following steps: Collecting a surface image of the sample optical element based on the principle of dark-field microscopy imaging to obtain an initial sample image; Filtering the initial sample image to obtain a sample filtered image; Labeling surface defects of the sample filtered image to obtain a label image; Performing the same data augmentation on the sample filtered image and the label image corresponding to the sample filtered image to obtain a target sample image and a target label image; Based on the target sample image, the target label image, and the target loss function, train the improved U 2 -Net network model to obtain the pre-trained improved U 2 -Net network model.
5. The method for detecting surface defects of an optical element according to claim 4, characterized in that, The target loss function is: Among them, L is the target loss function, and M is the number of encoders in the encoder-decoder stage of the improved U 2 -Net network model. m is a variable, and the is the loss function of the prediction result output by the m-th decoding unit of the improved U 2 -Net network model. The is the corresponding loss weight. The l fuse is the loss function of the prediction map after fusing the prediction results output by all decoding units of the improved U 2 -Net network model. The w fuse is the loss weight corresponding to the l fuse corresponding to it.
6. The method for detecting surface defects of an optical element according to claim 5, wherein The and the fuse is the binary cross-entropy loss function.
7. The method for detecting surface defects of an optical element according to claim 4, characterized in that, Based on the target sample image, the target label image, and the target loss function, training the improved U 2 -Net network model to obtain the pre-trained improved U 2 -Net network model, including: Obtaining a set target learning rate; Based on the target sample image, the target label image, and the target loss function, using the Adam optimizer with the target learning rate as the learning rate, train the improved U 2 -Net network model to obtain the pre-trained improved U 2 -Net network model.
8. An optical element surface defect detection device, characterized in that Including: A collecting module for collecting an initial surface image of the optical element to be measured based on the principle of dark-field microscopy imaging; The detection module is used to input the initial surface image into a pre-trained improved U 2 -Net network model to obtain the surface defect detection result output by the pre-trained improved U 2 -Net network model; Among them, the pre-trained improved U 2 -Net network model is used to detect surface defect features of the initial surface image, and the improved U 2 -Net network model is obtained by adding an effective channel attention (ECA) module between two adjacent encoding units of the encoder of the original U 2 -Net network model.
9. A computer device, characterized in that, The computer device includes a memory and a processor. When the computer program stored in the memory is executed by the processor, the processor executes the optical element surface defect detection method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, Storing a computer program, which when executed by a processor, causes the processor to execute the optical element surface defect detection method according to any one of claims 1 to 7.