Method for segmenting transparent optical element examples measured with auxiliary deflection in dark light scenarios

Abstract

The application provides a transparent optical element instance segmentation method for auxiliary deflection measurement in a dark light scene, and comprises the following steps: step S1, generating a single-frequency sinusoidal fringe image; step S2, under dark light conditions, collecting the sinusoidal fringe image based on a deflection measurement system; step S3, obtaining a phase image; step S4, changing the position or shape of the transparent optical element, and executing steps S2-S3; step S5, obtaining a data set, and training a neural network model; step S6, executing steps S1-S3 to obtain a phase image of a transparent optical element to be measured; step S7, inputting the phase image of the transparent optical element to be measured output by step S6 into the trained neural network model to obtain a segmentation result, that is, a mask image MASK1 of the transparent optical element to be measured; and step S8, multiplying the MASK1 and an original fringe image pixel by pixel for subsequent deflection measurement of a surface profile. The application can accurately and quickly perform transparent optical element instance segmentation in a dark light scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical measurement, and in particular relates to a transparent optical element instance segmentation method for assisting deflectometry measurement in dark light scenes. Background Art

[0002] Transparent components play a vital role in all aspects of society and life. In the field of optics, transparent components such as lenses, filters, and prisms are widely used in the manufacture of various optical instruments and devices, including cameras, telescopes, VR / AR devices, and lasers. Transparent components are also widely used in the automotive, electronics, aerospace, and medical device industries. For example, high-precision transparent components such as objective lenses and eyepieces are key components in high-end cameras and projectors. Transparent components used in instruments such as spectrometers and laser interferometers help scientists accurately measure and analyze the properties of light. In the aerospace field, aircraft cockpit windshields (windshield covers) must possess high strength and excellent transparency to provide pilots with an excellent field of view while also withstanding the high pressure, wind speed, and temperature fluctuations experienced during flight. Windows on passenger aircraft and spacecraft must be transparent, robust, and able to withstand extreme environmental conditions, making their surface shape crucial for assembly. Therefore, transparent components are an indispensable component of modern industry and scientific research. The surface accuracy of transparent components is crucial to improving the performance and stability of these devices. Deflectometry technology offers advantages such as high precision, high speed, and compatibility with complex shapes and large components. Therefore, it is widely used in optical manufacturing and optical inspection. However, in the automatic measurement scenario of deflectometry, automatically identifying transparent components and removing background is a difficult problem to solve.

[0003] Traditional image segmentation techniques are mostly based on edge detection, such as the Canny edge detector and the Sobel operator, which achieve segmentation by identifying edges within an image. These methods are effective for simple edge detection, but often struggle to accurately segment objects in complex backgrounds. Neural networks, with their superior ability to handle nonlinear problems and adaptability, are widely used in image segmentation.

[0004] The fully convolutional neural network (FCN) proposed by Long et al. [Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431-3440).] first applied convolutional neural networks to pixel-level prediction tasks and was successfully applied to image segmentation. FCNs achieve end-to-end training by directly upsampling in the convolutional layer to generate segmentation results of the same size as the input image. U-Net, proposed by Ronneberger et al., was originally used for biomedical image segmentation. [Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234-241). Springer, Cham.] U-Net uses a symmetrical U-shaped structure and concatenates the encoder's feature map with the decoder's feature map via skip connections, effectively preserving high-resolution feature information. Mask R-CNN, proposed by He et al., combines object detection with segmentation tasks, simultaneously outputting object bounding boxes and segmentation masks. [He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017). Mask R-CNN. In Proceedings of the IEEE international conference on computer vision (pp. 2961-2969).] By adding a segmentation branch to Faster R-CNN, Mask R-CNN achieves excellent performance in instance segmentation tasks and is widely used in various vision tasks. The DeepLab series of models is a series of image segmentation methods proposed by Chen et al., which use dilated convolution to increase the receptive field of the convolution layer, thereby improving the ability to capture details of the segmentation results.[Chen, LC, Papandreou, G., Schroff, F., & Adam, H. (2017). Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587.] The latest version, DeepLab V3+, incorporates an encoder-decoder architecture to further improve segmentation accuracy. Because the deflectometry system must be built in a darkroom, current neural network segmentation models have low segmentation accuracy in dimly lit scenes. For transparent objects, the image quality is further aggravated, making it difficult to meet measurement requirements.

[0005] In summary, the following challenges remain in the field of deflectometry: Transparent component segmentation is difficult during automated measurement, often leaking background and causing algorithm failure. Directly using existing deep learning models is difficult to train and cannot meet the requirements for high-precision segmentation. Summary of the Invention

[0006] The purpose of the present invention is to provide a method for segmenting transparent optical components using deflectometry in low-light conditions. This method can accurately and quickly segment transparent optical components in low-light conditions, facilitating subsequent surface reconstruction using deflectometry and other methods. The technical solutions employed are:

[0007] A method for segmenting transparent optical components to assist deflectometry measurement in dark light scenes comprises the following steps:

[0008] Step S1, generate a single-frequency sinusoidal fringe image: select the frequency f of the fringe according to the experimental requirements, set the three-step phase-shifted sinusoidal fringe images I1, I2 and I3, and their intensity is expressed as:

[0009]

[0010] I2=A(x,y)+B(x,y)cos[φ(x,y)]

[0011]

[0012] Among them, A(x,y) is the background light intensity, B(x,t) is the modulated light intensity, is the phase to be determined;

[0013] Step S2: Under dark light conditions, a sinusoidal fringe image is collected based on the deflectometry system:

[0014] A projector is used to project sinusoidal fringe images I1, I2, and I3 with a frequency of f onto the transparent optical element to be tested, and a camera is used to capture the three-step phase-shifted sinusoidal fringe images I1, I2, and I3 after they are reflected by the transparent optical element under dark light conditions.

[0015] Step S3: Acquire phase image:

[0016] The camera uses a phase shift algorithm to process the captured three-step phase-shifted sinusoidal fringe images I1, I2, and I3 to solve And output the corresponding phase image;

[0017]

[0018] Step S4, changing the position or shape of the transparent optical element and executing steps S2 to S3;

[0019] Step S5: Obtain a data set and train a neural network model;

[0020] The data set acquisition step includes: manually annotating the phase image to obtain a mask image of the transparent optical element, and constructing a data set using the phase image and the corresponding mask image of the transparent optical element;

[0021] The input of the neural network model is the phase image, and the output is the mask image MASK0 of the transparent optical element;

[0022] Step S6, executing steps S1 to S3 to obtain a phase image of the transparent optical element to be tested;

[0023] Step S7: inputting the phase image of the transparent optical element to be tested outputted in step S6 into the trained neural network model to obtain a segmentation result, i.e., a mask image MASK1 of the transparent optical element to be tested;

[0024] Step S8: Multiply MASK1 and the original fringe image pixel by pixel for subsequent deflectometry surface measurement.

[0025] Preferably, in the neural network model:

[0026] The input image passes through the 3*3 convolution module to generate 32 feature maps, which are then downsampled twice using the maximum pooling method. After downsampling, each image passes through the 3*3 convolution module twice.

[0027] Then, each feature map of different scales passes through the feature enhancement module and performs two 3*3 convolution module operations. The obtained feature map is upsampled by transposed convolution and concatenated with the feature map of the previous scale.

[0028] Then through 3*3 convolution module, 1*1 convolution module and four residual modules, finally two times of 2*2 transpose convolution and one time of 1*1 convolution are carried out to output single-channel mask image;

[0029] The 3*3 convolution module is composed of 3*3 convolution, batch normalization and Relu non-linear activation;

[0030] The 1*1 convolution module is composed of 1*1 convolution, batch normalization and Relu non-linear activation;

[0031] The residual module is composed of 3*3 convolution, batch normalization, Relu activation layer and dropout random inactivation.

[0032] Preferably, the feature enhancement module firstly performs global pooling on the input feature map, then obtains a feature map weight Scale through two fully connected layers and a Sigmoid activation function, and finally performs channel-by-channel multiplication on the Scale and the input feature map to obtain the output.

[0033] Preferably, the Adam optimizer is used for optimization in step S5, and the learning rate is set to 0.001; during training, the cross-entropy loss is used as the loss function.

[0034] Preferably, the deflection measurement system in step S2 comprises a camera, a projector and a transparent optical element to be measured.

[0035] The transparent optical element to be measured is located in the field of view of the camera, so that the camera can shoot the reflected image.

[0036] The projector projects a three-step phase-shifted sinusoidal fringe image onto the transparent optical element.

[0037] Preferably, the Zhang Zhengyou calibration method is used to calibrate the deflection measurement system.

[0038] Preferably, the transparent optical element in step S4 is a plano-convex lens.

[0039] Compared with the prior art, the present application has the following advantages:

[0040] The present application is different from the method of using target detection for transparent element segmentation. The proposed method uses a three-step phase-shifted phase diagram as the input of the neural network, which well overcomes the problem of low reflectivity of the transparent optical element in the dark scene and is difficult to segment. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The method flowchart of the present application is shown;

[0042] Figure 2 The main body of the neural network structure of the present application is shown;

[0043] Figure 3 It is the feature enhancement module structure of the present invention;

[0044] Figure 4 This is the sinusoidal fringe image of a transparent optical element taken by a camera;

[0045] Figure 5 Phase image of a transparent optical element obtained by three-step phase shift.

[0046] Figure 6 Binary mask image of the optical component during manual annotation. DETAILED DESCRIPTION

[0047] The following, with reference to schematic diagrams, provides a more detailed description of an example method for segmenting a transparent optical element to assist deflectometry measurements in dark light scenarios. This method illustrates a preferred embodiment of the present invention. It should be understood that those skilled in the art may modify the present invention described herein while still achieving its advantageous effects. Therefore, the following description should be understood as a general guide for those skilled in the art and not as a limitation of the present invention.

[0048] like Figures 1 to 5 A method for instance segmentation of transparent optical components for assisting deflectometry measurement in dark light scenes is performed in a darkroom and includes the following steps:

[0049] Step S0: Building a deflection measurement system.

[0050] The deflectometry system includes: a camera, a projector, a transparent optical element to be measured, and a computer.

[0051] The transparent optical component to be tested is located in the camera's field of view so that the camera can capture the image reflected by it;

[0052] The projector projects a three-step phase-shifted sinusoidal fringe image onto a transparent optical element.

[0053] A computer that performs neural network training.

[0054] The deflectometry system is calibrated using the Zhang Zhengyou calibration method.

[0055] Step S1, generate a single-frequency sinusoidal fringe image: select the frequency f of the fringe according to the experimental requirements, set the three-step phase-shifted sinusoidal fringe images I1, I2 and I3, and their intensity is expressed as:

[0056]

[0057] I2=A(x,y)+B(x,y)cos[φ(x,y)]

[0058]

[0059] Among them, A(x,y) is the background light intensity, B(x,t) is the modulated light intensity, is the phase to be determined;

[0060] x represents the horizontal coordinate of the pixel position of the display; y represents the vertical coordinate of the pixel position of the display;

[0061] Step S2: Under dark light conditions, a sinusoidal fringe image is collected based on a deflectometry system.

[0062] A projector is used to project sinusoidal fringe images I1, I2 and I3 with a frequency of f onto the transparent optical element to be tested, and a camera is used to capture the three-step phase-shifted sinusoidal fringe images I1, I2 and I3 after being reflected by the transparent optical element under dark light conditions.

[0063] Phase deflectometry measurement systems require measurements in near-no-light conditions. Dark light conditions are those where all external light sources are minimized, with the screen being the only source of light. In this case, ambient illumination is typically below 0.1 lux.

[0064] Step S3: Acquire a phase image.

[0065] The camera uses a phase shift algorithm to process the captured three-step phase-shifted sinusoidal fringe images I1, I2, and I3 to solve And output the corresponding phase image;

[0066]

[0067] According to the description in step S1, the specific steps for solving φ(x,y) are:

[0068]

[0069] Therefore:

[0070]

[0071] Step S4: Change the position or shape of the transparent optical element and execute steps S2 to S3. The number of phase images finally acquired is ≥100.

[0072] That is, the dataset in step S5 should contain a variety of transparent optical components with different shapes and positions to enhance the generalization ability of the model.

[0073] In this embodiment, the transparent optical element is a plano-convex lens. Figure 5 In the figure, the left picture shows a plano-convex cylindrical lens, and the right picture shows a plano-convex spherical lens.

[0074] Step S5: Obtain a data set and train a neural network model;

[0075] The data set acquisition step is: manually annotating the phase image to obtain the mask image of the transparent optical element, such as Figure 6 As shown. The data set is constructed using the phase image and the corresponding mask image of the transparent optical element;

[0076] The input of the neural network model is the phase image, and the output is the mask image MASK0 of the transparent optical element.

[0077] The light reflected from the front and back surfaces of the transparent component can obtain a relatively flat phase, while the background and tool fixture do not reflect light or the reflected light signal is extremely weak, resulting in a significant contrast between the phase information and the transparent component. This can be segmented using a neural network and achieve good segmentation results.

[0078] In step S5, the Adam optimizer is used for optimization, and the learning rate is set to 0.001; during training, the loss function adopts the cross entropy loss.

[0079]

[0080] Among them, y ic is an indicator function, indicating which category the element belongs to; p ic is the predicted probability, the predicted probability that the observed sample belongs to category c; N is the number of image pixels.

[0081] like Figures 2-3 ,The main body of the neural network model consists of a U-shaped structure and an ,enhancement module.

[0082] The input image passes through the 3*3 convolution module to generate 32 feature maps, which are then downsampled twice using the maximum pooling method. After downsampling, each image passes through the 3*3 convolution module twice.

[0083] Then, each feature map of different scales passes through the feature enhancement module and performs two 3*3 convolution module operations. The obtained feature map is upsampled by transposed convolution and concatenated with the feature map of the previous scale.

[0084] Then, by sequentially passing through the 3*3 convolution module, the 1*1 convolution module and the four residual modules, two 2*2 transposed convolutions and one 1*1 convolution are performed to output a single-channel mask image;

[0085] Among them, the 3*3 convolution module consists of 3*3 convolution, batch normalization and Relu nonlinear activation;

[0086] The 1*1 convolution module consists of 1*1 convolution, batch normalization and Relu nonlinear activation;

[0087] The residual module consists of a 3*3 convolution, batch normalization, a ReLU activation layer, and dropout random inactivation.

[0088] The feature enhancement module first performs global pooling on the input feature map, then obtains the feature map weight Scale through two fully connected layers and a Sigmoid activation function, and finally multiplies the Scale by the input feature map channel by channel to obtain the output.

[0089] Step S6, executing steps S1 to S3 to obtain a phase image of the transparent optical element to be tested.

[0090] Step S7: inputting the phase image of the transparent optical element to be tested outputted in step S6 into the trained neural network model to obtain a segmentation result, i.e., a mask image MASK1 of the transparent optical element to be tested;

[0091] The segmentation results, i.e. the output of the neural network model, can accurately locate and segment the edges and contours of transparent optical components. The segmentation results are similar to Figure 6 .

[0092] Step S8: multiplying MASK1 by the original fringe image pixel by pixel for subsequent deflectometry-based surface shape measurement.

[0093] The above description is merely a preferred embodiment of the present invention and does not limit the present invention in any way. Any person skilled in the art who, without departing from the scope of the present invention, makes any equivalent substitution, modification, or other changes to the technical solution and technical content disclosed in the present invention shall be deemed to be within the scope of the present invention and still fall within the scope of protection of the present invention.

Claims

1. A method for segmenting transparent optical components using deflectometry in dark light scenes, characterized in that: The following steps are involved: Step S1, generate a single-frequency sinusoidal fringe image: select the frequency f of the fringe according to the experimental requirements, set the three-step phase-shifted sinusoidal fringe images I1, I2 and I3, and their intensity is expressed as: I2=A(x,y)+B(x,y)cos[φ(x,y)] Among them, A(x,y) is the background light intensity, B(x,y) is the modulated light intensity, is the phase to be determined; Step S2: Under dark light conditions, a sinusoidal fringe image is collected based on the deflectometry system: A projector is used to project sinusoidal fringe images I1, I2, and I3 with a frequency of f onto the transparent optical element to be tested, and a camera is used to capture the three-step phase-shifted sinusoidal fringe images I1, I2, and I3 after they are reflected by the transparent optical element under dark light conditions. Step S3: Acquire phase image: The camera uses a phase shift algorithm to process the captured three-step phase-shifted sinusoidal fringe images I1, I2, and I3 to solve And output the corresponding phase image; Step S4, changing the position or shape of the transparent optical element and executing steps S2 to S3; Step S5: Obtain a data set and train a neural network model; The data set acquisition step includes: manually annotating the phase image to obtain a mask image of the transparent optical element, and constructing a data set using the phase image and the corresponding mask image of the transparent optical element; The input of the neural network model is the phase image, and the output is the mask image MASK0 of the transparent optical element; Step S6, executing steps S1 to S3 to obtain a phase image of the transparent optical element to be tested; Step S7: inputting the phase image of the transparent optical element to be tested outputted in step S6 into the trained neural network model to obtain a segmentation result, i.e., a mask image MASK1 of the transparent optical element to be tested; Step S8: Multiply MASK1 and the original fringe image pixel by pixel for subsequent deflectometry surface measurement.

2. The method for segmenting transparent optical components using deflectometry in dark light scenarios according to claim 1, wherein: In the neural network model: The input image passes through the 3*3 convolution module to generate 32 feature maps, which are then downsampled twice using the maximum pooling method. After downsampling, each image passes through the 3*3 convolution module twice. Then, each feature map of different scales passes through the feature enhancement module and performs two 3*3 convolution module operations. The obtained feature map is upsampled by transposed convolution and concatenated with the feature map of the previous scale. Then, by sequentially passing through the 3*3 convolution module, the 1*1 convolution module and the four residual modules, two 2*2 transposed convolutions and one 1*1 convolution are performed to output a single-channel mask image; Among them, the 3*3 convolution module consists of 3*3 convolution, batch normalization and Relu nonlinear activation; The 1*1 convolution module consists of 1*1 convolution, batch normalization and Relu nonlinear activation; The residual module consists of a 3*3 convolution, batch normalization, a ReLU activation layer, and dropout random inactivation.

3. The method for segmenting transparent optical components using deflectometry in dark light scenarios according to claim 2, wherein: The feature enhancement module first performs global pooling on the input feature map, then obtains the feature map weight Scale through two fully connected layers and a Sigmoid activation function, and finally multiplies the Scale by the input feature map channel by channel to obtain the output.

4. The method for segmenting transparent optical components using deflectometry in dark light scenarios according to claim 1, wherein: In step S5, the Adam optimizer is used for optimization, and the learning rate is set to 0.001; during training, the loss function adopts the cross entropy loss.

5. The method for segmenting transparent optical components using deflectometry in dark light scenarios according to claim 1, wherein: The deflection measurement system in step S2 includes: a camera, a projector, and a transparent optical element to be measured; The transparent optical component to be tested is located in the camera's field of view so that the camera can capture the image reflected by it; The projector projects a three-step phase-shifted sinusoidal fringe image onto a transparent optical element.

6. The method for segmenting transparent optical components using deflectometry in dark light scenarios according to claim 5, wherein: The deflectometry system is calibrated using the Zhang Zhengyou calibration method.

7. The method for segmenting transparent optical components using deflectometry in dark light scenarios according to claim 1, wherein: In step S4, the transparent optical element is a plano-convex lens.

Images