Transparent optical element instance segmentation method for assisting deflection measurement in dark light scene

Abstract

The invention provides a transparent optical element instance segmentation method for assisting deflection measurement in a dark light scene, and the method comprises the following steps: S1, generating a single-frequency sine stripe image: S2, collecting the sine stripe image based on a deflection measurement system under a dark light condition; s3, acquiring a phase image; s4, changing the position or shape of the transparent optical element, and executing the steps S2 to S3; s5, acquiring a data set, and training a neural network model; s6, executing the steps S1 to S3, and obtaining a phase image of the transparent optical element to be measured; step S7, inputting the phase image of the transparent optical element to be detected, which is output in the step S6, into the trained neural network model, and obtaining a segmentation result, namely a mask image MASK1 of the transparent optical element to be detected; and step S8, multiplying the MASK1 by the original fringe pattern pixel by pixel for subsequent deflection surface type measurement. According to the method, the transparent optical element instance segmentation can be accurately and quickly carried out in a dark light scene.

Description

Technical Field

[0001] The invention belongs to the technical field of optical measurement, and in particular relates to a transparent optical element instance segmentation method for assisting deflectometry measurement in a dark light scene. Background Art

[0002] Transparent components play an important role in all aspects of social production and life. In the field of optics, transparent components such as lenses, filters and prisms are widely used to manufacture various optical instruments and equipment, such as cameras, telescopes, VR / AR and lasers. In addition, transparent components are also widely used in industries such as automobiles, electronics, aerospace and medical devices. For example, high-precision transparent components such as objective lenses and eyepieces are key components of high-end cameras and projectors. Transparent components used in instruments such as spectrometers and laser interferometers help scientists accurately measure and analyze the characteristics of light. In the field of aerospace, the windshield (windshield cover) of the aircraft cockpit needs to have high strength and excellent transparency to provide pilots with a good field of vision while resisting high pressure, wind speed and temperature changes during flight. Windows on passenger aircraft and spacecraft are required to be transparent, strong and able to withstand extreme environmental conditions, and their surface shape is very important for assembly. In summary, transparent components are an indispensable part of modern industry and scientific research. The surface accuracy of transparent components is crucial to improving the performance and stability of these devices. Deflectometry technology has the advantages of high precision, fast speed, and applicability to complex shapes and large-size components. Therefore, it is widely used in optical manufacturing and optical inspection. However, in the automatic measurement scenario of deflectometry, it is a difficult problem to automatically identify transparent components and remove the background.

[0003] Traditional image segmentation techniques are mostly based on edge detection, such as the Canny edge detector and the Sobel operator, which achieve segmentation by identifying the edges in the image. These methods work well for simple edge recognition, but it is often difficult to accurately segment objects in complex backgrounds. Neural networks are widely used in image segmentation due to their superior ability to handle nonlinear problems and their adaptability.

[0004] The Fully Convolutional Network (FCN) proposed by Long et al. [Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431-3440).] first used convolutional neural networks for pixel-level prediction tasks and was successfully applied to the field of image segmentation. FCN achieves end-to-end training by directly upsampling in the convolutional layer to generate segmentation results of the same size as the input image. U-Net was proposed by Ronneberger et al. and was originally used for biomedical image segmentation. [Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234-241). Springer, Cham.] U-Net uses a symmetrical U-shaped structure, and concatenates the feature map of the encoder with the feature map of the decoder through skip connections, effectively retaining high-resolution feature information. Mask R-CNN proposed by He et al. combines target detection with segmentation tasks, and can simultaneously output the bounding box and segmentation mask of the object. [He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017). Mask R-CNN. In Proceedings of the IEEE international conference on computer vision (pp. 2961-2969).] By adding a segmentation branch on the basis of Faster R-CNN, Mask R-CNN performs well in instance segmentation tasks and is widely used in various visual tasks. The DeepLab series of models is a series of image segmentation methods proposed by Chen et al., which use dilated convolution to increase the receptive field of the convolution layer, thereby improving the ability to capture details of the segmentation results.[Chen, LC, Papandreou, G., Schroff, F., & Adam, H. (2017). Rethinking atrous convolution for semantic imagesegmentation. arXiv preprint arXiv: 1706.05587.] The latest version of DeepLab V3+ combines the encoder-decoder architecture to further improve the segmentation accuracy. Since the deflectometry system needs to be built in a dark room, the current neural network segmentation model has low segmentation accuracy in dark light scenes, and for transparent objects, the image is further aggravated, making it difficult to meet measurement requirements.

[0005] In summary, the following problems remain to be solved in the field of deflectometry: In the process of automated measurement, transparent components are difficult to segment, and the background often leaks, causing the algorithm to crash. Directly using existing deep learning models is difficult to train and cannot meet the requirements of high-precision segmentation. Summary of the invention

[0006] The purpose of the present invention is to provide a method for segmenting transparent optical components in dim light scenes using assisted deflectometry measurement, which can accurately and quickly segment transparent optical components in dim light scenes, and provide the possibility for subsequent reconstruction of transparent component surface using deflectometry and other methods. The technical solution adopted is:

[0007] A method for segmenting transparent optical components in a dark light scene to assist deflectometry measurement comprises the following steps:

[0008] Step S1, generate a single-frequency sinusoidal fringe image: select the frequency f of the fringe according to the experimental requirements, set the three-step phase-shifted sinusoidal fringe images I1, I2 and I3, and its intensity is expressed as:

[0009]

[0010] I2=A(x,y)+B(x,y)cos[φ(x,y)]

[0011]

[0012] Among them, A(x,y) is the background light intensity, B(x,t) is the modulated light intensity, is the phase to be sought;

[0013] Step S2: Under dark light conditions, a sinusoidal fringe image is collected based on a deflectometry system:

[0014] A projector is used to project sinusoidal fringe images I1, I2 and I3 with a frequency of f onto the transparent optical element to be tested, and a camera is used to collect the three-step phase-shifted sinusoidal fringe images I1, I2 and I3 after being reflected by the transparent optical element under dark light conditions;

[0015] Step S3, obtaining a phase image:

[0016] The camera uses a phase shift algorithm to process the captured three-step phase-shifted sinusoidal fringe images I1, I2, and I3 to solve And output the corresponding phase image;

[0017]

[0018] Step S4, changing the position or shape of the transparent optical element, and executing steps S2 to S3;

[0019] Step S5, obtaining a data set and training a neural network model;

[0020] The data set acquisition step includes: manually annotating the phase image to obtain a mask image of the transparent optical element, and constructing a data set using the phase image and the corresponding mask image of the transparent optical element;

[0021] The input of the neural network model is the phase image, and the output is the mask image MASK0 of the transparent optical element;

[0022] Step S6, executing steps S1 to S3 to obtain a phase image of the transparent optical element to be tested;

[0023] Step S7, inputting the phase image of the transparent optical element to be tested outputted in step S6 into the trained neural network model to obtain a segmentation result, i.e., a mask image MASK1 of the transparent optical element to be tested;

[0024] Step S8: multiplying MASK1 by the original fringe image pixel by pixel for subsequent deflectometry surface measurement.

[0025] Preferably, in the neural network model:

[0026] The input image is passed through a 3*3 convolution module to generate 32 feature maps, and then the image is downsampled twice using the maximum pooling method. After downsampling, the image is passed through a 3*3 convolution module twice.

[0027] Then, each feature map of different scales is subjected to two 3*3 convolution module operations after passing through the feature enhancement module, and the obtained feature map is concatenated with the feature map of the previous scale through transposed convolution upsampling;

[0028] Then, by sequentially passing through a 3*3 convolution module, a 1*1 convolution module, and four residual modules, two 2*2 transposed convolutions and one 1*1 convolution are performed to output a single-channel mask image;

[0029] Among them, the 3*3 convolution module consists of 3*3 convolution, batch normalization and Relu nonlinear activation;

[0030] The 1*1 convolution module consists of 1*1 convolution, batch normalization and Relu nonlinear activation;

[0031] The residual module consists of a 3*3 convolution, batch normalization, a Relu activation layer, and dropout random inactivation.

[0032] Preferably, the feature enhancement module first performs global pooling on the input feature map, then obtains the feature map weight Scale through two fully connected layers and a Sigmoid activation function, and finally multiplies the Scale by the input feature map channel by channel to obtain the output.

[0033] Preferably, in step S5, Adam optimizer is used for optimization, and the learning rate is set to 0.001; during training, the loss function adopts cross entropy loss.

[0034] Preferably, the deflection measurement system in step S2 comprises: a camera, a projector and a transparent optical element to be measured;

[0035] The transparent optical element to be tested is located in the camera's field of view, so that the camera can capture the image reflected by it;

[0036] The projector projects a three-step phase-shifted sinusoidal fringe image onto a transparent optical element.

[0037] Preferably, the deflection measurement system is calibrated using Zhang Zhengyou calibration method.

[0038] Preferably, in step S4, the transparent optical element is a plano-convex lens.

[0039] Compared with the prior art, the advantages of the present invention are:

[0040] The present invention is different from the method of using target detection to segment transparent originals. The proposed method uses the phase map of three-step phase shift as the input of the neural network, which well overcomes the problem that transparent optical elements are difficult to segment due to low reflectivity in dark light scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of the method of the present invention;

[0042] Figure 2 It is the main body of the neural network structure of the present invention;

[0043] Figure 3 It is the feature enhancement module structure in the present invention;

[0044] Figure 4 This is the sinusoidal fringe image of the transparent optical element taken by the camera;

[0045] Figure 5 Phase image of a transparent optical element obtained by three-step phase shift.

[0046] Figure 6 Binary mask image of optical component during manual annotation. DETAILED DESCRIPTION

[0047] The following is a more detailed description of the transparent optical element segmentation method for assisting deflectometry measurement in a dark light scene of the present invention in conjunction with a schematic diagram, wherein a preferred embodiment of the present invention is shown, and it should be understood that those skilled in the art can modify the present invention described herein while still achieving the advantageous effects of the present invention. Therefore, the following description should be understood as being widely known to those skilled in the art, and not as a limitation of the present invention.

[0048] like Figures 1 to 5 , a transparent optical element instance segmentation method for assisting deflectometry measurement in dark light scenes, performed in a dark room, comprising the following steps:

[0049] Step S0: construct a deflection measurement system.

[0050] The deflectometry system includes: a camera, a projector, a transparent optical element to be measured, and a computer.

[0051] The transparent optical element to be tested is located in the camera's field of view, so that the camera can capture the image reflected by it;

[0052] The projector projects a three-step phase-shifted sinusoidal fringe image onto a transparent optical element.

[0053] Computers are used to train neural networks.

[0054] The deflection measurement system is calibrated using Zhang Zhengyou calibration method.

[0055] Step S1, generate a single-frequency sinusoidal fringe image: select the frequency f of the fringe according to the experimental requirements, set the three-step phase-shifted sinusoidal fringe images I1, I2 and I3, and its intensity is expressed as:

[0056]

[0057] I2=A(x,y)+B(x,y)cos[φ(x,y)]

[0058]

[0059] Among them, A(x,y) is the background light intensity, B(x,t) is the modulated light intensity, is the phase to be sought;

[0060] x represents the horizontal coordinate of the pixel position of the display; y represents the vertical coordinate of the pixel position of the display;

[0061] Step S2: Under dark light conditions, a sinusoidal fringe image is collected based on a deflection measurement system.

[0062] A projector is used to project sinusoidal fringe images I1, I2 and I3 with a frequency of f onto the transparent optical element to be tested. A camera is used to collect the three-step phase-shifted sinusoidal fringe images I1, I2 and I3 after being reflected by the transparent optical element under dark light conditions.

[0063] Since the phase deflection measurement system needs to be measured under nearly no light conditions, dark light conditions refer to the situation where all external light sources are isolated as much as possible, and the only light source comes from the screen. At this time, the ambient illumination is usually below 0.1 lux.

[0064] Step S3: Acquire a phase image.

[0065] The camera uses a phase shift algorithm to process the captured three-step phase-shifted sinusoidal fringe images I1, I2, and I3 to solve And output the corresponding phase image;

[0066]

[0067] According to the description in step S1, the specific solution steps for φ(x,y) are:

[0068]

[0069] Therefore, there are:

[0070]

[0071] Step S4, changing the position or shape of the transparent optical element, and executing steps S2 to S3. The number of phase images finally acquired is ≥ 100.

[0072] That is, the data set in step S5 should contain a variety of transparent optical elements of different shapes and positions to enhance the generalization ability of the model.

[0073] In this embodiment, the transparent optical element is a plano-convex lens. Figure 5 In the figure, the left picture shows a plano-convex cylindrical lens, and the right picture shows a plano-convex spherical lens.

[0074] Step S5, obtaining a data set and training a neural network model;

[0075] The data set acquisition step is: manually annotating the phase image to obtain the mask image of the transparent optical element, such as Figure 6 As shown. A data set is constructed using the phase image and the corresponding mask image of the transparent optical element;

[0076] The input of the neural network model is the phase image, and the output is the mask image MASK0 of the transparent optical element.

[0077] The light reflected from the front and back surfaces of the transparent element can obtain a relatively flat phase, while the background and tool fixture do not reflect light or the reflected light signal is extremely weak, resulting in a significant contrast between the phase information and the transparent element. This can be segmented through a neural network and achieve good segmentation effects.

[0078] In step S5, the Adam optimizer is used for optimization, and the learning rate is set to 0.001; during training, the loss function adopts the cross entropy loss.

[0079]

[0080] Among them, y ic is an indicative function, indicating which category the element belongs to; p ic is the predicted probability, the predicted probability that the observed sample belongs to category c; N is the number of image pixels.

[0081] like Figures 2-3 ,The main body of the neural network model consists of a U-shaped structure and an ,enhancement module.

[0082] The input image is passed through a 3*3 convolution module to generate 32 feature maps, and then the image is downsampled twice using the maximum pooling method. After downsampling, the image is passed through a 3*3 convolution module twice.

[0083] Then, each feature map of different scales is subjected to two 3*3 convolution module operations after passing through the feature enhancement module, and the obtained feature map is concatenated with the feature map of the previous scale through transposed convolution upsampling;

[0084] Then, by sequentially passing through a 3*3 convolution module, a 1*1 convolution module, and four residual modules, two 2*2 transposed convolutions and one 1*1 convolution are performed to output a single-channel mask image;

[0085] Among them, the 3*3 convolution module consists of 3*3 convolution, batch normalization and Relu nonlinear activation;

[0086] The 1*1 convolution module consists of 1*1 convolution, batch normalization and Relu nonlinear activation;

[0087] The residual module consists of a 3*3 convolution, batch normalization, a Relu activation layer, and dropout random inactivation.

[0088] The feature enhancement module first performs global pooling on the input feature map, then obtains the feature map weight Scale through two fully connected layers and a Sigmoid activation function, and finally multiplies the Scale by the input feature map channel by channel to obtain the output.

[0089] Step S6, executing steps S1 to S3 to obtain a phase image of the transparent optical element to be tested.

[0090] Step S7, inputting the phase image of the transparent optical element to be tested outputted in step S6 into the trained neural network model to obtain a segmentation result, i.e., a mask image MASK1 of the transparent optical element to be tested;

[0091] The segmentation results, i.e. the output of the neural network model, can accurately locate and segment the edges and contours of transparent optical components. The segmentation results are similar to Figure 6 .

[0092] Step S8: multiplying MASK1 by the original fringe image pixel by pixel for subsequent surface shape measurement based on deflectometry.

[0093] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A method for segmenting transparent optical element instances in dark light scenes to assist deflectometry measurement, characterized in that: The following steps are involved: Step S1, generate a single-frequency sinusoidal fringe image: select the frequency f of the fringe according to the experimental requirements, set the three-step phase-shifted sinusoidal fringe images I1, I2 and I3, and its intensity is expressed as: I2=A(x,y)+B(x,y)cos[φ(x,y)] Among them, A(x,y) is the background light intensity, B(x,y) is the modulated light intensity, is the phase to be sought; Step S2: Under dark light conditions, a sinusoidal fringe image is collected based on a deflectometry system: A projector is used to project sinusoidal fringe images I1, I2 and I3 with a frequency of f onto the transparent optical element to be tested, and a camera is used to collect the three-step phase-shifted sinusoidal fringe images I1, I2 and I3 after being reflected by the transparent optical element under dark light conditions; Step S3, obtaining a phase image: The camera uses a phase shift algorithm to process the captured three-step phase-shifted sinusoidal fringe images I1, I2, and I3 to solve And output the corresponding phase image; Step S4, changing the position or shape of the transparent optical element, and executing steps S2 to S3; Step S5, obtaining a data set and training a neural network model; The data set acquisition step includes: manually annotating the phase image to obtain a mask image of the transparent optical element, and constructing a data set using the phase image and the corresponding mask image of the transparent optical element; The input of the neural network model is the phase image, and the output is the mask image MASK0 of the transparent optical element; Step S6, executing steps S1 to S3 to obtain a phase image of the transparent optical element to be tested; Step S7, inputting the phase image of the transparent optical element to be tested outputted in step S6 into the trained neural network model to obtain a segmentation result, namely, a mask image MASK1 of the transparent optical element to be tested; Step S8: multiplying MASK1 by the original fringe image pixel by pixel for subsequent deflectometry surface measurement.

2. The method for segmenting transparent optical element instances in dark light scene-assisted deflectometry measurement according to claim 1, characterized in that: In the neural network model: The input image is passed through a 3*3 convolution module to generate 32 feature maps, which are then downsampled twice using the maximum pooling method. After downsampling, the image is passed through a 3*3 convolution module twice. Then, each feature map of different scales is subjected to two 3*3 convolution module operations after passing through the feature enhancement module, and the obtained feature map is concatenated with the feature map of the previous scale through transposed convolution upsampling; Then, by sequentially passing through a 3*3 convolution module, a 1*1 convolution module, and four residual modules, two 2*2 transposed convolutions and one 1*1 convolution are performed to output a single-channel mask image; Among them, the 3*3 convolution module consists of 3*3 convolution, batch normalization and Relu nonlinear activation; The 1*1 convolution module consists of 1*1 convolution, batch normalization and Relu nonlinear activation; The residual module consists of a 3*3 convolution, batch normalization, a Relu activation layer, and dropout random inactivation.

3. The method for segmenting transparent optical element instances in dark light scene-assisted deflectometry measurement according to claim 2, characterized in that: The feature enhancement module first performs global pooling on the input feature map, then obtains the feature map weight Scale through two fully connected layers and a Sigmoid activation function, and finally multiplies the Scale by the input feature map channel by channel to obtain the output.

4. The method for segmenting transparent optical element instances in dark light scene-assisted deflectometry measurement according to claim 1, characterized in that: In step S5, the Adam optimizer is used for optimization, and the learning rate is set to 0.001; during training, the loss function adopts the cross entropy loss.

5. The method for segmenting transparent optical element instances in dark light scene-assisted deflectometry measurement according to claim 1, characterized in that: The deflection measurement system in step S2 includes: a camera, a projector and a transparent optical element to be measured; The transparent optical element to be tested is located in the camera's field of view, so that the camera can capture the image reflected by it; The projector projects a three-step phase-shifted sinusoidal fringe image onto a transparent optical element.

6. The method for segmenting transparent optical element instances in dark light scene-assisted deflectometry measurement according to claim 5, characterized in that: The deflection measurement system is calibrated using Zhang Zhengyou calibration method.

7. The method for segmenting transparent optical element instances in dark light scene-assisted deflectometry measurement according to claim 1, characterized in that: In step S4, the transparent optical element is a plano-convex lens.

Images