Optical imaging system quality evaluation method based on multi-task deep learning

Abstract

The present application belongs to the technical field of optical imaging system quality evaluation, and particularly relates to an optical imaging system quality evaluation method based on multi-task deep learning. The method comprises the following steps: S1: constructing a training data set based on the quality values of different optical imaging systems; S2: constructing a multi-task deep learning model, wherein the multi-task deep learning model comprises a shared encoder and a multi-task decoder; S3: training the multi-task deep learning model using the training data set to obtain a trained multi-task deep learning model; and S4: inputting an image captured by a to-be-detected optical imaging system into the trained multi-task deep learning model for processing to obtain a quality evaluation value of the to-be-detected optical imaging system. The present application greatly simplifies the test process, improves the test efficiency, and provides more accurate and reliable technical support for performance evaluation of optical systems.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of optical imaging system quality evaluation, and particularly relates to an optical imaging system quality evaluation method based on multi-task deep learning. BACKGROUND

[0002] The quality of an optical imaging system is a core factor determining the performance of the optical imaging system, and the quality is usually measured by a series of precise optical indicators, such as a modulation transfer function (MTF) (used to evaluate the transmission ability of an optical system to different spatial frequency signals), distortion (used to evaluate the geometric deformation degree of an image), field curvature (used to evaluate the bending degree of an image plane), and astigmatism (used to evaluate the directional blur degree caused by the focusing difference of a light beam in the meridian and sagittal directions). However, the traditional detection method relies on special hardware equipment (such as an MTF measuring instrument, a distortion test bench) and complex software algorithms. An operator needs to precisely install the optical imaging system to be measured on the corresponding equipment, collect images of a standard calibration plate or a high-precision target (star point, slit, knife-edge target), and then calculate each indicator through a series of digital image processing algorithms (such as edge detection, line spread function calculation, Fourier transform, etc.). The overall test process is complicated and inefficient, and each measurement can only obtain a single indicator, while comprehensive detection requires frequent equipment replacement or configuration adjustment, greatly increasing the time and labor cost; at the same time, the test process has extremely harsh requirements on the running environment, not only relying on a stable optical platform and precise calibration and adjustment, but also being highly sensitive to external disturbances such as vibration and illumination.

[0003] In addition, the detection process of the traditional optical imaging system has many limitations:

[0004] (1) Complex process and low efficiency: each measurement can only target a single indicator, and comprehensive detection requires equipment replacement or parameter configuration adjustment, which is time-consuming and labor-intensive;

[0005] (2) Strict environmental requirements: a stable optical platform and strict calibration are required, and the process is sensitive to environmental vibration, illumination and other disturbances;

[0006] (3) Strong dependence on professional knowledge: operation and result interpretation require professional technical personnel;

[0007] (4) Difficult to integrate into online detection: traditional test equipment is large in size and slow in speed, and it is difficult to integrate into a high-speed production line for 100% full inspection.

[0008] With the promotion of industrial 4.0 and intelligent manufacturing, higher speed, higher efficiency and online detection of optical imaging systems are required. SUMMARY

[0009] In view of this, the present application aims to provide a multi-task deep learning-based optical imaging system quality evaluation method to solve the problems of complex detection process, low efficiency, high environmental requirements and other issues of traditional optical imaging system quality detection, and break through the bottleneck of traditional optical imaging system detection technology. The present application directly learns the deep features related to the quality indicators of the optical imaging system from the images of the optical imaging system by training a deep learning model, and gives the evaluation results of multiple quality indicators at one time, greatly simplifies the test process, improves the test efficiency, and provides more accurate and reliable technical support for the performance evaluation of the optical system.

[0010] To achieve the above-mentioned purpose, the technical scheme of the present application is realized as follows:

[0011] A multi-task deep learning-based optical imaging system quality evaluation method, specifically comprising the following steps:

[0012] S1: Based on the quality values of different optical imaging systems, a training data set is constructed;

[0013] S2: A multi-task deep learning model is constructed, which includes a shared encoder and a multi-task decoder;

[0014] S3: The multi-task deep learning model is trained using the training data set to obtain a trained multi-task deep learning model;

[0015] S4: The image taken by the optical imaging system to be detected is input into the trained multi-task deep learning model for processing to obtain the quality evaluation value of the optical imaging system to be detected.

[0016] Further, step S1 specifically comprises the following steps:

[0017] S11: Use a high-precision optical imaging system index measurement device to collect images of optical imaging systems of different types and different quality levels under standard calibration plates or high-precision targets, and record the quality values of the corresponding optical imaging systems output by the high-precision optical imaging system index measurement device;

[0018] The quality values of the optical imaging system include MTF, distortion, field curvature and astigmatism;

[0019] S12: Based on the images and corresponding quality values collected by each optical imaging system, a training data set is constructed, and each training sample contained in the training data set is in the format of {image, [MTF, distortion, field curvature, astigmatism]}.

[0020] Further, in step S11, the high-precision optical imaging system index measurement device at least includes an MTF measuring instrument and a distortion analyzer.

[0021] Further, in step S2, the multi-task decoder includes an MTF decoder, a distortion decoder, a stigmation decoder and a field curvature decoder.

[0022] Further, in step S2, the shared encoder receives an image of a current training sample and encodes the image to obtain a feature map containing MTF information, distortion information, field curvature information and stigmation information;

[0023] The MTF decoder extracts the MTF information in the feature map and decodes the MTF information to obtain an MTF prediction evaluation value corresponding to the current image;

[0024] The distortion decoder extracts the distortion information in the feature map and decodes the distortion information to obtain a distortion prediction evaluation value corresponding to the current image;

[0025] The stigmation decoder extracts the stigmation information in the feature map and decodes the stigmation information to obtain a stigmation prediction evaluation value corresponding to the current image;

[0026] The field curvature decoder extracts the field curvature information in the feature map and decodes the field curvature information to obtain a field curvature prediction evaluation value corresponding to the current image.

[0027] Further, a weighted comprehensive loss function is used The multi-task deep learning model is trained:

[0028] ;

[0029] Wherein, task is the MTF decoder, the distortion decoder, the stigmation decoder or the field curvature decoder, T is a task set, is a weight coefficient of each task, is an MSE loss value, is an estimated value output by the current decoder, is a real value corresponding to the current decoder.

[0030] Further, the MSE loss function is:

[0031] ;

[0032] Wherein, is an MSE loss value, is an estimated value output by the current decoder, is a real value corresponding to the current decoder, task is the MTF decoder, the distortion decoder, the stigmation decoder or the field curvature decoder, N is the total number of training samples, i is the serial number of the training sample, is a real value, is an estimated value.

[0033] Further, in step S4, the quality evaluation values include an MTF prediction evaluation value, a distortion prediction evaluation value, a field curvature prediction evaluation value, and a coma prediction evaluation value.

[0034] Compared with the prior art, the application can achieve the following beneficial effects:

[0035] (1) The optical imaging system quality evaluation method based on multi-task deep learning provided by the application uses powerful end-to-end feature learning and non-linear mapping capabilities, learns deep features related to optical quality indicators from optical imaging system images directly through training of a multi-task deep learning model, and gives evaluation results of multiple indicators at one time, greatly simplifying the test process and improving the test efficiency.

[0036] (2) The optical imaging system quality evaluation method based on multi-task deep learning provided by the application uses an end-to-end evaluation method, outputs multiple optical imaging system quality indicators at one time, completes second-level detection, far exceeds the detection speed of traditional methods, and can realize online real-time detection.

[0037] (3) The optical imaging system quality evaluation method based on multi-task deep learning provided by the application does not need expensive and complex special measuring equipment, but only needs a target generating mechanism (light source + target plate), a set of parallel light collimation system and a standard image acquisition device, thereby reducing the hardware cost and system complexity.

[0038] (4) The optical imaging system quality evaluation method based on multi-task deep learning provided by the application has strong robustness, and the deep learning model has certain fault tolerance to slight environmental disturbances and mechanical adjustment errors through learning of a large amount of data.

[0039] (5) The optical imaging system quality evaluation method based on multi-task deep learning provided by the application is easy to integrate and automate, and the multi-task deep learning model can be easily deployed on an industrial computer or a cloud server, realizing seamless integration with a production line.

[0040] (6) The optical imaging system quality evaluation method based on multi-task deep learning provided by the application can mine deep quality information hidden in images through a deep learning model, and even find defects that are difficult to quantify by human eyes or traditional algorithms. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the present application and serve to explain the principles of the present application. In the drawings:

[0042] Figure 1A flowchart of a method for evaluating the quality of an optical imaging system based on multi-task deep learning according to an embodiment of the present invention is shown in FIG. 1.

[0043] Figure 2 An architecture diagram of a multi-task deep learning model according to an embodiment of the present invention is shown in FIG. 2.

[0044] Figure 3 A training flowchart of a multi-task deep learning model according to an embodiment of the present invention is shown in FIG. 3. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and do not constitute a limitation on the present invention.

[0046] It should be noted that the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.

[0047] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second" and the like are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited by "first", "second" and the like can explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0048] In the description of the present invention, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connection" should be understood broadly, for example, it can be a fixed connection, or a detachable connection, or an integral connection; it can be a mechanical connection, or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or a communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present invention can be understood according to the specific circumstances.

[0049] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0050] As Figure 1 shown, the present application provides a multi-task deep learning-based optical imaging system quality evaluation method, specifically comprising the following steps:

[0051] S1: Based on the quality values of different optical imaging systems, a training data set is constructed;

[0052] S2: A multi-task deep learning model is constructed, which includes a shared encoder and a multi-task decoder;

[0053] S3: The multi-task deep learning model is trained using the training data set to obtain a trained multi-task deep learning model;

[0054] S4: The image taken by the optical imaging system to be detected is input into the trained multi-task deep learning model for processing to obtain the quality evaluation value of the optical imaging system to be detected.

[0055] It should be noted that the present application innovates the traditional optical detection process. The traditional detection paradigm is a serial, multi-stage complex process that relies on manually designed algorithms to extract features and calculate indicators step by step, which is low in efficiency and poor in fault tolerance. The paradigm of the present application, however, constructs a highly integrated intelligent evaluation system. After inputting the original image taken by the optical imaging system into the multi-task deep learning model, all quality indicator results can be output in parallel at once, greatly simplifying the process, improving the efficiency and automation, and realizing true "end-to-end" intelligent evaluation.

[0056] In some embodiments, step S1 specifically comprises the following steps:

[0057] S11: Use a high-precision optical imaging system index measurement device to collect images of optical imaging systems of different models and different quality levels under standard calibration plates or high-precision targets, and record the quality values of the corresponding optical imaging systems output by the high-precision optical imaging system index measurement device;

[0058] The quality values of the optical imaging system include MTF, distortion, field curvature, and astigmatism;

[0059] S12: Based on the images and corresponding quality values collected by each optical imaging system, a training data set is constructed, and each training sample contained in the training data set is in the format of {image, [MTF, distortion, field curvature, astigmatism]}.

[0060] In some embodiments, in step S11, the high-precision optical imaging system index measurement device at least includes an MTF measuring instrument and a distortion analyzer.

[0061] It should be noted that (1) using the existing high-precision optical imaging system index measurement equipment in the laboratory (such as MTF measuring instrument and distortion analyzer), a large number of imaging images of optical imaging systems of different models and quality levels under standard plates or high-precision targets were collected; (2) the real optical imaging system quality index values ​​(MTF value, distortion rate, field curvature value, astigmatism value at different spatial frequencies) of the corresponding optical imaging system given by the high-precision optical imaging system index measurement equipment were recorded to form a massive, high-quality training dataset; (3) the training samples were labeled, each training sample containing one image and four labels, in the following format: {image, [MTF, distortion, field curvature, astigmatism]}.

[0062] In some embodiments, in step S2, the multi-task decoder includes an MTF decoder, a distortion decoder, an astigmatism decoder, and a field curvature decoder.

[0063] It should be noted that the multi-task deep learning model adopts a "shared encoder-multi-task decoder" architecture. The shared encoder acts as a general feature extractor, learning low-level representations from the image that are beneficial for all metrics; multiple specialized decoders (including MTF decoder, distortion decoder, astigmatism decoder, and field curvature decoder) work in parallel, each focusing on learning a refined mapping for a specific metric. The entire multi-task deep learning model can be viewed as a function F, which maps the input image X to a set of predicted evaluation values ​​as output, with the following functional relationship:

[0064] ;

[0065] Where W represents all parameters of the multi-task deep learning model. The predicted evaluation value representing MTF, The predicted value representing distortion. The predicted evaluation value representing the performance. F represents the predicted evaluation value for astigmatism, and F is the output function for the predicted evaluation value.

[0066] In some embodiments, such as Figure 2 As shown, in step S2, the shared encoder receives the image of the current training sample and encodes the image to obtain a feature map containing MTF information, distortion information, field curvature information and astigmatism information.

[0067] The MTF decoder extracts MTF information from the feature map and decodes the MTF information to obtain the MTF prediction evaluation value corresponding to the current image.

[0068] The distortion decoder extracts distortion information from the feature map and decodes the distortion information to obtain the distortion prediction evaluation value corresponding to the current image.

[0069] The astigmatism decoder extracts astigmatism information in the feature map and decodes the astigmatism information to obtain an astigmatism prediction evaluation value corresponding to the current image.

[0070] The curvature of field decoder extracts the curvature of field information in the feature map and decodes the curvature of field information to obtain a curvature of field prediction evaluation value corresponding to the current image.

[0071] It should be noted that the function E corresponding to the shared encoder converts the input image X into a high-dimensional feature tensor-feature map Z, and the function relationship is as follows:

[0072]

[0073] Where We is the parameter of the shared encoder, C is the number of image channels, and H and W are the height and width of the feature map.

[0074] The detailed construction scheme of the shared encoder is as follows:

[0075] (a) Load a 34-layer residual pre-trained on a dataset (ImageNet) in an open-source deep learning framework PyTorch.

[0076] (b) Remove the original network's last global average pooling layer (avgpool) and fully connected layer (fc), so that the output of the model is no longer a 1000-class classification score, but a feature map of the last convolutional layer.

[0077] The original network uses a 34-layer residual in the paper "Deep Residual Learning for Image Recognition" by Kaiming He et al. published in the journal "Proceedings of the IEEE conference on computer vision and pattern recognition" in 2016.

[0078] (c) Output feature map: [batch_size, 512, 7, 7].

[0079] Where: batch_size represents the number of samples obtained at one time, generally 32 or 64 according to the GPU condition; the model outputs 512 different "feature maps" of the required evaluation optical imaging system indicators, and the size of each feature map is 7x7.

[0080] In some embodiments, a weighted comprehensive loss function is used The multi-task deep learning model is trained as follows: ​​

[0081] ;

[0082] wherein task is an MTF decoder, a distortion decoder, a distortion decoder, or a field curvature decoder, T is a task set, is a weight coefficient of each task, is an MSE loss value, is an estimated value output by a current decoder, is a real value corresponding to the current decoder.

[0083] In some embodiments, the MSE loss function is:

[0084] ;

[0085] wherein, is an MSE loss value, is an estimated value output by a current decoder, is a real value corresponding to the current decoder, task is an MTF decoder, a distortion decoder, a distortion decoder, or a field curvature decoder, N is a total number of training samples, and i is a serial number of a training sample, is a real value, is an estimated value.

[0086] The function D of each decoder maps the shared feature map Z to the output of a specific task indicator, and the function relationship is as follows:

[0087] ;

[0088] wherein, is a parameter of a current decoder, is an indicator prediction evaluation value output by the decoder under the corresponding task, and task includes an MTF decoder, a distortion decoder, a distortion decoder, and a field curvature decoder.

[0089] The detailed construction scheme of the multi-task decoder is as follows:

[0090] Define structure: use nn.Sequential in PyTorch to construct a multi-task learning framework-Plain Network network containing a linear layer, an activation function, a Dropout, and a batch normalization;

[0091] Integrate into model: instantiate a regression head (MTF head, distortion head, field curvature head, and distortion head) for each regression task, and call it in the forward function;

[0092] The design of the regression head is improved based on the Plain Network network in the paper "U-net: Convolutional networks for biomedical image segmentation" by Ronneberger et al. published in 2015 in the journal "International Conference on Medical image computing and computer-assisted intervention. Cham: Springer international publishing". The output of the last convolutional layer of the Plain Network network is globally averaged pooled to obtain a 512-dimensional vector, and a fully connected layer is added to make the number of output layer nodes equal to the number of indicators to be regressed. The regression indicators are four (MTF, distortion, field curvature, and astigmatism), and the output layer is four nodes.

[0093] Selecting the loss function: Selecting mean square error (MSE) as the loss function, the function relationship is as follows:

[0094] ;

[0095] Where N is the size of the training sample (batch_size) of the task, is the true value, is the predicted value.

[0096] Constructing a weighted comprehensive loss function to train the multi-task deep learning model:

[0097]

[0098] Where T is the task set, including predicting and estimating MTF, distortion, field curvature, and astigmatism, is the weight coefficient of each task, generally taken in equal proportion.

[0099] Further, the present application proposes an innovative "shared encoder-multi-task decoder" architecture. The shared encoder serves as a general feature extractor, learning a bottom-up representation from images that is beneficial for all metrics; the multiple dedicated decoders work in parallel, each focusing on learning the fine mapping for a specific metric. The advantages of this architecture are: one stone, many birds (output all metric evaluation results at once, extremely efficient), thorough understanding (shared features make the model more robust and more general), and mutual reinforcement (related tasks promote each other through feature sharing, potentially improving overall performance).

[0100] As shown in Figure 3 , the training process of the multi-task deep learning model is as follows:

[0101] A1: Data set division: divide the training data set into training set, validation set and test set according to the number ratio of 8:1:1;

[0102] A2: Training parameter setting: generally set the batch size (Batch Size) to 32, the training period (Epoch) to 50, and the training times n to 0.

[0103] A3: Forward propagation: input the image data of the training set into the multi-task deep learning model in turn, and obtain the prediction output of all tasks correspondingly;

[0104] A4: Loss calculation: calculate the independent loss of each index prediction task according to the loss function (MSE), and weight sum the loss of each task according to the preset weight to obtain the comprehensive loss;

[0105] A5: Back propagation:

[0106] (a) Clear gradient: use optimizer.zero_grad() in PyTorch to clear the gradient to prevent gradient accumulation.

[0107] (b) Gradient calculation: use total_loss.backward() in PyTorch to automatically calculate the gradient of the loss function to all model parameters.

[0108] A6: Model parameter update: use optimizer.step() in PyTorch to update the model parameters according to the calculated gradient to reduce the total loss.

[0109] A7: Training termination judgment: when the condition n>Epoch is met, save the current model parameters, and the model training is completed; if the condition is not met, let n=n+1, and return to execute A3 forward propagation process.

[0110] Finally, for the quality evaluation of the optical imaging system: install the optical imaging system to be tested in a fixed, simplified optical imaging system (a target generating mechanism (light source + target plate), a set of parallel light collimation system and a standard image acquisition device), collect its image; input the image into the trained multi-task deep learning model, and the model can quickly and in parallel output all the trained quality evaluation indexes of the optical imaging system.

[0111] It should be understood that the various forms of flow shown above can be reordered, steps added or deleted. For example, the steps described in the present disclosure can be performed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which is not limited herein.

[0112] The above specific embodiments do not constitute a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A quality assessment method for optical imaging systems based on multi-task deep learning, characterized in that: Specifically, the steps include the following: S1: Construct a training dataset based on the quality values ​​of different optical imaging systems; Step S1 specifically includes the following steps: S11: Use high-precision optical imaging system index measurement equipment to acquire images of different models and quality levels of optical imaging systems under a standard target or high-precision target, and record the corresponding quality value of the optical imaging system output by the high-precision optical imaging system index measurement equipment. The quality values ​​of an optical imaging system include MTF, distortion, field curvature, and astigmatism; S12: Construct a training dataset based on the images acquired by each optical imaging system and their corresponding quality values. The format of each training sample in the training dataset is {image, [MTF, distortion, field curvature, astigmatism]}. S2: Construct a multi-task deep learning model, which includes a shared encoder and a multi-task decoder; In step S2, the multi-task decoder includes an MTF decoder, a distortion decoder, an astigmatism decoder, and a field curvature decoder. S3: Train the multi-task deep learning model using the training dataset to obtain a well-trained multi-task deep learning model; S4: Input the image captured by the optical imaging system to be tested into the trained multi-task deep learning model for processing to obtain the quality evaluation value of the optical imaging system to be tested.

2. The method for quality assessment of optical imaging systems based on multi-task deep learning according to claim 1, characterized in that: In step S11, the high-precision optical imaging system index measurement equipment includes at least an MTF measuring instrument and a distortion analyzer.

3. The method for quality assessment of optical imaging systems based on multi-task deep learning according to claim 1, characterized in that: In step S2, the shared encoder receives the image of the current training sample and encodes the image to obtain a feature map containing MTF information, distortion information, field curvature information and astigmatism information; The MTF decoder extracts MTF information from the feature map and decodes the MTF information to obtain the MTF prediction evaluation value corresponding to the current image. The distortion decoder extracts distortion information from the feature map and decodes the distortion information to obtain the distortion prediction evaluation value corresponding to the current image. The astigmatism decoder extracts astigmatism information from the feature map and decodes the astigmatism information to obtain the astigmatism prediction evaluation value corresponding to the current image. The field curvature decoder extracts field curvature information from the feature map and decodes the field area information to obtain the field curvature prediction evaluation value corresponding to the current image.

4. The method for quality assessment of optical imaging systems based on multi-task deep learning according to claim 1, characterized in that: Using a weighted comprehensive loss function Training a multi-task deep learning model: ； Where task is an MTF decoder, distortion decoder, astigmatism decoder, or field curvature decoder, and T is the set of tasks. The weighting coefficient for each task. This is the MSE loss value. This is an estimate of the current decoder output. This is the actual value corresponding to the current decoder.

5. The method for quality assessment of optical imaging systems based on multi-task deep learning according to claim 4, characterized in that: The MSE loss function is: ； in, This is the MSE loss value. This is an estimate of the current decoder output. This represents the true value corresponding to the current decoder. `task` can be an MTF decoder, distortion decoder, astigmatic decoder, or field curvature decoder. `N` represents the total number of training samples, and `i` is the index of the training sample. For the true value, This is an estimated value.

6. The method for quality assessment of optical imaging systems based on multi-task deep learning according to claim 1, characterized in that: In step S4, the quality assessment values ​​include MTF prediction assessment value, distortion prediction assessment value, field curvature prediction assessment value, and astigmatism prediction assessment value.

Images