Optical imaging system tolerance correction method and device based on neural network

Abstract

The invention discloses an optical imaging system tolerance correction method and device based on a neural network, and the method comprises the steps: building an optical imaging system degradation model under an assembly tolerance condition on the basis of considering a plurality of assembly working conditions; the optical imaging system degradation model is used for simulating the spatial change degradation process of the optical system under different assembly working conditions and generating a data set, each sample is an image pair composed of a clear image and a degraded image of a target, and the degraded image is marked with a working condition label; building a neural network model comprising a working condition estimation module and an image restoration module; and training the neural network model based on the data set and a preset loss function to obtain a trained working condition estimation-image restoration model which is used for restoring a clear image from the degraded image to be detected. The problem of imaging quality nonlinear degradation caused by eccentricity and inclination of the optical element in the assembling process is solved from the imaging end, the influence of assembling tolerance can be restrained, and the imaging quality is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of optical imaging, and specifically relates to a tolerance correction method and apparatus for an optical imaging system based on a neural network. Background Technology

[0002] With the rapid development of technologies such as drones, electric vehicles, remote sensing satellites, mobile phones, and virtual reality headsets, the market is placing higher demands on the application scope and imaging performance of camera modules. For traditional optical design methods, the increasing imaging needs and wider application range mean a need for longer focal lengths, wider fields of view, and higher resolutions, which in turn leads to more complex lens structures. However, transforming a complex optical system from a theoretical design into a practical lens with theoretical imaging quality is no easy task; the key lies in solving tolerance issues.

[0003] All optical systems are inevitably affected by tolerances, which mainly include manufacturing errors, assembly errors, and environmental errors. Manufacturing errors originate from the difference between optical elements and theoretical design parameters; environmental errors are caused by deformation of optical elements and lens structures due to temperature changes; and assembly errors are mainly due to spatial displacement of optical elements during assembly, and are a major factor leading to a decrease in the lens's modulation transfer function (MTF). During assembly, optical elements often exhibit eccentricity and tilt, which introduce off-axis aberrations such as coma, astigmatism, and field curvature, exhibiting significant spatial variation characteristics. Therefore, in optical systems with a large field of view, the edge image quality is more susceptible to the effects of assembly tolerances. For optical systems with complex structures, increasing the number of lenses may lead to more eccentricity and tilt, further exacerbating image quality degradation.

[0004] In traditional optical system design, perturbation mitigation is typically added to improve the system's tolerance to tolerances, ensuring that the lens meets minimum imaging requirements even with assembly tolerances. However, adding perturbation mitigation means requiring more lens elements or additional coatings, increasing lens structure complexity, manufacturing, and assembly costs, and placing higher demands on designers' technical skills. Currently, the primary method to avoid the impact of assembly tolerances still relies on experienced manual adjustments, resolving tolerance issues through precise manual adjustments. Introducing digital assembly tolerance correction methods can not only reduce costs but also improve system robustness, adapt to more diverse assembly conditions, and broaden its application range.

[0005] In summary, existing tolerance correction methods for optical imaging systems mainly include assembly process optimization, design process optimization, adaptive optics compensation, and numerical compensation based on optical models. 1. Precision mechanical assembly optimization By employing ultra-high precision machining technology and high-precision mechanical assembly, along with extensive manual experience, the lens spacing and tilt angle are calibrated step by step, thereby reducing aberrations introduced by assembly tolerances and improving image quality. This method relies on manual fine-tuning, typically requiring multiple iterations and adjustments, and is relatively costly.

[0006] 2. Numerical compensation algorithm based on optical model An imaging model of the optical system under assembly tolerance conditions is established by measuring the point spread function (PSF). Based on this model, the imaging results are processed using a deconvolution algorithm to inversely restore image sharpness. Although this method can compensate for assembly tolerances to some extent, its computational complexity is high and its effectiveness is limited by the accuracy of the imaging model.

[0007] 3. Tolerance sensitivity analysis and design redundancy reservation Monte Carlo simulations were used to perform sensitivity analysis on image quality variations caused by assembly tolerances, evaluating the impact of optical element tilt and offset on the imaging performance (such as MTF) of the optical system. Based on the analysis results, during the optical design phase, the tolerance of the optical system to assembly tolerances can be increased by increasing the number of lenses, lens combinations, or introducing additional coatings. While this method allows for the provisioning of potential assembly tolerances during the design phase, adding redundant components also increases the complexity of the lens structure and cost.

[0008] 4. Active optics and deformable element technology Wavefront distortion can be dynamically corrected by introducing deformable mirrors or spatial light modulators. This technique can compensate for aberrations caused by assembly tolerances in real time, making it particularly suitable for imaging systems with high precision requirements. However, the implementation of this type of technology is relatively complex and requires a high-precision control system and real-time feedback mechanism, resulting in high costs and technical barriers.

[0009] Therefore, in the face of various problems caused by the reliance of traditional assembly tolerance correction methods on high-precision measurement, processing equipment and human experience, there is an urgent need for a new tolerance correction method for optical imaging systems. Summary of the Invention

[0010] To address the aforementioned problems in the prior art, this invention provides a tolerance correction method and apparatus for optical imaging systems based on neural networks. The technical problem to be solved by this invention is achieved through the following technical solution: In a first aspect, embodiments of the present invention provide a tolerance correction method for an optical imaging system based on a neural network, the method comprising: Based on considering various assembly conditions, a degradation model of the optical imaging system under assembly tolerance conditions is built. Using the aforementioned optical imaging system degradation model, the spatial variation degradation process of the optical system under different assembly conditions is simulated, and a dataset is generated. Each sample in the dataset is an image pair consisting of a clear image of the target and a degraded image, and the degraded image is labeled with an operating condition label. A neural network model is constructed, comprising a working condition estimation module and an image restoration module. The working condition estimation module is used to predict the assembly working condition based on the information of the input degraded image and output a feature map. The image restoration module is used to output a corresponding restored image based on the feature map output by the working condition estimation module. Based on the dataset and a preset loss function, the neural network model is trained to obtain a trained condition estimation-image restoration model; wherein, the trained condition estimation-image restoration model is used to restore a clear image from the degraded image under test.

[0011] In one embodiment of the present invention, the step of building a degradation model of the optical imaging system under assembly tolerance conditions, considering various assembly conditions, includes: The PSF of the optical system is regarded as a spatially varying PSF, and the traditional optical imaging system model is modified based on this to obtain the modified optical imaging system model. For the modified optical imaging system model, the imaging process is extended from a two-dimensional plane to a three-dimensional space in order to accurately describe the degradation process of the spatially varying PSF under different assembly conditions, and thus obtain the optical imaging system degradation model under assembly tolerance conditions.

[0012] In one embodiment of the present invention, the modified optical imaging system model is expressed by the formula: ; in, Indicates the location in the degraded image The degradation result at the site; P , Q These respectively represent the clear image The total number of rows and columns divided into image blocks; Represents a clear image exist The pixel at location is associated with the image patch number . ; Optical systems with assembly tolerances are indicated by the corresponding number. PSF block at the image block; Indicates position Additive noise at the location.

[0013] In one embodiment of the present invention, the degradation model of the optical imaging system under assembly tolerance conditions is expressed by the formula: ; in, Indicates the label is Assembly conditions; Indicates the label is Pixel positions in degraded images under assembly conditions The result of degradation at that location.

[0014] In one embodiment of the present invention, the working condition estimation module includes: Main structure and probability prediction structure; The main structure includes four main modules, each of which includes a batch normalization layer, a convolutional layer, and a nonlinear activation layer connected in sequence; the probability prediction structure includes a global pooling layer, a fully connected layer, and a softmax layer connected in sequence.

[0015] In one embodiment of the present invention, the image restoration module is obtained by adding a coordinate attention mechanism module to each skip connection in the encoding-decoding step of the UNet network.

[0016] In one embodiment of the present invention, for the neural network model, the information of the input degraded image includes the RGB three channels and X and Y coordinate information of the degraded image.

[0017] In one embodiment of the present invention, the preset loss function is obtained by weighting the working condition classification loss function and the image restoration loss function, and the preset loss function is expressed as follows: ; in, This represents the preset loss function; This represents the loss function for classifying operating conditions. This represents the image restoration loss function; This represents the hyperparameters that control the balance between the two loss functions; Indicates the number of samples; Indicates the total number of assembly operation condition categories; Indicates sample Condition labels for medium-degraded images; This indicates that the working condition estimation module obtains samples. This belongs to the assembly process. The probability of; Indicates the height of the input image; Indicates the width of the input image; Indicates location in a clear image Pixels at that location; This indicates the location in the restored image predicted by the image restoration module. The pixels at that location. In one embodiment of the present invention, the neural network model is trained based on the dataset and a preset loss function to obtain a trained condition estimation-image restoration model, including: Based on the training set divided from the dataset, the neural network model is iteratively trained until the preset loss function converges, thus obtaining a pre-trained neural network model. The performance of the pre-trained neural network model is verified using the validation set derived from the dataset. Once the requirements are met, the trained condition estimation-image restoration model is obtained.

[0018] Secondly, embodiments of the present invention provide a tolerance correction device for an optical imaging system based on a neural network, the device comprising: The optical imaging system degradation model building module is used to build an optical imaging system degradation model under assembly tolerance conditions, taking into account various assembly conditions. The dataset generation module is used to simulate the spatial variation degradation process of the optical system under different assembly conditions using the degradation model of the optical imaging system, and generate a dataset. Each sample in the dataset is an image pair consisting of a clear image and a degraded image of the target, and the degraded image is labeled with an operating condition label. A neural network model building module is used to build a neural network model that includes a working condition estimation module and an image restoration module. The working condition estimation module is used to predict the assembly working condition based on the information of the input degraded image and output a feature map. The image restoration module is used to output a corresponding restored image based on the feature map output by the working condition estimation module. The model training module is used to train the neural network model based on the dataset and a preset loss function to obtain a trained condition estimation-image restoration model; wherein, the trained condition estimation-image restoration model is used to restore a clear image from the degraded image to be tested.

[0019] To address the reliance of traditional assembly tolerance correction methods on high-precision measurement, processing equipment, and human experience, this invention proposes a neural network-based tolerance correction scheme for optical imaging systems. This method utilizes a Convolutional Neural Network (CNN) to correct spatially varying aberrations caused by assembly tolerances, thereby achieving high-quality imaging. By studying the imaging degradation mechanism caused by assembly tolerances, an imaging model of the optical system under assembly tolerance conditions was established. Through simulation of imaging degradation results under various assembly conditions, degraded images with assembly tolerance perturbations were obtained and paired with clear images without tolerance perturbations to create corresponding datasets. Based on these datasets, leveraging the fitting ability of CNNs for complex nonlinear processes, a mapping process between clear and degraded images was established. By fully utilizing the fitting ability of the convolutional neural network model for complex nonlinear mapping processes, the nonlinear degradation of imaging quality caused by eccentricity and tilt of optical components during assembly is solved from the imaging end. The influence of assembly tolerances is suppressed at the algorithmic level, effectively improving imaging quality.

[0020] The present invention has the following beneficial effects: 1. No additional human experience or expensive equipment required. This invention directly corrects image quality degradation caused by assembly tolerances at the imaging end, eliminating the need for expensive equipment and calibration methods that heavily rely on human experience, thus significantly reducing production costs and technical difficulty.

[0021] 2. Calibration is not dependent on additional measuring equipment. This invention performs tolerance correction directly at the imaging end using a deep learning algorithm, without the need for additional measurement equipment or complex post-processing steps. This avoids the reliance on additional hardware in the deblurring process of traditional methods, thereby simplifying the system structure and operation process.

[0022] 3. Combined with the tolerance tolerance in the design, the impact of tolerances is further reduced. This invention can not only perform tolerance correction at the imaging end, but also combine with tolerance tolerance optimization at the design end to further reduce the impact of assembly tolerances on imaging quality and improve the robustness and adaptability of the system. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a tolerance correction method for an optical imaging system based on a neural network, provided in an embodiment of the present invention. Figure 2 A comparison of the imaging process of an ideal system and the tolerance correction method for an optical imaging system based on neural networks proposed in this invention embodiment; Figure 3This is a schematic diagram of the imaging process of two-dimensional spatial sampling PSF in an embodiment of the present invention; Figure 4 This is a schematic diagram of the imaging process of two-dimensional spatial PSF sampling and adding the working condition dimension in an embodiment of the present invention; Figure 5 This is a schematic diagram of the imaging process under the influence of multiple assembly tolerances in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the structure and training process of the neural network model in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure and prediction process of the trained working condition estimation-image restoration model in an embodiment of the present invention. Figure 8 The working condition estimation-image restoration model designed for embodiments of the present invention demonstrates the restoration effect of a two-dimensional spatially degraded image under the same working condition. Figure 9 The working condition estimation-image restoration model designed for embodiments of the present invention demonstrates the restoration effect of a two-dimensional spatially degraded image under different assembly tolerance conditions; Figure 10 This is a schematic diagram of the structure of a tolerance correction device for an optical imaging system based on a neural network, provided in an embodiment of the present invention. Detailed Implementation

[0024] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0025] Currently, traditional methods for assembly tolerance correction have the following drawbacks: 1. High cost and low efficiency Traditional optical assembly methods rely on sophisticated assembly equipment and experienced operators, resulting in high equipment and labor costs, as well as low assembly efficiency. A single assembly adjustment can take several hours for each assembly condition, and even longer for adjustments across multiple conditions. This not only increases production cycles but also limits the feasibility of large-scale production.

[0026] 2. Dependence on accurate optical models and parameter differences Traditional methods rely on precise optical model parameters, but discrepancies exist between actual assembly tolerances and theoretical model parameters. This necessitates the use of physical measurements to obtain the actual parameters of the optical system under tolerance conditions. In particular, numerical compensation algorithms typically require known PSFs of the optical system for deconvolution processing, but deconvolution algorithms are highly sensitive to noise and prone to ringing effects in complex scenes. For spatially varying aberrations, multiple PSF measurements across different fields of view are required, which is time-consuming. Furthermore, for perturbations caused by different assembly tolerances, traditional methods require repeated measurements of several PSFs of the optical system under different assembly conditions, a complex process with low processing efficiency.

[0027] 3. Negative impacts of increased design redundancy In the optical system design phase, to improve tolerance, it is often necessary to compensate for design deficiencies by adding lens elements or coating layers. However, this approach not only increases the system's size, weight, and overall length but also raises processing, assembly, and manufacturing costs. More importantly, this redundant design is only suitable for the initial design optimization stage. For already assembled systems, effective tolerance correction is impossible, and reliance must be placed on later numerical compensation algorithms or other hardware methods, which are more complex and have limited effectiveness.

[0028] 4. High hardware requirements and slow response speed Some assembly tolerance correction methods, especially those based on active optics and deformable optics, involve high hardware complexity and place high demands on sensors and feedback control systems. The hardware costs far exceed the budget of consumer-grade lenses, hindering the widespread adoption of this technology. Furthermore, the response speed of the feedback control system is limited by hardware performance, often failing to meet the demands for rapid correction in real-time, dynamic, and complex aberration scenarios, thus limiting its application in high-dynamic environments.

[0029] 5. Lacks adaptive correction capability for different tolerance conditions. Existing technologies generally lack adaptive correction capabilities that can automatically adjust to different assembly tolerances and environmental changes. When faced with various assembly conditions or complex environmental conditions, the effects of traditional methods are often unstable and difficult to adjust, especially in scenarios that require rapid response to real-time changes, where the correction effect may not meet expectations.

[0030] To address the aforementioned shortcomings, embodiments of the present invention provide a tolerance correction method and apparatus for optical imaging systems based on neural networks.

[0031] Firstly, please see Figure 1 and Figure 2 Understanding this, the tolerance correction method for an optical imaging system based on neural networks may include the following steps: S1. Based on considering various assembly conditions, a degradation model of the optical imaging system under assembly tolerance conditions is built. In one optional implementation, S1 may include the following steps: S11, the PSF of the optical system is regarded as a spatially varying PSF, and the traditional optical imaging system model is modified based on this to obtain the modified optical imaging system model. Specifically, image quality degradation caused by assembly tolerances mainly stems from errors generated during the mechanical or manual assembly of optical components, manifested as eccentricity and tilt. Eccentricity refers to the deviation between the optical axis of the optical component and the optical axis of the lens, while tilt refers to the angle formed between the optical axis of the optical component and the optical axis of the lens. These two errors lead to changes in the ideal optical path difference in geometric optics, thus causing image blurring. Specifically, eccentricity and tilt typically cause off-axis aberrations such as coma, astigmatism, and field curvature, resulting in a decrease in image quality.

[0032] In traditional optical imaging models, the PSF at the center field of view of an optical system is typically considered equivalent to the PSF across the entire field of view. Therefore, the PSF in traditional optical imaging models can be regarded as spatially invariant. This description of the imaging process is only effective for optical systems with small fields of view and small aberrations. It becomes ineffective for optical systems with large fields of view or those with large aberrations due to tolerances or design flaws. Specifically, the PSF at the edge of the field of view differs significantly from the PSF at the center field of view. In this case, the PSF of the optical system is no longer spatially invariant but spatially variable. If the traditional optical imaging model is still used to describe the imaging process, there will be significant errors. Therefore, optical imaging systems under tolerance conditions should be considered as spatially variable degenerate models. To accurately describe this process, this invention considers the PSF of the optical system as spatially variable. Please refer to [link to relevant documentation]. Figure 3 The schematic diagram of the imaging process of a two-dimensional spatial sampling PSF is shown below, and the traditional optical imaging system model is modified based on this. The modified optical imaging system model is expressed by the formula: (1); in, Indicates the location in the degraded image The degradation result at the site; P , Q These respectively represent the clear image The total number of rows and columns divided into image blocks; Represents a clear image exist The pixel at location is associated with the image patch number . ; Optical systems with assembly tolerances are indicated by the corresponding number. PSF block at the image block; Indicates position Additive noise at the location.

[0033] S12, for the modified optical imaging system model, the imaging process is extended from a two-dimensional plane to a three-dimensional space in order to accurately describe the degradation process of the spatially varying PSF under different assembly conditions, and to obtain the optical imaging system degradation model under assembly tolerance conditions.

[0034] It can be seen that the degradation process described by formula (1) is based on the degradation of PSF sampled at different two-dimensional spatial positions, that is, the degradation process is limited to parameter changes on the two-dimensional plane. However, for optical systems with significantly different assembly tolerances, even if the PSF is sampled at the same two-dimensional spatial position, the degradation result will still be significantly affected by the different assembly tolerances. In this case, formula (1) cannot effectively describe the imaging process under different assembly tolerance conditions, especially when the assembly tolerance range is large. Therefore, in order to overcome this limitation, this invention extends the imaging process from the two-dimensional plane to three-dimensional space, so as to more accurately describe the degradation process of spatially varying PSF under different working conditions. Please refer to [link to relevant documentation]. Figure 4 The diagram shows the imaging process of sampling two-dimensional space PSF and adding the working condition dimension, and... Figure 5 The diagram illustrates the imaging process under the influence of multiple assembly tolerances. The modified degradation model of the optical imaging system under assembly tolerance conditions is expressed by the following formula: (2); in, Indicates the label is Assembly conditions; Indicates the label is Pixel positions in degraded images under assembly conditions The degradation result at the point. The modified formula (2) is actually closely related to the mass production of lenses. Lenses with the same theoretical parameters produced in different batches often have more similar assembly conditions within the batch, while the assembly conditions between batches often differ greatly. By expanding the assembly condition dimension of the optical imaging model, formula (2) can be expressed in the first... The physical meaning of a batch of lenses produced under certain assembly conditions is to classify the imaging performance (PSF) of different batches of lenses under different assembly conditions.

[0035] S2, using the optical imaging system degradation model, simulate the spatial variation degradation process of the optical system under different assembly conditions, and generate a dataset. Each sample in the dataset is an image pair consisting of a clear image of the target and a degraded image, and the degraded image is labeled with an operating condition label. This invention embodiment can utilize the optical imaging system degradation model under assembly tolerance conditions shown in formula (2) to simulate the imaging process of the optical system under different assembly conditions. First, based on the actual assembly level, corresponding ranges of tilt and eccentricity tolerances are added to the optical system in the optical design software to simulate the assembly conditions. The optical system is described below. By sampling the PSF of this optical system at different fields of view, the first... Under various assembly conditions A PSF (as shown in Equation (2)). The finely tuned optical system can be used to image various scenes as clear images. Then, a clear image will be obtained. Divide the image into blocks corresponding to PSF and number them as follows: Image blocks and numbered The PSF is convolved and noise is added according to the pixel position to obtain the degraded image I under the k-th assembly condition, and the clear image is then obtained. Degraded images Operating condition number The dataset is a set of data. To facilitate subsequent use, the dataset can be divided into training, validation, and test sets according to a certain ratio.

[0036] S3, Construct a neural network model that includes a working condition estimation module and an image restoration module; wherein, the working condition estimation module is used to predict the assembly working condition and output a feature map based on the information of the input degraded image; the image restoration module is used to output the corresponding restored image based on the feature map output by the working condition estimation module; In this embodiment of the invention, the constructed neural network model is used to solve the aberrations under different assembly conditions. The neural network model includes a condition estimation module and an image restoration module connected in sequence.

[0037] The image restoration module uses the UNet network as its basic framework. Since the aberrations caused by the tilt and eccentricity of optical elements introduced by assembly tolerances exhibit typical spatial variation characteristics, the constructed neural network model needs to be able to handle spatially varied aberrations. Therefore, this embodiment of the invention modifies the input of the neural network model by adding two additional one-dimensional horizontal and vertical coordinate dimensions to calibrate the spatial position of pixels. Each pixel has corresponding X and Y coordinates. The RGB three channels of the image are input along with the X and Y coordinates. In other words, for the neural network model, the information of the degraded image input includes the RGB three channels of the degraded image and the X and Y coordinate information. Adding X and Y coordinate information enhances the network model's perception of pixel-level spatial position information.

[0038] Since assembly tolerances are not static in actual operation but exist under various assembly conditions, training a neural network model for only one assembly condition is insufficient. To enhance the generalization ability of the network model and enable it to cope with different assembly tolerance conditions, this embodiment of the invention adds an additional condition estimation module before the image restoration module.

[0039] For the structure and training process of the neural network model in this embodiment of the invention, please refer to [link / reference]. Figure 6 As shown.

[0040] The operating condition estimation module includes: Main structure and probability prediction structure; The main structure includes four main modules, each of which includes a batch normalization layer, a convolutional layer, and a nonlinear activation layer connected in sequence; the probability prediction structure includes a global pooling layer, a fully connected layer, and a softmax layer connected in sequence.

[0041] Figure 6 Within the medium-load estimation module, the blue rectangles represent the main module, the batch normalization layer (BN layer), and the nonlinear activation layer (ReLU activation layer). The green rectangles represent the probability prediction structure, the global pooling layer uses global average pooling, and the Softmax layer is the Softmax activation layer.

[0042] It should be noted that the probabilistic prediction structure is only used during network model training, and not during the prediction phase after the network model training is completed.

[0043] The main structure of the condition estimation module extracts aberration image features and pixel location features from the input image. The probability prediction structure outputs estimated probabilities of assembly conditions based on these extracted features, and calculates a condition classification loss function using the actual condition labels. This helps the main structure simultaneously capture aberration features from different conditions. Therefore, the condition estimation module has two outputs during training: the condition label of the degraded image and the feature map passed to the image restoration module. This design allows the subsequent image restoration module to receive not only spatial degradation information but also pixel spatial coordinate information and condition information, enabling the neural network model to effectively correct spatially varying aberrations and aberrations from different assembly tolerance conditions.

[0044] The image restoration module is obtained by adding a coordinate attention mechanism module to each skip connection in the encoding to decoding step of the UNet network.

[0045] Those skilled in the art will understand that in the four-layer encoder-decoder structure of the UNet network, each layer has skip connections for the encoding to decoding steps. This invention adds a coordinate attention mechanism module to each layer's skip connections, such as... Figure 6 As shown in the purple rectangle, this coordinate attention mechanism module can pass the spatial features of each layer to the corresponding layer in the decoding step through skip connections, thereby avoiding the loss of spatial location information during downsampling and improving the network's ability to perceive spatial degradation regions. Through training, the network model becomes sensitive to changes in spatial information and has the ability to perceive different ranges of fuzziness and degradation, effectively handling the influence of different assembly conditions.

[0046] To enhance the network model's perception of working conditions and its correction of spatially degraded images, the preset loss function of this invention is obtained by weighting the working condition classification loss function and the image restoration loss function. Specifically, The working condition classification loss is used to measure the difference between the output of the working condition estimation module and the true working condition label, and is used to enhance the network model's ability to distinguish different working conditions; the image restoration loss is used to compare the pixel differences between blurry and clear images, and is used to enhance the network model's ability to restore degraded images.

[0047] The preset loss function is expressed as: (3); in, This represents the preset loss function; This represents the loss function for classifying operating conditions. This represents the image restoration loss function; This represents the hyperparameters that control the balance between the two loss functions; Indicates the number of samples; Indicates the total number of assembly operation condition categories; Indicates sample Condition labels for medium-degraded images; This indicates that the working condition estimation module obtains samples. This belongs to the assembly process. The probability of; Indicates the height of the input image; Indicates the width of the input image; Indicates location in a clear image Pixels at that location; This indicates the location in the restored image predicted by the image restoration module. The pixels at that location. S4. Based on the dataset and the preset loss function, the neural network model is trained to obtain the trained working condition estimation-image restoration model; Specifically, based on the dataset and a preset loss function, the neural network model is trained to obtain a trained condition estimation-image restoration model, including: Based on the training set divided from the dataset, the neural network model is iteratively trained until the preset loss function converges, thus obtaining a pre-trained neural network model. The performance of the pre-trained neural network model is verified using the validation set derived from the dataset. Once the requirements are met, the trained condition estimation-image restoration model is obtained.

[0048] The above process can be understood by referring to the training and validation process of a conventional neural network, and will not be explained in detail here.

[0049] The trained condition estimation-image restoration model is used to restore a clear image from the degraded image under test.

[0050] For the prediction process of the trained condition estimation-image restoration model, please refer to [link to relevant documentation]. Figure 7 The diagram shown illustrates the structure and prediction process of the completed training condition estimation-image restoration model. During training, the work condition estimation module uses a training set containing work condition labels for degraded images. The classification loss function is calculated by comparing this loss function with the label probabilities output by the module, helping it to fit the correct work conditions. Therefore, after the neural network model is trained, the loss function does not need to be calculated during prediction, and the estimated work condition label probabilities no longer need to be output. Furthermore, since the neural network model has converged, the work condition estimation module can be considered to have correctly fitted the work conditions. Therefore, the probability prediction structure in the work condition estimation module can be removed during the prediction phase. However, the model input for the prediction phase remains the same as during training: the RGB three channels and X and Y coordinate information of the input degraded image. During the experiment, degraded images in the test set can be used as degraded images to be tested. That is, the degraded images in the test set are used to test the restoration performance of the trained condition estimation-image restoration model for unknown input degraded images, and the corresponding clear images are used to evaluate the quality of the restored images, thereby detecting the generalization ability of the network model. Therefore, the test set can be set without condition labels. Of course, in the face of actual use scenarios, the degraded images to be tested can be degraded images of actual collected targets. Through the method of the embodiments of the present invention, they can be restored to obtain clear images.

[0051] Please see Figure 8 and Figure 9 The experimental results shown are illustrated below. Figure 8 The working condition estimation-image restoration model designed for embodiments of the present invention demonstrates the restoration effect of a two-dimensional spatially degraded image under the same working condition. Figure 9 The working condition estimation-image restoration model designed for embodiments of the present invention demonstrates its restoration effect on two-dimensional spatially degraded images under different assembly tolerance conditions. It is evident that the method of the present invention can solve the problem of nonlinear degradation of imaging quality caused by eccentricity and tilt of optical components during assembly, effectively improving imaging quality.

[0052] In summary, this invention addresses the image quality degradation caused by eccentricity and tilt during the assembly of optical systems by proposing a tolerance correction method for optical imaging systems based on a neural network model. Assembly tolerances, especially eccentricity and tilt, are major factors affecting the imaging performance of optical systems. They typically manifest as optical lenses deviating from the optical axis or tilting, leading to optical path deviations and consequently affecting image quality. Because these aberrations exhibit significant spatial variation, image blurring is particularly pronounced for lenses with a large field of view or a large number of lenses.

[0053] Therefore, this invention designs a region-sensitive neural network model to correct image blurring problems caused by different spatial variations.

[0054] To address various operational issues that may arise during assembly, this invention incorporates an operational condition estimation module into the network model framework. This module effectively distinguishes the imaging characteristics of the same optical system under different tolerance conditions, implicitly estimating the degree of blurring and differentiating between different conditions. Through this module, the optical system avoids directly restoring imaging results from different tolerance conditions, thus effectively preventing ringing effects caused by varying tolerances. It better distinguishes between different tolerance conditions, ensuring the accuracy and stability of the correction process, significantly improving the adaptability and generalization ability of tolerance correction, and enhancing the ability to handle various complex scenarios. Compared to directly training the network model using an image restoration module, the introduction of the operational condition estimation module greatly reduces the number of parameters required to train the network model, improving its computational efficiency.

[0055] Furthermore, this invention incorporates absolute spatial location information into the image restoration module within the network model framework and introduces an attention mechanism in the skip connections during the encoding and decoding stages. This effectively avoids the loss of spatial location information during continuous downsampling and enhances the network's ability to perceive spatially degraded regions. Simultaneously, to improve the network's ability to recognize different operating conditions, this invention designs a weighted loss function that combines operating condition classification loss and image restoration loss. This weighted loss function can more accurately guide the model's correction ability under multiple operating conditions, improving the restoration effect.

[0056] This invention establishes an imaging model under tolerance conditions, generates datasets under various assembly conditions, and trains a neural network model to identify and compensate for the influence of assembly tolerances under different conditions. It is applicable to assembly tolerance correction under various conditions, no longer limited to single-condition tolerances, and reduces the tolerance adjustment cost under multiple conditions.

[0057] In summary, compared with the prior art, the present invention has the following beneficial effects: 1. No additional human experience or expensive equipment required. This invention directly corrects image quality degradation caused by assembly tolerances at the imaging end, eliminating the need for expensive equipment and calibration methods that heavily rely on human experience, thus significantly reducing production costs and technical difficulty.

[0058] 2. Calibration is not dependent on additional measuring equipment. This invention performs tolerance correction directly at the imaging end using a deep learning algorithm, without the need for additional measurement equipment or complex post-processing steps. This avoids the reliance on additional hardware in the deblurring process of traditional methods, thereby simplifying the system structure and operation process.

[0059] 3. Combined with the tolerance tolerance in the design, the impact of tolerances is further reduced. This invention can not only perform tolerance correction at the imaging end, but also combine with tolerance tolerance optimization at the design end to further reduce the impact of assembly tolerances on imaging quality and improve the robustness and adaptability of the system.

[0060] Secondly, corresponding to the above method embodiments, this invention also provides a tolerance correction device for an optical imaging system based on a neural network, such as... Figure 10 As shown, the device includes: The optical imaging system degradation model building module is used to build an optical imaging system degradation model under assembly tolerance conditions, taking into account various assembly conditions. The dataset generation module is used to simulate the spatial variation degradation process of the optical system under different assembly conditions using the degradation model of the optical imaging system, and generate a dataset. Each sample in the dataset is an image pair consisting of a clear image and a degraded image of the target, and the degraded image is labeled with an operating condition label. A neural network model building module is used to build a neural network model that includes a working condition estimation module and an image restoration module. The working condition estimation module is used to predict the assembly working condition based on the information of the input degraded image and output a feature map. The image restoration module is used to output a corresponding restored image based on the feature map output by the working condition estimation module. The model training module is used to train the neural network model based on the dataset and a preset loss function to obtain a trained condition estimation-image restoration model; wherein, the trained condition estimation-image restoration model is used to restore a clear image from the degraded image to be tested.

[0061] For details on the specific processing procedures of each module of the device, please refer to the relevant content in the first section, which will not be repeated here.

[0062] To address the reliance of traditional assembly tolerance correction methods on high-precision measurement, processing equipment, and human experience, this invention proposes a neural network-based tolerance correction scheme for optical imaging systems. This method utilizes a Convolutional Neural Network (CNN) to correct spatially varying aberrations caused by assembly tolerances, thereby achieving high-quality imaging. By studying the imaging degradation mechanism caused by assembly tolerances, an imaging model of the optical system under assembly tolerance conditions was established. Through simulation of imaging degradation results under various working conditions, degraded images with assembly tolerance perturbations were obtained and paired with clear images without tolerance perturbations to create corresponding datasets. Based on these datasets, leveraging the fitting ability of CNNs for complex nonlinear processes, a mapping process between clear and degraded images was established. By fully utilizing the fitting ability of the convolutional neural network model for complex nonlinear mapping processes, the nonlinear degradation problem of imaging quality caused by eccentricity and tilt of optical components during assembly was solved from the imaging end. This algorithmically suppresses the influence of assembly tolerances and effectively improves imaging quality. Traditional methods for mitigating aberrations introduced during optical system assembly typically rely on precision machining and experienced manual calibration, which are costly and have poor robustness to assembly tolerances. In contrast, the deep learning-based tolerance correction method for optical imaging systems proposed in this invention offers significant advantages over traditional methods in terms of hardware, labor costs, algorithm response speed, and complexity, providing strong technical support for research and applications in the field of tolerance correction.

[0063] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0064] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A tolerance correction method for an optical imaging system based on a neural network, characterized in that, include: Based on considering various assembly conditions, a degradation model of the optical imaging system under assembly tolerance conditions is built. Using the aforementioned optical imaging system degradation model, the spatial variation degradation process of the optical system under different assembly conditions is simulated, and a dataset is generated. Each sample in the dataset is an image pair consisting of a clear image of the target and a degraded image, and the degraded image is labeled with an operating condition label. A neural network model is constructed, comprising a working condition estimation module and an image restoration module. The working condition estimation module is used to predict the assembly working condition based on the information of the input degraded image and output a feature map. The image restoration module is used to output a corresponding restored image based on the feature map output by the working condition estimation module. Based on the dataset and a preset loss function, the neural network model is trained to obtain a trained condition estimation-image restoration model; wherein, the trained condition estimation-image restoration model is used to restore a clear image from the degraded image under test.

2. The method according to claim 1, characterized in that, The aforementioned degradation model of the optical imaging system under assembly tolerance conditions, considering various assembly conditions, includes: The PSF of the optical system is regarded as a spatially varying PSF, and the traditional optical imaging system model is modified based on this to obtain the modified optical imaging system model. For the modified optical imaging system model, the imaging process is extended from a two-dimensional plane to a three-dimensional space in order to accurately describe the degradation process of the spatially varying PSF under different assembly conditions, and thus obtain the optical imaging system degradation model under assembly tolerance conditions.

3. The method according to claim 2, characterized in that, The corrected optical imaging system model is expressed by the following formula: ； in, Indicates the location in the degraded image The degradation result at the site; P , Q These respectively represent the clear image The total number of rows and columns divided into image blocks; Represents a clear image exist The pixel at location is associated with the image patch number . ; Optical systems with assembly tolerances are indicated by the corresponding number. PSF block at the image block; Indicates position Additive noise at the location.

4. The method according to claim 3, characterized in that, The degradation model of the optical imaging system under the assembly tolerance conditions is expressed by the following formula: ； in, Indicates the label is Assembly conditions; Indicates the label is Pixel positions in degraded images under assembly conditions The result of degradation at that location.

5. The method according to claim 1, characterized in that, The operating condition estimation module includes: Main structure and probability prediction structure; The main structure includes four main modules, each of which includes a batch normalization layer, a convolutional layer, and a nonlinear activation layer connected in sequence; the probability prediction structure includes a global pooling layer, a fully connected layer, and a softmax layer connected in sequence.

6. The method according to claim 5, characterized in that, The image restoration module is obtained by adding a coordinate attention mechanism module to each skip connection in the encoding-decoding step of the UNet network.

7. The method according to claim 1, characterized in that, For the neural network model, the information of the input degraded image includes the RGB three channels and X and Y coordinate information of the degraded image.

8. The method according to claim 1, characterized in that, The preset loss function is obtained by weighting the working condition classification loss function and the image restoration loss function, and the preset loss function is expressed as follows: ； in, This represents the preset loss function; This represents the loss function for classifying operating conditions. This represents the image restoration loss function; This represents the hyperparameters that control the balance between the two loss functions; Indicates the number of samples; Indicates the total number of assembly operation condition categories; Indicates sample Condition labels for medium-degraded images; This indicates that the working condition estimation module obtains samples. This belongs to the assembly process. The probability of; Indicates the height of the input image; Indicates the width of the input image; Indicates location in a clear image Pixels at that location; This indicates the location in the restored image predicted by the image restoration module. The pixels at that location.

9. The method according to any one of claims 1-8, characterized in that, Based on the dataset and a preset loss function, the neural network model is trained to obtain a trained condition estimation-image restoration model, including: Based on the training set divided from the dataset, the neural network model is iteratively trained until the preset loss function converges, thus obtaining a pre-trained neural network model. The performance of the pre-trained neural network model is verified using the validation set derived from the dataset. Once the requirements are met, the trained condition estimation-image restoration model is obtained.

10. A tolerance correction device for an optical imaging system based on a neural network, characterized in that, include: The optical imaging system degradation model building module is used to build an optical imaging system degradation model under assembly tolerance conditions, taking into account various assembly conditions. The dataset generation module is used to simulate the spatial variation degradation process of the optical system under different assembly conditions using the degradation model of the optical imaging system, and generate a dataset. Each sample in the dataset is an image pair consisting of a clear image and a degraded image of the target, and the degraded image is labeled with an operating condition label. A neural network model building module is used to build a neural network model that includes a working condition estimation module and an image restoration module. The working condition estimation module is used to predict the assembly working condition based on the information of the input degraded image and output a feature map. The image restoration module is used to output a corresponding restored image based on the feature map output by the working condition estimation module. The model training module is used to train the neural network model based on the dataset and a preset loss function to obtain a trained condition estimation-image restoration model; wherein, the trained condition estimation-image restoration model is used to restore a clear image from the degraded image to be tested.

Images