Method, computer program, device and neural network for evaluating effective wavefront aberrations of an optical system

Abstract

The present invention relates to a method, a computer program with instructions, and a device for evaluating the effective wavefront aberrations of an optical system, in particular an optical system for use in a means of transportation. In a first step, the effective wavefront aberrations of the optical system are estimated from image recordings of the optical system using a trained neural network (10). Output parameters are then determined from the estimated effective wavefront aberrations (11). The determined output parameters are output to a processing system (12). The invention also relates to a neural network trained for this application.

Description

[0001] The present invention relates to a method, a computer program with instructions, and a device for evaluating effective wavefront aberrations of a complete optical system, in particular a complete optical system for use in a means of transportation. The invention also relates to a neural network trained for this application.

[0002] The optical quality of windshields is relevant for the use of camera systems behind the glass, for example, in driver assistance systems or even self-driving vehicles. For instance, a windshield must have sufficient quality to ensure that, for example, the traffic sign recognition of the driver assistance system functions as specified throughout the vehicle's lifespan. Therefore, the quality of the entire optical system must be checked at the latest at the end of the production line, but preferably before assembly, by measuring the individual optical components of the system.

[0003] However, after delivery of the vehicle, the quality of the glass can degrade, for example due to aging or thermal effects. Such degradation can lead to safety-relevant functions no longer working as specified and can result in safety-hazardous situations.

[0004] It is an object of the invention to provide suitable solutions for monitoring the image quality of an overall optical system.

[0005] This problem is solved by a method having the features of claim 1, by a computer program with instructions according to claim 13, by a device having the features of claim 14, and by a neural network according to claim 15. Preferred embodiments of the invention are the subject of the dependent claims.

[0006] According to a first aspect of the invention, a method for evaluating effective wavefront aberrations of an overall optical system comprises the following steps: - Estimating the effective wavefront aberrations of the entire optical system from images of the entire optical system using a trained neural network; - Determining output parameters from the estimated effective wavefront aberrations; and - Outputting the output values ​​to a processing system.

[0007] According to another aspect of the invention, a computer program contains instructions which, when executed by a computer, cause the computer to perform the following steps for evaluating effective wavefront aberrations of an overall optical system: - Estimating the effective wavefront aberrations of the entire optical system from images of the entire optical system using a trained neural network; - Determining output parameters from the estimated effective wavefront aberrations; and - Outputting the output values ​​to a processing system.

[0008] The term "computer" is to be understood broadly. In particular, it also includes control units, embedded systems, and other processor-based data processing devices. Furthermore, the individual steps are not necessarily executed directly by the computer. It is equally possible that the computer controls or relies on external components to perform individual steps.

[0009] The computer program can, for example, be made available for electronic retrieval or be stored on a computer-readable storage medium.

[0010] According to another aspect of the invention, a device for evaluating effective wavefront aberrations of an overall optical system comprises: - a trained neural network for estimating the effective wavefront aberrations of the entire optical system from image recordings of the entire optical system; - an evaluation module for determining output values ​​from the estimated effective wavefront aberrations; and - an output module for outputting the output values ​​to a processing system.

[0011] According to another aspect of the invention, a neural network is trained to estimate effective wavefront aberrations of an overall optical system from image recordings of the overall optical system.

[0012] In the solution according to the invention, a trained neural network is used to estimate the effective wavefront aberrations of the overall optical system during operation. The effective wavefront aberrations allow for continuous monitoring of the optical quality during operation. This enables monitoring of the quality of the overall optical system throughout its entire service life.

[0013] According to one aspect of the invention, the output parameters are coefficients that parameterize the effective wavefront aberrations using a decomposition into an orthonormal functional basis. Assuming a circular aperture, the effective wavefront aberrations are expediently parameterized by Zernike coefficients. For a rectangular aperture, parameterization by Legendre coefficients is more suitable.

[0014] According to one aspect of the invention, the processing system is a monitoring system for detecting degradation of the overall optical system. The monitoring system is preferably designed to issue a warning when the degradation exceeds a tolerance threshold. The solution according to the invention allows for the early detection of lens degradation, external influences such as weather phenomena (i.e., rain, snow, ice, etc.), or impacts from stones. This reduces potentially hazardous situations. The safety mechanism based on image quality enables the dynamic monitoring of the hazard potential.

[0015] According to one aspect of the invention, the processing system is an automated driving system of a means of transportation. Preferably, the output variables serve as an additional input signal for providing calibrated confidence levels. The estimated wavefront aberrations of the overall optical system allow for the calibration of the predicted confidence levels of the network. The class-wise confidence levels can be used to determine the prediction uncertainty. This measure of uncertainty is important for decision-making regarding the subsequent action of the automated driving system, e.g., braking, swerving, etc. If the uncertainty is low and the situation or status is captured with sufficient confidence, a reliable decision can be made.If, however, the confidence is insufficiently low, either an independent secondary system must contribute to the decision-making process or responsibility must be returned to the driver and the autonomous driving process interrupted.

[0016] According to one aspect of the invention, the calibrated confidences are determined using parameterized temperature scaling. The determined confidences can be calibrated using modern methods such as temperature scaling or parameterized temperature scaling. However, if a data shift is induced by the degradation of the image data due to the windshield, the calibration of the confidences or uncertainties of the grid can break down. The grid becomes increasingly overconfident. To avoid this, the estimated wavefront aberrations of the overall optical system can be used as an additional input signal for the parameterized temperature scaling. With such an approach, referred to as physics-informed parameterized temperature scaling, the calibration of the predicted confidences of the base grid can be maintained even when a data shift occurs due to internal factors, such as...The aging of the glass or thermal effects, and external factors such as weather conditions, are taken into account. Thermal effects can be caused, for example, by sunlight or the glass heating system.

[0017] According to one aspect of the invention, the complete optical system comprises a disc and a camera lens. Complete optical systems consisting of a disc and a camera are increasingly being installed in means of transportation, particularly in motor vehicles.

[0018] According to one aspect of the invention, the trained neural network is a convolutional neural network, preferably a U-NET-based convolutional neural network. A convolutional neural network (CNN) is well suited for estimating the effective wavefront aberrations of the overall optical system. A U-NET-based convolutional neural network is particularly advantageous for applications requiring semantic segmentation.

[0019] According to one aspect of the invention, the neural convolutional network has a coupled decoder head. A coupled decoder head allows for increased performance with respect to the objective function, e.g., semantic segmentation.

[0020] According to one aspect of the invention, the neural convolutional network includes a ResNet encoder. The use of a ResNet encoder, preferably in combination with a coupled decoder head, allows for particularly effective compensation of the nonlinearity of the overall optical system.

[0021] Further features of the present invention will become apparent from the following description and the attached claims in conjunction with the figures. Fig. Figure 1 schematically shows a perception system of a means of transportation; Fig. Figure 2 schematically shows a method for evaluating effective wavefront aberrations of an entire optical system; Fig. Figure 3 shows a first embodiment of a device for evaluating effective wavefront aberrations of an overall optical system; Fig. Figure 4 shows a second embodiment of a device for evaluating effective wavefront aberrations of an overall optical system; Fig. Figure 5 schematically represents a means of transport in which a solution according to the invention is implemented; and Fig. Figure 6 shows an embodiment of a neural network suitable for implementing a solution according to the invention.

[0022] To better understand the principles of the present invention, embodiments of the invention are explained in more detail below with reference to the figures. It is understood that the invention is not limited to these embodiments and that the described features can also be combined or modified without leaving the scope of protection of the invention as defined in the appended claims.

[0023] Fig. Figure 1 schematically shows a perception system 1 of a means of transport 40. In this example, the perception system 1 comprises an optical system 2 and a data processing system 5, which performs an assistance or driving function of the means of transport 40. The optical system 2 comprises a first optical component, here a windshield 3, and a second optical component, here a camera lens 4. The images provided by the camera lens 4 can be used by the data processing system 5, for example, for lane detection or the detection of pedestrians or other road users.

[0024] Fig. Figure 2 schematically shows a method for evaluating the effective wavefront aberrations of a complete optical system. The complete optical system can comprise a disk and a camera lens and be part of a perception system that, for example, serves an assistance function or a driving function in a means of transportation. In a first step, the effective wavefront aberrations of the complete optical system are estimated from image recordings of the complete optical system using a trained neural network.10 The neural network can, for example, be a convolutional neural network, preferably a U-NET-based convolutional neural network. Preferably, such a convolutional neural network has a coupled decoder head and / or a ResNet encoder. Output parameters are then determined from the estimated effective wavefront aberrations.11 The output parameters can be, for example,These can be Zernike coefficients or Legendre coefficients that parameterize the effective wavefront aberrations. The determined output values ​​are output to a processing system. 12. The processing system can be, for example, a monitoring system for detecting degradation of the overall optical system. This system is preferably designed to issue a warning if the degradation exceeds a tolerance threshold. Alternatively, the processing system can be an automated driving system of a means of transportation. In this case, the output values ​​can serve as an additional input signal for providing calibrated confidence levels. These can be determined, for example, using parameterized temperature scaling.

[0025] Fig. Figure 3 shows a simplified schematic representation of a first embodiment of a device 20 for evaluating effective wavefront aberrations of an optical system. The optical system can comprise a disk and a camera lens and be part of a perception system that, for example, serves an assistance function or driving function in a means of transportation. The device 20 has an input 21 through which image recordings A B The data from the entire optical system is read in. A computing module 22 provides a trained neural network N. This network is configured to extract data from the image recordings A. B of the overall optical system effective wavefront aberrations W pto estimate the overall optical system. The neural network N can, for example, be a convolutional neural network, preferably a U-NET-based convolutional network. Preferably, such a convolutional network has a coupled decoder head and / or a ResNet encoder. An evaluation module 23 is set up to calculate the effective wavefront aberrations W from the estimated values. p To determine output quantities A. The output quantities A can be, for example, Zernike coefficients or Legendre coefficients, which represent the effective wavefront aberrations W. pParameterize. An output module 24 is configured to output the output variables A via an output 27 of the device 20 to a processing system 41 for further use. The processing system 41 can, for example, be a monitoring system 410 for monitoring degradation of the overall optical system. This system is preferably configured to issue a warning H if the degradation exceeds a tolerance threshold. The processing system 41 can also be an automated driving system 411 of a means of transportation. In this case, the output variables A can serve as an additional input signal for providing calibrated confidence levels K. These can be determined, for example, using parameterized temperature scaling.

[0026] The computing module 22, the evaluation module 23, and the output module 24 can be controlled by a control module 25. Settings of the computing module 22, the evaluation module 23, the output module 24, or the control module 25 can be changed via a user interface 28. The data generated in the device 20 can be stored in a memory 26 of the device 20 as needed, for example, for later evaluation or for use by the components of the device 20. The computing module 22, the evaluation module 23, the output module 24, and the control module 25 can be implemented as dedicated hardware, for example, as integrated circuits. Of course, they can also be partially or completely combined or implemented as software running on a suitable processor, such as a CPU or a GPU.Input 21 and output 27 can be implemented as separate interfaces or as a combined interface.

[0027] Fig. Figure 4 shows a simplified schematic representation of a second embodiment of a device 30 for evaluating effective wavefront aberrations of an optical system. The device 30 comprises a processor 32 and a memory 31. For example, the device 30 is a computer, a control unit, or an embedded system. Instructions are stored in the memory 31 which, when executed by the processor 32, cause the device 30 to perform the steps according to one of the described methods. The instructions stored in the memory 31 thus embody a program executable by the processor 32, which implements the method according to the invention. The device 30 has an input 33 for receiving information.Data generated by processor 32, for example, data concerning estimated effective wavefront aberrations of the overall optical system or output quantities derived therefrom, are provided via output 34. In addition, data can be stored in memory 31. Input 33 and output 34 can be combined into a bidirectional interface.

[0028] The processor 32 can comprise one or more processor units, such as microprocessors, digital signal processors, or combinations thereof.

[0029] The memory elements 26, 31 of the described embodiments can have both volatile and non-volatile memory areas and can include a wide variety of storage devices and storage media, for example hard disks, optical storage media or semiconductor memory.

[0030] Fig. Figure 5 schematically represents a means of transport 40 in which a solution according to the invention is implemented. In the illustrated example, the means of transport 40 is a motor vehicle. The motor vehicle has a camera lens 4 arranged behind a windshield 3 for capturing environmental information. The windshield 3 and the camera lens 4 together form a complete optical system 2. A device 20 according to the invention is configured to use image recordings A BThe device 20 evaluates the effective wavefront aberrations of the optical system 2 and outputs the derived output variables A to a processing system 41. The processing system 41 can be, for example, a monitoring system 410 for monitoring degradation of the optical system 2. It can also be an automated driving system 411 of the vehicle. The device 20 can, for example, be implemented in a computer 42 of the vehicle. Data generated in the vehicle can be stored in a memory 43. A connection to a backend can be established via a data transmission unit 44, for example, to retrieve updated software for vehicle components. Data exchange between the various vehicle components takes place via a network 45.

[0031] A preferred embodiment of the invention will now be explained. The optical system under consideration includes, by way of example, a windshield and a camera lens. The objective function is also, by way of example, semantic segmentation.

[0032] Fig. Figure 6 shows an embodiment of a neural network N suitable for implementing a solution according to the invention. This is, by way of example, a multifunctional neural network N which, in addition to predicting wavefront aberrations W, p It also performs semantic segmentation. Of course, other objective functions can be implemented instead of semantic segmentation, or the function of the neural network N can be limited solely to predicting wavefront aberrations W. p be limited.

[0033] In the example shown, the neural network N is a U-NET-based convolutional network for estimating the effective wavefront aberrations W. p of the overall optical system, consisting of the windshield and camera lens. The neural network N comprises an encoder ENC, a bottleneck BN, a coupled decoder head consisting of a decoder DEC and a restorer RES, as well as skip connections SC for the features of the different resolution levels. Additionally, the architecture includes a residual encoder RN for identifying the second-order Zernike coefficients (Z3, Z4, Z5). The input for the neural network is a batch B. d degraded images.

[0034] The DEC decoder for semantic segmentation uses pre-trained weights from training on undistorted raw images without optical aberrations. The RES restorer, designed to restore undistorted images from distorted ones, is added in parallel. The RN residual encoder, for identifying second-order Zernike coefficients, consists of several ResNet cells that calculate the difference between the neural network input N and the batch B. d degraded images, and the output of the restorer RES, a batch B w recovered images, process, i.e., a batch B r of residues.

[0035] The form of the input to the neural network N, i.e., batch B dThe number of degraded images is assumed to be: number of images times image height in pixels times image width in pixels times number of color channels. Several hyperparameters can be adjusted if necessary before training the model: - N_classes: Number of classes for the DEC decoder for semantic segmentation - conv2D_kernel: Size of the convolution kernel tensor in pixels - conv2D_increment: Increment of the convolution operation in pixels - regu_strength: Weight of the regularization term - learning_rate: Weighting factor for the overall loss function - batch_size: Number of processed images before the trainable variables are updated - class_weights: Weighting factors for the individual classes based on their occurrence in the dataset - loss_focus: Exponent that forces a concentration of the learning process on classes that are difficult to learn. - loss_balance_restorer: Weighting factor for the loss of the restorer RES (L1 loss) - loss_balance_analyzer: Weighting factor for the loss of the analyzer, i.e., the residual encoder RN (L2 loss)

[0036] The encoder ENC consists of five encoder blocks, with the number of filters doubling in each subsequent block. Each encoder block comprises two convolutional layers, followed by a batch normalization layer and a max-pooling layer for downsampling. After the batch normalization layer, the computational graph is split into two branches to pass information to the decoder DEC, thus mitigating the vanishing gradient problem.

[0037] The Bottleneck BN consists of two convolution layers and subsequent batch normalization layers.

[0038] The restorer RES consists of five transposed convolution blocks, with the number of filters halved in each subsequent block. Each transposed convolution block comprises a transposed convolution layer, followed by a batch normalization layer and a merge node to incorporate, with high reliability, the information provided by the skip connection SC of the corresponding encoder block. The chained tensor is then smoothed by two convolution layers with a step size of one to preserve the dimensions.

[0039] The residual encoder RN, used to identify Zernike coefficients, consists of five ResNet cells for classifying aberrations based on second-order Zernike coefficients. Each ResNet cell comprises two convolution layers, two batch normalization layers, and a concatenation layer that combines the convolutional signal with the input signal to mitigate the vanishing gradient problem. Finally, a Max pooling layer is employed for downsampling. After the ResNet cells, the signal is smoothed and divided into five dense layers with batch normalization and dropout for regularization.

[0040] The DEC decoder for semantic segmentation consists of five transposed convolution blocks, with the number of filters halved in each subsequent block. These transposed convolution blocks for semantic segmentation are structured similarly to the transposed convolution blocks of the RES restorer.

[0041] The network N is trained using a fitted loss function. The loss function consists of three components. The first component is the negative log-likelihood for semantic segmentation, which corresponds to the cross-entropy. The second term quantifies the deviation between the undistorted image and the recovered image using the L1 norm. The third term quantifies the deviation between the predicted vector of Zernike coefficients and the vector of basic truth using the L2 norm. All components are summed, taking into account a weighting factor for hypertuning.

[0042] For training the neural network N, a physically realistic degradation model F based on Fourier optics is used. Such a model F requires wavefront measurement results W. gregarding a representative number of windshields and camera lenses that correspond to the project status. With this model F, the training data of the base network are now degraded, i.e., a batch B. c of undistorted images from a set of A T of images taken without a windshield. To strengthen the robustness of the N network against optical aberrations, the degraded training data can be used for transfer learning. The components required for training are in Fig. 6 enclosed by a dashed box.

[0043] To create a batch B dTo obtain degraded image data, both the aberrations of the lens and the aberrations of the windshield must be taken into account. Measurement results from a wavefront measurement system, e.g., based on the background-oriented schlieren method, serve as the basis for the total aberrations to be learned. In the illustrated embodiment, aberrations of individual components of the overall optical system are incorporated by a Fourier optical model F. The batch of degraded image data B d It consists of a whole ensemble of different lenses and windscreens, so that the definition range of the input of the network N is sufficiently covered.

[0044] In the example shown, the predicted total aberrations W serve as pHere, the predicted Zernike coefficients are used as an additional input signal to provide calibrated confidence levels K. A batch B serves again as the input for the neural network. d degraded images, which now consist of a lot of A B The images are derived from images taken with the windshield visible. The estimated wavefront aberrations of the overall optical system allow for the calibration of the predicted confidences of the network N. The class-wise confidences can be used to determine the prediction uncertainty. This measure of uncertainty is important for decision-making regarding the subsequent action of the automated driving system, e.g., braking, swerving, etc. To determine the calibrated confidences K, the logit tensor LT output by the decoder DEC and the predicted total aberrations W are used. pThe data is fed into a calibration block P, which determines calibration factors for the predicted confidences using parameterized temperature scaling. The calibrated confidences K obtained in this way are transferred to a segmentation map S, which is output to downstream systems.

[0045] The implementation described above assumes a UNET as the basic architecture; however, the approach can also be applied to other network architectures. The proposed architecture can be constructed in a modified form with the same objective. Generally, the network architecture of a neural network can be adjusted in three different dimensions: depth (more or fewer layers), width (more or fewer channels), or resolution (higher or lower resolution training images). Reference symbol list 1 Perceptual system 2 Optical system 3 discs 4 camera lenses 5 Data processing system 10 Estimating effective wavefront aberrations 11 Determining output sizes 12. Outputting the output sizes 20 Device Entrance 21 22 Computing module 23 Evaluation module 24 Output module 25 Control module 26 storage 27 Exit 28 User interface 30 Device 31 storage 32 processor 33 Entrance 34 Exit 40 means of transport 41 Processing system 410 Monitoring system 411 Automated Driving System 42 computers 43 storage 44 Data transmission unit 45 Network A Output size A B Shots with windshield AT Shots without windshield B rating B c Batch of non-degraded images B d Batch of degraded images B r Batch with residuals B w Batch of recovered images BN Bottleneck DEC decoder ENC Encoder F Fourier-optical degradation model H Warning K Calibrated Confidences LT Logit Tensor N Neural Network P Calibration block RES Restorer RN ResNet encoder S Segmentation map W g Measured wavefront aberrations W p Predicted wavefront aberrations

Claims

Method for evaluating effective wavefront aberrations (W) of an optical system (2), comprising the steps: - Estimating (10) the effective wavefront aberrations (Wp) of the optical system (2) from image acquisitions (AB) of the optical system (2) using a trained neural network (N); - Determining (11) output quantities (A) from the estimated effective wavefront aberrations (Wp); and - Outputting (12) the output quantities (A) to a processing system (41). Method according to claim 1, wherein the output variables (A) are coefficients of a parameterizing the effective wavefront aberrations (Wp) using a decomposition into an orthonormal functional basis. Method according to claim 1 or 2, wherein the processing system (41) is a monitoring system (410) for monitoring degradation of the overall optical system (2). Method according to claim 3, wherein the monitoring system (410) is configured to issue a warning (H) when the degradation exceeds a tolerance threshold. Method according to claim 1 or 2, wherein the processing system (41) is an automated driving system (411) of a means of transport (40). Method according to claim 5, wherein the output variables (A) serve as an additional input signal to provide calibrated confidences (K). Method according to claim 6, wherein the calibrated confidences (K) are determined using parameterized temperature scaling. Method according to one of the preceding claims, wherein the overall optical system (2) comprises a disk (3) and a camera lens (4). Method according to one of the preceding claims, wherein the trained neural network (N) is a convolutional neural network. Method according to claim 9, wherein the neural convolutional network is a U-NET-based convolutional network. Method according to claim 9 or 10, wherein the neural convolutional network has a coupled decoder head. Method according to one of claims 9 to 11, wherein the neural convolutional network comprises a ResNet encoder (RN). Computer program with instructions which, when executed by a computer, cause the computer to perform the steps of a method according to one of claims 1 to 12 for evaluating effective wavefront aberrations (Wp) of an optical system (2). Device (20) for evaluating effective wavefront aberrations (Wp) of an optical system (2), comprising: - a trained neural network (N) for estimating (10) the effective wavefront aberrations (Wp) of the optical system (2) from image acquisitions (AB) of the optical system (2); - an evaluation module (23) for determining (11) output quantities (A) from the estimated effective wavefront aberrations (Wp); and - an output module (24) for outputting (12) the output quantities (A) to a processing system (41). Neural network (N) for use in a method according to one of claims 1 to 12 or a device (20) according to claim 14, wherein the neural network (N) is trained to estimate effective wavefront aberrations (Wp) of an optical system (2) from image recordings (AB) of the optical system (2).

Images