Method for adapting a computer vision unit for environmental sensing to be carried out from a motor vehicle, as well as optical model, computer vision unit and motor vehicle

Abstract

The invention relates to a method for adapting a computer vision unit (12) for environmental sensing to be performed from a motor vehicle. The method provides that, in a predetermined viewing area (16) of a windshield (15) of the motor vehicle, characteristic data (22) of a manufacturing-related waviness (32) of the viewing area (16) are determined by means of a wavefront measurement (21), and from the characteristic data (22) an optical model (31) of an optical disturbance caused by the waviness (32) is formed, and training image data are generated from environmental image data of an environmental camera (14) using the optical model (31), and a machine learning model (13), ML model, is generated using the training image data.the computer vision unit (12) is trained and the environment camera (14) or another environment camera (14) is provided in an interior of the motor vehicle and its detection range is directed through the viewing area of ​​the windshield (15) of the motor vehicle into the environment of the motor vehicle and the computer vision unit (12) is coupled with the environment camera (14) for environment detection.

Description

[0001] The invention relates to a method for adapting a computer vision unit for environmental detection to be carried out from a motor vehicle. By means of the computer vision unit, objects in the environment of the motor vehicle can be recognized or detected based on environmental image data from an environmental camera of the motor vehicle. Depending on such objects detected in the environment, a driver assistance function can guide or steer the motor vehicle. The invention also includes an optical model for simulating optical disturbances or aberrations in the form of loss of contrast / loss of sharpness (e.g.: a point is blurred into a spot or edge, but remains a line) of a windshield of a motor vehicle, as well as the computer vision unit with a machine learning model, ML model, and finally a motor vehicle with the computer vision unit.

[0002] A motor vehicle can be provided with an environment-view camera whose detection range is directed towards the environment of the motor vehicle (in particular in a front area ahead) in order to capture environment image data of this environment. Using a computer vision unit, objects in the environment can be detected based on the environment image data. The computer vision unit can have a machine learning model, for example an artificial neural network, for the necessary processing of the environment image data. Based on the detected objects, a driver assistance function, for example an autonomous driving function, can then automatically guide or control the motor vehicle, for example plan and execute a driving trajectory for a journey through the environment.

[0003] To further develop such object recognition based on a computer vision unit, it is planned to increase the resolution of the environment camera, for example, to use a horizontal resolution of more than 2,000 pixels, in particular an environment camera with a value range of around 4,000 pixels (so-called 4k cameras). With an increased resolution, object recognition can also be performed for distant objects in the environment images, in particular objects at a distance beyond a distance value in a range of 50 meters to 100 meters from the environment camera, since such an object, for example a traffic sign or another road user, is represented in the environment image data with a sufficient number of image pixels to enable object recognition.

[0004] However, tests on prototypes have shown that when computer vision units are used with a respective machine learning model (ML model), particularly an artificial neural network, the recognition performance (e.g., the proportion of correctly recognized objects in environmental image data) decreases or decreases. For these tests, the ML model was trained with environmental images such as those generated by an environmental camera during test drives with a test vehicle. The trained ML model was then used in a computer vision unit in a production vehicle, where the environmental camera captured the environment through the vehicle's windshield.

[0005] EP 3 982 328 A1 discloses the measurement of optical interference caused by a windshield when capturing images of the surroundings. This method then determines a quality metric that indicates whether the windshield is even suitable for image processing in a driver assistance system. If the interference is too severe, the windshield is marked as unsuitable. While this method allows unsuitable windshields to be identified, they are treated as scrap, increasing material costs.

[0006] US 2022 / 0412897 A1 discloses a measurement method for optical wavefront analysis, which allows optical disturbances in a windshield to be measured and described, for example, using a PSF (point spread function). Corresponding parameters can also be provided together with the measured windshield, for example, via a database or as a QR code.

[0007] In order to prevent the influence of optical disturbances from a windscreen, it is also known to mount the surround camera for surroundings detection on the outside of a motor vehicle, for example on a roof.

[0008] The invention is based on the object of minimizing the influence of optical disturbances (loss of contrast / loss of sharpness), such as can be caused by a windscreen of a motor vehicle, during environmental detection for object recognition of a computer vision unit.

[0009] The problem is solved by the subject matter of the independent claims. Advantageous further developments are described by the dependent claims, the following description, and the figures.

[0010] As one solution, the invention comprises a method for adapting or adjusting a computer vision unit for environmental detection to be performed from a motor vehicle. The computer vision unit can be based, in a conventional manner, on a processor circuit that can receive environmental image data and, based on the environmental image data, detects or recognizes at least one object in the environment of the motor vehicle, which is referred to collectively here as "environment detection."

[0011] The surrounding image data can be provided by a surrounding camera, e.g., a video camera, which can be installed in the motor vehicle and records the surroundings from the motor vehicle. To enable the surrounding camera to be arranged in the interior of the motor vehicle and to record the surroundings through the windshield of the motor vehicle, the invention provides for a wavefront measurement of the view-through area to be performed in a predetermined light transmission area or view-through area of ​​a windshield of the motor vehicle. In particular, this involves recording the optical path difference across the aperture surface of the camera.This can be the windshield that is actually intended to be installed in a specific motor vehicle, for which a specific computer vision unit that needs to be adapted is also provided, or it can be one or more other windshields that are generally intended for a specific vehicle type, for example one or more windshields in a delivery or batch for the production of several motor vehicles of the same vehicle type. The “view-through area” refers to the geometric area of ​​the windshield through which the surround-view camera is intended to capture the surroundings. The wavefront measurement can, for example, be carried out using the wavefront measurement described above. The wavefront measurement yields characteristic data about a production-related waviness of the view-through area, for example a geometric shape of the surface and / or at least an optical transition area within the windshield.

[0012] From these characteristics, an optical model is used to describe or simulate the optical aberration, or disturbance, of a light wavefront caused by the waviness as it occurs when the light wavefront passes through the viewport. This always occurs when light from the environment passes through the front window to the surround camera.

[0013] Using the optical model, training image data is generated by modulating environmental images. In other words, the optical model is applied to environmental image data from an environmental camera. Thus, the environmental images are modulated using the optical model in the same way as they would be if the environmental images had been recorded or captured through the view-through area of ​​the windshield. The optical model is, in particular, a digital and / or numerical model.

[0014] Using this generated training image data, a machine learning model (ML model) of the computer vision unit is trained. Thus, objects contained or depicted in the environmental images are represented or presented to the ML model during its training for object recognition or environmental sensing in the same way they would appear under the influence of the optical disturbance of the view-through area.

[0015] Accordingly, the surround camera in the vehicle or (if the surround camera used for the training images was that of a test vehicle) a different surround camera (e.g. of the same camera type) can be used in the interior of the motor vehicle. In other words, the surround camera does not have to be operated outside on the roof of the motor vehicle, for example, but can film through the windshield. For this purpose, the detection area of ​​the surround camera is aligned through the view-through area of ​​the motor vehicle's windshield into the surroundings of the motor vehicle. In other words, the surround camera records the surroundings through the view-through area of ​​the windshield. This results in optical interference in the surround images (even without the optical model) because it is caused by the view-through area of ​​the windshield.

[0016] The computer vision unit for environment detection is coupled to the environment camera. Since the computer vision unit contains the trained machine learning model, which performs environment detection or object recognition, this ML model is now able to take into account the optical disturbance of the windshield in the view-through area. In other words, this optical disturbance is "trained" or "retrained." The latter is the case when you want to increase the robustness against optical aberrations of an already trained ML model through retraining, or you want to train in individual aberrations of a specific windshield.

[0017] The invention provides the advantage that the surround-view camera can be operated in the interior of the motor vehicle and the surroundings can be captured through the windshield. This also makes it possible, for example, to utilize the advantage that the windshield can be cleaned using a windshield wiper and / or heated and / or dried by an air conditioning system in the motor vehicle. Nevertheless, the influence of the optical interference in the view-through area of ​​the windshield can be compensated for or reduced by training the ML model with training image data in order to take or be able to take the influence of the optical interference into account during object recognition. Furthermore, this eliminates the need for additional computing during operation of the motor vehicle, as would be the case, for example, with online image rectification.

[0018] The computer vision unit coupled to the vehicle's surround-view camera can be provided in the vehicle itself or it can be operated externally in a so-called backend, i.e., an internet server. The surround-view camera, as described above, is in particular a so-called high-resolution camera, i.e., a camera with a horizontal resolution of preferably more than 2,000 pixels, in particular more than 3,500 pixels. A vertical resolution can be more than 1,000 pixels. A corresponding pixel matrix can be provided by a CCD (charged coupled device). Using a high-resolution camera, even objects at a distance of more than 40 meters, in particular more than 60 meters or more than 70 meters, can be detected in the object recognition by the ML model.Such objects are still represented by a sufficient number of pixels, even though the optical interference from the windshield has a greater impact than for closer objects of the same size, which are represented by even more pixels. This allows, for example, traffic sign recognition to be performed even for traffic signs located more than 50 meters away.

[0019] The invention also includes further developments whose features provide additional advantages.

[0020] According to a further development, a Zernike analysis (Zernike decomposition) is performed using the wavefront measurement. Zernike coefficients are generated accordingly as characteristic data. A corresponding wavefront measurement can be taken from the state of the art, where Zernike analysis is known, for example, in the context of ophthalmology. Zernike coefficients have proven particularly advantageous in that, in the optical model, different forms of the waviness are described or defined separately or orthogonally to one another by the Zernike coefficients, so that the different forms of the optical disturbance can be adjusted separately in the optical model.

[0021] In particular, a further development provides for only the Zernike coefficients Z3, Z4, Z5, or one or two of these Zernike coefficients to be used as characteristic data. This simplifies the optical model and allows it to be calculated and applied accordingly efficiently. Nevertheless, the Zernike coefficients mentioned have proven to be a sufficient description of the influence of optical interference on the recognition performance of an ML model. Thus, by determining only up to three Zernike coefficients, a computer vision unit can be efficiently adapted to a specific windshield or windshield type of a motor vehicle. Z3-Z5 have proven to be the dominant coefficients, however, higher orders can also play a significant role and can be implemented into the described optical model without additional effort.Here, the expert can find out through simple experiments which coefficients are advantageous for a specific type of pane.

[0022] According to a further development, an incoherent PSF (point spread function) is used as the optical model. More generally, a convolution kernel or a Fourier transform of a convolution kernel can be generated as the optical model. As a Fourier transform, the optical model can be applied particularly efficiently to ambient images, since the 2D Fourier coefficients of the ambient image can be multiplied by the Fourier transform of the convolution kernel. "Incoherent PSF" refers to a PSF in which the characteristics of the optical disturbance, such as the Zernike coefficients, are included. A corresponding calculation can be found, for example, in the prior art document described above.

[0023] The optical model can be applied to all environmental images used to train the ML model. The "training" can be performed in a conventional manner, for example, based on a loss function for object detection and the backpropagation algorithm. An alternative development provides for the ML model to first be trained using undistorted environmental image data. This can be the same environmental image data to which the optical model is also applied, and / or it can be additionally or alternatively other environmental image data. This results in pre-training that ignores the optical interference from the windshield.For example, such environmental image data can be generated using an environmental camera that captures the environment through a different windshield or that is operated in a test vehicle where it is located outside, i.e., not through a windshield. This pre-trained ML model is then adapted to the windshield in a second training phase by adapting the training using the described training image data (i.e., using distorted environmental images) by retraining the trained ML model. Such retraining can be implemented, for example, by applying the backpropagation algorithm to the pre-trained ML model again using the training image data.For example, a step size or a maximum amount of a change in the ML model's coefficients per iteration step can be limited to a maximum value to avoid completely "overwriting" the pre-training. This division into pre-training and post-training has the advantage that the number of environment images for pre-training can be larger, in particular, more than a factor of 5 or 10 larger than the number of distorted environment images. This makes it possible to apply the optical model to generate the distorted environment images with minimal computational effort.

[0024] To obtain the characteristics of the optical disturbance, the wavefront measurement can be performed on several windshields, and the resulting characteristic values ​​can be averaged, i.e., an average value can be calculated from the corresponding characteristic values ​​of the multiple wavefront measurements. This is particularly possible when several windshields are manufactured using a consistent or regular manufacturing process, so that the optical disturbances caused by the different windshields in their viewing area are comparable, i.e., an average value can be used. If, on the other hand, the manufacturing process is subject to greater variance, it may be advantageous to adapt the computer vision unit to the specific vehicle.To this end, a further development provides that during the production of the motor vehicle, the windshield intended for installation in the motor vehicle or already installed in it is detected, determined, or identified, and the wavefront measurement is carried out on a vehicle-specific basis for this vehicle only on this windshield. Thus, the computer vision unit for this vehicle is individually adapted to the windshield of the vehicle through training. This can be implemented efficiently, especially in conjunction with the described pre-training and post-training, since post-training with the distorted environmental images does not represent a complete training of the ML model and can therefore be performed with little computational effort.

[0025] As already explained, the environmental images to be distorted using the optical model can be provided by environmental image data that is still interference-free, i.e., that is ...

[0026] When the computer vision unit then performs environmental detection during operation of the motor vehicle, environmental image data from the specific environmental camera of the motor vehicle (which is, specifically, captured through the windshield of the motor vehicle) is analyzed or processed using the machine learning model to detect or recognize objects located in the environment depicted by the environmental image data and depicted in the environmental image data. Such objects can be other road users (e.g., motor vehicles and / or pedestrians and / or cyclists and / or motorcyclists) and / or road markings and / or traffic signs (traffic signs and / or traffic lights).

[0027] Based on the objects detected in this way, i.e., based on corresponding object data describing the detected objects, a driver assistance function of the motor vehicle is then operated, which controls the motor vehicle depending on the detected objects. Such a driver assistance function can, for example, be an autonomous driving function that plans a driving trajectory for collision-free operation through the environment and implements it using longitudinal and lateral guidance of the motor vehicle. Another possible driver assistance function can plan longitudinal and / or lateral guidance for a driving intervention and, for example, avoid an object on a collision course. A driver assistance function can, for example, also provide lane markings for lane keeping functionality and / or traffic sign recognition for cruising control.The object detection carried out here can then also be used by the adapted computer vision unit to detect objects that are far away, especially at a distance greater than 40 meters or greater than 60 meters, since the influence of the optical interference from the windscreen is correspondingly reduced.

[0028] For use cases or application situations that may arise during the method and which are not explicitly described here, it may be provided that, in accordance with the method, an error message and / or a request to enter user feedback is issued and / or a default setting and / or a predetermined initial state is set.

[0029] A further component of the invention is the described trained machine learning model, as it can be stored, for example, in a digital storage medium by model data.

[0030] A further aspect of the invention is also the optical model of an optical disturbance of a viewing area of ​​a windscreen of a motor vehicle, as is also generated in the manner described within the scope of the method.

[0031] The computer vision unit adapted or configured using the method for environmental detection to be performed from a motor vehicle is also considered a component of the invention. This computer vision unit is characterized by the trained machine learning model, which is trained using the distorted environmental images.

[0032] In the manner described, the computer vision unit can be implemented as a processor circuit.

[0033] Finally, a motor vehicle also represents an aspect of the invention, as can be provided as follows. The motor vehicle has a surround-view camera arranged in an interior of the motor vehicle and whose detection range is directed through a viewing area of ​​a windshield of the motor vehicle into the surroundings of the motor vehicle. In particular, the detection range is directed toward a front area of ​​the motor vehicle, i.e., toward the front in the forward direction of travel. The surround-view camera can be arranged, for example, in the area of ​​a rearview mirror of the motor vehicle, for example, in a horizontal center of the windshield at the upper edge of the windshield.The environmental camera is coupled to a computer vision unit of the type described, wherein the computer vision unit is then also additionally coupled to a driver assistance system of the motor vehicle, so that the computer vision unit can receive environmental image data from the environmental camera, can perform the described object recognition and can signal object data from recognized objects to the driver assistance system.

[0034] The control device for the motor vehicle, which can be configured as a computer vision unit, can have a processor circuit or processor device with at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). Furthermore, the processor device can have program code that is configured to carry out the embodiment of the method according to the invention when executed by the processor device. The program code can be stored in a data memory of the processor device. The processor device can be based, for example, on at least one circuit board and / or on at least one SoC (System on Chip).

[0035] The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle.

[0036] As a further solution, the invention also encompasses a computer-readable storage medium comprising program code which, when executed by a computer or computer network, causes the computer to carry out an embodiment of the method according to the invention. The storage medium can be provided at least partially as a non-volatile data memory (e.g. as a flash memory and / or as an SSD - solid state drive) and / or at least partially as a volatile data memory (e.g. as a RAM - random access memory). The storage medium can be arranged in the computer or computer network. However, the storage medium can also be operated on the Internet, for example, as a so-called app store server and / or cloud server. The computer or computer network can provide a processor circuit with, for example, at least one microprocessor.The program code may be provided as binary code and / or as assembly code and / or as source code of a programming language (e.g. C) and / or as a program script (e.g. Python).

[0037] The invention also encompasses combinations of the features of the described embodiments. The invention therefore also encompasses implementations that each comprise a combination of the features of several of the described embodiments, unless the embodiments are described as mutually exclusive.

[0038] Exemplary embodiments of the invention are described below. Shown are: Fig. 1 a schematic representation of a system for adapting a computer vision unit to an optical characteristic of a windscreen of a motor vehicle, and Fig. 2 a sketch illustrating a generation of training image data for retraining a machine learning model of the computer vision unit of Fig. 1.

[0039] The exemplary embodiments explained below are preferred embodiments of the invention. In the exemplary embodiments, the described components of the embodiments each represent individual features of the invention that can be considered independently of one another, each of which also develops the invention independently of one another. Therefore, the disclosure is intended to encompass combinations of the features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0040] In the figures, the same reference symbols designate elements with the same function.

[0041] Fig. 1 shows a system 10 by means of which a machine learning model 13 can be provided for motor vehicles 11 in a respective computer vision unit 12, which takes into account that an environmental camera 14 of the respective motor vehicle 11 must capture or film an environment U outside the motor vehicle 11 through a windshield or front window 15 of the motor vehicle 11 in order to generate environmental images I. For this purpose, a detection area B of the environmental camera 14 captures light from the environment U through a viewing area 16 of the front window 15 during operation and generates the environmental image data I, in which objects O from the environment U can be depicted, for example other road users and / or traffic signs and / or one of the other examples already described.

[0042] The computer vision unit 12 can be coupled to the environmental camera 14 to receive the environmental image data of the environmental images I. The environmental image data can be, for example, raw image data or MPEG image data. The computer vision unit 12 can detect or recognize the objects O in the environmental images I by operating the machine learning model 13 in a manner known per se and signal corresponding object data 17 to a driver assistance function 18 of the motor vehicle 11. For example, the object data 17 can signal an object type of the respective detected object (e.g., "traffic sign" or "road user") together with environmental coordinates of a relative position of the detected object O with respect to the motor vehicle 11 or absolute geocoordinates.

[0043] The driver assistance function 18 can calculate a travel trajectory T of the motor vehicle 11 based on the object data 17, and in doing so, consider or include the detected objects O according to the object data 17. For example, the driver assistance function 18 can generate control commands 20 for at least one actuator 19 of the motor vehicle 11 in order to carry out longitudinal and / or transverse guidance of the motor vehicle 11 by means of the actuator 19 according to or to implement the calculated trajectory T. The at least one actuator 19 can, for example, comprise a steering system and / or a drive system and / or a braking system of the motor vehicle.

[0044] For the recognition of the objects O in the environmental image data I of the environmental camera 14, the machine learning model 13 of the computer vision unit 12 of the motor vehicle 11 can take into account that the light L from the environment U can be deflected when passing through the viewing area 16 of the windscreen 15 by surface structures in a range of 0.5 µm to 1 cm, for example surface waviness and / or transitions between layers within the windscreen 15.

[0045] In order to adapt the machine learning model 13 of the computer vision unit 12 to the optical transmission path as it is impaired or influenced by the viewing area 16 of the front pane 15, the system 10 can be provided for performing a wavefront measurement 21 for the viewing area 16 of the front pane 15. Such a waviness measurement 21 can be performed, for example, using a known measuring method, which can be based on a known method, for example, a measurement of a wavefront of light passing through the viewing area 16. It has proven particularly advantageous if the waviness measurement 21 generates characteristic data 22 that determines the waviness of the front pane 15 on the basis of Zernike coefficients Z iwhere i is the index for the different Zernike coefficients defined in the prior art. In particular, Zernike coefficients with indices 3, 4, and 5, i.e., Z3, Z4, and Z5, have proven advantageous.

[0046] Based on the characteristic data 22, a development platform 23 can adapt or configure a training T of the machine learning model 13 such that the influence or optical disturbance or aberration caused by the windshield 15 in the viewing area 16 is taken into account in the machine learning model 13. For this purpose, the machine learning model can be based, for example, on one or more artificial neural networks.

[0047] The training T can be based on environmental images 25 of objects, which can be used to generate a trained machine learning model 27 in a pre-training step 26, in which the influence of the optical disturbance is not yet taken into account. Subsequently, in a post-training step 28, the influence of the optical disturbance can be trained into the machine learning model 13 based on distorted environmental images 29.

[0048] The development platform 23 can now provide a trained machine learning model 24, which can be installed or operated in the motor vehicle 11 as the machine learning model 13 for the computer vision unit 12.

[0049] The distorted environmental images 29 can be generated on the basis of the characteristic data 22 by the development platform 23 generating an optical model 31 from the characteristic data 22. This optical model 31 simulates the ripple 32, as described by the characteristic data 22, in such a way that the optical disturbance caused by the ripple 32 upon passage of the light L is modeled. The environmental images 25, which were also used for the pre-training 26, a subset thereof, or other environmental images can be converted into the distorted environmental images 29 by means of a filter 30, which are then used as training data D for the post-training 28.

[0050] The filter 30 can, for example, be based on a convolution operation 33, which can alternatively also be implemented in the Fourier domain as a multiplication of the Fourier transform of the environmental images 25 on the one hand and the optical model 31 on the other. The post-training 28 can, for example, be based on the backpropagation algorithm. In order to retain a result of the pre-training 26 in the pre-trained learning model 27, a step size of a change in coefficients of the pre-trained learning model 27 can be limited in magnitude. The number of distorted environmental images 29 can be smaller than the number of environmental images 25 used for the pre-training 26; in particular, it can require fewer than half as many environmental images. Alternatively, it can be provided to distort all environmental images 25 using the optical model 31 in a single training session.

[0051] The characteristic data 22 can be determined individually for the respective windshield 15 of the motor vehicle 11 whose computer vision unit 12 is to be adapted to the windshield 15. Alternatively, the characteristic data 22 can also represent average values ​​from several windshields that are used during a production run M for the manufacture of several motor vehicles 11. Alternatively, characteristic data from prototype windshields can also be used.

[0052] Fig. 2 illustrates how the distorted environmental images 29 can be generated from the environmental images 25 using the environmental image data thereof by means of the filter 30 and the optical model 31.

[0053] It is shown how a distant object 35, which in the surroundings image 25, i.e. in the detection range B of a surroundings camera 14, only occupies an area of ​​less than 10 percent due to its distance, can be depicted with sharp edges in an image section 36. The waviness 32 of the view-through area 16 of the windscreen 15 results in an optical disturbance of the passing light L, as simulated by the optical model 31. The optical model 31 is represented here as a convolution kernel 37, which can be described, for example, by means of the characteristic data 22 by Zernike coefficients, in particular the three Zernike coefficients Z3, Z4 and Z5. The respective distorted surroundings image 29 can be generated by means of the filter 30, for example as a convolution or multiplication in the 2D Fourier domain.For a distant object 35, the result shown in image section 38 is a blurred edge, as an illustrative example. Given the percentage of pixels in the edge area, this can represent a higher percentage of the total pixel count of the imaged object 35 than for larger, closer objects of the same actual size. It should be noted again: The edge is primarily blurred, which is only symbolically illustrated as hatching in the figure.

[0054] With the deployment of high-resolution ADAS cameras (e.g., 4K cameras), this problem of optical aberrations through the windshield has become increasingly important, especially for Level 4 autonomous driving, which requires the localization of potential highway hazards from a distance of 60 meters. In the future, it will be crucial to develop effective and cost-efficient methods to collect individual training data for different vehicle models and optimize the PSF to improve the generalization ability of machine learning models.

[0055] The solution described here is a cost-effective method for improving the generalization capabilities of machine learning models for different windshield configurations. It generates artificially degenerated environmental images that reflect the optical aberrations induced by different windshield types, which are then used for data augmentation in the training and evaluation of machine learning models for autonomous driving. The optical threat model described in the invention is based on optical Fourier principles and assumes that the wavefront aberration map of the windshield is known in advance, either through measurement or optical simulation. The wavefront aberrations are represented by a set of Zernike coefficients, which are widely used in optics to describe the shape of optical surfaces.The wavefront aberration mapping is used to construct a generalized aperture function that represents the combined effects of the windshield and the camera aperture stop.

[0056] Based on the generalized aperture function, the incoherent, non-diffraction-limited point spread function (PSF) is preferably calculated, which fully characterizes the optical system. The PSF describes how light from a point source in object space spreads into image space. The PSF is perturbed to reflect the optical deflections caused by different types of windshields. The perturbed PSF is used to convolve the ground-truth image, resulting in an artificially degenerated image that reflects the optical aberrations caused by the windshield.

[0057] During this process, the sampling rate dictated by the camera image can be taken into account to ensure that the aperture-plane resolution requirements are met. The initial wavefront measurements are performed under this condition, and the perturbed images are generated at the same sampling rate as the ground-truth images. The perturbed images can then be used for data augmentation to train and evaluate machine learning models for autonomous driving on new windshield configurations, eliminating the need to collect additional road data for each vehicle model.

[0058] Thus, the solution provides a cost-effective method to close the training data gap caused by optical variations in different windshield designs and improves the generalization capabilities of machine learning models to unknown windshield configurations.

[0059] In particular with the Zernike coefficients ω n a measured wavefront W can be expressed by the following Zernike decomposition as a weighted sum of the Zernike polynomials Z n (ϕ is the azimuthal angle and ρ is the normalized radial distance): W(ρ,ϕ)=∑n=0∞ωnZn(ρ,ϕ) ie the Zernike coefficient results from the projection of the measured wavefront W onto the individual Zernike polynomials: ωn:=〈W,Zn〉 Starting from the PSF, point spread function, P of the figure, the distorted point spread function (according to Feynman notation) can be calculated: P(x→a):=P(x→a)⋅exp[2πiλ⋅W(x→a)] h(x→o)=F[P(λdzk→a¯)] F[ϕ](k→)=F[|h|2](k→)⋅F[ρ](k→) with the following variables: Position vector in the aperture plane. Position vector in the image plane. λ Average wavelength based on the sensitivity range of the CMOS sensor (see quantum efficiency curve). W Measured wavefront difference with respect to a plane reference wave. P Diffraction-limited aperture function of the camera lens. d z Distance between aperture and observation plane (approximately equal to the focal length f of the lens). Spatial frequency vector in the scaled Fourier space of the aperture plane. Spatial frequency vector in the Fourier space of the image plane. coherent PSF. Generalized aperture function. incoherent PSF ρ Electromagnetic intensity distribution in the image plane without optical aberrations (diffraction-limited case). Electromagnetic intensity distribution in the image plane with optical aberrations (diffraction-limited case) i imaginary number exp() exponential function F Fourier transform

[0060] Forming an optical model from Zernike coefficients can be implemented, for example, based on the publication by J. Goodman, "Introduction to Fourier optics," 3rd ed., by J.W. Goodman. Englewood, CO: Roberts & Co. Publishers, 2005.

[0061] Overall, the examples show how a Fourier-based optical model can be used to make the recognition performance of a computer vision system robust against optical disturbances from a windshield. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] EP 3 982 328 A1

[0005] US 2022 / 0412897 A1

[0006] Zitierte Nicht-Patentliteratur

[0000] J. Goodman, „Introduction to Fourier optics“, 3rd ed., by JW Goodman. Englewood, CO: Roberts & Co. Publishers, 2005

[0060]

Claims

[1] Method for adapting a computer vision unit (12) for environmental detection to be carried out from a motor vehicle, characterized bythat in a predetermined viewing area (16) of a windscreen (15) of the motor vehicle and / or the viewing area of at least one windscreen (15) provided for a vehicle type of the motor vehicle, characteristic data (22) of a manufacturing-related waviness (32) of the viewing area (16) are determined, and from the characteristic data (22) an optical model (31) of an optical disturbance caused by the waviness (32) of a light wave front passing through the viewing area (16) is formed, and by means of the optical model (31) by distorting environmental images from environmental image data of an environmental camera (14), training image data is generated, and by means of the training image data a machine learning model (13), ML model,the computer vision unit (12) is trained and in the motor vehicle the surroundings camera (14) or another surroundings camera (14) is provided in an interior of the motor vehicle and its detection range is aligned through the viewing area of the windscreen (15) of the motor vehicle into the surroundings of the motor vehicle and the computer vision unit (12) is coupled to the surroundings camera (14) for detecting the surroundings., [2] Method according to claim 1, wherein a Zernike analysis is carried out to determine the characteristic data (22) by means of a wavefront measurement (21) and Zernike coefficients are generated as the characteristic data (22). [3] Method according to claim 2, wherein only the Zernike coefficients Z3, Z4, Z5 or one or two thereof are used as characteristic data (22). [4] Method according to one of the preceding claims, wherein an incoherent PSF, point spread function, and / or a convolution kernel (37) or a Fourier transform of a convolution kernel (37) is generated as the optical model (31). [5] Method according to one of the preceding claims, wherein the ML model is first trained (pre-trained) using the environmental image data and / or environmental image data different therefrom, and the training using the training image data is carried out as a post-training of the trained ML model. [6] Method according to one of the preceding claims, wherein during production of the motor vehicle, the windscreen (15) which is intended for installation in the motor vehicle or which is already installed therein is identified and the characteristic data (22) are determined individually for this motor vehicle only on this windscreen (15). [7] Method according to one of the preceding claims, wherein the environmental images (25) which are distorted by means of the optical model (31) are provided by environmental image data which are captured free from an optical disturbance of a windscreen (15). [8] Method according to one of the preceding claims, wherein during operation of the motor vehicle, the computer vision unit (12) receives environmental image data from the environmental camera (14) of the motor vehicle and, by means of the machine learning model, objects (35) depicted in the environmental image data are recognized and a driver assistance function of the motor vehicle is controlled as a function of the recognized objects (35). [9] Machine learning model (13), ML model, or digital storage medium with model data of the ML model, wherein the ML model is trained using a method according to one of the preceding claims. [10] Optical model (31) of an optical disturbance of a viewing area (16) of a windscreen (15) of a motor vehicle, wherein the optical model (31) is generated by means of a method according to one of the preceding method claims. [11] Computer vision unit (12) for environmental detection to be carried out from a motor vehicle, wherein the computer vision unit (12) has a machine learning model (13) according to claim 9. [12] Motor vehicle with an environment camera (14) which is arranged in an interior of the motor vehicle and whose detection range is directed through a viewing area of a windscreen (15) of the motor vehicle into the environment of the motor vehicle, wherein the environment camera (14) is coupled to a computer vision unit (12) according to claim 11, wherein the computer vision unit (12) is coupled to a driver assistance system (10) of the motor vehicle.

Images