Method for recalibrating a monitoring system and monitoring system

Abstract

In a method for recalibrating a monitoring system, the monitoring system comprises a camera, a light source and a dot projector. The monitoring system is arranged to monitor an environment. The dot projector is adapted to project a dot pattern with known dot pattern parameters into the environment. A set of calibration parameters describes properties of the monitoring system. The method comprises illuminating the environment using the light source and acquiring a first 2D image of the environment using the camera. The method further comprises reconstructing a 3D model of the environment from the first 2D image with a trained neural network. The method further comprises projecting the dot pattern into the environment using the dot projector and acquiring a second 2D image of the environment using the camera. The method further comprises rendering a third 2D image from the 3D model using the calibration parameters and the dot pattern parameters. The method further comprises adjusting the calibration parameters based on a comparison between the second 2D image and the third 2D image.

Description

[0001] METHOD FOR RECALIBRATING A MONITORING SYSTEM AND MONITORING

[0002] SYSTEM

[0003] DESCRIPTION

[0004] The present invention relates to a method for recal ibrating a monitoring system and to a monitoring system .

[0005] This patent application claims the priority of German patent application 10 2023 134 154 . 5 , the disclosure content of which is hereby incorporated by reference .

[0006] Monitoring systems for monitoring a driver of a vehicle are known in the state of the art . Known monitoring systems comprise an infrared camera and an infrared light source . Single dot proj ectors are known in the state of the art for depth sensing .

[0007] It is an obj ect of the present invention to provide a method for recalibrating a monitoring system . It is a further obj ect of the present invention to provide a monitoring system .

[0008] These obj ectives are achieved by a method for recal ibrating a monitoring system and by a monitoring system as claimed in the independent claims . Further variants are disclosed in the dependent claims .

[0009] In a method for recalibrating a monitoring system, the monitoring system comprises a camera, a light source , and a dot proj ector . The monitoring system is arranged to monitor an environment . The dot proj ector is adapted to proj ect a dot pattern with known dot pattern parameters into the environment . A set of calibration parameters describes properties of the monitoring system . The method comprises illuminating the environment using the light source and acquiring a first 2D image of the environment using the camera . The method further comprises reconstructing a 3D model of the environment from the first 2D image with a trained neural network . The method further comprises proj ecting the dot pattern into the environment using the dot proj ector and acquiring a second 2D im- age of the environment using the camera . The method further comprises rendering a third 2D image from the 3D model using the calibration parameters and the dot pattern parameters .

[0010] The method further comprises adj usting the calibration parameters based on a comparison between the second 2D image and the third 2D image .

[0011] This method allows for an automatic recalibration o f the monitoring system . Advantageously, the method does not require any human participation, in particular no participation of an expert . The method does not require any additional external equipment either . The method allows for a regular recalibration of the monitoring system which may ensure an accurate and trustworthy function of the monitoring system . One advantage of the method is that it allows a j oint optimization of all calibration parameters describing properties of the monitoring system . This may leverage interdependencies and relationships between these parameters , leading to a more accurate and consistent calibration . The method may allow to find an optimal combination of all calibration parameters .

[0012] In a variant of the method, rendering the third 2D image and adj usting the calibration parameters is carried out repeatedly . This allows for an incrementary optimi zation of the calibration parameters .

[0013] In a variant of the method, rendering the third 2D image and adj usting the calibration parameters is carried out according to a di f ferentiable rendering algorithm . Di f ferentiable rendering is a powerful approach that enables to optimi ze the calibration parameters by iteratively adj usting them and observing the impact on the rendered third 2D image . By comparing the rendered third 2D image with the acquired second 2D image , a feedback loop can be established that guides the adj ustment of the calibration parameters . This iterative approach allows to fine-tune the calibration parameters to minimi ze any discrepancies between the second 2D image and the third 2D image . This optimi zation process may signi ficantly enhance the accuracy of the calibration of the monitoring system .

[0014] In a variant of the method, the environment is a cabin of a vehicle . In this variant , the monitoring system may be provided for monitoring a driver of the vehicle during the normal operation of the monitoring system . Advantageously, the method allows for an automatic recalibration of the monitoring system after the installation of the monitoring system in the cabin of the vehicle . A recalibration of the monitoring system may be useful for compensating ef fects of a prolonged exposure of the monitoring system to mechanical vibrations and temperature fluctuations .

[0015] A variant of the method is initiated after a determined amount of time after a driver of the vehicle has le ft the vehicle . This allows to perform a recalibration of the monitoring system right after the driver of the vehicle has left the vehicle and before shutting down the monitoring system. This allows to perform the recalibration under suitable conditions , particularly avoiding any interference from moving parts or occlusions caused by occupants of the vehicle .

[0016] In a variant of the method, reconstructing the 3D model of the environment from the first 2D image includes determining depth values from the first 2D image with the trained neural network . Advantageously, this allows for a simple , fast , and reliable reconstruction of the 3D model .

[0017] In a variant of the method, reconstructing the 3D model from the first 2D image uses additional information . Advantageously, this may increase the accuracy and reliability of the reconstruction of the 3D model .

[0018] In a variant of the method, the additional information includes a known position of at least one obj ect visible in the first 2D image . This variant takes advantage of the fact that the monitoring system is installed in a known environment . The at least one obj ect visible in the first 2D image may be any obj ect such as a headrest in a cabin of a vehicle or a light source such as an LED or a VCSEL .

[0019] In a variant of the method, the set of calibration parameters includes at least one of intrinsic parameters and extrinsic parameters of the camera . For example , the set of calibration parameters may include at least one of lens parameters of the camera and a spatial arrangement of the camera . Advantageously, the method allows for recalibrating these parameters of the camera of the monitoring system .

[0020] In a variant of the method, the set of calibration parameters includes at least one of intrinsic parameters and extrinsic parameters of the dot proj ector . For example , the set of calibration parameters may include at least one of optical parameters and a spatial arrangement of the dot proj ector . Advantageously, the method allows for recalibrating these parameters of the dot proj ector of the monitoring system.

[0021] A variant of the method further comprises creating a training dataset by generating multiple 2D training images o f virtual environments along with corresponding ground truth depth values and training the neural network to predict the corresponding depth values from the 2D training images . These steps may be carried out once before performing the other steps of the method, for example .

[0022] This method allows to train the neural network based on artificially generated 2D training images and corresponding ground truth depth values . This allows for a simple and cost- effective creation of a training dataset of any des ired si ze , without having to obtain training images and corresponding ground truth depth values from real environments .

[0023] Creating the training dataset from virtual environments is particularly well suited for applications where the monitor- ing system is installed in a known and constrained environment such as a cabin of a vehicle .

[0024] This method may allow to train the neural network such that the trained neural network is capable of reconstructing the 3D model of the environment from the first 2D image with a high degree of reliability and precision later in the execution of the method .

[0025] In a variant of the method, the multiple 2D training images are generated using di f ferent capture parameters . This allows to train the neural network using a training dataset that covers di f ferent capture situations which may occur later in the field .

[0026] In a variant of the method, the virtual environments are simulated cabins of vehicles . This allows to train the neural network to reconstruct a 3D model of a cabin of a vehicle from a first 2D image of the cabin of the vehicle .

[0027] In a variant of the method, the camera is an IR ( infrared) camera, the light source is an IR light source and the dot proj ector is an IR dot proj ector . This allows to operate the monitoring system in a way that is not disturbing to people in the environment that is monitored by the monitoring system .

[0028] A monitoring system comprises a camera, a light source, and a dot proj ector . The monitoring system is adapted to monitor an environment . The dot proj ector is adapted to proj ect a dot pattern with known dot pattern parameters into the environment . A set of calibration parameters describes properties of the monitoring system . The monitoring system is adapted for illuminating the environment using the light source and acquiring a first 2D image of the environment using the camera . The monitoring system is further adapted for reconstructing a 3D model of the environment from the first 2D image with a trained neural network . The monitoring system is further adapted for proj ecting the dot pattern into the environment using the dot proj ector and acquiring a second 2D image of the environment using the camera . The monitoring system is further adapted for rendering a third 2D image from the 3D model using the calibration parameters and the dot pattern parameters . The monitoring system is further adapted for adj usting the calibration parameters based on a comparison between the second 2D image and the third 2D image .

[0029] This monitoring system allows for an automatic recalibration of the calibration parameters . Advantageously, the recalibration does not require any human participation, in particular no participation of an expert . The recalibration does not require any additional external equipment either . The recalibration allows for a regular recalibration of the monitoring system which may ensure an accurate and trustworthy function of the monitoring system . One advantage of the monitoring system is that it allows a j oint optimi zation of al l calibration parameters describing properties of the monitoring system . This may leverage interdependencies and relationships between these parameters , leading to a more accurate and consistent calibration . Recalibrating the monitoring system may allow to find an optimal combination of all calibration parameters .

[0030] The above-described properties , features , and advantages of the invention, as well as the way in which they are achieved, will become more clearly and comprehensively understandable in connection with the following description of exemplary variants , which will be explained in more detail in connection with the drawings in which in schematic representation :

[0031] Fig . 1 shows a monitoring system for monitoring an environment ;

[0032] Fig . 2 shows a sequence diagram of a method for recalibrating the monitoring system; and Fig . 3 shows a sequence diagram of method steps for training a neural network of the monitoring system .

[0033] Fig . 1 shows a highly schematic depiction of a monitoring system 100 that is arranged to monitor an environment 200 .

[0034] In the example of Fig . 1 , the environment 200 is a cabin 201 of a vehicle . In this example , a driver seat 202 and a passenger seat 203 are arranged in the environment 200 .

[0035] The monitoring system 100 may serve to monitor a driver of the vehicle to veri fy that the driver is awake and attentive , for example . The monitoring system 100 may provide additional functionalities such as a secure face authentication of the driver of the vehicle , a measurement of a distance between the driver and an airbag of the vehicle , and a tracking of a position of the head of the driver in 3D space .

[0036] The monitoring system 100 comprises a camera 110 , a light source 120 , and a dot proj ector 130 . The camera 110 is adapted for acquiring 2D images of the environment 200 . The light source 120 is a flood light source that is adapted to illuminate the environment 200 . The dot proj ector 130 is adapted to proj ect a dot pattern 132 with known dot pattern parameters 131 into the environment 200 . The dot pattern parameters 131 describe the geometry of the dot pattern 132 , for example the number and angular positions of the dots . The camera 110 may be an IR ( infrared) camera . The light source 120 may be an IR light source . The dot proj ector 130 may be an IR dot proj ector .

[0037] The monitoring system 100 comprises a set of calibration parameters 101 . The calibration parameters 101 may include intrinsic parameters of the camera 110 such as lens parameters of the camera 110 including, for example , a focal length, a principal point , and distortion characteristics of the camera 110 . The calibration parameters 101 may also include extrinsic parameters of the camera 110 such as a spatial relation- ship between a proj ection centre of the camera 110 and the environment 200 .

[0038] The calibration parameters 101 may further include intrinsic parameters of the dot proj ector 130 , such as , for example , optical parameters like a focal length, a principal point , distortion characteristics , and a virtual view . The calibration parameters 101 may also include extrinsic parameters of the dot proj ector 130 , such as an alignment and a positioning of the dot proj ector 130 with respect to the environment 200 and the camera 110 .

[0039] For proper operation and to obtain high quality results , it is necessary that the monitoring system 100 holds accurate values of the calibration parameters 101 . In the environment 200 , however, the monitoring system 100 may be exposed to factors such as mechanical vibrations and temperature fluctuations that may af fect properties of the monitoring system 100 . To this end, it may be necessary to recalibrate the set of calibration parameters 101 once or regularly after the installation of the monitoring system 100 in the environment 200 .

[0040] Recalibration of the monitoring system 100 can be carried out automatically using a recalibration method 300 that is depicted schematically in Fig . 2 . The recalibration method 300 may be carried out by a control unit 150 of the monitoring system 100 , for example .

[0041] In the case that the environment 200 is the cabin 201 of the vehicle , the recalibration method 300 may be initiated after a determined amount of time after a driver of the vehicle has left the vehicle , for example . It is convenient to initiate the recalibration method 300 after all occupants of the vehicle have left the vehicle , but before completely shutting down the vehicle . This ensures that the monitoring system 100 is recalibrated under ideal conditions , avoiding any inter- ference from moving parts or occlusions caused by occupants of the vehicle .

[0042] The recalibration method 300 starts with a first step 310 to illuminate the environment 200 using the light source 120 and acquire a first 2D image 311 of the environment 200 using the camera 110 . In the case that the environment 200 is the cabin 201 of a vehicle , the first 2D image 311 may show the driver seat 202 and the passenger seat 203 , for example .

[0043] In a second step 320 , a 3D model 321 is reconstructed from the first 2D image 311 with a trained neural network 160 . An example of a suitable neural network is described in the article "Depth Map Prediction from a Single Image using a Multi-Scale Deep Network" , D . Eigen et al . , Advances in neural information processing systems 27 ( 2014 ) .

[0044] Reconstructing the 3D model 321 of the environment 200 from the first 2D image 311 may include determining depth values 322 from the first 2D image 311 for each pixel of the first 2D image 311 with the trained neural network 160 . This allows to use all pixels of the first 2D image 311 for recalibrating the monitoring system 100 , allowing for a reliable calibration, in particular a reliable calibration of distortion parameters .

[0045] Reconstructing the 3D model 321 from the first 2D image 311 may use additional information 312 to further enhance an accuracy of the reconstruction of the 3D model 321 . This is described in "Estimating Depth from RGB and Sparse Sensing" , Z . Chen et al . , Proceedings of the European Conference on Computer Vision (ECCV) ( 2018 ) , for example . The additional information 312 may include a known position 211 of one or more obj ects 210 that are located in the environment 200 and visible in the first 2D image 311 . Such obj ects 210 may be headrests , pillars , or light sources such as IR LEDs or IR VCSELs , for example . In a third step 330 of the recalibration method 300 , the dot pattern 132 is proj ected into the environment 200 using the dot proj ector 130 , and a second 2D image 331 of the environment 200 is acquired using the camera 110 .

[0046] In a fourth step 340 of the recalibration method 300 , a third 2D image 341 is rendered from the 3D model 321 using the calibration parameters 101 and the dot pattern parameters 131 .

[0047] The second 2D image 331 shows a real image acquired with the camera 110 of the environment 200 with the dot pattern 132 proj ected into the environment 200 using the dot proj ector 130 .

[0048] The third 2D image 341 shows a virtual image that s imulates the same situation based on the 3D model 321 , the dot pattern parameters 131 and the calibration parameters 101 . The third 2D image 341 simulates proj ecting the dot pattern 132 into the environment 200 using the dot proj ector 130 and acquiring a 2D image of the environment 200 using the camera 110 . The third 2D image 341 shows how that image would look in case that the current set of calibration parameters 101 would correctly describe the properties of the monitoring system 100 .

[0049] In a fi fth step 350 of the recalibration method 300 , the calibration parameters 101 are adj usted based on a comparison between the second 2D image 331 and the third 2D image 341 .

[0050] Inaccuracies of the calibration parameters 101 cause di f ferences between the second 2D image 331 and the third 2D image 341 . A comparison between the second 2D image 331 and the third 2D image 341 thus allows to assess the accuracy of the calibration parameters 101 and to adj ust the calibration parameters 101 .

[0051] The fourth step 340 of rendering the third 2D image 341 and the fi fth step 350 of adj usting the calibration parameters 101 may be carried out repeatedly to adj ust the cal ibration parameters 101 such that the third 2D image 341 matches the second 2D image 331 increasingly well.

[0052] The fourth step 340 of rendering the third 2D image 341 and the fifth step 350 of adjusting the calibration parameters 101 may be carried out according to a differentiable rendering algorithm 360, as described in M. Hannemose et al., "Su- peraccurate Camera Calibration via Inverse Rendering", Modeling Aspects in Optical Metrology VII, Vol. 11057, SPIE (2019) and H. Kato, "Differentiable Rendering: A Survey", arXiv preprint arXiv: 2006.12057 (2020) , for example.

[0053] In case that the dot pattern parameters 131 are also subject to change, the dot pattern parameters 131 may be adjusted along with the calibration parameters 101 in the fifth step 350 of the recalibration method 300. In this case, the dot pattern parameters 131 may be regarded as belonging to the calibration parameters 101 of the monitoring system 100.

[0054] The second step 320 of the recalibration method 300 uses the trained neural network 160 to reconstruct the 3D model 321 of the environment 200 from the first 2D image 311. Since the environment 200 may be a highly constrained environment such as the cabin 201 of a vehicle, the neural network 160 can be trained to perform the reconstruction of the 3D model 321 in a reliable and accurate way.

[0055] Training the neural network 160 may be carried out using a training method 400 that is schematically depicted in Fig. 3, for example. The training method 400 may be carried out once, for example, before installing the trained neural network 160 in the monitoring system 100. Copies of the same trained neural network 160 may be installed in a plurality of instances of the monitoring system 100.

[0056] In a first step 410 of the training method 400, a training dataset 413 is created by generating multiple 2D training im- ages 414 of virtual environments 411 along with corresponding ground truth depth values 415 .

[0057] This leverages the fact that the environment 200 may be highly constrained, such that the expected geometry and appearance of the environment 200 is largely known . This allows to construct one or more virtual environments 411 that simulate the geometry and appearance of the real environment 200 .

[0058] For example , in the case that the real environment 200 is the cabin 201 of a vehicle , the virtual environments 411 are simulated cabins of vehicles . Di f ferent virtual environments 411 may simulate di f ferent models of vehicles or vehicle having dif ferent outfits and configurations , for example .

[0059] The 2D training images 414 can be generated by rendering views of the virtual environments 411 . Multiple 2D training images 414 can be generated using di f ferent capture parameters 412 to simulate di f ferent camera positions , di f ferent camera parameters or di f ferent lighting situations , for example .

[0060] Since the 2D training images 414 are generated from the virtual environments 411 , the corresponding ground truth depth values 415 can be derived in a mathematical way from the virtual environments 411 and the capture parameters 412 .

[0061] In a second step 420 of the training method 400 , the neural network 160 is trained to predict the corresponding ground truth depth values 415 from the 2D training images 414 .

[0062] Since the training dataset 413 can be generated in any si ze without much ef fort , the training dataset 413 can be made large enough to fully train the neural network 160 to a high degree of reliabilty .

[0063] The invention has been illustrated and described in more detail with the aid of exemplary variants . The invention is not , however, restricted to the examples disclosed . Rather, other variants may be derived therefrom by the person skilled in the art .

[0064] REFERENCE SYMBOLS monitoring system set of calibration parameters camera light source dot proj ector dot pattern parameters dot pattern control unit trained neural network environment cabin driver seat passenger seat obj ect position recalibration method first step first 2D image additional information second step 3D model depth values third step second 2D image fourth step third 2D image fi fth step 360 di f ferentiable rendering algorithm

[0065] 400 training method

[0066] 410 first step

[0067] 411 virtual environment

[0068] 412 capture parameters

[0069] 413 training dataset 414 2D training images

[0070] 415 corresponding ground truth depth values

[0071] 420 second step

Claims

CLAIMS1. A method for recalibrating a monitoring system (100) , wherein the monitoring system (100) comprises a camera (110) , a light source (120) , and a dot projector (130) , wherein the monitoring system (100) is arranged to monitor an environment (200) , wherein the dot projector (130) is adapted to project a dot pattern (132) with known dot pattern parameters (131) into the environment (200) , wherein a set of calibration parameters (101) describes properties of the monitoring system (100) , the method comprising:- illuminating the environment (200) using the light source (120) and acquiring a first 2D image (311) of the environment (200) using the camera (110) ;- reconstructing a 3D model (321) of the environment (200) from the first 2D image (311) with a trained neural network (160) ;- projecting the dot pattern (132) into the environment (200) using the dot projector (130) and acquiring a second 2D image (331) of the environment (200) using the camera (110) ;- rendering a third 2D image (341) from the 3D model (321) using the calibration parameters (101) and the dot pattern parameters (131) ;- adjusting the calibration parameters (101) based on a comparison between the second 2D image (331) and the third 2D image (341) .

2. The method according to claim 1, wherein rendering the third 2D image (341) and adjusting the calibration parameters (101) is carried out repeatedly.

3. The method according to claim 2, wherein rendering the third 2D image (341) and adjustingthe calibration parameters (101) is carried out according to a differentiable rendering algorithm (360) .

4. The method according to one of the previous claims, wherein the environment (200) is a cabin (201) of a vehicle .

5. The method according to claim 4, wherein the method is initiated after a determined amount of time after a driver of the vehicle has left the vehicle .

6. The method according to one of the previous claims, wherein reconstructing the 3D model (321) of the environment (200) from the first 2D image (311) includes determining depth values (322) from the first 2D image (311) with the trained neural network (160) .

7. The method according to one of the previous claims, wherein reconstructing the 3D model (321) from the first 2D image (311) uses additional information (312) .

8. The method according to claim 7, wherein the additional information (312) includes a known position (211) of at least one object (210) visible in the first 2D image (311) .

9. The method according to one of the previous claims, wherein the set of calibration parameters (101) includes at least one of intrinsic parameters and extrinsic parameters of the camera (110) .

10. The method according to claim 9, wherein the set of calibration parameters (101) includes at least one of lens parameters of the camera (110) and a spatial arrangement of the camera (110) .

11. The method according to one of the previous claims, wherein the set of calibration parameters (101) includes at least one of intrinsic parameters and extrinsic parameters of the dot projector (130) .

12. The method according to claim 11, wherein the set of calibration parameters (101) includes at least one of optical parameters and a spatial arrangement of the dot projector (130) .

13. The method according to one of the previous claims, the method further comprising- creating a training dataset (413) by generating multiple 2D training images (414) of virtual environments(411) along with corresponding ground truth depth values (415) ;- training the neural network (160) to predict the corresponding depth values (415) from the 2D training images (414) .

14. The method according to claim 13, wherein the multiple 2D training images (414) are generated using different capture parameters (412) .

15. The method according to one of claims 13 and 14, wherein the virtual environments (411) are simulated cabins of vehicles.

16. The method according to one of the previous claims, wherein the camera (110) is an IR camera, the light source (120) is an IR light source and the dot projector (130) is an IR dot projector.

17. A monitoring system (100) comprising a camera (110) , a light source (120) and a dot pro j ector ( 130 ) , wherein the monitoring system (100) is adapted to monitor an environment (200) ,wherein the dot projector (130) is adapted to project a dot pattern (132) with known dot pattern parameters (131) into the environment (200) , wherein a set of calibration parameters (101) describes properties of the monitoring system (100) , wherein the monitoring system (100) is adapted for:- illuminating the environment (200) using the light source (120) and acquiring a first 2D image (311) of the environment (200) using the camera (110) ;- reconstructing a 3D model (321) of the environment (200) from the first 2D image (311) with a trained neural network (160) ;- projecting the dot pattern (132) into the environment (200) using the dot projector (130) and acquiring a second 2D image (331) of the environment (200) using the camera (110) ;- rendering a third 2D image (341) from the 3D model (321) using the calibration parameters (101) and the dot pattern parameters (131) ;- adjusting the calibration parameters (101) based on a comparison between the second 2D image (331) and the third 2D image (341) .

Images