Method for operating a driver monitoring system and driver monitoring system

Abstract

A method for operating a driver monitoring system with the following steps is provided: - defining a plurality of predetermined points (8) within a vehicle cabin (4), - illuminating the plurality of predetermined points (8) with the plurality of light dots (7) projected by the dot projector (2) according to a predetermined pattern (12), - detecting light of the light dots (7) reflected by the predetermined points (8) with the camera (3) such that a calibration pattern (9) is generated, - illuminating the vehicle cabin (4) with the plurality of light dots (7) projected by the dot projector (2) according to the predetermined pattern (12), - detecting the light of the light dots (7) reflected within the vehicle cabin (4) with the camera (3) such that an operation pattern (10) is generated, - comparing the operation pattern (10) with the calibration pattern (9), and - calibrating extrinsic parameters and / or intrinsic parameters of the dot projector (2) and / or extrinsic parameters and / or intrinsic parameters of the camera (3) when the operation pattern (10) differs from the calibration pattern (9). Further, a driver monitoring system is provided.

Description

[0001] Description

[0002] METHOD FOR OPERATING A DRIVER MONITORING SYSTEM AND DRIVER MONITORING SYSTEM

[0003] A method for operating a driver monitoring system and a driver monitoring system are provided .

[0004] An improved method for operating a driver monitoring system, particularly with a 3D sensitivity, is to be provided . Further, an improved driver monitoring system, particularly with a 3D sensitivity, is to be provided .

[0005] These obj ects are achieved with a method for operating a driver monitoring system with the steps of claim 1 and a driver monitoring system with the features of claim 17 .

[0006] Developments and improvements of the method and the driver monitoring system are given in the respective dependent claims .

[0007] The driver monitoring system to be operated by the present method comprises a dot proj ector configured for proj ecting a plurality of light dots and a camera configured for detecting light of the light dots . Particularly, the dot proj ector and the camera are comprised or form a structured light system . Particularly, at present a camera is used for detection of light and not a time-of- f light sensor .

[0008] For example , the dot proj ector comprises a VCSEL-chip (VCSEL short for "vertical cavity surface emitting laser" ) or an LED-chip ( LED short for " light emitting diode" ) as light source . Further, the dot proj ector comprises for example a lens array, such as a microlens array for the generation of a plurality of light dots from the light of the VSCEL-chip or the LED-chip . The camera is particularly configured for detecting the light of the light dots emitted by the dot proj ector . Particularly, the camera has a plurality of pixels and is configured to capture an image of a scene within a field of view of the camera .

[0009] According to an embodiment of the method, a plurality of predetermined points within the environment of the driver monitoring system, such as a vehicle cabin, is defined . Particularly, the driver monitoring system is configured to be mounted in a vehicle cabin, for example in front of the driver seat . The predetermined points are defined by 3D coordinates of a vehicle coordinate system, which is particularly a three-dimensional Cartesian coordinate system fixed to the vehicle comprising the vehicle cabin . For example , an origin of the vehicle coordinate system is arranged within the vehicle , for example in a center of gravity of the vehicle .

[0010] According to a further embodiment of the method, the plurality of predetermined points is illuminated with the plurality of light dots proj ected by the dot proj ector according to a predetermined pattern . Particularly, each predetermined point within the vehicle cabin is illuminated by one light dot of the plurality of light dots proj ected by the dot proj ector according to the predetermined pattern . Particularly, the predetermined pattern is factory-set .

[0011] According to a further embodiment of the method, light of the light dots reflected by the predetermined points is detected by the camera such that a calibration pattern is generated . In other words , the predetermined points reflect the light of the light dots and the reflected light is detected by the camera, for example by capturing an image . Particularly, calibration pattern is stored by the driver monitoring system .

[0012] Particularly, the calibration pattern is a two-dimensional image of the scene within the field of view of the camera and the reflected light of the light dots forms dots within the image . Particularly, the calibration pattern is formed by the dots of the reflected light recorded by the camera in the form of an image . The dots of the calibration pattern have two-dimensional coordinates within the image . For example , x- coordinates of an x-axis are the number of the pixels of the camera in x-direction and y-coordinates of a y-axis are the number of the pixels of the camera in y-direction .

[0013] According to a further embodiment of the method, the vehicle cabin is illuminated with the plurality of light dots proj ected by the dot proj ector according to the predetermined pattern and the light of the light dots reflected within the vehicle cabin are detected with the camera such that an operation pattern is generated . Particularly, these steps are carried out after the generation of the calibration pattern . For example , the operation pattern is generated during driving of the vehicle or during an interruption of a drive of the vehicle .

[0014] Particularly, the operation pattern is , as the calibration pattern, a two-dimensional image of the scene within the field of view of the camera and the reflected light of the light dots forms dots within the image . Particularly, the operation pattern is formed by the dots of the reflected light recorded by the camera in the form of an image . The dots of the operation pattern have two-dimensional coordinates within the image as described above in connection with the calibration pattern .

[0015] According to a further embodiment , the operation pattern is compared with the calibration pattern . Particularly, during comparison of the operation pattern and the calibration pattern, the two-dimensional coordinates of the operation pattern and the calibration pattern are compared .

[0016] According to a further embodiment of the method, extrinsic parameters and / or intrinsic parameters of the dot proj ector and / or extrinsic parameters and / or intrinsic parameters of the camera are calibrated when the operation pattern di f fers from the calibration pattern . Particularly, calibration is carried out , when a di f ference of the two-dimensional coordinates of the image of the calibration pattern and the two-dimensional coordinates of the image of the operation pattern exceeds a predetermined threshold .

[0017] Intrinsic camera parameters are particularly a focal length, a principal point , or a distortion characteristic of a lens of the camera . Intrinsic parameters of the lens of the camera, for example , can change over time due , for example due to change in temperature . The intrinsic parameters of the camera can be , for example , calibrate with the help of a Levenberg-Marquardt algorithm .

[0018] Extrinsic parameters of the camera are particularly a spatial relationship between a proj ection center of the camera and the vehicle coordinate system . In other words , extrinsic parameters of the camera are dependent on the position of the camera within the vehicle cabin, for example characteri zed by 3D coordinates of the camera within the vehicle coordinate system . Particularly, the extrinsic parameters of the camera can change due to mechanical impact , such as mechanical vibrations and / or mechanical shock events of the vehicle cabin . Intrinsic parameters of the dot proj ector are , for example , the focal length, the principal point , a distortion and a virtual view . The extrinsic parameters of the dot proj ector are defined as , for example , the alignment and positioning of the dot proj ector with respect to the camera . For example , extrinsic parameters of the dot proj ector are defined with respect to the position of the camera .

[0019] From the two-dimensional coordinates of an image of the camera, 3D coordinates , for example of a vehicle coordinate system can be derived by a PnP algorithm ( short for : Perspective-n-Point algorithm) during calibrating extrinsic parameters and / or intrinsic parameters of the camera and / or the dot proj ector .

[0020] The method for operating the driver monitoring system comprising the dot proj ector configured for proj ecting the plurality of light dots and the camera configured for detecting light of the light dots comprises the following steps :

[0021] - defining the plurality of predetermined points within the vehicle cabin,

[0022] - illuminating the plurality of predetermined points with the plurality of the light dots proj ected by the dot proj ector according to the predetermined pattern,

[0023] - detecting light of the light dots reflected by the predetermined points with the camera such that the calibration pattern is generated, - illuminating the vehicle cabin, particularly again with the plurality of light dots proj ected by the dot proj ector according to the predetermined pattern,

[0024] - detecting the light of the light dots reflected within the vehicle cabin with the camera such that the operation pattern is generated,

[0025] - comparing the operation pattern with the calibration pattern, and

[0026] - calibrating the extrinsic parameters and / or the intrinsic parameters of the dot proj ector and / or the extrinsic parameters and / or the intrinsic parameters of the camera, when the operation pattern di f fers from the calibration pattern .

[0027] Preferably, the above-mentioned steps are carried out in the given order . Particularly, with the present method, the driver monitoring system is calibrated .

[0028] The present method particularly provides a possibility of recalibrating the driver monitoring system when extrinsic or intrinsic parameters of the dot proj ector and / or the camera change during operation of the driver monitoring system, for example due to temperature variations and / or mechanical impact such as vibrations or shocks . Change of intrinsic or extrinsic parameters due to temperature variation are often elastic which means that the sensor can recover its original state . Particularly, mechanical vibrations or shocks can lead to inelastic changes such that a recalibration is necessary .

[0029] Particularly, the method ensures continuous recalibration of the system leading to a reliable performance assessment . Further, deviation or miscalibrations can be detected promptly, leading to timely corrective measures . In particular, it is not necessary to use an external device , such as a calibration board, for the generation of the calibration pattern during the present method . Instead, the predetermined points are defined in the vehicle cabin as reference points during calibration . Therefore , calibration can be carried out by a non-expert person, such as the driver, for example , on the street . It is not necessary that an expert conducts calibration in a garage .

[0030] According to a further embodiment of the method, the dot proj ector emits infrared light and the camera detects the infrared light . Particularly, the infrared light is not visible to the human eye . Therefore , the light dots are not visible for the driver with advantage . For example , the infrared light has a wavelength between and including 780 nanometer and 1400 nanometer .

[0031] According to a further embodiment of the method, the predetermined points are arranged within the field of view of the camera at di f ferent depths . Particularly, the predetermined points have di f ferent distances to the camera . Particularly, the predetermined points are not comprised by one common plane . In such a way, the operation method, and particularly recalibration, can be carried out in a more robust way . Particularly, performance of the method is improved by this feature .

[0032] According to a further embodiment of the method, calibrating the extrinsic parameters and / or the intrinsic parameters of the dot proj ector and / or the extrinsic parameters and / or the intrinsic parameters of the camera is carried out in iterative steps . Particularly, the following steps are repeated iteratively until a deviation between the operation pattern and the calibration pattern does not exceed a predetermined threshold :

[0033] - illuminating the vehicle cabin with the plurality of light dots proj ected by the dot proj ector according to the predetermined pattern,

[0034] - detecting the light of the light dots reflected within the vehicle cabin with the camera such that an operation pattern is generated,

[0035] - comparing the operation pattern with the calibration pattern, and

[0036] - calibrating the extrinsic parameters and / or the intrinsic parameters of the dot proj ector and / or the extrinsic parameters and / or the intrinsic parameters of the camera, when the operation pattern di f fers from the calibration pattern .

[0037] According to a further embodiment of the method, the driver monitoring system comprises at least one additional light source configured for emitting light of the light dots .

[0038] Therefore , the light of the light source can be detected by the camera . Particularly, the at least one additional light source is arranged in the vehicle cabin, for example in the field of view of the camera . In particular, the driver monitoring system comprises a plurality of additional light sources configured for emitting light of the light dots .

[0039] According to a further embodiment of the method, at least one predetermined point within the vehicle cabin is provided with the additional light source . In other words , at least one predetermined point within the vehicle cabin has an additional light source , which emits the same light as the dot proj ector . Particularly, all predetermined points are provided with an additional light source . For example , the at least one additional light source comprises an LED-chip or a VCSEL-chip .

[0040] It is also possible that the additional light sources are placed on locations being di f ferent from the predetermined points . For example , the additional light sources are arranged within a car headliner in a part of the car which is adj acent to the rear window .

[0041] According to a further embodiment of the method, the at least one additional light source is switched on during detecting the light of the light dots reflected within the vehicle cabin with the camera such that the operation pattern is generated . In this embodiment the predetermined points do not only reflect light of the dot proj ector, but actively emit light of the additional light sources , which is detected by the camera . In such a way recalibration can be improved .

[0042] According to a further embodiment of the method, the predetermined points are located on fixed elements within the vehicle cabin . Particularly, the fixed elements are fixed to the vehicle . For example , the predetermined points are located on a car headliner and / or on a car trim and / or a back seat bench as fixed elements . I f the predetermined points are located on fixed elements , it is not necessary to take a movement of the fixed elements on which the predetermined points are located into account .

[0043] According to a further embodiment of the method, the predetermined points are located on movable elements configured to be moved by a motor . For example , the movable elements are a headrest and / or a front seat and / or a pillar . Within this embodiment of the method, the operation pattern is corrected by a feedback signal of a control unit controlling the motor . In such a way targeted movement of the movable elements can be taken into account during recalibration .

[0044] According to a further embodiment of the method, the generation of the calibration pattern is started when initiali zing the driver monitoring system . Particularly, the generation of the calibration pattern takes place automatically when initiali zing the driver monitoring system and does not have to be triggered externally, for example by a driver of the vehicle . Particularly, when starting the motor, initiali zing the driver monitoring system is started . During initiali zing the driver monitoring system, particularly, the following method steps are carried out :

[0045] - defining the plurality of predetermined points within the vehicle cabin,

[0046] - illuminating the plurality of predetermined points with the plurality of light dots proj ected by the dot proj ector according to the predetermined pattern,

[0047] - detecting light of the light dots reflected by the predetermined points with the camera such that the calibration pattern is generated .

[0048] According to a further embodiment of the method, the generation of the operation pattern is carried out , when the vehicle cabin is free of occupants . This has the advantage that the predetermined points are clearly visible for the camera . For example , i f the predetermined points are located on movable elements it is possible to reposition the movable elements to a predetermined original position, which was present when the calibration pattern was determined . According to a further embodiment of the method, the following steps for recalibration are configured to be triggered by a human user :

[0049] - detecting the light of the light dots reflected within the vehicle cabin with the camera such that an operation pattern is generated,

[0050] - comparing the operation pattern with the calibration pattern, and

[0051] - calibrating the extrinsic parameters and / or the intrinsic parameters of the dot proj ector and / or the extrinsic parameters and / or the intrinsic parameters of the camera, when the operation pattern di f fers from the calibration pattern .

[0052] For example , the human user triggers the recalibration steps before leaving the car and the recalibration steps are carried out when the driver has left the car . This has the advantage that no occupants hinder the view . It is also possible that the recalibration steps are carried out automatically in a periodic manner .

[0053] According to a further embodiment of the method, extrinsic parameters and / or intrinsic parameters of the camera are calibrated when deviations of two-dimensional coordinates of light dots of the operation patterns and two-dimensional coordinates of light dots of the calibration pattern increase a predetermined threshold .

[0054] Particularly, when both intrinsic and extrinsic parameters are to be calibrated, we apply classic non-linear methods considering potential degeneracy issues and marker placement for calibration . According to a further embodiment of the method, two- dimensional coordinates of light dots of a pitch of the operation pattern are compared with two-dimensional coordinates of light dots of the pitch of the calibration pattern and extrinsic parameters of the dot proj ector are calibrated when a deviation between the two-dimensional coordinates of the light dots of the pitch of the operation pattern and the two-dimensional coordinates of the light dots s of the pitch of the calibration pattern exceeds a predetermined threshed .

[0055] A driver monitoring system to be operated with the method desribed is disclosed in the following . Features and embodiments described in connection with the method can be also embodied within the driver monitoring system and vice versa .

[0056] According to an embodiment , the driver monitoring system comprises a dot proj ector configured for proj ecting a plurality of light dots illuminating a plurality of predetermined points according to a predetermined pattern .

[0057] According to a further embodiment , the driver monitoring system comprises a camera configured for detecting light of the light dots reflected by the predetermined points with the camera such that a calibration pattern and / or an operation pattern is generated .

[0058] According to a further embodiment of the driver monitoring system extrinsic parameters and / or intrinsic parameters of the dot proj ector and / or extrinsic parameters and / or intrinsic parameters of the camera are calibrateable , when the operation pattern di f fers from the calibration pattern .

[0059] Particularly, the driver monitoring system comprises :

[0060] - the dot proj ector configured for proj ecting the plurality of light dots illuminating the plurality of predetermined points according to the predetermined pattern,

[0061] - the camera configured for detecting light of the light dots reflected by the predetermined points such that a calibration pattern and / or an operation pattern is generated, wherein extrinsic parameters and / or intrinsic parameters of the dot proj ector and / or extrinsic parameters and / or intrinsic parameters of the camera are calibrateable , when the operation pattern di f fers from the calibration pattern .

[0062] According to a further embodiment , the driver monitoring system comprises a flood illuminator configured for illuminating a field of view of the camera . Particularly, the flood illuminator emits visible light during operation . For example , during operation of the driver monitoring system images of the scene within the field of view of the camera are taken, wherein the scene is alternatingly illuminated with the flood illuminator and the dot proj ector in order to achieve visible images of the scene besides the images for calibration .

[0063] Particularly, the driver monitoring system is part of a vehicle cabin .

[0064] Further advantageous embodiments and developments of the method for operating a driver monitoring system and the driver monitoring system result from the exemplary embodiment described below in connection with the Figures . Figure 1 shows a schematic view on a driver cabin with a driver monitoring system according to an exemplary embodiment .

[0065] Figure 2 shows schematically a flow diagram of a method for operating a driver monitoring system according to an exemplary embodiment .

[0066] Figure 3 shows a schematic view on a driver cabin with a driver monitoring system according to a further exemplary embodiment .

[0067] Figures 4 to 6 show stages of a method for operating a driver monitoring system according to an exemplary embodiment .

[0068] Figure 7 shows a stage of a method for operating a driver monitoring system according to a further exemplary embodiment .

[0069] Figure 8 shows a stage of a method for operating a driver monitoring system according to a further exemplary embodiment .

[0070] Figures 9 to 15 show stages of a method for operating a driver monitoring system according to an exemplary embodiment .

[0071] Equal or similar elements as well as elements of equal function are designated with the same reference signs in the Figures . The Figures and the proportions of the elements shown in the Figures are not regarded as being shown to scale . Rather, single elements , in particular layers , can be shown exaggerated in magnitude for the sake of better presentation and / or better understanding .

[0072] The driver monitoring system 1 according to the exemplary embodiment of Figure 1 comprises a dot proj ector 2 and a camera 3 . The dot proj ector 2 and the camera 3 are comprised by a structured light system . The driver monitoring system 1 is installed in a vehicle cabin 4 in front of the head of a driver . Figure 1 shows the driver monitoring system 1 schematically in plan view inserted in a vehicle cabin 4 .

[0073] Particularly, the driver monitoring system 1 is able to detect , for example , 3D obj ects , such as the head of the driver 5 . Detection of the 3D obj ects , such as the head of the driver 5 , are based on a dynamical distance measurement with the help of the reflected light . At present , the method for operating the driver monitoring system 1 is particularly configured to calibrate the driver monitoring system 1 .

[0074] For example , the dot proj ector 2 comprises an LED-chip or an a VCSEL-chip for producing light rays generating a predermined pattern of light dots 7 , when impinging on fixed or movable elements of the vehicle cabin . Further, the driver monitoring system 1 comprises a flood illuminator 6 for illumination of the scene of the field of view of the camera 3 .

[0075] The light dots 7 emitted by the dot proj ector 2 have infrared light at present and the camera 3 is configured to detect the infrared light emitted by the dot proj ector 2 . The flood illuminator 6 emits visible light in order to achieve visible images of the scene within the field of view of the camera 3 . For example , the camera 3 has a high resolution such as 2592 pixels x 1944 pixels and a wide hori zontal field of view, for example with an opening angle of about 110 ° .

[0076] For example , the driver monitoring system 1 has a base calibration, for example carried out in the factory after manufacturing and before delivery to the customer using the method disclosed in PCT SG2023 / 050173 , the disclosure content of which is incorporated herein by reference . The base calibration at the factory determines initial intrinsic and extrinsic parameters of the camera 3 as well as of the dot proj ector 2 . Particularly, the base calibration defines the predetermined pattern 12 .

[0077] A plurality of predetermined points 8 is illuminated with the plurality of light dots 7 proj ected by the dot proj ector 2 according to a predetermined pattern 12 . The light dots 7 are generated when the light rays emitted by the dot proj ector 2 impinge in surfaces within the vehicle cabin 4 . For example , the surfaces within the vehicle cabin 4 are surfaces of fixed elements like a back seat bench or surfaces of movable elements such as a front seat or a head rest . Particularly, the predetermined points 8 are arranged in a di f ferent depth from the camera 3 and have particularly a di f ferent distance from the driver monitoring system 1 .

[0078] The predermined points 8 reflect the light of the light dots 7 to the camera 3 and the camera 3 detects the reflected light . In Figure 1 only one reflected light ray is shown for the sake of clarity . However, all predermined points 8 shown in Figure 1 reflect light emitted from the dot proj ector 2 to the camera 3 . The light of the light dots 7 reflected to the camera 3 is detected by the camera 3 such that a calibration pattern 9 or an operation pattern 10 is generated ( see Figures 6 , 7 and 8 ) .

[0079] Further, the driver monitoring system 1 comprises several additional light sources 11 arranged in the vehicle cabin 4 . The additional light sources 11 emit also infrared light as the dot proj ector 2 . Particularly, the additional light sources 11 are controllable independently from each other and independently from the dot proj ector 2 and the camera 3 . Therefore , they selectively enhance the visibility during recalibration .

[0080] The driver monitoring system 1 according to the exemplary embodiment of Figure 1 can be , for example , operated with the method according to the exemplary embodiment of Figure 2 .

[0081] In a first step S I of the method for operating a driver monitoring system 1 according to the exemplary embodiment of Figure 2 a plurality of predetermined points 8 is defined, particularly within a vehicle cabin 4 . The plurality of predetermined points 8 have 3D coordinates of a vehicle coordinate system . The predetermined points 8 are distributed within the vehicle cabin 4 and have di f ferent depths with respect to a camera 3 of the driver monitoring system 1 and are , particularly i f the vehicle cabin 4 is free of occupants , within a field of view of the camera 3 .

[0082] In a next step S2 , the plurality of predetermined points 8 is illuminated with a plurality of light dots 7 proj ected by a dot proj ector 2 according to a predetermined pattern 12 . Particularly, each predetermined point 8 has a corresponding point within the predetermined pattern 12 and is illuminated by one light dot 7 proj ected by the dot proj ector 2 . Particularly, the light dots 7 proj ected by the dot proj ector 2 cover the predetermined points 8 of the predetermined pattern 12 in the first illumination step S2 .

[0083] In a next step S3 subsequent to step S2 , the light of the light dots 7 reflected within the vehicle cabin 4 is detected with the camera 3 such that a calibration pattern 9 is generated . Particularly, the light dots 7 are positioned on fixed elements or movable elements within the vehicle cabin 4 and are reflected by the surfaces of these elements . The camera 3 captures an image of the reflected light of the light dots 7 , showing the calibration pattern 9 . Two- dimensional coordinates , such as the numbers of the pixels of the camera 3 , can be assigned to the light dots 7 of the calibration pattern 9 .

[0084] In a next step S4 subsequent to step S3 , the vehicle cabin 4 is again illuminated with the plurality of light dots 7 proj ected by the dot proj ector! according to the predetermined pattern 12 and the light of the light dots 7 reflected within the vehicle cabin 4 is detected with the camera 3 again in a step S5 such that an operation pattern 10 is generated . The camera 3 captures an image of the reflected light of the light dots 7 , showing the operation pattern 10 . Two-dimensional coordinates , such as the numbers of the pixels of the camera 3 , can be assigned to the light dots 7 of the operation pattern 10 .

[0085] In a next step S 6 , the operation pattern 10 is compared with the calibration pattern 9 . For example , the two-dimension coordinates of the light dots 7 of the calibration pattern 9 and the two-dimension coordinates of the light dots 7 of the operation pattern 10 are compared, when the operation pattern 10 and the calibration pattern 9 are compared with each other .

[0086] In a next step S7 , extrinsic parameters and / or intrinsic parameters of the dot proj ector 2 and / or extrinsic parameters and / or intrinsic parameters of the camera 3 are calibrated when a di f ference between the operation pattern 10 and the calibration pattern 9 exceeds a threshold .

[0087] Particularly, the operation pattern 10 di f fers from the calibration pattern 9 when the camera 3 and / or the dot proj ector 2 are subj ected to temperature variation . Temperature variations often lead to changes in intrinsic properties of camera 3 and / or the dot proj ector 2 . Further, it is possible that the operation pattern 10 di f fers from the calibration pattern 9 when the dot proj ector 2 and / or the camera 3 are subj ect to a mechanical impact such as mechanical vibration and / or mechanical shocks .

[0088] As the driver monitoring system 1 according to the exemplary embodiment of Figure 1 , the driver monitoring system 1 according to the exemplary embodiment of Figure 3 comprises additional light sources 11 . The additional light sources 11 are arranged on the locations of the vehicle cabin 4 , where the light dots 7 of the dot proj ector 2 are located in an initially calibrated state according to the predetermined pattern 12 . Particularly, the predetermined points 8 within the vehicle cabin 4 are each provided with an additional light source 11 emitting light of the same wavelength range as the light dots 7 . Particularly, the camera 3 is configured to detect the light of the additional light sources 11 . For example , the additional light sources 11 comprise an LED- chip or VCSEL-chip generating the light emitted by the additional light sources 11 . The additional light sources 11 arranged at the predetermined 8 are switched on during steps S4 and S5 of the method according to the exemplary embodiment of Figure 2 , wherein the vehicle cabin 4 is illuminated with the plurality of the light dots 7 proj ected by the dot proj ector 2 according to the predetermined pattern 12 and wherein the light of the light dots 7 reflected within the vehicle cabin 4 is detected with the camera 3 such that an operation pattern 10 is generated . In that case , the image of the operation pattern 10 taken by the camera 3 , comprises also the calibration pattern 9 . In such a way quality of calibration can be improved .

[0089] Figures 4 and 5 show simulated images of a vehicle cabin 4 with two passengers , a driver on the driver' s seat and a further passenger on the co-driver' s seat taken by a camera 3 of a driving monitoring system 1 according to an exemplary embodiment . The driver monitoring system 1 is at present located about 40 centimeter away from the passengers in front of them . Figure 4 shows an image captured by the camera 3 when the scene within the vehicle cabin 4 is illuminated with a flood illuminator 6 with visible light , while Figure 5 shows the scene within the vehicle cabin 4 illuminated with a dot proj ector 2 .

[0090] Figure 6 shows a simulation of a predetermined pattern 12 of light dots 7 illuminating the vehicle cabin 4 . Particularly, Figure 6 shows an image with a calibration pattern 9 of the light dots 7 according to the predetermined pattern 12 as for example achieved in step S3 of the method according to the exemplary embodiment of Figure 2 . For example , this image is taken when initiali zing the driver monitoring system 1 with starting the engine of the vehicle .

[0091] At a later stage , the vehicle cabin 4 is again illuminated with the plurality of light dots 7 , which are proj ected by the dot proj ector 2 , again according to the predetermined pattern 12 such that an operation pattern 10 is generated ( compared steps S4 and S5 of the method of Figure 2 ) . I f the driver monitoring system 1 has changed its intrinsic and / or extrinsic parameters , for example due to temperature variation and / or mechanical impact , the operation pattern 10 di f fers from the calibration pattern .

[0092] Figure 7 shows an image of the camera 3 when the camera 3 of the driver monitoring system 1 has moved laterally, for example in y-direction of the vehicle coordinate system, leading to an operation pattern 20 with a plurality of light dots 7 with locations di f fering from the light dots 7 of the calibration pattern 9 .

[0093] Therefore , when the operation pattern 10 and the calibration pattern 9 re compared, as for example described in S 6 of the method of Figure 2 , a di f ference can be detected in the two- dimensional coordinates of the light dots 7 of the calibration pattern 9 and the light dots 7 of the operation pattern 10 . I f this di f ference exceeds a threshold, extrinsic parameters and / or intrinsic parameters of the dot proj ector 2 and / or extrinsic parameters and / or intrinsic parameters of the camera 3 are calibrated ( step S7 of the method of Figure 2 ) . Particularly, here the extrinsic parameters of the camera 3 are calibrated, for example by using a PnP algorithm . It is also possible to calibrate the camera 3 in an iterative way, wherein steps S4 to S7 are carried out several times until the operation pattern 10 and the calibration pattern 9 match within a predetermined threshold .

[0094] Figure 8 shows a di f ferent operation pattern 10 than Figure 7 . The operation pattern 10 according to Figure 8 is due to change of a focal length of the camera 3 , which is an intrinsic parameter . The change in focal length expands the operation pattern 10 compared to the calibration pattern 9 . Calibration of the intrinsic parameters of the camera 3 , such as the focal length, can be carried out , for example , correspondingly to a calibration with the help of a checkerboard pattern . Particularly, intrinsic and / or extrinsic parameters can be calibrated using a non-linear minimi zation of proj ection equations .

[0095] During the method according to the exemplary embodiment of Figures 9 to 15 , a calibration of extrinsic parameters of the dot proj ector 2 is described in order to compensate a translation of the dot proj ector 2 relative to the camera 3 .

[0096] Figure 9 shows an image of a vehicle cabin 4 with a driver and a co-driver illuminated with light dots 7 according to a predetermined pattern 12 . The light dots 7 of the image of Figure 9 form a calibration pattern 9 . The image has two- dimension coordinates , wherein the unit of the x-axis is the number of the pixels of the camera in x-direction and the y- axis is the number of the pixels of the camera in y- direction .

[0097] Figure 10 shows the image of the same scene as Figure 9 and particularly the vehicle cabin 4 with the driver and the codriver illuminated with the light dots 7 according to the predetermined pattern 12 . However, the dot proj ector 2 has changed its position relative to the position of the camera 3 , for example due to a mechanical shock during driving . Particularly, the operation pattern 10 formed by the light dots 7 shown in Figure 10 is shi fted in an x-direction .

[0098] In the next steps a template matching process is carried out starting with a pitch of the image of Figure 10 .

[0099] Figure 11 shows the pitch 13 of the image of Figure 10 after the mechanical shock used as a starting point for the template matching process . Particularly, the pitch 13 of Figure 11 comprises an environment of one or more detected known elements within the driver cabin 4 .

[0100] Figure 12 shows a pitch 13 of Figure 9 before the mechanical shock around the same detected known elements as shown in the image of Figure 11 . The image of Figures 11 after the shock and image of Figure 12 before the shock are matched with each other . Particularly, Figure 13 shows a cost image when performing template matching of Figures 11 and 12 .

[0101] Figure 14 is an image showing the best match providing information about movement of the known elements within the vehicle cabin 4 , which corresponds to a movement of the dot proj ector 2 due to the mechanical shock .

[0102] Figure 15 shows how the image of Figure 11 looks like within the image of Figure 9 after identi fying it .

[0103] The present application claims priority of the German application DE 102023126431 . 1 , the disclosure content of which is incorporated herein by reference . The invention is not limited to the description of the exemplary embodiments . Rather, the invention comprises each new feature as well as each combination of features , particularly each combination of features of the claims , even i f the feature or the combination of features itsel f is not explicitly given in the claims or the exemplary embodiments .

[0104] References

[0105] 1 driver monitoring system

[0106] 2 dot projector 3 camera

[0107] 4 vehicle cabin

[0108] 5 head of a driver

[0109] 6 flood illuminator

[0110] 7 light dot 8 predetermined point

[0111] 9 calibration pattern

[0112] 10 operation pattern

[0113] 11 additional light source

[0114] 12 predetermined pattern 13 pitch

[0115] SI, S2, S3, S4, S5, S6, S7: method step

Claims

Claims1. Method for operating a driver monitoring system (1) comprising a dot projector (2) configured for projecting a plurality of light dots (7) and a camera (3) configured for detecting light of the light dots (7) , the method comprises the steps:- defining a plurality of predetermined points (8) within a vehicle cabin (4) ,- illuminating the plurality of predetermined points (8) with the plurality of light dots (7) projected by the dot projector (2) according to a predetermined pattern (12) ,- detecting light of the light dots (7) reflected by the predetermined points (8) with the camera (3) such that a calibration pattern (9) is generated,- illuminating the vehicle cabin (4) with the plurality of light dots (7) projected by the dot projector (2) according to the predetermined pattern (12) ,- detecting the light of the light dots (7) reflected within the vehicle cabin (4) with the camera (3) such that an operation pattern (10) is generated,- comparing the operation pattern (10) with the calibration pattern ( 9 ) , and- calibrating extrinsic parameters and / or intrinsic parameters of the dot projector (2) and / or extrinsic parameters and / or intrinsic parameters of the camera (3) , when the operation pattern (10) differs from the calibration pattern ( 9 ) .

2. Method according to the previous claim, wherein the dot projector (2) emits infrared light and the camera (3) detects the infrared light.

3. Method according to any of the previous claims, wherein the predetermined points (8) are arranged within the field of view of the camera (3) at different depths.

4. Method according to any of the previous claims, wherein calibrating the extrinsic parameters and / or the intrinsic parameters of the dot projector (2) and / or the extrinsic parameters and / or the intrinsic parameters of the camera (3) is carried out in iterative steps.

5. Method according to any of the previous claims, wherein the driver monitoring system (1) comprises at least one additional light source (11) configured for emitting light of the light dots (7) .

6. Method according to the previous claim, wherein at least one predetermined point (8) within the vehicle cabin(8) is provided with the additional light source (11) .

7. Method according to any of claims 5 to 6, wherein the additional light source (11) comprises a LED-chip or a VCSEL-chip.

8. Method according to any of claims 5 to 7, wherein the at least one additional light source (11) is switched on during detecting the light of the light dots (7) reflected within the vehicle cabin (4) with the camera (3) such that the operation pattern (10) is generated.

9. Method according to any of the previous claims, wherein the predetermined points (8) are located on fixed elements within the vehicle cabin (4) .

10. Method according to the previous claims, wherein the predetermined points (8) are located on a car headliner and / or a car trim.

11. Method according to any of claims 1 to 9, wherein- the predetermined points (8) are located on movable elements configured to be moved by a motor, and- the operation pattern (10) is corrected by a feedback signal of a control unit controlling the motor.

12. Method according to any of the previous claims, wherein the generation of the calibration pattern (9) is started, when initializing the driver monitoring system (1) .

13. Method according to any of the previous claims, wherein the generation of the operation pattern (10) is carried out, when the vehicle cabin (4) is free of occupants.

14. Method according to any of the previous claims, wherein the following steps are configured to be triggered by a human user :- detecting light of the light dots (7) reflected within the vehicle cabin (4) with the camera (4) such that the operation pattern (10) is generated,- comparing the operation pattern (10) with the calibration pattern ( 9 ) , and- calibrating the extrinsic parameters and / or the intrinsic parameters of the dot projector (2) and / or the extrinsic parameters and / or the intrinsic parameters of the camera (3) .

15. Method according to any of the previous claims, wherein extrinsic parameters and / or intrinsic parameters of the camera (3) are calibrated when deviations of two-dimensionalcoordinates of light dots (7) of the operation pattern (10) and two-dimensional coordinates of light dots (7) of the calibration pattern (9) exceeds a predetermined threshold.

16. Method according to any of the previous claims, wherein- two-dimensional coordinates of light dots (7) of a pitch (13) of the operation pattern (10) are compared with two- dimensional coordinates of light dots (7) of the pitch (13) of the calibration pattern (9) , and- extrinsic parameters of the dot projector (2) are calibrated when a deviation between the two-dimensional coordinates of the light dots (7) of the pitch (13) of the operation pattern (10) and the two-dimensional coordinates of the light dots (7) of the pitch (13) of the calibration pattern (9) exceeds a predetermined threshold.

17. Driver monitoring system (1) comprising:- a dot projector (2) configured for projecting a plurality of light dots (7) illuminating a plurality of predetermined points (8) according to a predetermined pattern (12) ,- a camera (3) configured for detecting light of the light dots (7) reflected by the predetermined points (8) such that a calibration pattern (9) and / or an operation pattern (10) is generated, wherein extrinsic parameters and / or intrinsic parameters of the dot projector (2) and / or extrinsic parameters and / or intrinsic parameters of the camera (3) are calibrateable, when the operation pattern (10) differs from the calibration pattern (9) .

18. Driver monitoring system according to the previous claim further comprising a flood illuminator (6) configured for illuminating a field of view of the camera (3) .

19. Vehicle cabin (4) with the driver monitoring system (1) according to any of claims 17 and 18.

Images