Method and apparatus for processing data associated with two interior camera systems for a vehicle

Abstract

A method, for example a computer-implemented method, for processing data associated with at least two interior camera devices for a vehicle, comprising: determining first information that characterizes an object pose of an object located in the interior of the vehicle, for example the head pose of a driver of the vehicle, using a first interior camera device of the at least two interior camera devices; determining second information that characterizes the object pose of the object located in the interior of the vehicle, for example the head pose of the driver of the vehicle, using a second interior camera device of the at least two interior camera devices; determining a first value that characterizes a possible decalibration of at least one of the two interior camera devices, based at least on the first information and the second information.

Description

State of the art

[0001] The disclosure relates to a method for processing data associated with at least two interior camera systems for a vehicle.

[0002] The disclosure further relates to a device for processing data associated with at least two interior camera systems for a vehicle. Disclosure of the invention

[0003] Some examples relate to a method, for example a computer-implemented method, for processing data associated with at least two interior camera systems for a vehicle, comprising: determining first information that characterizes an object pose of an object located in the interior of the vehicle, for example a head pose of the driver of the vehicle, using a first interior camera system of the at least two interior camera systems; determining second information that characterizes the object pose of the object located in the interior of the vehicle, for example the head pose of the driver of the vehicle, using a second interior camera system of the at least two interior camera systems; determining a first value that characterizes a possible decalibration of at least one of the two interior camera systems.based at least on the first and second pieces of information. In some examples, this allows for the detection of a decalibration based, for example, solely on the first and second pieces of information.

[0004] For example, the first interior camera device is associated with a driver monitoring system for the vehicle, where, for example, the first interior camera device is a component of the driver monitoring system.

[0005] For example, the second interior camera system is associated with an occupant monitoring system for the vehicle, where, for example, the second interior camera system is a component of the occupant monitoring system.

[0006] In some examples, the procedure includes at least one of the following elements: a) receiving the first information, for example from the first interior camera device or from a driver monitoring system, or b) receiving the second information, for example from the second interior camera device or from an occupant monitoring system.

[0007] For example, determining the cause includes at least one of the following elements: a) comparing the first piece of information with the second piece of information, for example, comparing a time series of the first piece of information with a time series of the second piece of information, or b) calculating averages for the first and second pieces of information and comparing the averages. In some examples, this can enable the detection of a decalibration.

[0008] In some examples, the procedure has at least one of the following elements: a) Organizing the initial information in the form of an initial vector, for example according to A i , where, for example, the first vector has six components, where, for example, the first three components of the first vector are associated with a translation of the object pose, for example, head pose, where, for example, three further components of the first vector are associated with a rotation of the object pose, for example, head pose, with respect to the first indoor camera setup, where, for example, i characterizes a time index, or b) organizing the second piece of information in the form of a second vector, for example, according to B i, where, for example, the second vector has six components, where, for example, the first three components of the second vector are associated with a translation of the object pose, for example, the head pose, where, for example, three further components of the second vector are associated with a rotation of the object pose, for example, the head pose, with respect to the second indoor camera setup, where, for example, i characterizes a time index, or c) forming an average of the first vector over a predefined period, for example, a predefined time window, whereby, for example, an averaged first vector A is obtained, or d) forming an average of the second vector over a predefined period, for example, a predefined time window, whereby, for example, an averaged second vector B is obtained.

[0009] For example, the method includes at least one of the following elements: a) Determining transformation matrices, for example according to or as T a and T b , based on the means of the first vector and the second vector, or b) determining a relative position between the at least two indoor camera devices based on the transformation matrices, for example according to Tab=TbTa−1, where tab a transformation matrix that characterizes a transformation from the first interior camera unit to the second interior camera unit, or c) determining an offset, for example according to offset=tb−Tabta, where t a a translation associated with the first vector, for example with the averaged first vector A, is characterized, where t ba translation associated with the second vector, for example with the averaged second vector B.

[0010] In further examples, the procedure includes at least one of the following elements: a) Determining a deviation for a predefined time index i, for example according to error=tb_i−Tabta_i, where t a_i a translation associated with the first vector at time index i, where t b_i a) a translation associated with the second vector to the time index i, or b) using a difference between the offset and the deviation as the first quantity, or c) inferring a decalibration of at least one of the two indoor camera devices if the difference exceeds a predefinable threshold, or d) collecting several values ​​for the deviation and comparing a median value of the several values ​​for the deviation with the predefinable threshold.

[0011] Further examples relate to a device for a vehicle, for example a motor vehicle, wherein the device is designed to perform the method according to the disclosure.

[0012] In further examples, a system for a vehicle, for example a motor vehicle, is proposed, comprising the device according to the disclosure, the first interior camera device and the second interior camera device.

[0013] Some examples refer to a vehicle having at least one device according to the disclosure and / or a system according to the disclosure.

[0014] Other examples relate to a computer-readable storage medium comprising instructions which, when executed by a computer, cause it to perform the procedure according to the disclosure.

[0015] Other examples relate to a computer program, comprising instructions which, when the program is executed by a computer, cause it to perform the procedure according to the disclosure.

[0016] Other examples relate to a data carrier signal that transmits and / or characterizes the computer program according to the disclosure.

[0017] Some examples relate to a use of the method according to the disclosure and / or the device according to the disclosure and / or the system according to the disclosure and / or the computer-readable storage medium according to the disclosure and / or the computer program according to the disclosure and / or the data carrier signal according to the disclosure for at least one of the following elements: a) monitoring for a decalibration of at least one of the at least two indoor camera devices, or b) repeated, for example periodic, for example continuous, checking for a decalibration of at least one of the at least two indoor camera devices, or c) avoiding the use of information other than the first and second pieces of information for the detection of a decalibration, or d) initiating a recalibration, for example recalibration, or e) increasing safety, for example with regard to the operation of aof the driver monitoring system and / or of the occupant monitoring system.

[0018] Further features, applications, and advantages will become apparent from the following description of examples illustrated in the figures of the drawing. All described or illustrated features, individually or in any combination, constitute the subject matter of the disclosure, irrespective of their aggregation in the claims or their cross-reference, and irrespective of their formulation or representation in the description or in the drawing.

[0019] The drawing shows: Fig. 1. A simplified flowchart (schematical). Fig. 2. A simplified block diagram (schematically). Fig. 3. A simplified flowchart (schematically). Fig. 4. A simplified flowchart (schematically). Fig. 5. A simplified block diagram (schematically). Fig. 6 schematic examples of uses.

[0020] Some examples, e.g. Fig. 1, Fig. 2, refer to a method, for example a computer-implemented method, for processing data associated with at least two interior camera devices 10a, 10b for a vehicle 1, comprising: 100 determining first information 1-1 that characterizes an object pose OP of an object OBJ located in an interior IR of the vehicle 1, for example a head pose KP of a driver F of the vehicle 1, using a first interior camera device 10a of the at least two interior camera devices; 102 determining second information I-2 that characterizes the object pose OP of the object OBJ located in the interior IR of the vehicle 1, for example the head pose KP of the driver F of the vehicle 1, using a second interior camera device 10b of the at least two interior camera devices; 104 determining a first parameter G-1 that indicates a possible decalibration of at least one of the two interior camera devices 10a.10b is characterized based on at least the first information I-1 and the second information 1-2. In some examples, this allows for the detection of a decalibration based, for example, solely on the first and second information 1-1, 1-2.

[0021] For example, Fig. 2, at least some aspects according to the disclosure can be implemented by a device 200, which may for example be a control unit or whose functionality may for example be integrated into a control unit for the vehicle 1.

[0022] For example, the first interior camera system 10a is equipped with a driver monitoring system 5a ( Fig. 2) associated with the vehicle 1, wherein, for example, the first interior camera device 10a is a component of the driver monitoring system 5a.

[0023] For example, the second interior camera device 10b is associated with an occupant monitoring system 5b for the vehicle 1, wherein, for example, the second interior camera device 10b is a component of the occupant monitoring system 5b.

[0024] In some examples, Fig. 1, the method comprises at least one of the following elements: a) Receiving 100a the first information I-1, for example from the first interior camera device 10a or from a driver monitoring system 5a, or b) Receiving 102a the second information I-2, for example from the second interior camera device 10b or from a passenger monitoring system 5b.

[0025] For example, Fig. 1, the determination 104 includes at least one of the following elements: a) Comparing 104a the first piece of information I-1 with the second piece of information I-2, for example, comparing a time course of the first piece of information I-1 with a time course of the second piece of information I-2, or b) Calculating 104b average values ​​for the first and second pieces of information and comparing the average values. In some examples, this may enable the detection of a decalibration.

[0026] In some examples, Fig. 3, the method includes at least one of the following elements: a) Organizing 110 of the first information 1-1 in the form of a first vector V1, for example according to A i, where, for example, the first vector V1 has six components, where, for example, the first three components of the first vector are associated with a translation of the object pose OP, for example, head pose KP, where, for example, three further components of the first vector are associated with a rotation of the object pose OP, for example, head pose KP, with respect to the first indoor camera device 10a, where, for example, i characterizes a time index, or b) organizing 111 the second information I-2 in the form of a second vector V2, for example, according to B i, wherein, for example, the second vector V2 has six components, wherein, for example, the first three components of the second vector are associated with a translation of the object pose OP, for example, head pose KP, wherein, for example, three further components of the second vector V2 are associated with a rotation of the object pose OP, for example, head pose KP, with respect to the second indoor camera device 10b, wherein, for example, i characterizes a time index, or c) forming a mean value MW1 of the first vector V1 over a predefinable period, for example, a predefinable time window, whereby, for example, an averaged first vector A is obtained, or d) forming a mean value MW2 of the second vector V2 over a predefinable period, for example, a predefinable time window, whereby, for example, an averaged second vector B is obtained.

[0027] For example, Fig. 3, the method includes at least one of the following elements: a) Determining 114 transformation matrices TM, for example according to or as T a and T b , based on the mean values ​​MW1, MW2 of the first vector and the second vector, or b) Determine a relative position between the at least two indoor camera devices based on the transformation matrices TM, for example according to Tab=TbTa−1, where tab a transformation matrix that characterizes a transformation from the first interior camera unit 10a to the second interior camera unit 10b, or c) Determine 116 an offset OFS, for example according to offset=tb−Tabta, where t a a translation associated with the first vector V1, for example with the averaged first vector A, is characterized, where t ba translation associated with the second vector V2, for example with the averaged second vector B.

[0028] For further examples, Fig. 3, the procedure includes at least one of the following elements: a) Determining a deviation ERR for a predefinable time index i, for example according to error=tb_i−Tabta_i, where t a_i characterizes a translation associated with the first vector V1 at the time index i, where t b_i a translation associated with the second vector V2 to the time index i, or b) using a difference DIFF between the offset OFS and the deviation ERR as the first quantity G-1 (see Fig. 1), or c) decalibrate at least one of the two interior camera devices 10a, 10b if the difference DIFF exceeds a predefinable threshold, or d) collect several values ​​for the deviation ERR and compare a median value of the several values ​​for the deviation ERR with the predefinable threshold.

[0029] In some examples, using the median value can increase the robustness or stability of the method.

[0030] Fig. Figure 4 schematically shows a simplified flowchart according to further examples. Element E1 symbolizes the driver monitoring system 5a according to Fig. 2. Element E2 symbolizes a determination of the head position KP based on information, for example the initial information 1-1, from the driver monitoring system E1. Element E3 symbolizes the occupant monitoring system 5b according to Fig. 2. Element E4 symbolizes a determination of the head position KP based on information, for example the second piece of information I-2, from the occupant monitoring system E4. Element E5 symbolizes a decalibration detection for at least one of the interior camera devices 10a, 10b based on the information from elements E2, E4, wherein, for example, the above with reference to Fig. 1, Fig. 2 to Fig. The methods described in the disclosure are usable. For example, the decalibration detection E5 is designed to determine the first quantity G-1 based on the information from elements E2 and E4; see also... Fig. 1. To determine. Element E6 symbolizes a determination of whether a decalibration has occurred, for example by evaluating the first parameter G-1, for example by comparing the first parameter G-1 with a predefined threshold. If a decalibration has occurred, i.e., if the first parameter G-1 exceeds the predefined threshold, a recalibration E7 is triggered. If no decalibration has occurred, i.e., if the first parameter G-1 does not exceed the predefined threshold, no recalibration is triggered, but rather no action is performed, see element E8.

[0031] Further examples, Fig. 5, refer to a device 200 for a vehicle 1, for example a motor vehicle, wherein the device 200 is configured to perform the method according to the disclosure.

[0032] In some examples, Fig. 5, it is provided that the device 200 comprises: a computing device (“computer”) 202 having at least one computing core 202a, a storage device 204 associated with the computing device 202 for at least temporary storage of at least one of the following elements: a) data DAT (e.g., data associated with the first information I-1 and / or with further information I-2 or size G-1, V1, V2, ...), b) computer program PRG, for example, for carrying out the method according to the disclosure.

[0033] For further examples, Fig. 5, the memory device 204 includes volatile memory (e.g., RAM) 204a, and / or non-volatile (NVM) memory (e.g., Flash EEPROM) 204b, or a combination thereof or with other memory types not explicitly mentioned.

[0034] Further examples, Fig. 5, refer to a computer-readable storage medium SM, comprising instructions PRG which, when executed by a computer 202, cause it to execute the procedure according to the disclosure.

[0035] Further examples, Fig. 5, refer to a computer program PRG, comprising instructions which, when the program PRG is executed by a computer 202, cause it to execute the procedure according to the disclosure.

[0036] Further examples, Fig. References 5 to a data carrier signal DCS, which characterizes and / or transmits the computer program PRG according to the disclosure. The data carrier signal DCS can be transmitted (e.g., sent and / or received) via an optional data interface 206 of the device 200.

[0037] In some examples, Fig. 5. The device 200 can also be designed, e.g., purely hardware-based, e.g., as a hardware circuit, or the functionality of the device 200 can be realized by means of a, e.g., purely hardware circuit.

[0038] For further examples, Fig. 2, a system 1000 for a vehicle, for example a motor vehicle, 1, is proposed, comprising the device 200 according to the disclosure, the first interior camera device 10a and the second interior camera device 10b. Optionally, in some examples, the system 1000 may also comprise further components or systems, such as at least one of the systems 5a, 5b.

[0039] Some examples, Fig. 2, refer to a vehicle 1 having at least one device 200 according to the disclosure and / or a system 1000 according to the disclosure.

[0040] Further aspects and examples are described below, which – in the case of further examples – can each be combined individually or in any combination with at least one of the aspects and / or examples described above.

[0041] In some examples, the principle according to the disclosure can be used with one or more indoor camera devices (“indoor cameras”) 10a, which are located, for example, at different points in the interior IR ( Fig. 2) may be attached to or installed in the vehicle, e.g., to record the number of occupants and / or the condition of the driver and, if applicable, the occupants. For example, interior cameras are used for safety-relevant driver functions and can trigger an alarm if driver distraction is detected (e.g., using a mobile phone while driving). A driver monitoring system 5a (e.g., Driver Monitoring System, DMS), which may use one or more interior cameras, has, for example, optically or to humans invisible IR (infrared) illumination units and one or more interior cameras, e.g., to enable invisible integration and operation within the interior IR. An occupant monitoring system 5b (e.g., Occupant Monitoring System, OMS) may, for example, have a similar function to the driver monitoring system 5a and also enables, for example, corresponding monitoring of other occupants.

[0042] In some examples, the DMS 5a and the OMS 5b estimate and / or track, for example repeatedly, periodically, or continuously, the head position or head pose KP (characterized, for example, by three translations and three rotations) of the driver F, e.g., to ensure that the driver F is looking forward. To ensure that the driver F is not distracted, the 3D head position is calculated, for example, by the cameras or the DMS and the OMS. The head position is, for example, a safety-critical parameter, which is why it is calculated by two cameras or the two systems DMS 5a and OMS 5b, i.e., at two different locations. The accuracy of the head position depends, for example, on the accuracy of the calibration parameters for the interior camera systems 10a and 10b or their systems 5a and 5b. Therefore, in some examples, it may be important to know whether the cameras 10a and 10b are calibrated (i.e., whether they are calibrated).The principle addresses this issue by requiring the system to check whether the calibration parameters are valid and to detect a decalibration event (where the calibration parameters are invalid), for example, when a steering column is moved up and down. In some examples, the two cameras 10a and 10b cannot be calibrated to each other at the factory during production, as one camera is typically mounted on the steering column. Therefore, adjusting the steering column up / down or sideways necessitates recalibration of the cameras in some cases. The principle, as disclosed, addresses this decalibration problem and describes, among other things, an algorithm that considers the head position KP in DMS and OMS cameras (e.g., with three translations and three rotations each) to detect the decalibration event, for example, while the system is running.

[0043] In some examples, the principle, according to the disclosure, can detect a decalibration event solely based on the head position KP or another position vector OP, which can be determined using DMS 5a and OMS 5b, e.g., to continuously check whether cameras 10a, 10b are calibrated. A significant advantage, according to some examples, is that no additional information (e.g., beyond information I-1, I-2) is required, such as camera extrinsics, known 3D points in the world coordinate system, the distance between DMS and OMS cameras, or a sophisticated online calibration.

[0044] Some examples, Fig.6, refer to a use 300 of the method according to the disclosure and / or the device 200 according to the disclosure and / or the system 1000 according to the disclosure and / or the computer-readable storage medium SM according to the disclosure and / or the computer program PRG according to the disclosure and / or the data carrier signal DCS according to the disclosure for at least one of the following elements: a) monitoring 301 for a decalibration of at least one of the at least two indoor camera devices 10a, 10b, or b) repeatedly, for example periodically, for example continuously, checking 302 for a decalibration of at least one of the at least two indoor camera devices 10a, 10b, or c) avoiding 303 the use of information other than the first and second information I-1, I-2 for the detection of a decalibration, or d) initiating 304, E7 a recalibration, for example, recalibration, or e) increasing 305 a level of security,for example, regarding the operation of a driver monitoring system 5a and / or an occupant monitoring system 10a.

Claims

[1] A method, for example a computer-implemented method, for processing data associated with at least two interior camera devices (10a, 10b) for a vehicle (1), comprising: determining (100) first information (I-1) that characterizes an object pose (OP) of an object (OBJ) located in an interior (IR) of the vehicle (1), for example the head pose (KP) of a driver (F) of the vehicle (1), using a first interior camera device (10a) of the at least two interior camera devices (10a, 10b); determining (102) second information (I-2) that characterizes the object pose (OP) of the object (OBJ) located in the interior (IR) of the vehicle (1), for example the head pose (KP) of the driver (F) of the vehicle (1), using a second interior camera device (10b) of the at least two interior camera devices (10a, 10b); determining (104) a first quantity (G1);which characterizes a possible decalibration of at least one of the two interior camera systems (10a, 10b), based at least on the first information (I-1) and the second information (I-2). [2] Method according to claim 1, wherein the first interior camera device (10a) is associated with a driver monitoring system (5a) for the vehicle (1), wherein, for example, the first interior camera device (10a) is a component of the driver monitoring system (5a), and / or wherein the second interior camera device (10b) is associated with an occupant monitoring system (5b) for the vehicle (1), wherein, for example, the second interior camera device (10b) is a component of the occupant monitoring system (5b). [3] Method according to at least one of the preceding claims, comprising at least one of the following elements: a) Receiving (100a) the first information (I-1), for example from the first interior camera device (10a) or from a driver monitoring system (5a), or b) Receiving (102a) the second information (I-2), for example from the second interior camera device (10b) or from an occupant monitoring system (5b). [4] Method according to at least one of the preceding claims, wherein the determination (104) comprises at least one of the following elements: a) comparing (104a) the first information (I-1) with the second information (I-2), for example comparing a time course of the first information (I-1) with a time course of the second information (I-2), or b) forming (104b) average values ​​for the first and second information (I-1, 1-2) and comparing (104c) the average values. [5] Method according to at least one of the preceding claims, comprising at least one of the following elements: a) Organizing (110) the first information (1-1) in the form of a first vector (V1), for example according to A i , wherein, for example, the first vector (V1) has six components, wherein, for example, the first three components of the first vector (V1) are associated with a translation of the object pose (OP), for example, head pose (KP), wherein, for example, three further components of the first vector (V1) are associated with a rotation of the object pose (OP), for example, head pose (KP), with respect to the first indoor camera setup (10a), wherein, for example, i characterizes a time index, or b) organizing (111) the second information (I-2) in the form of a second vector (V2), for example, according to B i, wherein, for example, the second vector (V2) has six components, wherein, for example, the first three components of the second vector (V2) are associated with a translation of the object pose (OP), for example, head pose (KP), wherein, for example, three further components of the second vector (V2) are associated with a rotation of the object pose (OP), for example, head pose (KP), with respect to the second indoor camera device (10a), wherein, for example, i characterizes a time index, or c) forming (112) a mean (MW1) of the first vector (V1) over a predefinable period, for example, a predefinable time window, whereby, for example, an averaged first vector A is obtained, or d) forming (113) a mean (MW2) of the second vector (V2) over a predefinable period, for example, a predefinable time window, whereby, for example, an averaged second vector B is obtained. [6] Method according to claim 5, comprising at least one of the following elements: a) Determining (114) transformation matrices (TM), for example according to or as T a and T b , based on the means (MW1, MW2) of the first vector (V1) and the second vector (V2), or b) determining (115) a relative position between the at least two interior camera devices (10a, 10b) based on the transformation matrices (TM), for example according to Tab=TbTa−1, where tab a transformation matrix that characterizes a transformation from the first interior camera device (10a) to the second interior camera device (10b), or c) determining (116) an offset (OFS), for example according to offset=tb−Tabta, where t a a translation associated with the first vector (V1), for example with the averaged first vector A, is characterized, where t ba translation associated with the second vector (V2), for example with the averaged second vector B. [7] Method according to claim 6, comprising at least one of the following elements: a) Determining (117) a deviation (ERR) for a predefinable time index i, for example according to error=tb_i−Tabta_i, where t a_i a translation associated with the first vector (V1) at the time index i is characterized, where t b_ia translation associated with the second vector (V2) to the time index i, or b) using (118) a difference (DIFF) between the offset (OFS) and the deviation (ERR) as the first quantity (G-1), or c) inferring (119) a decalibration (DECAL) of at least one of the two interior camera devices (10a, 10b) if the difference (DIFF) exceeds a predefinable threshold, or d) collecting (120a) several values ​​for the deviation (ERR) and comparing (120b) a median value of the several values ​​for the deviation (ERR) with the predefinable threshold. [8] Device (200) for a vehicle (1), for example a motor vehicle, wherein the device (200) is configured to perform the method according to at least one of the preceding claims. [9] System (1000) for a vehicle (1), for example motor vehicle, comprising the device (200) according to claim 8, the first interior camera device (10a) and the second interior camera device (10b). [10] Vehicle (1; 1a) comprising at least one device (200) according to claim 8 and / or a system (1000) according to claim 9. [11] Computer-readable storage medium (SM) comprising instructions (PRG) which, when executed by a computer (202), cause it to execute the method according to at least one of claims 1 to 7. [12] Computer program (PRG) comprising instructions which, when the program (PRG) is executed by a computer (202), cause it to execute the method according to at least one of claims 1 to 7. [13] Data carrier signal (DCS) that transmits and / or characterizes the computer program (PRG) according to claim 12. [14] Use (300) of the method according to at least one of claims 1 to 7 and / or the device (200) according to claim 8 and / or the system (1000) according to claim 9 and / or the computer-readable storage medium (SM) according to claim 11 and / or the computer program (PRG) according to claim 12 and / or the data carrier signal (DCS) according to claim 13 for at least one of the following elements: a) monitoring (301) for a decalibration of at least one of the at least two indoor camera devices (10a, 10b), or b) repeatedly, for example periodically, for example continuously, checking (302) for a decalibration of at least one of the at least two indoor camera devices (10a, 10b), or c) avoiding (303) the use of information other than the first and second information (I-1, I-2) for the detection of a decalibration, or d) initiating (304) a recalibration, for example Recalibration, or e) increasing (305) a safety,for example, with regard to the operation of a driver monitoring system (5a) and / or an occupant monitoring system (5b).

Images