Device and method for checking the plausibility of a position of at least one object, said position being based on the image data of an interior camera provided in the interior of a vehicle

Abstract

In a device and a method for checking the plausibility of a position of at least one object, said position being based on the image data of an interior camera provided in the interior of a vehicle, an object comprising at least one radio-based communication unit is located in the interior of the vehicle by radio; position data for determining the located position of the object in the interior of the vehicle is generated; the position data is processed; the located position of the object is determined; and on the basis of the located position, the plausibility of the image-data-based position of the object is checked.

Description

[0001] Device and method for verifying the plausibility of the position of at least one object based on image data from an interior camera arranged in the interior of a vehicle

[0002] The invention relates to a device and a method for verifying the plausibility of the position of at least one object based on image data from an interior camera arranged in the interior of a vehicle. It is known in the prior art to capture at least one image depicting an area of ​​the interior and to generate corresponding image data. The image data is processed by a processing unit, which determines the object's position within the vehicle's interior. Particularly for autonomous or semi-autonomous vehicle operation, it is necessary to verify the plausibility of detection results from sensors, especially cameras. For this purpose, it is known to evaluate detection results from two sources, in particular the detection results from two independent camera systems. The camera systems preferably each comprise a camera and a processing unit for processing the image data provided by the camera.

[0003] Document US 2024 / 0045426 A1 reveals that autonomous driving is one of the world's most demanding computational challenges. It requires processing vast amounts of data from cameras, radars, lidar, and high-definition maps to generate commands for safe and comfortable real-time vehicle control. This demanding task requires a dedicated, energy-efficient supercomputer, complex high-performance software, and groundbreaking deep learning algorithms. To meet this challenge, advanced systems and methods are known to facilitate autonomous driving functionality, including Level 3, 4, and / or 5 autonomous driving platforms.In preferred embodiments, the technology offers an end-to-end platform with a flexible architecture, including an architecture for autonomous vehicles that utilizes computer vision and well-known ADAS techniques, provides diversity and redundancy, and meets functional safety standards.

[0004] Document US 2023 / 0286520 A1 discloses an automated driving system (ADS) for an autonomous vehicle, comprising a communication module, a perception module, a maladaptive module, and a processor. The communication module is configured to receive a vehicle-to-vehicle (V2V) message containing message-based vehicle data. The perception module is configured to receive sensor data from at least one vehicle sensor device. The maladaptive module is configured to determine whether the V2V message is a serious or malicious message, at least in part, by comparing the message-based vehicle data with sensor-based vehicle data generated from the sensor data.

[0005] The document "Decimeter-Level Localization with a Single WiFi Access Point," by Deepak Vasisht, Swarun Kumar, and Dina Katabi (ISBN 978-1-931971-29-4), reveals a system that allows a single WiFi access point to locate connected communication devices with centimeter-level accuracy. Such a system can enable indoor location tracking for homes and small businesses, which typically have only one access point. The key is a novel algorithm that can calculate time-of-flight times of less than one nanosecond using commercially available WiFi network adapters. By multiplying the time-of-flight by the speed of light, an access point calculates the distance between each of its antennas and the communication device, thus locating it.

[0006] Starting from the known state of the art, it is an object of the invention to specify a device, an arrangement and a method for verifying the plausibility of a position of at least one object based on the image data of an interior camera arranged in the interior of a vehicle.

[0007] This problem is solved by a device having the features of claim 1, by an arrangement having the features of the arrangement claim, and by a method having the features of the independent method claim. Advantageous embodiments are specified in the dependent claims.

[0008] The device according to claim 1 enables the plausibility check of the position of at least one object, based on image data from an interior camera located inside a vehicle, using a system independent of the interior camera. The location can be determined simply and relatively accurately using the tracking unit for radio-based location of the same object inside the vehicle. This location can then be used to verify whether the object's position based on the image data is plausible.The processing unit can be configured to prevent the object's position, determined based on the image data, from being plausible due to the location data. This configuration also prevents the processing unit from further using the object's position, based on the image data, to provide assistance and / or safety functions, and / or from providing the object's position, based on the image data, to other vehicle assistance systems. This ensures that only plausibly determined positions of the object and the positions of other objects, based on the image data, are made available to assistance systems.

[0009] The processing unit can also be configured to initiate a calibration of the indoor camera if the test reveals that the object's position, based on the image data, is implausible due to the location tracking. This allows for improved position determination after calibration, and the plausibility check can be performed again.

[0010] The processing unit can also be configured if the test has shown that the object's position based on the image data is plausible based on the location position, to use the object's position based on the image data from the processing unit to provide assistance and / or safety functions and / or to provide the object's position based on the image data from the processing unit to other assistance systems of the vehicle.

[0011] The object comprising the radio-based communication unit can be a mobile phone, wireless headphones, a smartwatch, a fitness tracker, VR glasses, smart glasses, or a tablet computer. This allows objects often brought into the vehicle interior by occupants to be validated by the image data from the interior camera, and the positions of other objects determined using image data from this interior camera to be assessed as plausible.

[0012] It is particularly advantageous if the tracking unit is designed to perform radio-based tracking of the object inside the vehicle using a triangulation method. This allows for the easy determination of a precise location. Furthermore, it is advantageous if the tracking unit is designed to establish at least one wireless communication with the object. This enables simple tracking and determination of the location.

[0013] Wireless communication can be established via at least one Wi-Fi connection, at least one Bluetooth connection, and / or at least one Ultra Wide Band (UWB) connection. Ultra Wide Band (UWB) connectivity is already integrated into current smartphones, such as iPhones. This enables simple communication and object tracking.

[0014] It is advantageous if the processing unit, during plausibility checks, determines whether the position based on the image data is credible, preferably likely correct, based on the determined location position. This ensures that further positions of objects determined from the image data of the indoor camera are also likely to be correct. These objects then do not need to include a radio-based communication unit.

[0015] A second aspect concerns an arrangement for reliable object detection in the interior of a vehicle, comprising a device according to the invention with or without the aforementioned further developments and a camera. The camera is arranged in the interior of the vehicle and is configured to capture at least one image depicting an area of ​​the interior and to generate corresponding image data. The processing unit is configured to process the image data and determine the position of the object in the interior of the vehicle based on the image data. This allows the position determined from the image data to be easily verified by the determined location position of the same object.

[0016] It is advantageous if the object is visible in the image. Then the object's position can be easily determined based on the image data.

[0017] It is particularly advantageous if the camera captures a 2D image and / or if it is an RGB-IR camera. This allows the camera to capture images suitable for further processing in both daylight and darkness.

[0018] The processing unit can also be configured to correct the object's position determined from the image data, based on the position determined by the positioning unit. Furthermore, it can determine or, if necessary, correct at least one correct distance between two key points in a human pose estimation procedure.

[0019] It is also advantageous to determine the position of at least one of the driver's hands from the image data in order to check whether the determined position means the hand is touching the steering wheel or not, or whether the driver is operating a switch or indicator lever.

[0020] The camera can be integrated into the vehicle's interior rearview mirror or its housing, or it can be positioned above the driver and / or front passenger in the roof area and / or in the B-pillar on the driver's and / or front passenger's side. This allows for the detection of objects that are touched by an occupant. Furthermore, it is advantageous if the camera's field of view has a detection angle of 100° to 150°, and the camera is preferably a monocular camera and / or an RGB-IR camera. This allows for image processing by the processing unit based on the captured IR images. The use of IR images is advantageous because these images are independent of ambient light. The objects to be detected, such as switches, the steering wheel, and / or levers, can also have special infrared markers that are invisible to the human eye.These can then be easily captured from a camera image.

[0021] The method with the features of the dependent claim has the same advantages as the claimed device. In particular, the method can be further developed with the features of the dependent claims directed to the device. The method is carried out especially whenever the camera has been recalibrated, after a journey has commenced, and / or at predetermined time intervals in the range of 10 seconds to 2 hours, preferably in the range of 1 minute to 30 minutes.

[0022] This method also allows for the capture of at least one image depicting an area of ​​the interior, and the generation of corresponding image data. The image data is then processed to determine the object's position within the vehicle's interior.

[0023] This method is also advantageous for detecting not only incorrect calibration but also whether the system is operating reliably. For example, if discrepancies arise between a localized device and its position in the camera image, or between the position of an operated switch or lever and the hand position determined by the camera image when the switch or lever is operated, it can be assumed that the system has a fault, at least if the discrepancy exceeds a preset limit. This is particularly beneficial for safety-critical (ASIL-classified) applications.

[0024] Exemplary embodiments of the invention are explained in more detail below with reference to the figures. These show:

[0025] Figure 1 is a perspective schematic representation of a vehicle cockpit;

[0026] Figure 2 shows a schematic representation of a first image of the vehicle's interior taken by an interior camera of the vehicle;

[0027] Figure 3 shows a schematic representation of a second image of the vehicle's interior taken by an interior camera; and Figure 4 shows a flowchart for verifying the plausibility of the position of at least one object based on the image data from an interior camera located inside a vehicle.

[0028] Figure 1 shows a perspective schematic representation of the cockpit 100 of a vehicle 102. A driver 104 is seated in a driver's seat 106 of the vehicle 102. The cockpit 100 also includes a central information display (CID) 108, a head-up display 110, and a graphic instrument cluster 112, which are arranged in a dashboard 114 of the vehicle 102. The aforementioned display elements 108, 110, and 112 each form a functional unit of an output unit of the vehicle 102, which is designed to output information to the driver 104 and / or another occupant 206 (see Figure 2). At least one loudspeaker 116 of an entertainment system of the vehicle 102 is also arranged in the cockpit 100. The loudspeaker 116 also forms a functional unit of the output unit. The functional units of the output unit also serve as a playback unit for audio and / or video playback.

[0029] The cockpit 100 further comprises a steering wheel 118 with controls 120, a gear selector 122, pedals 124, and an input unit 126 with a rotary dial and push-button function and / or a touch input panel. This input unit 126 is also referred to as the Ergo Commander. An interior rearview mirror 132 and an interior camera 134 integrated into this interior rearview mirror 132 are arranged in the upper area of ​​the windshield 128 of the vehicle 102. The camera 134 is designed and integrated into the rearview mirror 132 in such a way that it captures images depicting at least one area of ​​the interior of the vehicle 102. Specifically, the field of view of the camera 134 is directed towards the driver's seat 106 and a passenger seat 202 of the vehicle 102.

[0030] An exemplary image 200, 300 taken with the aid of camera 134 is shown in Figures 2 and 3, respectively. Camera 134 is specifically designed to capture several successive images, particularly in the form of a video stream, to generate image data corresponding to images 200, 300, and to transmit this data to a control unit 138 serving as a processing unit.

[0031] The control unit 138 has data outputs 140 and data inputs 142, which serve to connect to other units of the vehicle 102, for example with other cameras, sensors, input and output units and control units of assistance systems.

[0032] The control unit 138 also includes a communication module 144, which is designed to establish a connection with a telecommunications network, in particular a mobile communications network.

[0033] Furthermore, additional cameras are arranged in the interior of the vehicle 102 in the B-pillars and above the driver 104 and the front passenger. Using camera 134 and the other interior cameras, several images can be recorded sequentially, preferably as a sequence of images or a video sequence, and corresponding image data can be generated for each image. This data is then transmitted to and processed by the control unit 138.

[0034] Figures 2 and 3 show schematic representations of images 200 and 300, respectively, of the interior of the vehicle 102, each captured by the interior camera 134. The camera 134 has, by way of example, a field of view of 120 degrees. It is particularly advantageous if the camera 134 is an RGB-IR interior camera. In other embodiments, the camera 134 can also have a field of view ranging from 100 degrees to 150 degrees or more. The camera 134 is oriented towards the driver's seat 106, a passenger seat 202, and a rear bench seat 204 of the vehicle 102. The camera 134 generates image data corresponding to the respective images 200 and 300, which are then processed by the control unit 138. During the processing of the image data, the control unit 138 determines the position of the object in the interior of the vehicle 102.

[0035] The cockpit 100 of vehicle 102 contains several radio units 150 to 174 for radio-based localization of at least one object comprising a radio-based communication unit inside vehicle 102. At least one of these radio units 150 to 174, together with the control unit 138, forms a localization unit. The localization unit generates position data to determine the location of the object inside vehicle 102. The control unit 138 processes the position data and thereby determines the location of the object. For this purpose, the radio units 150 to 174 of the localization unit establish at least one wireless communication link with the object.

[0036] Wireless communication can take place in particular via at least one WLAN connection, at least one Bluetooth connection and / or at least one UWB connection.

[0037] Starting from the location position, the control unit 138 checks the plausibility of the object's position based on the image data. In particular, during this plausibility check, the control unit 138 determines whether the position based on the image data is credible, preferably likely correct, based on the determined location position.

[0038] The object comprising a radio-based communication unit can be a mobile phone, wireless headphones, a smartwatch, a fitness tracker, VR glasses, smart glasses or a tablet computer, as will be explained below in conjunction with Figures 2 and 3.

[0039] If the plausibility check reveals that the object's position based on the image data is implausible due to the location tracking, the control unit 138 will no longer use the object's position based on the image data to provide assistance and / or safety functions. Furthermore, the control unit 138 will not provide the object's position based on the image data to any other assistance systems of the vehicle 102. Additionally or alternatively, a calibration of the interior camera can be initiated and / or performed.

[0040] If, however, the plausibility check has shown that the position of the object based on the image data is plausible due to the location position, the position of the object based on the image data can be used by the control unit 138 to provide assistance and / or safety functions and / or the position of the object based on the image data can be provided by the control unit 138 to other assistance systems of the vehicle 102.

[0041] The radio-based location of the object inside the vehicle 102 can be carried out using a triangulation method. For this purpose, the data from at least two radio units 150 to 174 can be processed by the control unit 138 in order to determine the location position simply and reliably.

[0042] Figure 2 schematically shows image 200 captured by the interior camera 134 of the vehicle 102. In the situation shown in this image 200, the driver 104 is sitting in the driver's seat 106 and the other occupant 206 is sitting in the passenger seat 202. Thus, in the present embodiment, the other occupant is the passenger 206. The passenger 206 is holding a tablet computer 208. The driver 104 is wearing smart glasses 210, a fitness tracker 212, and wireless headphones 214. The tablet computer 208, the smart glasses 210, the fitness tracker 212, and the wireless headphones 214 are all objects with a radio-based communication unit located inside the vehicle 102. In Figure 2, the positions of further radio units 176 to 190 are schematically indicated by circles.These radio units 176 to 190, like the radio units 150 to 174, are integrated into the interior of the vehicle 102 and are preferably not visible to the occupants 104, 204.

[0043] The tracking unit can easily determine the location position of these objects 208 to 214 inside the vehicle 102 using its radio units 150 to 190. Furthermore, the objects 208 to 214 are visible in image 200, so that the control unit can also determine the position of these objects using the image data from camera 134 and check and verify the plausibility of this determined position using the tracking position.

[0044] Figure 3 shows a schematic representation of image 300 taken by the interior camera 134 of the vehicle 102. In the situation shown in this image 300, the driver 104 is again sitting in the driver's seat 106, and the other occupant 206 is sitting in the front passenger seat 202. The front passenger 206 has a mobile phone 414 in his left hand, a smartwatch 416 on his right wrist, and a wireless earphone 420 in his right ear. The driver 104 has a smartwatch 418 on his right wrist. The mobile phone 414, the smartwatches 416, 418 and the wireless headphones 420 are each objects with a radio-based communication unit in the interior of the vehicle 102. Figure 4 shows a flowchart for validating the position of at least one object 208 to 214, 414 to 420 based on the image data from an interior camera 134 arranged in the interior of a vehicle 102.The process is described for object 208. However, the process can alternatively or additionally be carried out for the other objects 210 to 214 and 414 to 420. The process starts in step S100. Subsequently, in step S102, the position of object 208 is determined based on the image data from camera 134. The tracking unit then determines the location of object 208, for which a radio unit 150 to 190 establishes a wireless communication link to object 208. Then, in step S106, the plausibility of the location of object 208, based on the image data, is checked. The process then ends in step S108.

[0045] In other embodiments, steps S102 and S104 are carried out in reverse order or in parallel.

[0046] The plausibility check can be performed for any additional interior cameras of the vehicle 102, either alternatively or additionally.

[0047] In the embodiments described with reference to Figures 1 to 4, at least the radio units 150 to 190 and the control unit 138 form a device for verifying the position of at least one object 208 to 214, 414 to 420 based on image data from an interior camera 134 arranged in the interior of a vehicle 102. The control unit 138 is also referred to as the processing unit. Further elements and features shown in Figures 1 to 4 and mentioned in the preceding description can be part of the device for verifying the position of at least one object 208 to 214 based on image data from an interior camera 134 arranged in the interior of a vehicle 102. Likewise, method steps described with reference to the device can be part of the claimed method. List of reference numerals

[0048] 100 Cockpit

[0049] 102 vehicles

[0050] 104 drivers

[0051] 106 Driver's seat

[0052] 108 central information display

[0053] 110 Head Up Display

[0054] 112 graphic instrument cluster

[0055] 114 Dashboard

[0056] 116 speakers

[0057] 118 Steering wheel

[0058] 120 Control element

[0059] 122 Gear selector

[0060] 124 Pedals

[0061] 126 Input unit

[0062] 128 Windscreen

[0063] 130 microphone

[0064] 132 Interior rearview mirror

[0065] 134 Interior camera

[0066] 136 Emergency call bracelet

[0067] 138 Control unit

[0068] 140 Data output

[0069] 142 Data input

[0070] 144 Communication interface

[0071] 150 to 190 radio units

[0072] 200, 300 image

[0073] 202 Passenger seat

[0074] 204 Rear seat

[0075] 206 passengers

[0076] 208 tablet computers

[0077] 210 Smart glasses

[0078] 212 Fitness tracker

[0079] 214, 420 wireless headphones

[0080] 414 Smartphone

[0081] 416, 418 Smartwatch

[0082] S100- S108 Procedure steps

Claims

Claims:

1. Device for verifying the plausibility of a position of at least one object based on image data from an interior camera arranged in the interior of a vehicle, comprising a tracking unit for radio-based tracking of at least one object (208 to 214, 414 to 420) comprising a radio-based communication unit in the interior of the vehicle (102), wherein the tracking unit is configured to generate position data for determining the location position of the object (208 to 214, 414 to 420) in the interior of the vehicle (102), comprising a processing unit (138) configured to process the position data and determine the location position of the object (208 to 214, 414 to 420), wherein the processing unit (138) is configured to verify the plausibility of the position of the object (208 to 214, 414 to 420) based on the image data, starting from the location position. check.

2. Device according to claim 1, wherein the processing unit is configured if the test has shown that the position of the object (208 to 214, 414 to 420) based on the image data is not plausible due to the location position, the processing unit (138) does not further use the position of the object (208 to 214, 414 to 420) based on the image data to provide assistance and / or safety functions and / or the processing unit (138) does not provide the position of the object (208 to 214, 414 to 420) based on the image data to further assistance systems of the vehicle (102).

3. Device according to claim 1 or 2, wherein the processing unit is configured to initiate a calibration of the indoor camera (134) if the test has shown that the position of the object (208 to 214, 414 to 420) based on the image data is not plausible due to the location position.

4. Device according to one of the preceding claims, wherein the processing unit (138) is configured, if the test has shown that the position of the object (208 to 214, 414 to 420) based on the image data is plausible on the basis of the location position, to use the position of the object (208 to 214, 414 to 420) based on the image data by the processing unit (138) to provide assistance and / or safety functions and / or to provide the position of the object (208 to 214, 414 to 420) based on the image data by the processing unit (138) to further assistance systems of the vehicle (102).

5. Device according to one of the preceding claims, wherein the object (208 to 214, 414 to 420) comprising a radio-based communication unit is a mobile phone (414), a wireless headphone (214, 420), a smartwatch (416, 418), a fitness tracker, a VR headset, smart glasses (214) or a tablet computer (208).

6. Device according to one of the preceding claims, wherein the locating unit is configured to perform radio-based locating of the object (208 to 214, 414 to 420) in the interior of the vehicle (102) using a triangulation method.

7. Device according to one of the preceding claims, wherein the locating unit is configured to establish at least one wireless communication with the object (208 to 214, 414 to 420).

8. Device according to claim 7, wherein wireless communication is carried out via at least one WLAN connection, at least one Bluetooth connection and / or at least one UWB connection.

9. Device according to one of the preceding claims, wherein the processing unit determines during plausibility checks whether the position based on the image data is credible, preferably probably correct, based on the determined location position.

10. Arrangement for reliable object detection in the interior of a vehicle, comprising a device according to one of the preceding claims, and with a camera (134) which is arranged in the interior of the vehicle (102) and which is configured to capture at least one image (200) depicting an area of ​​the interior and to generate image data corresponding to the image (200), wherein the processing unit (138) is configured to process the image data and to determine the position of the object (208 to 214, 414 to 420) in the interior of the vehicle (102) based on the image data.

11. Arrangement according to claim 10, wherein the object is an object (208 to 214, 414 to 420) visible in the image (200, 300).

12. Arrangement according to one of the preceding claims, wherein the camera (134) captures a 2D image and / or wherein the camera (134) is an RGB-IR camera.

13. Arrangement according to one of the preceding claims, characterized in that the processing unit (138) is configured to correct the position of the object (208 to 214, 414 to 420) determined from the image data, starting from the position of an object (208 to 214, 414 to 420) determined with the aid of the locating unit.

14. Method for verifying the plausibility of a position of at least one object based on image data from an interior camera arranged in the interior of a vehicle, wherein at least one object (208 to 214, 414 to 420) comprising a radio-based communication unit is located in the interior of the vehicle (102) using radio-based location, wherein position data for determining the location position of the object (208 to 214, 414 to 420) in the interior of the vehicle (102) are generated, wherein the position data are processed and the location position of the object (208 to 214, 414 to 420) is determined, and wherein, starting from the location position, the plausibility of the position of the object (208 to 214, 414 to 420) based on the image data is checked.

15. Method according to claim 14, wherein at least one image (200, 300) is captured with a representation of an area of ​​the interior and the image (200, 300) corresponding image data is generated, and the image data is processed and the position of the object (208 to 214, 414 to 420) in the interior of the vehicle (102) is determined.

Images