Motor vehicle and method for providing vehicle-external camera data for a driving task in a motor vehicle

Abstract

The invention relates to a motor vehicle (10) and a method for providing vehicle-external camera data for a driving task in a motor vehicle (10). The method comprises the following steps: receiving vehicle-external camera data originating from a camera device (12) not mounted on the vehicle by a receiving device (14) of the motor vehicle (10); determining by a computing device (16) of the motor vehicle (10) whether the received vehicle-external camera data is suitable for supporting a driving task of the motor vehicle (10), whereby the received vehicle-external camera data is checked according to predefined criteria; and displaying the vehicle-external camera data on a display device (18) of the motor vehicle (10) only if the received vehicle-external camera data is suitable for supporting the driving task.

Description

[0001] The invention relates to a method for providing vehicle-external camera data for a driving task in a motor vehicle. Furthermore, the invention relates to a motor vehicle configured to perform the method.

[0002] Vehicles often feature video visualization of their surroundings, captured by in-vehicle cameras. These cameras are typically located at the rear of the vehicle to assist the driver when reversing. The footage is usually captured from a rear-view perspective and displayed on a vehicle display, such as an infotainment system screen. Alternatively, or additionally, 360° video images around the vehicle may also be displayed.

[0003] However, it is possible that the vehicle cameras may not capture certain areas or perspectives, for example because trailers or caravans block the view.

[0004] From DE 10 2021 128 255 A1, a device and a method for the safety-optimized control of infotainment content in a vehicle are known. The device comprises a detection unit that is configured to detect a request to activate non-driving-related infotainment content in the vehicle and to determine a basic requirement for activating non-driving-related infotainment content in the vehicle.The device also includes a control unit which, upon detection of a request to activate non-driving-related infotainment content in the vehicle and upon determination of a basic requirement by the detection unit, is configured to automatically activate a safety function in the vehicle, wherein the safety function in the vehicle for activating non-driving-related infotainment content for a driver of the vehicle can be specified, and after activation of the safety function in the vehicle, to output non-driving-related infotainment content for the driver of the vehicle in accordance with the detected requirements.

[0005] From DE 10 2023 106 108 A1, a device and a method for determining and displaying an area observed by a driver in the interior of a vehicle are known, wherein at least one camera of the vehicle is used, which captures at least one image depicting an area of ​​the interior and generates corresponding image data, which are processed by a processing unit. The processing unit detects an adjustment of a rearview mirror located in the interior of the vehicle. After the adjustment of the rearview mirror, the area of ​​the interior visible to the driver in the rearview mirror is determined, and at least one object present in this area is detected by the processing unit.Based on the image data from the camera, the processing unit outputs an image of at least one part of the interior area visible to the driver in the adjusted rearview mirror as the observed area on a display unit of the vehicle. This part comprises at least one object.

[0006] From EP 3 978 296 A1 a vehicle display system is known, comprising a vehicle information receiver configured to receive vehicle information from a motor vehicle and an image stream selector configured to select an image stream source from at least two image stream sources based on the received vehicle information in order to display an image stream from the image stream source selected by the image stream selector.

[0007] From DE 10 2021 105 051 A1, a method is known for generating at least one warning information for a user of a motor vehicle by means of an assistance system of the motor vehicle, in which environmental information generated by a detection device of another motor vehicle in the immediate vicinity of the motor vehicle is transmitted to a first electronic computing device of the assistance system of the motor vehicle, wherein the transmitted environmental information of the other motor vehicle is evaluated internally in the motor vehicle and, depending on the evaluation, the warning information regarding the environment is output to the user via an output device of the assistance system.

[0008] From DE 10 2021 207 258 B3, a method for the automatic control of at least one vehicle function of a vehicle is known on the basis of at least one optical notification signal in relation to a traffic situation in a vehicle environment of the vehicle, wherein the optical notification signal is output by an optical output device in a vehicle environment detected by at least one sensor device for detecting the vehicle environment of the vehicle.The method comprises receiving environmental data determined by the sensor device with respect to the detected vehicle environment and processing data characteristic of the environmental data by means of an evaluation device using a machine learning model, wherein the model comprises a set of parameters which are set to values ​​learned as a result of a training process, and thereby determining at least one notification parameter which is characteristic of the optical notification signal, in particular of a meaning of the optical notification signal with respect to the traffic situation.

[0009] From DE 10 2018 206 591 A1, an assistance system for supporting contactless charging of a motor vehicle battery, a corresponding method, and a corresponding motor vehicle with such an assistance system are known. To support contactless charging, a camera is used to capture a current image of the base plate of an external device that provides electrical energy for charging. The captured image is compared by a data processing unit with a predefined reference image, which shows the base plate in a state suitable for charging. If a difference is detected, a prompt is automatically issued to the driver of the motor vehicle, requesting the driver to inspect the base plate and confirm that it is free of any foreign objects that could impede charging.

[0010] The object of the invention is to assist a user of the motor vehicle in a driving task.

[0011] This problem is solved by the independent patent claims. Advantageous embodiments of the invention are disclosed in the dependent patent claims, the following description, and the figures.

[0012] The invention is based on the idea that video images from external vehicle cameras can be displayed as embedded streams on a screen of the vehicle, whereby a suitable stream is selected situationally and, if necessary, based on AI-based user preferences.

[0013] One aspect of the invention relates to a method for providing vehicle-external camera data for a driving task in a motor vehicle. The method comprises the following steps: receiving vehicle-external camera data originating from a camera device not mounted on the vehicle, by a receiving device of the motor vehicle; determining by a computing device of the motor vehicle whether the received vehicle-external camera data is suitable for supporting a driving task of the motor vehicle, whereby the received vehicle-external camera data is checked according to predefined criteria; and displaying the vehicle-external camera data on a display device of the motor vehicle only if the received vehicle-external camera data is suitable for supporting the driving task.

[0014] In other words, it may be possible to display camera data from a mobile or external camera, which could, for example, be from a third-party provider, on a vehicle display to assist the user with a driving task. For this purpose, the external camera device may have one or more cameras directed into or from the vehicle's surroundings. The camera device can be coupled with the vehicle's receiver, preferably via a wireless or radio connection. That is to say, the camera device can be a stationary camera or a mobile camera, such as a GoPro™, a smartphone, or similar device.

[0015] The camera device can, for example, record the vehicle's surroundings from inside the vehicle or be reversibly mounted on the vehicle or a trailer. For instance, cameras of the camera device can record the wheels and / or undercarriage of the vehicle off-road, be mounted on a trailer or caravan to capture a rear view, be mounted on wing mirrors to provide other perspectives, and / or the camera device can be a drone from which the external camera data can be obtained.

[0016] To prevent a vehicle user from being distracted by camera data from an external source, such as one that merely provides entertainment, it may be possible to display only camera data relevant to driving the vehicle on the vehicle's display. For this purpose, the received external camera data can be analyzed by a computer, checking against a number of predefined criteria. These criteria might include, for example, whether the camera data depicts a driving situation, whether the vehicle is recognized in the camera data, and / or whether the camera data originates from a position near the vehicle.To evaluate the camera data, a trained artificial intelligence, in particular an image analysis algorithm, can be used to identify driving situations and / or objects within the camera data. Alternatively, the user can select a suitable image stream from received external camera data. Furthermore, the decision as to whether the camera data supports the user in the driving task can take into account the current state of the vehicle, such as speed and / or direction of travel.

[0017] The computing device could, for example, be a vehicle computer on which a program for evaluating the camera data can be run. Evaluation and differentiation of camera data according to its suitability for supporting a driving task can preferably take place in real time, so that a user receives the camera data that is most helpful to them.

[0018] Finally, the appropriate external camera data can be displayed on a vehicle display unit. This display unit could, for example, be a screen in an instrument cluster, center console, or head-up display. A user can configure the position of the camera data display on the screen and specify when it should be shown.

[0019] The invention offers the advantage that additional camera data or video recordings, particularly those from sources outside the vehicle, can be used for multiple situations and applications within the motor vehicle. Furthermore, it prevents the user from being distracted by moving images unrelated to driving, thus increasing safety.

[0020] Furthermore, according to the invention, a predetermined criterion for verifying the suitability of the camera data is that the external camera data originates from a third-party camera device mounted on the motor vehicle. This means that if it is determined that the third-party camera device has been mounted on the motor vehicle, particularly reversibly, a criterion can be met that indicates the suitability of the external camera data. In particular, it can be provided that at least one criterion, or preferably several criteria, must be met to indicate the suitability of the camera data for supporting a driving task.

[0021] Furthermore, the invention provides that, to determine whether the external camera device is mounted on the motor vehicle, a location signal from the camera device is compared with a location signal from the motor vehicle. In other words, it can be checked, for example, whether the GPS location of the camera device is similar to that of the motor vehicle. The location can be checked, for example, over a predetermined period, particularly over the last few minutes. It can be checked whether the camera device and the motor vehicle are moving in the same position and / or at the same speed to determine whether the external camera device is mounted on the motor vehicle.

[0022] The invention also includes embodiments that offer additional advantages.

[0023] One embodiment involves a trained artificial intelligence (AI) that checks whether the received external camera data is suitable for supporting the vehicle's driving task. This means the AI ​​can, for example, recognize objects in the video footage, as well as scenes, exposure, and perspectives, and use this information to determine predefined driving situations. The AI ​​can be trained using camera data or a stream of images from different driving situations and / or driving tasks. Furthermore, the AI ​​can be configured to censor areas or an entire image stream if it determines that the corresponding camera data does not support the current driving task.This design offers the advantage that it allows for a quick response to different situations in the vehicle-external camera data.

[0024] Another embodiment provides that one criterion is that the vehicle, or at least a part of it, is recognized in the external camera data. This means that a program or algorithm of the computing device can check whether the vehicle, or a part of it, was recorded by the external camera device and is recognized in the camera data. For this purpose, for example, the shape or color of the vehicle can be stored in a memory of the computing device to recognize the vehicle from the camera data.

[0025] Another embodiment stipulates that one criterion is that the external camera data must contain driving-related content. This means that only camera data containing driving-related content is classified as suitable for supporting the driving task. This implies that the camera data must contain at least road users, such as vehicles or pedestrians, and / or roads, in order for them to be recognized as driving-related content. In contrast, video content originating from an entertainment source, for example, can be classified as entertainment content or as non-driving-related content, which could be distracting for a driver and should therefore not be displayed on the screen. Driving-related content can also be identified based on perspective and / or focus.

[0026] Another embodiment provides that the system uses external camera data to determine whether it is suitable for driving or maneuvering the vehicle, with the external camera data being made available only for the relevant driving task. This means, for example, that it can be checked situationally whether the camera data is suitable for driving or primarily for maneuvering, which can be determined, for instance, by the camera's orientation and the vehicle's direction of travel and / or speed. Maneuvering could, for example, involve parking, and driving could include, for example, changing lanes or turning. If, for instance, it is determined during a drive that camera data suitable for maneuvering is available, it will not be displayed, as it is not appropriate for the current driving task.

[0027] In particular, the system is designed to determine whether a vehicle is driving or maneuvering based on its speed. The vehicle's direction of travel can also be taken into account, with, for example, reversing indicating maneuvering. Threshold values ​​can be defined to differentiate between driving and maneuvering. For instance, a threshold of 15 km / h could be set, with speeds below this threshold still indicating maneuvering. Above this threshold, or at a further threshold, such as 80 km / h, it can be reliably assumed that the situation is driving, allowing the corresponding camera data to be displayed. This design offers the advantage that camera data is only displayed for relevant driving tasks, thus reducing user distraction.This can increase security.

[0028] Another embodiment provides that the receiving device has a wireless interface and / or a wired interface. Furthermore, it can be provided that logging in or pairing the camera device with the receiving device is carried out using a password or a scannable QR code, which authorizes the connection of the vehicle to the camera ecosystem. For example, Bluetooth, WLAN, and / or a mobile communication standard, such as 5G, can be used as the wireless interface. Alternatively or additionally, a wired interface can be provided on the vehicle, by means of which the camera device can be connected to the vehicle's receiving device via a cable.

[0029] Another aspect of the invention relates to a motor vehicle comprising a receiving device, a computing device, and a display device, wherein the motor vehicle is configured to carry out a previously described method. This offers the same advantages and possibilities for variation as the method itself. The motor vehicle according to the invention is preferably configured as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle.

[0030] For use cases or application situations that may arise during the procedure and are not explicitly described here, it may be provided that, according to the procedure, an error message and / or a request for user feedback is issued and / or a default setting and / or a predetermined initial state is set.

[0031] The invention also includes the control device for the motor vehicle. The control device can comprise a data processing device or a processor circuit configured to carry out an embodiment of the method according to the invention. For this purpose, the processor circuit can comprise at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). In particular, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Processing Unit) can be used as the microprocessor. Furthermore, the processor circuit can comprise program code configured to carry out the embodiment of the method according to the invention when executed by the processor circuit.The program code can be stored in a data memory of the processor device. The processor device can be based, for example, on at least one circuit board and / or on at least one SoC (System on Chip).

[0032] The invention also includes further developments of the motor vehicle according to the invention, which have features already described in connection with the further developments of the method according to the invention. For this reason, the corresponding further developments of the motor vehicle according to the invention are not described again here.

[0033] As a further solution, the invention also includes a computer-readable storage medium comprising program code which, when executed by a computer or a computer network, causes it to execute an embodiment of the method according to the invention. The storage medium can be provided at least partially as a non-volatile data storage medium (e.g., as flash memory and / or as an SSD - solid state drive) and / or at least partially as a volatile data storage medium (e.g., as RAM - random access memory). The storage medium can be located within the computer or computer network. However, the storage medium can also be operated, for example, as an app store server and / or cloud server on the internet. The computer or computer network can provide a processor circuit with, for example, at least one microprocessor.The program code can be provided as binary code, assembly code, source code in a programming language (e.g., C), or a program script (e.g., Python). Alternatively, the computer-readable storage medium can be implemented as a signal containing computer-readable data, such as a time-varying voltage signal or a radio signal.

[0034] The invention also includes combinations of the features of the described embodiments. The invention therefore also includes realizations that each exhibit a combination of the features of several of the described embodiments, provided that the embodiments have not been described as mutually exclusive.

[0035] The following are exemplary embodiments of the invention described. This is illustrated by: Fig. 1 a schematically represented motor vehicle according to an exemplary embodiment; Fig. 2 a schematic process diagram for a procedure for providing vehicle-external camera data for a driving task.

[0036] The exemplary embodiments described below are preferred embodiments of the invention. In these exemplary embodiments, the described components each represent individual features of the invention, which can be considered independently of one another and each further develops the invention independently. Therefore, the disclosure is intended to include combinations of features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0037] In the figures, identical reference symbols denote functionally equivalent elements.

[0038] In Fig. Figure 1 shows a schematic representation of a motor vehicle 10 which can be designed to process external camera data from a camera device 12 outside the vehicle for a driving task.

[0039] The motor vehicle 10 can have a receiving device 14, a computing device 16 and a display device 18.

[0040] The external camera device 12 can, for example, be a mobile or stationary camera that does not originally belong to the motor vehicle 10, i.e., that is not permanently installed in the motor vehicle 10. The external camera device 12 can therefore be an external camera that can, for example, be reversibly attached to the motor vehicle 10. Alternatively or additionally, the camera device 12 can be a camera in the vicinity of the motor vehicle 10 that can record a driving situation or the motor vehicle 10 itself.

[0041] The camera data recorded by the camera device 12 from outside the vehicle can be received by the receiver 14 of the motor vehicle 10, for example via a radio connection, in particular WLAN, Bluetooth and / or mobile network. Alternatively or additionally, sockets may also be provided in or on the motor vehicle 10 via which at least one camera mountable on the motor vehicle 10 can be connected to the receiver 14.

[0042] To prevent the user of the vehicle 10 from being distracted by unnecessary camera data or moving images, the received external camera data can first be examined by the computing unit 16 to determine whether it supports the user in a driving task. Various criteria can be checked for this purpose, whereby at least one, or preferably several, of these criteria must be met before the external camera data is made available to a user of the vehicle 10. This check can be performed, for example, by a trained artificial intelligence, which can also be specifically trained to distinguish the camera data, i.e., the image data, according to objects and / or situations.

[0043] In particular, the external camera data can be checked to determine whether it originates from a camera device 12 mounted on the vehicle 10. For this purpose, a location signal from the camera device 12 is compared with a location signal from the vehicle. This means, for example, that the respective GPS locations of the preceding minutes can be compared. If the locations match, it can be assumed that the camera device is mounted on the vehicle 10. Alternatively or additionally, it can be checked whether the vehicle 10, or at least a part of it, is visible in the camera data, which would establish a connection to the vehicle 10 or a driving task of the vehicle 10.

[0044] It is also possible to check whether driving content, or a driving situation, is present in the camera data, which suggests the potential use of the camera data to support the driving task. Furthermore, it can be checked whether the provided camera data corresponds to a current driving task of the vehicle 10, for example, whether it can support the vehicle 10 while driving or during maneuvering or parking, whereby the camera data can only be released for the appropriate case. A distinction between normal driving and maneuvering can be made, for example, based on vehicle speed and / or direction of travel.

[0045] If the check by the computing unit 16 determines that the vehicle-external camera data is suitable for supporting the driving task of the motor vehicle 10, it can be enabled for display on the display unit 18 of the motor vehicle 10. However, if it is determined that it is not suitable for supporting the driving task of the motor vehicle 10, it can be censored or blocked so that no display takes place on the display unit 18.

[0046] In Fig.Figure 2 shows a schematic process diagram for a method for providing vehicle-external camera data for a driving task in a motor vehicle 10. In step S10, vehicle-external camera data can be received by a receiving device 14 of the motor vehicle 10, wherein the vehicle-external camera data can originate from a camera device 12 not belonging to the vehicle. For example, the camera device 12 not belonging to the vehicle can be a mobile device used by a vehicle occupant or a pedestrian outside the motor vehicle 10.

[0047] In step S12, a computing unit 16 of the vehicle 10 can determine whether the received external camera data is suitable for supporting a driving task of the vehicle 10, whereby the received external camera data is checked according to predefined criteria. Since some content can be distracting, it can be verified whether the video content of the camera data is driving-related. For example, if vehicle-mounted cameras are connected to the receiving unit 14 via Bluetooth, WLAN, or LAN, a direct vehicle reference can be established. A direct vehicle reference can also be established if the camera device 12 and the vehicle 10 have similar GPS locations over the last few minutes of driving.If the camera data or recording contains driving-related content, such as roads or vehicles, it can be assessed whether this is suitable for driving or primarily for maneuvering, based on perspective, focus, field of view, or depth of field. Based on this assessment, camera data relevant to maneuvering can be deactivated, for example, above a certain speed, particularly above 15 km / h. Other camera data, perhaps only relevant for maneuvering, can only be activated above a certain speed, such as 80 km / h.Other camera data or video images from front doors, refrigerators, or surveillance systems can be displayed, for example, when the vehicle is stationary or at autonomous driving levels 3 to 5. Otherwise, this data is censored due to potential distractions, and a user can only receive notifications about recorded activity while driving. It can also be configured that a dashcam is only used to set a perspective when stationary. Video content can be classified as infotainment content and thus excluded if it originates from such a source or contains non-driving-related content that could be distracting for the user.

[0048] Discrimination between useful and distracting camera data can be performed in real time, using artificial intelligence that can recognize objects in the camera data, as well as scene exposure and perspectives.

[0049] Finally, in step S14, the camera data can only be displayed on a display device 18 of the motor vehicle if it is suitable to support the current driving task.

[0050] Overall, the examples show how a strategy for displaying video recordings in a vehicle can be implemented.

Claims

[1] Method for providing vehicle-external camera data for a driving task in a motor vehicle (10), comprising as steps: - Receiving external camera data originating from a camera device (12) not belonging to the vehicle by a receiving device (14) of the motor vehicle (10); - Determining by a computing device (16) of the motor vehicle (10) whether the received vehicle-external camera data are suitable to support the driving task of the motor vehicle (10), whereby the received vehicle-external camera data are checked according to predefined criteria; - Displaying the vehicle-external camera data on a display device (18) of the motor vehicle (10) only if the received vehicle-external camera data is suitable to support the driving task; - wherein a predefined criterion is that the vehicle-external camera data originates from a vehicle-external camera device (12) mounted on the motor vehicle (10); and - wherein, in order to determine that the camera device (12) not belonging to the vehicle is mounted on the motor vehicle (10), a location signal of the camera device (12) is compared with a motor vehicle location signal. [2] Method according to claim 1, wherein a trained artificial intelligence checks whether the received vehicle-external camera data are suitable to support the driving task of the motor vehicle (10). [3] Method according to one of the preceding claims, wherein a criterion is that the motor vehicle (10) or at least a part of the motor vehicle (10) is recognized in the vehicle-external camera data. [4] Method according to one of the preceding claims, wherein a criterion is that driving content is provided in the vehicle-external camera data. [5] Method according to one of the preceding claims, wherein the vehicle-external camera data is used to determine whether it is suitable for driving or maneuvering the motor vehicle (10), wherein the vehicle-external camera data is provided only for the corresponding driving task. [6] Method according to claim 5, wherein it is determined on the basis of a vehicle speed of the motor vehicle (10) whether a journey or a maneuvering is taking place. [7] Method according to any of the preceding claims, wherein the receiving device (14) has a radio interface and / or a cable interface. [8] Motor vehicle (10) comprising a receiving device (14), a computing device (16) and a display device (18), wherein the motor vehicle (10) is configured to perform a method according to one of the preceding claims.

Images