Method for providing dashcam-like playback functionality in a vehicle display unit

Abstract

The invention relates to a method (100) for providing a dashcam-like playback functionality in a vehicle display unit (50) by generating a visual re-simulation from individual video recordings of a multi-camera system (60) of a vehicle (30), comprising the following steps, wherein the steps are performed by a vehicle data processing device (10): - Providing (101) video recordings from various cameras (61) of the multi-camera system (60), wherein the video recordings originate from a camera recording of an environment of the cameras (61), - Providing (102) vehicle signal data, wherein the vehicle signal data is specific to a state of the vehicle (30), - Providing (103) environment model data, wherein the environment model data are specific for modeling an environment around the vehicle (30), - Synchronizing (104) a playback based on the provided environment model data and the provided vehicle signal data and the provided video recordings, - Generating (105) a three-dimensional visual reconstruction of the recorded environment based on synchronized playback, - Providing (106) a control for an interactive change of perspective of the generated three-dimensional visual reconstruction, which is temporally coupled with a control of the playback.

Description

[0001] The invention relates to a method for providing dashcam-like playback functionality in a vehicle display unit. The invention further relates to a computer program, a device, and a storage medium for this purpose. State of the art

[0002] Drive recorder (DVR) systems are similar to dashcams and allow the recording of video footage by one or more in-vehicle cameras. Such recordings are invaluable for purposes such as documenting accidents, monitoring parking lots, ensuring vehicle and passenger compartment security, and preserving memories of a trip. A typical DVR system includes a recorder, a video / media player, and an interface for seamless export or transfer of the recordings.

[0003] Typically, the recorder captures video content and saves it as files with a specific file format in the system's accessible memory. Recording is triggered either manually or automatically by an event such as a collision or threat detection. Upon triggering an event, the recorder should continue recording video for the duration specified in the settings. Advanced DVR systems should offer the ability to buffer video content, allowing for the recording of past video footage related to the triggering event.

[0004] A video player must offer the ability to play back recorded video with functions such as play, pause, stop, rewind, etc. Current state-of-the-art DVR systems for the automotive market allow users to view video content on the infotainment display. Car cameras are typically fisheye cameras, which capture undistorted images across a wide field of view (FoV). This means that the recordings are distorted, and playback may be either distorted or undistorted, depending on the system's supported features. Current state-of-the-art systems play back content from either a single camera or a cascaded video composed of multiple camera images. Disclosure of the invention

[0005] According to aspects of the invention, a method with the features of claim 1, a computer program with the features of claim 9, a data processing device with the features of claim 10, and a computer-readable storage medium with the features of claim 11 are provided. Further features and details of the invention are disclosed in the corresponding dependent claims, the description, and the drawings. Features and details described in connection with the method according to the invention also correspond to the computer program, the data processing device, and the computer-readable storage medium, and vice versa.

[0006] In one aspect of the invention, a method for providing dashcam-like playback functionality for a vehicle, particularly in a vehicle display unit, by generating a visual re-simulation from individual video recordings of a vehicle's multi-camera system, comprises the following steps. These steps are preferably performed by a vehicle data processing device, i.e., a data processing device of the vehicle, particularly a data processing device according to the invention. The vehicle display unit can be designed as a device within the vehicle that visualizes information and data about the vehicle's condition and other relevant parameters. It can, in particular, comprise a digital display in an area in or around the vehicle's center console. This unit can enable the driver to quickly and clearly understand important vehicle data.One possible design for the vehicle display unit could offer enhanced interaction options, such as touchscreen functionality. Additionally, features like voice control or haptic feedback could be integrated.

[0007] The "dashcam-like playback functionality" can refer to a vehicle feature that generally allows the user to play back recorded data and video sequences from the vehicle's cameras. This feature enhances safety by enabling the user to analyze critical driving situations after the fact. A possible extension could be an analysis function that automatically highlights critical moments in the video footage. Alternatively, the dashcam-like playback functionality could be combined with a cloud storage solution to store recordings long-term and make them accessible from various devices, such as a mobile device.

[0008] According to a first step of the invention, video recordings from various cameras of the multi-camera system can be provided. These video recordings can originate from camera recordings of the cameras' surroundings. The cameras' surroundings can include the vehicle's environment and optionally also the vehicle's interior.

[0009] A further step of the invention may involve providing vehicle signal data. This vehicle signal data may be specific to a particular state of the vehicle. A vehicle data processing device, and in particular vehicle sensors, may provide this signal data. The signal data may indicate the vehicle's speed, steering angle, braking activity, and / or location (e.g., GPS data). Providing this vehicle signal data enables precise monitoring and analysis of the vehicle's behavior. Furthermore, the signal data could also include information such as acceleration or engine speed. An extension of the invention could involve integrating real-time traffic data or weather information into the vehicle signal data.

[0010] A further step of the invention may involve providing environmental model data. This environmental model data may be specific to modeling the environment around the vehicle. It may be based on measurement data from the vehicle's ultrasonic sensors, lidar, radar, and / or cameras. This environmental model data can be used to understand the vehicle's surroundings, and in particular, a current traffic scene. The environmental model data may also be referred to as environmental measurement data.

[0011] A further step of the invention may involve synchronizing playback based on the provided environmental model data and / or the provided vehicle signal data and / or the provided video recordings. This allows the various recordings to be combined, taking into account the time of each recording. The synchronization can be performed such that the synchronized playback displays each recording with respect to a common time reference and is thus time-synchronized. This can be achieved by reading timestamps from each recording and adding a time offset for each recording to display each recording at the same time.

[0012] A further step of the invention can include generating a three-dimensional visual reconstruction of the recorded environment based on synchronized playback. This makes it possible not only to display each recording in isolation with a two-dimensional video playback, as with conventional dashcams, but also to combine the recordings to create a common three-dimensional visual reconstruction for playback. This generation can be achieved by synchronizing the recording sequences temporally or aligning them temporally, thus incorporating temporal information such as timestamps of the recordings.

[0013] A further step of the invention may consist of providing a control for interactively changing the perspective of the generated three-dimensional visual reconstruction, which is coupled to a temporal control of the playback. The control may be designed as a user interface that allows a user to control the playback and / or the visual display of the generated three-dimensional visual reconstruction. The control may utilize visual elements such as buttons on a touchscreen, physical buttons, voice control, or other controls within the vehicle.These controls for the generated three-dimensional visual reconstruction make it possible to view recorded videos in 3D and interact with the recordings during playback. This allows users to change virtual camera perspectives, zoom in, zoom out, drag, pan, and tilt—all using playback functions such as play, pause, resume, rewind, etc. (and can therefore be linked to playback controls). The interactive perspective changes can also be linked to playback timing controls, allowing the perspective to be adjusted during playback. Visual elements for both controls can be displayed on a shared screen of the display unit and / or mobile device.

[0014] The method according to the invention can enable a more detailed visualization of the vehicle's condition, its surroundings, and events that occurred at a past time. While conventional video playback systems focus on replaying only recorded content captured by a single vehicle camera, the solution according to the invention allows the use of a vehicle's multi-camera system with the added complexity of synchronizing the playback of recordings captured from multiple camera streams. This enables the creation of a three-dimensional visual reconstruction within the vehicle.

[0015] The invention can not only enable simple playback, but also offer the possibility of performing in-vehicle simulations with the same recordings, but with different perspectives, zoom, tilt, pan, and drag factors. Furthermore, playback in the form of an interactive 3D rendering can offer the user the opportunity to create creative 3D perspectives and animations of the recorded content, which can then be saved as a new file and shared with others (e.g., via the internet, particularly on social media).

[0016] Along with the recordings, particularly in the form of recorded video camera data, the method can also include recorded vehicle signal data from various sensors in the vehicle, preferably together with inputs from various automated functions such as perception, environmental modeling, situation analysis, and interior monitoring, to support a black-box-like analysis in the event of accidents. The vehicle signal data can also be used for automatic camera flight animations and displays during interactive playback. The recorded data can also be used to train the various algorithms to better avoid collisions or breakdown scenarios.

[0017] It may also be possible to display the generated three-dimensional visual reconstruction on a mobile device such as a user's smartphone. For this purpose, a wireless data connection can be established between a data processing device in the vehicle and the mobile device.

[0018] In connection with the invention, and in particular with an enhanced 3D surround view application, visual resimulation can refer to the process of visualizing and displaying previously acquired environmental data. This allows for the re-observation and analysis of environmental data without the need to re-acquire the data. A possible extension is the integration of real-time modification tools into the resimulation to make changes during playback. In this context, the term "playback" can refer to the display or representation of the previously acquired environmental data by the 3D visualization application. This can be a process in which the visual resimulation of the data, as it was originally recorded or generated, is reviewed again.

[0019] Furthermore, it is possible that the following steps will be provided: - Receiving any of the video and data recordings, especially in the form of recording files, - Decoding the received video and data recordings, - Performing the step of synchronizing playback based on the decoded video and data recordings.

[0020] The individual video recordings from each camera in a vehicle's multi-camera system can be stored as separate files using a special file encoding format that efficiently compresses the raw video data. The various cameras in the multi-camera system can be mounted on different parts and sides of the vehicle and oriented in different directions to capture the most complete view of the camera's surroundings, specifically the vehicle's environment and / or its interior. Video recording decoding refers to the process of converting compressed video file information into a format suitable for playback. Decoding allows video recordings to be played back in their original quality and resolution. One example of an approach to this could be the use of a decoding algorithm or protocol.The enhancements could include the integration of error correction mechanisms to improve playback quality during disrupted transmissions. For example, each video recording could be compressed using a suitable compression format such as H.265 and stored in an appropriate video container format like MPEG4. The environmental model and vehicle status data would be stored in an efficient data container format.

[0021] It is also possible to provide video recordings from various cameras of the multi-camera system in the form of buffered video recordings, allowing the recordings to be played back a predetermined time in the past using the generated three-dimensional visual reconstruction. The buffered video recordings enable playback of recordings after a predetermined period from the past. This allows for the replay of past events. Timestamps can be used for each recording to allow for specific playback times.

[0022] It is also possible to initiate a display of the generated three-dimensional visual reconstruction on the vehicle's display unit, particularly in a center console, and / or on a mobile device. The multi-camera system can be designed as an in-vehicle camera system.

[0023] It is also possible that the following steps will be provided: - Receiving a recording trigger signal specific to an event relating to the vehicle, such as an accident, a specific scenario, or through manual activation by the user in the vehicle, - Performing at least the steps of synchronizing playback and / or generating the three-dimensional visual reconstruction upon receiving the playback selection from the list of recordings listed in the human-machine interface.

[0024] It is also possible for the three-dimensional visual reconstruction to be generated using at least one algorithm from an Enhanced 3D Surround View application. Enhanced 3D Surround View applications can generate three-dimensional reconstruction based on environmental data created using techniques such as stereo vision algorithms, which extract depth from multiple camera images, and Structure from Motion (SfM), which derives 3D structures by analyzing object movements across frames. Environmental modeling is often facilitated by in-vehicle ultrasonic and / or radar and / or LiDAR sensors that capture the 3D representation of the environment. Point cloud processing creates coherent 3D models from sets of vertices in three-dimensional coordinates, while convolutional neural networks (CNNs) predict depth from images, thus aiding in the creation of 3D models.The chosen method often depends on the requirements of the application, the availability of the hardware, and the desired level of detail and accuracy.

[0025] It is also possible that the provided control system includes at least one of the following control functions: - An interaction of the generated three-dimensional visual reconstruction during playback, - Changing a virtual camera perspective of the generated three-dimensional visual reconstruction, - Zooming in on the generated three-dimensional visual reconstruction, - Zooming out from the generated three-dimensional visual reconstruction, - Dragging and / or panning and / or tilting the three-dimensional visual reconstruction, pausing and resuming playback, - Search in playback.

[0026] This allows the user to analyze the playback in order to reliably identify an event or to check the vehicle's condition in the past.

[0027] In another aspect of the invention, a computer program, in particular a computer program product, can be provided that includes instructions which, when the computer program is executed by a computer, cause the computer to perform the method according to the invention. In this way, the computer program according to the invention can have the same advantages as have been described in detail with reference to a method according to the invention.

[0028] In a further aspect of the invention, a data processing device can be provided that is designed to carry out the method according to the invention. For example, a computer that executes the computer program according to the invention can be provided as the device. The computer can include at least one processor that can be used to execute the computer program. A non-volatile data storage medium can also be provided in which the computer program can be stored and from which the computer program can be read for execution by the processor.

[0029] According to a further aspect of the invention, a computer-readable storage medium can be provided which contains the computer program according to the invention and / or instructions which, when executed by a computer, cause the computer to perform the steps of the method according to the invention. The storage medium can be designed as a data storage device such as a hard disk and / or non-volatile memory and / or a memory card and / or a solid-state drive. The storage medium can, for example, be integrated into the computer.

[0030] Furthermore, the method according to the invention can be implemented as a computer-implemented method.

[0031] Further advantages, features, and details of the invention will become apparent from the following description, in which embodiments of the invention are described in detail with reference to the drawings. In this context, the features mentioned in the claims and in the description can be essential to the invention, either individually or in any combination. The drawings show: Fig. 1 a method, a computer program, a storage medium and a device according to an embodiment of the invention. Fig. 2 a top view of a vehicle with the camera system according to embodiments of the invention. Fig. 3. An exemplary visualization of the use of a trigger event. Fig. 4 an exemplary visualization of a data processing device according to embodiments of the invention.

[0032] Fig. Figure 1 shows a method 100 for providing dashcam playback functionality in a vehicle display unit 50 (see Fig. 2) by generating a visual simulation from individual video recordings of a multi-camera system 60 of a vehicle 30. The method 100 can be carried out at least partially by a data processing device 10, in particular by a computer program 20 stored in a computer-readable storage medium 15.

[0033] According to a first process step 101, the video recordings from various cameras 61 of the multi-camera system 60 can be provided, e.g., read from the computer-readable storage medium 15. The video recordings can result from a camera recording of the environment of the cameras 61. Furthermore, according to a second and third process step 102, 103, vehicle signal data and environment model data can also be provided. A fourth process step 104 then comprises the synchronization of a playback based on the provided environment model data, the provided vehicle signal data, and the provided video recordings. This enables the generation of a three-dimensional visual reconstruction of the recorded environment based on the synchronized playback according to a fifth process step 105.In a sixth process step 106, a control for an interactive change of a perspective of the generated three-dimensional visual reconstruction can be provided, which can be coupled with a control of the temporal playback.

[0034] According to Fig. 2. A display of the generated three-dimensional visual reconstruction can be initiated on the vehicle display unit 50, in particular in a center console of the vehicle 30, and / or on a mobile device 40.

[0035] In Fig. Figure 3 shows that a trigger event 201 can at least partially trigger the process steps. For this purpose, the data and video from the various cameras 61 of the multi-camera system 60, buffered over a predetermined period, are to be used to generate the recordings. In the example shown, the period can be 60 s, in particular 30 s before the trigger event (from the buffered recordings) and 30 s after the trigger event.

[0036] The available recordings are then listed on the human-machine interface 17. Selecting a specific recording from the list should initiate the procedural steps for playing back the corresponding recording as a three-dimensional visual reconstruction.

[0037] Fig.Figure 4 shows an exemplary configuration of components of a device 10 according to the invention. The device 10 can include an encoder 11 that receives the recordings from the cameras 61 of the vehicle 30. The encoder 11 can encode the recordings and store them in a data storage device 15, in particular within the vehicle 30. This enables the recordings to be made available for dashcam-like functionality according to the embodiments of method 100. A decoder 12 can then decode the provided recordings to enable the generation of the three-dimensional visual reconstruction using a three-dimensional visualization software unit 13. This task can be accomplished with the aid of a graphics processing unit 14.The controls can be provided by a human interface unit 15 and displayed together with the three-dimensional visual reconstruction on a display device 50.

[0038] The preceding explanation of the embodiments describes the present invention by way of examples. Naturally, individual features of the embodiments can be freely combined with one another, provided this is technically feasible without deviating from the scope of the present invention.

Claims

[1] A method (100) for providing dashcam-like playback functionality in a vehicle display unit (50) by generating a visual resimulation from individual video recordings of a multi-camera system (60) of a vehicle (30), comprising the following steps, wherein the steps are performed by a vehicle data processing device (10): - providing (101) the video recordings of various cameras (61) of the multi-camera system (60), wherein the video recordings originate from a camera recording of an environment of the cameras (61), - providing (102) vehicle signal data, wherein the vehicle signal data is specific to a state of the vehicle (30), - providing (103) environmental model data, wherein the environmental model data is specific for modeling an environment around the vehicle (30), - synchronizing (104) a playback based on the provided environment model data and the provided vehicle signal data and the provided video recordings, - generating (105) a three-dimensional visual reconstruction of the recorded environment based on the synchronized playback, - Providing (106) a control for an interactive change of a perspective of the generated three-dimensional visual reconstruction, which is temporally coupled to a control of the reproduction. [2] Method (100) according to claim 1, characterized by that the following steps are provided: - Receiving each of the video and data recordings, in particular in the form of recording files, - Decoding the received video and data recordings, - Perform playback synchronization based on the decoded video and data recordings. [3] Method (100) according to one of the preceding claims, characterized by that the video recordings of various cameras (61) of the multi-camera system (60) and additional data from the vehicle (30) are provided in the form of buffered recordings so that they are reproduced at a predetermined time in the past using the generated three-dimensional visual reconstruction. [4] Method (100) according to one of the preceding claims, characterized by that a display of the generated three-dimensional visual reconstruction is initiated on the vehicle display unit (50), in particular in a center console of the vehicle (30), and / or on a mobile device (40). [5] Method (100) according to one of the preceding claims, characterized by that the following steps are provided: - receiving a trigger signal (201) specific to an event relating to the vehicle (30) for storing the recording. [6] Method (100) according to one of the preceding claims, characterized by that the following steps are provided: - performing at least the steps of synchronizing (104) the playback and / or generating the three-dimensional visual reconstruction upon selection of a specific recording by the user for playback. [7] Method (100) according to one of the preceding claims, characterized by that the three-dimensional visual reconstruction is generated using at least one Enhanced 3D Surround View application algorithm. [8] Method (100) according to one of the preceding claims, characterized by that the control provided includes at least one of the following control functions: - an interaction of the generated three-dimensional visual reconstruction during playback, - Changing a virtual camera perspective of the generated three-dimensional visual reconstruction, - Zooming into the generated three-dimensional visual reconstruction, - Zooming out of the generated three-dimensional visual reconstruction, - Dragging and / or panning and / or tilting the three-dimensional visual reconstruction, - Pause and resume playback, - Search in playback. [9] A computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer (10) to carry out the method (100) according to any one of the preceding claims. [10] Data processing device (10) comprising a device for carrying out the method (100) according to one of claims 1 to 8. [11] A computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer (10) to perform the steps of the method (100) according to any one of claims 1 to 8.

Images