Method for providing a driver-type reproduction function in an in-vehicle display unit

Abstract

The invention relates to a method (100) for providing an automobile data recorder playback function in an on-board display unit (50) by generating a visual resimulation from individual video recordings of a multi-camera system (60) of a vehicle (30), the method comprising the following steps: providing a plurality of video recorders (60); the step is performed by the vehicle data processing device (10): (101) providing video recordings of different cameras (61) of the multi-camera system (60), the video recordings being obtained from camera recordings of the environment of the cameras (61); providing (102) vehicle signal data, the vehicle signal data being specific for a state of the vehicle (30); (103) providing environmental model data, which is specifically for the modeling of the surroundings of the vehicle (30); (104) performing synchronous playback based on the provided environmental model data, the provided vehicle signal data and the provided video record; (105) generating a three-dimensional visual reconstruction model of the recorded environment based on the synchronized playback; a control for interactive transformation of the perspective of the generated three-dimensional visual reconstruction model is provided (106), the control being temporally coupled with the playback control.

Description

Technical Field

[0001] The present invention relates to a method for providing a playback function of a driving record in a vehicle display unit. In addition, the present invention relates to a computer program, a device and a storage medium for this purpose. Background Art

[0002] A drive recorder (DVR) system is similar to a dash cam and is able to capture video footage from one or more onboard cameras. These recordings are useful for purposes such as accident documentation, parking protection, vehicle and cabin safety, and preserving journey memories. A typical DVR system includes a recorder, a video / media player, and an interface for seamless export or transfer of recordings.

[0003] Typically, a recorder captures video content and stores it in a system-accessible memory as a file with a specific file format. Recording is triggered manually or automatically based on an event such as collision / threat detection. Once triggered, the recorder continues to capture video for the duration specified in the settings. Advanced DVR systems may offer the possibility to buffer video content, thereby recording video content from the past relative to the triggering time point.

[0004] The video player may provide functions for playback of recorded video, such as play, pause, stop, seek, etc. The most advanced car DVR systems on the market today allow users to playback video content on the infotainment display. Car cameras are typically fisheye cameras with uncorrected content over a large field of view (FoV). This means that the recording will be distorted, and the playback may be distorted or already corrected, depending on the features supported by the system. The most advanced systems today can playback content from a single camera or a cascaded video of multiple camera images together. Summary of the invention

[0005] According to aspects of the invention, a method having the features of claim 1, a computer program having the features of claim 9, a data processing device having the features of claim 10 and a computer-readable storage medium having the features of claim 11 are provided. Further features and details of the invention are disclosed in the respective dependent claims, the description and the drawings. Features and details described here for the method according to the invention also correspond to the computer program, the data processing device and the computer-readable storage medium and vice versa.

[0006] In one aspect of the present invention, a method for providing a driving record replay function for a vehicle, in particular in an on-board display unit, by generating a visual re-simulation from individual video recordings of a multi-camera system of the vehicle comprises the following steps. These steps are preferably performed by an on-board data processing device, i.e., a data processing device of the vehicle, in particular a data processing device according to the present invention. The on-board display unit can be configured as a device in the vehicle for visualizing information and data about the vehicle status and other relevant parameters. It can particularly include a digital display located in or in the surrounding area of ​​the vehicle's center console. The unit enables the driver to quickly and clearly understand important vehicle data. A possible design of the on-board display unit can provide enhanced interactive capabilities, such as touch screen functionality. Additionally, extended functionality in the form of voice control or tactile feedback can be integrated.

[0007] The "dashcam replay function" may refer to a function of a vehicle that generally allows a user to replay recorded data and video sequences from a camera installed in the vehicle. The function improves safety by allowing the user to retrospectively analyze critical driving situations. A potential extended function may be an analysis function that automatically highlights key moments in video material. Alternatively, the dashcam replay function may be combined with a cloud storage solution to store the recordings for a long term and make them accessible from a variety of devices such as mobile devices.

[0008] According to the first method step of the present invention, video recordings of different cameras of a multi-camera system can be provided. These video recordings can be video recordings of the environment of the camera. The environment of the camera can include the vehicle surroundings and optionally also the vehicle interior.

[0009] Another method step of the present invention may include providing vehicle signal data. The vehicle signal data may be specific to the state of the vehicle. Data processing equipment of the vehicle and in particular vehicle sensors may provide the signal data. The signal data may indicate speed and / or steering angle and / or braking activity and / or the position of the vehicle (e.g., GPS data). Providing these vehicle signal data allows accurate monitoring and analysis of vehicle behavior. In addition, the signal data may also contain information such as acceleration or engine speed. Extended functionality may include integrating real-time traffic data or weather information into the vehicle signal data.

[0010] Another method step of the present invention may include providing environmental model data. The environmental model data may be specifically for modeling the vehicle's surroundings. The environmental model data may be based on measurement data from ultrasonic sensors and / or lidar and / or radar and / or cameras of the vehicle. Thus, the environmental model data may be used to understand the vehicle's environment, in particular the current traffic scene. The environmental model data may also be referred to as environmental measurement data.

[0011] Another method step of the present invention may include synchronizing playback based on the provided environment model data and / or the provided vehicle signal data and / or the provided video recording. This allows different recordings to be combined together taking into account the time of each recording. The synchronization operation can be performed in such a way that the synchronized playback displays each recording with a common time reference and is thus synchronized in time. This can be achieved by reading the timestamp of each recording and adding a time offset to each recording to display each recording at the same point in time.

[0012] Another method step of the invention may include generating a 3D visual reconstruction model of the recorded environment based on the synchronized playback. This allows each recording to be played back not only individually using 2D video like a normal dashcam, but also to combine the recordings together to build a common 3D visual reconstruction model for playback. The generation may be performed in such a way that the recording sequences are timed or ordered with respect to each other, taking into account temporal information such as the timestamps of the recordings.

[0013] Another method step of the present invention may include providing a control for interactive transformation of the viewing angle of the generated three-dimensional visual reconstruction model, which is coupled in time with the playback control. The control can be configured as a user interface that enables a user to control the playback and / or visual presentation of the generated three-dimensional visual reconstruction model. The control can utilize visual elements, such as buttons on a touch screen or physical buttons or voice controls or other controls in the car. Using these controls for the generated three-dimensional visual reconstruction model, the recorded 3D video can be viewed and interacted with the record during playback, so that the virtual camera viewing angle, zoom in, zoom out, drag, pan and tilt can be transformed-all of which use the playback function of the record at the same time, such as play, pause, resume, search, etc. (thus, it can be coupled with the playback control). The control for interactive transformation of the viewing angle can also be coupled in time with the playback control, so that the viewing angle can be controlled during playback, and for these two controls, the visual element can be displayed on the common screen of the display unit and / or mobile device.

[0014] The method according to the invention can provide a more detailed visualization of the vehicle state, environment and events that occurred at past points in time. However, conventional video playback systems focus on simply playing back content recorded by a single vehicle camera, while the solution according to the invention can use the vehicle's multi-camera system, thus having the additional complexity of synchronizing the playback of recordings recorded by multiple cameras. This allows the generation of a three-dimensional visual reconstruction model in the vehicle.

[0015] The present invention enables not only simple playback, but also on-board re-simulation using the same recording but with different perspectives, zoom, tilt, pan and drag elements. Additionally, playback in the form of 3D interactive playback can provide users with the opportunity to generate creative 3D perspectives and animations on the recorded content, which can then be saved as a new file to be shared (e.g., via the Internet, particularly in social media).

[0016] In addition to recording, in particular in the form of video camera data, the method may also include recording vehicle signal data from different sensors in the vehicle, preferably together with inputs from different automated functions (such as perception, environmental modeling, situation analysis and internal monitoring) to help perform black box-like analysis in the event of an accident. The vehicle signal data may also be used for automatic camera fly-through animations and presentations during interactive playback. The recorded data may also be used to train different algorithms to better avoid collision or accident scenarios.

[0017] The generated 3D visual reconstruction model can also be visualized in a mobile device such as a user's smartphone. To this end, a wireless data connection can be provided by the vehicle's data processing device and the mobile device.

[0018] In the present invention, particularly in enhanced 3D surround view applications, visual re-simulation can refer to the process of visualizing and displaying previously captured environmental data. This allows the environmental data to be re-observed and analyzed without re-capturing the data. Potential extended functionality can be to integrate real-time modification tools into the re-simulation model to transform during playback. In this case, "playback" can refer to the act of displaying or presenting previously captured environmental data through a 3D visualization application. It can involve the process of visual re-simulation of re-accessing the originally recorded or generated data.

[0019] In addition, the method of the present invention can also provide the following steps:

[0020] - receiving each video and data record, in particular in the form of a record file,

[0021] -decode received video and data records,

[0022] - Based on the decoded video and data records, a synchronous playback step is performed.

[0023] The individual video records of each camera of the vehicle multi-camera system can be stored as individual files using a certain file encoding format that can efficiently compress the original video data. Different cameras of the multi-camera system can be installed in different parts and sides of the vehicle and arranged in different directions to record the most complete view of the camera environment, especially the environment around the vehicle and / or the environment inside the vehicle. The decoding of the video record can refer to the process of converting the compressed video file information into a format suitable for playback. Through decoding, the video record can be played back with its original quality and resolution. The exemplary method may involve the use of a decoding algorithm or protocol for this purpose. Extended functionality may include an integrated error correction mechanism to enhance the playback quality in the event of a transmission interruption. For example, each video record can be compressed using a suitable compression format (such as H.265 format) and stored in a suitable video container format (such as MPEG4). The environmental model and vehicle state data should be stored in an efficient data container format.

[0024] The video recordings from the different cameras of the multi-camera system can also be provided in the form of buffered video recordings, so that the recordings of a predetermined period of time in the past can be replayed using the generated 3D visual reconstruction model. The buffered video recording can replay the recordings of a predetermined period of time in the past. This allows the replay of past events. Each recording can be timestamped to allow a specific playback time.

[0025] Display of the generated 3D visual reconstruction model may also be initiated on an onboard display unit, in particular on a center console of the vehicle, and / or on a mobile device, the onboard display unit may be part of an infotainment system of the vehicle. The multi-camera system may be configured as an onboard camera system.

[0026] The method of the present invention may also provide the following steps:

[0027] - receiving a recording trigger signal specifically for an event at the vehicle (e.g. an accident, a special scenario or manual activation by a user in the vehicle),

[0028] -When a playback selection is received from the list of records listed in the human-machine interface, at least the steps of synchronous playback and / or generating a three-dimensional visual reconstruction model are performed.

[0029] The 3D visual reconstruction model may also be generated using at least one enhanced 3D surround view application algorithm. The enhanced 3D surround view application may generate the 3D reconstruction model based on environmental data generated using techniques such as stereo vision algorithms that extract depth from multiple camera images and structure from motion (SfM) that infers 3D structure by analyzing object motion between frames. Environmental modeling is typically accomplished using airborne ultrasonic and / or radar and / or lidar sensors that capture a 3D representation of the environment. Point cloud processing creates a cohesive 3D model from a set of vertices in 3D coordinates, while convolutional neural networks (CNNs) predict depth from images to help generate the 3D model. The method selected typically depends on the application requirements, hardware availability, and the desired detail and accuracy.

[0030] The provided controls may also include at least one of the following control functions:

[0031] - interaction of the generated 3D visual reconstruction model during playback,

[0032] - changing the virtual camera perspective of the generated 3D visual reconstruction model,

[0033] - zooming in on the generated three-dimensional visual reconstruction model,

[0034] - Reduce the generated 3D visual reconstruction model,

[0035] - drag and / or translate and / or tilt the 3D visual reconstruction model,

[0036] - pause and resume said playback,

[0037] - Seek within said playback.

[0038] This enables users to analyze playback to reliably identify events or examine the vehicle's past condition.

[0039] In another aspect of the present invention, a computer program, in particular a computer program product, can be provided, which includes instructions, when a computer executes the computer program, the instructions cause the computer to perform the method according to the present invention. Thus, the computer program according to the present invention can have the same advantages as described in detail with reference to the method according to the present invention.

[0040] In another aspect of the present invention, a data processing device can be provided, which is configured to be able to perform the method according to the present invention. As the device, for example, a computer that executes the computer program according to the present invention can be provided. The computer may include at least one processor, which can be used to execute the computer program. The present invention can also provide a non-volatile data memory, in which the computer program can be stored and the computer program can be read from it by the processor for execution.

[0041] According to another aspect of the present invention, a computer readable storage medium may be provided, comprising a computer program and / or instructions according to the present invention, which, when executed by a computer, causes the computer to perform the steps of the method according to the present invention. The storage medium may be formed as a data storage device, such as a hard disk and / or a non-volatile memory and / or a memory card and / or a solid state drive. The storage medium may, for example, be integrated into a computer.

[0042] Furthermore, the method according to the present invention may be realized as a computer-implemented method. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Further advantages, features and details of the invention will emerge from the following description, in which embodiments of the invention will be described in detail with reference to the accompanying drawings. Here, the features mentioned in the claims and the description may be essential to the invention individually or in any combination. The following shows:

[0044] Figure 1 A method, a computer program, a storage medium and an apparatus according to embodiments of the present invention are shown.

[0045] Figure 2 A top view of a vehicle having a camera system according to an embodiment of the present invention is shown.

[0046] Figure 3 An exemplary visualization of the use of triggers is shown.

[0047] Figure 4 An exemplary visualization diagram of a data processing device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0048] Figure 1 A method for generating a visual re-simulation by using various video recordings of a multi-camera system 60 of a vehicle 30 to display a video on an onboard display unit 50 (see Figure 2 ) provides a method 100 for providing a driving recorder playback function. The method 100 can be at least partially executed by a data processing device 10, in particular, by a computer program 20 stored in a computer-readable storage medium 15.

[0049] According to the first method step 101, video records of different cameras 61 of the multi-camera system 60 can be provided, for example read out from a computer-readable storage medium 15. The video record can come from a camera recording of the environment of the camera 61. In addition, according to the second and third method steps 102, 103, vehicle signal data and environmental model data can also be provided. Then, a fourth method step 104 includes synchronized playback based on the provided environmental model data and the provided vehicle signal data and the provided video record. In this way, a three-dimensional visual reconstruction model of the recorded environment can be generated based on the synchronized playback according to the fifth method step 105. In a sixth method step 106, controls for interactive transformation of the viewing angle of the generated three-dimensional visual reconstruction model can be provided, and the controls can be temporally coupled with the playback controls.

[0050] according to Figure 2 The display of the generated three-dimensional visual reconstruction model can be initiated on an on-board display unit 50 , in particular in a central console of the vehicle 30 , and / or on the mobile device 40 .

[0051] exist Figure 3 In the example shown, it is shown that a triggering event 201 can at least partially start the method steps. To this end, data and videos buffered for a predetermined time period of different cameras 61 of the multi-camera system 60 are used to generate a record. In the example shown, the time period can be 60 seconds, in particular 30 seconds before the triggering event (obtained from the buffered record) and 30 seconds after the triggering event.

[0052] The available recordings are then listed on the human-machine interface 17. Selecting a particular recording from the list will initiate the method steps for replaying the corresponding recording as a three-dimensional visual reconstruction model.

[0053] Figure 4 An exemplary construction of the components of the device 10 according to the invention is shown. The device 10 may comprise an encoder 11 which receives a recording from a camera 61 of a vehicle 30. The encoder 11 may encode the recording and store it in a data storage device 15, in particular within the vehicle 30. This enables the recording to be provided for the function of a driving recording ritual according to an embodiment of the method 100. The decoder 12 may then decode the provided recording so that a three-dimensional visual reconstruction model can be generated using a three-dimensional visualization software unit 13. This task may be performed with the help of a graphics processing unit 14. Controls may be provided via a human-machine interface unit 17 and displayed on a display 50 together with the three-dimensional visual reconstruction model.

[0054] The above description of the embodiments illustrates the present invention by way of example. Of course, as long as it is technically reasonable, the features of the various embodiments can be freely combined with each other without departing from the scope of the present invention.

Claims

1. A method (100) for providing a driving record replay function in an on-board display unit (50) by generating a visual re-simulation from each video recording of a multi-camera system (60) of a vehicle (30), the method comprising the following steps, which are performed by a vehicle data processing device (10): (101) providing video recordings of different cameras (61) of a multi-camera system (60), said video recordings being video recordings of the environment of the cameras (61), - (102) providing vehicle signal data, the vehicle signal data being specific to the state of the vehicle (30), - (103) providing environmental model data, the environmental model data being specifically aimed at modeling the environment surrounding the vehicle (30), -(104) synchronously playing back based on the provided environment model data, the provided vehicle signal data and the provided video recording, -(105) generating a three-dimensional visual reconstruction model of the recorded environment based on the synchronized playback, -(106) providing controls for interactive transformation of the viewing angle of the generated three-dimensional visual reconstruction model, said controls being temporally coupled with the playback controls.

2. The method (100) according to claim 1, characterized in that: The method provides the following steps: - receiving each video and data record, in particular in the form of a record file, -decode received video and data records, - Based on the decoded video and data records, synchronization of the playback is performed.

3. The method (100) according to claim 1 or 2, characterized in that: The video recordings of the different cameras (61) of the multi-camera system (60) and additional data from the vehicle (30) are provided in the form of buffered recordings, so that the recordings of a predetermined period of time in the past can be replayed using the generated three-dimensional visual reconstruction model.

4. The method (100) according to any one of the preceding claims, characterized in that The display of the generated three-dimensional visual reconstruction model can be initiated on an on-board display unit (50), in particular in a central console of the vehicle (30), and / or on a mobile device (40).

5. The method (100) according to any one of the preceding claims, characterized in that The method provides the following steps: - receiving a trigger signal (201) specific to an event at the vehicle (30) to store a record.

6. The method (100) according to any one of the preceding claims, characterized in that The method provides the following steps: - When the user selects a specific recording for playback, at least the steps of synchronous playback (104) and / or generating a three-dimensional visual reconstruction model are performed.

7. The method (100) according to any one of the preceding claims, characterized in that The method uses at least one enhanced 3D surround view application algorithm to generate a three-dimensional visual reconstruction model.

8. The method (100) according to any one of the preceding claims, characterized in that The provided controls include at least one of the following control functions: - interaction of the generated 3D visual reconstruction model during playback, - changing the virtual camera perspective of the generated three-dimensional visual reconstruction model, - zooming in on the generated three-dimensional visual reconstruction model, - reducing the generated three-dimensional visual reconstruction model, - drag and / or translate and / or tilt the 3D visual reconstruction model, - pause and resume said playback, - Seek within said playback.

9. A computer program (20) comprising instructions which, when executed by a computer (10), cause the computer (10) to perform the method (100) of any one of the preceding claims.

10. A data processing device (10) comprising means for performing the method (100) of any one of claims 1 to 8.

11. A computer-readable storage medium (15), the computer-readable storage medium comprising instructions, which, when executed by a computer (10), cause the computer (10) to perform the steps of the method (100) of any one of claims 1 to 8.

Images