Method to operate a vehicle, vehicle, perception device, computer program product, and non-transitory computer-readable storage medium

Abstract

A method for capturing images of interesting moments in and outside a vehicle, comprising; providing a perception device 28 and sensor data 14 of happy moments in a vehicle and of interesting scenic a

Description

FIELD OF THE INVENTION

[0001] The present invention relates to the field of automobiles. More specifically, the present invention relates to a method to operate a vehicle. Furthermore, the present invention relates to a corresponding vehicle, a corresponding perception device, a corresponding computer program product, as well as to a corresponding non-transitory computer-readable storage medium. BACKGROUND INFORMATION

[0002] Our vehicles have a significant degree of attachment with the social life of their owners. Users spend some of their best social hours in their vehicles while travelling to scenic places, eating, drinking, conversing, meeting new people and more. Current vehicle systems do not possess any abilities to capture these beautiful moments missing some of best candid moments that people can never plan to take.

[0003] Nowadays, vehicles are equipped with a huge number of cameras and other sensors that may be used to capture snapshots or memories. Vehicles may comprise an in-car camera, similar to a dashboard camera, that may capture these memories. However, people today are used to using their mobile phones to plan a photo shoot and do not associate their personal vehicle with this task. This may lead to users not using the built-in camera to take photos. SUMMARY OF THE INVENTION

[0004] It is an object of the present invention to provide a method, a corresponding vehicle, a corresponding perception device, a corresponding computer program product, as well as to a corresponding non-transitory computer-readable storage medium, by which a user is provided with suitable media documenting an activity of the user.

[0005] This object is solved by a method, a corresponding vehicle, a corresponding perception device, a corresponding computer program product, as well as to a corresponding non-transitory computer-readable storage medium according to the independent claims. Advantageous embodiments are presented in the dependent claims.

[0006] A first aspect of the present invention is related to a method to operate a vehicle. The vehicle may be a car or truck. The vehicle may be configured to transport passengers. The method comprises the following steps.

[0007] A first step comprises providing to a perception device of the vehicle a sensor data history of sensor data recorded at respective record times over a time period by a sensor device of the vehicle. The sensor data comprising media recorded by recording units of the sensor device. The perception device may be a memory auto capture system, configured to provide a media file for a user.

[0008] In other words the sensor device of the vehicle records sensor data at respective record times. The sensor device records the sensor data over the time period. The time period may be, for example, a time of a journey of the vehicle. The sensor device generates the sensor data history comprising the sensor data. In the sensor data history the sensor data are linked to the respective record time at which the respective sensor data was recorded by the sensor device. The sensor data comprise the media recorded by the sensor device. The media may be pictures or audio recorded at the respective record times. The media may describe images, videos and / or audios of an environment of the vehicle or of a cabin of the vehicle.

[0009] A next step comprises a generation of a classification history of a classification at the respective times over the time period by the perception device according to a predefined classification procedure as a function of the sensor data history. In other words, the perception device analyses that sensor data history to generate the classification history. The classification history describes the classification at respective times over the time period. The classification history is generated based on the sensor data history.

[0010] The method comprises a selection of media as a function of the classification of the respective record times of the images according to a predefined selection procedure by the perception device. In other words, each of the media has a respective record time. The respective record times are attributed the classification of the classification history. The perception device selects the media according to the classification of their respective record time according to the classification history.

[0011] A next step comprises generating a media file based on the selected media according to a predefined media generation procedure and saving the media file on a memory device by the perception device. In other words, the perception device may generate the media file and store the media file on the memory device.

[0012] The invention has the advantage that media recorded by the vehicle are provided to the memory device. Therefore the user may use the media recorded by the vehicle. Because the vehicle records a large number of media a message is provided that provides media of the specific classification. Therefore only interesting media are provided to the user.

[0013] According to a further embodiment of the present invention, the predefined classification procedure comprises a determination of predefined motion features in the sensor data. The classification at the respective times is determined as a function of the determined predefined motion features.

[0014] According to a further embodiment of the present invention, the predefined classification procedure comprises a determination of predefined activity features in the sensor data. The classification of the respective times is determined as a function of the determined predefined activity features.

[0015] According to a further embodiment of the present invention, the predefined classification procedure comprises a determination of predefined image features in the sensor data. Classification of the respective times is determined as a function of the predetermined predefined image features.

[0016] According to a further embodiment of the present invention, the predefined classification procedure comprises an identification of predefined identities in the sensor data. The classification the respective times is determined as a function of the identified identities.

[0017] According to a further embodiment of the present invention, the predefined selection procedure comprises an evaluation of quality characteristics of the media. The media are selected as a function of the evaluated quality characteristics.

[0018] According to a further embodiment of the present invention, the media comprise images of the cabin of the vehicle and / or images of an environment of the vehicle.

[0019] A second aspect of the present invention is related to a vehicle comprising at least a perception device, configured to perform a method according to the first aspect of the present invention.

[0020] The vehicle comprises a sensor device configured to record sensor data at respective record times over a time period, and to provide a sensor data history of the sensor data to a perception device of the vehicle, wherein the sensor data comprise media recorded by recording units of the sensor device.

[0021] The perception device is configured to generate a classification history of a classification at respective times over the time period according to a predefined classification procedure as a function of the sensor data history.

[0022] The perception device is configured to select media as a function of the classification of the respective record times of the images according to a predefined selection procedure.

[0023] The perception device is configured to generate a media file based on the selected media according to a predefined media generation procedure and to save the media file on a memory device.

[0024] A third aspect of the present invention is related to a perception device, configured to perform a method according to the first aspect of the present invention.

[0025] The perception device is configured to receive a sensor data history of sensor data recorded at respective record times over a time period by a sensor device of the vehicle, the sensor data comprising media recorded by recording units of the sensor device.

[0026] The perception device is configured to generate a classification history of a classification at respective times over the time period according to a predefined classification procedure as a function of the sensor data history.

[0027] The perception device is configured to select media as a function of the classification of the respective record times of the images according to a predefined selection procedure.

[0028] The perception device is configured to generate a media file based on the selected media according to a predefined media generation procedure and to save the media file on a memory device.

[0029] A fourth aspect of the present invention is related to a computer program product comprising program code for performing a method according to the first aspect of the present invention.

[0030] A fifth aspect of the present invention relates to a non-transitory computer-readable storage medium comprising at least the computer program product according to the fourth aspect of the present invention.

[0031] Advantageous embodiments of the method are to be regarded as advantageous embodiments of the vehicle, the perception device, the computer program product, as well as the non-transitory computer-readable storage medium. The vehicle as well as the perception device therefore comprise computing units for performing the method.

[0032] A computing unit may in particular be understood as a data processing device, which comprises processing circuitry. The computing unit can therefore in particular process data to perform computing operations. This may also include operations to perform indexed accesses to a data structure, for example a look-up table, LUT.

[0033] In particular, the computing unit may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits, ASIC, one or more field-programmable gate arrays, FPGA, and / or one or more systems on a chip, SoC. The computing unit may also include one or more processors, for example one or more microprocessors, one or more central processing units, CPU, one or more graphics processing units, GPU, and / or one or more signal processors, in particular one or more digital signal processors, DSP. The computing unit may also include a physical or a virtual cluster of computers or other of said units.

[0034] In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0035] A memory unit may be implemented as a volatile data memory, for example a dynamic random access memory, DRAM, or a static random access memory, SRAM, or as a non-volatile data memory, for example a read-only memory, ROM, a programmable read-only memory, PROM, an erasable programmable read-only memory, EPROM, an electrically erasable programmable read-only memory, EEPROM, a flash memory or flash EEPROM, a ferroelectric random access memory, FRAM, a magnetoresistive random access memory, MRAM, or a phase-change random access memory, PCRAM.

[0036] Further advantages, features, and details of the invention derive from the following description of preferred embodiments as well as from the drawings. The features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figures and / or shown in the figures alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figures.

[0038] The drawings show in:

[0039] Fig. 1 a schematic illustration of a vehicle; and

[0040] Fig. 2 a schematic illustration of a method to operate a vehicle.

[0041] In the figures the same elements or elements having the same function are indicated by the same reference signs. DETAILED DESCRIPTION

[0042] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0043] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0044] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0045] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0046] Fig. 1 shows a schematic illustration of a vehicle.

[0047] The vehicle 10 may comprise a sensor device 12 configured to provide the sensor data 14. The sensor device 12 may be configured to record the sensor data 14 during an operation of the vehicle 10. The sensor data 14 may be recorded at respective recording times over a time period. The time period may be a travel time of the vehicle 10. The sensor device 12 may comprise a plurality of sensor units such as a GPS sensor unit 16, a temperature sensor unit 18, a microphone unit 20, an interior camera unit and an exterior camera unit 24. The cabin camera unit 22 may be configured to take videos and / or pictures of the cabin of the vehicle 10. The microphone unit 20 may be configured to record sounds from the cabin. The exterior camera unit 24 may be configured to take pictures and / or videos of an environment around the vehicle 10. The sensor data 14 may comprise media. The media may comprise the pictures and / or videos taken by the camera units and / or sounds recorded by the microphone unit 20. The sensor device 12 may be configured to generate a sensor data history of the provided sensor data 14. The sensor data history may comprise the sensor data 14 recorded at the respective recording times.

[0048] Passengers 26 of the vehicle 10 wish to receive a media file 30 as a reminder of the journey. The media file 30 could be generated from selected media recorded by the sensor device 12 during the time period. However, the selected media should be limited to media showing interesting situations.

[0049] To provide the media, the vehicle 10 comprises a perception device 28 and the sensor device 12 is configured to provide the sensor data history to the perception device 28. The perception device 28 is configured to receive the sensor data history and generate a classification history of a classification at respective times over the time period according to a predefined classification procedure as a function of the sensor data 14.

[0050] For example, the perception device 28 may be configured to analyse the sensor data 14. For example, the perception device 28 may be configured to identify predefined emotion features 34 in the sensor data 14. The emotion features 34 may comprise smiling faces in images of the cabin or cheering in sounds recorded in the cabin. The perception device 28 may also identify predefined activity features 34 in the sensor data 14. For example, the perception device 28 may be configured to identify singing by passengers 26 in the cabin in images, videos and sounds of the cabin. The perception device 28 may be configured to identify predefined image features 34 of the sensor data 14. The predefined image features 34 may include, for example, predefined known landscape scenes in images taken by the external camera unit of the vehicle 10. The perception device 28 may also be configured to identify predefined identities in the sensor data 14. For example, the perception device 28 may be configured to identify known identities of passengers 26 in the images of the vehicle 10. Based on the various features 34, the perception device 28 may provide the classification history describing a classification at respective times over the time period. The classification may describe how interesting a particular situation is.

[0051] The perception device 28 may be configured to select media as a function of the classification of the respective recording times of the images according to the predefined selection procedure. In other words, some times may be classified as interesting while other times may be classified as uninteresting. The perception device 28 may examine the media recorded during the times classified as interesting. The perception device 28 may select the media according to a predefined selection procedure. The selection procedure may include an evaluation of quality characteristics 40 of the media. In other words, the perception device 28 may be configured to determine the quality of the media. The quality characteristics 40 may be evaluated based on the noise level of an image, the illumination of the image, the blurriness of the image, and / or the contrast of the image. The perception device 28 may select some of the media and create a media file 30 based on the selected media. The media file 30 may be a folder containing the media, a slide show, or a video generated using the selected media.

[0052] Fig 2 shows a schematic illustration of a method to operate a vehicle.

[0053] In a first step S1 of the method, sensor units of the sensor device 12 may record sensor data 14 during a time period. The sensor data 14 may be recorded at respective recording times. The sensor units may comprise an internal camera unit, a microphone unit 20, an external camera unit and / or a GPS device. The sensor device 12 may generate a sensor data history of the sensor data 14.

[0054] In a next step S2, the sensor device 12 may provide the sensor data history to the perception device 28. The perception device 28 may receive the sensor data history and may perform a classification procedure to classify times of the time period in a classification history based on the sensor data history. The classification procedure may comprise identification of emotion features 34 and / or activity features 34 and / or identities and / or image features 34 in the sensor data 14 by a recognition module 36. The recognition module 36 may provide the features 34 identified in the sensor data 14.

[0055] The activity features 34 may comprise predefined movements of passengers 26 or a sporty driving style that can be identified in the sensor data 14. The identity features 34 may comprise passengers 26 that can be identified in the sensor data 14. The identification may be based on images or sounds.

[0056] A classifier module 42 may classify times of the time period based on the detected features 34. For example, if sensor data 14 recorded at a particular time includes the predefined activity features 34, the particular time may be classified as interesting.

[0057] In a next step S3, the perception device 28 may select media as a function of the classification of the respective recording times of the media according to a predefined selection procedure. In other words, the perception device 28 may select media recorded at recording times classified as interesting according to the classification history. The selection may comprise an evaluation of quality characteristics 40 of the media by the quality assessment module 38 to evaluate quality values of the media. For example, the quality value may describe a noise level of an image. The media may be selected as a function of the evaluated quality values. It is therefore possible for only media of a predefined quality to be selected.

[0058] In a next step S4, the perception device 28 may use the selected media as input for a predefined media generation procedure to generate a media file 30. The perception device 28 may store the media file 30 on a memory device 32 and / or provide the media file 30 to a server external to the vehicle 10.

[0059] Example User Story:

[0060] Imagine a user who visits Lake Tahoe with their friends. This user had an amazing time with their friends travelling in a brand-new vehicle and gets back home only to know that they did not have enough pictures clicks throughout the trip.

[0061] Luckily enough, the user is driving a vehicle equipped with the perception device. The vehicle has clicked amazing panoramic / 360 degree pictures of scenic locations the user visited. The vehicle has taken a beautiful timelapse of the beach view sunset when the user parked it near the shore. The user had a blast singing songs, eating food and stargazing all night with their friends. All interesting moments have been captured and saved at a location which they can find and post on online platforms. signs vehicle sensor device sensor data GPS sensor unit temperature sensor unit microphone unit cabin camera unit exterior camera unit passenger perception device media file memory device feature recognition module quality assessment module quality characteristic classifier module steps

Claims

1. A method to operate a vehicle (10) comprising the following steps:- providing to a perception device (28) of the vehicle (10) a sensor data history of sensor data (14) recorded at respective record times over a time period by a sensor device (12) of the vehicle (10), the sensor data (14) comprising media recorded by recording units of the sensor device (12);- generating a classification history of a classification at respective times over the time period by the perception device (28) according to a predefined classification procedure as a function of the sensor data history;- selecting media as a function of the classification of the respective record times of the images according to a predefined selection procedure by the perception device (28); and- generating a media file (30) based on the selected media according to a predefined media generation procedure and saving the media file (30) on a memory device (32) by the perception device (28).

2. The method according to claim 1, characterized in thatthe predefined classification procedure comprises a determination of predefined emotion features (34) in the sensor data (14), wherein the classification at the respective times is determined as a function of the determined predefined emotion features (34).

3. The method according to claim 1 or 2,characterized in thatthe predefined classification procedure comprises a determination of predefined activity features (34) in the sensor data (14), wherein the classification at the respective times is determined as a function of the determined predefined activity features (34).

4. The method according to any one of the preceding claims, characterized in thatthe predefined classification procedure comprises a determination of predefined image features (34) in the sensor data (14), wherein the classification at the respective times is determined as a function of the determined predefined image features (34).

5. The method according to any one of the preceding claims, characterized in thatthe predefined classification procedure comprises an identification of predefined identities in the sensor data (14), wherein the classification at the respective times is determined as a function of the detected identities.

6. The method according to any one of the preceding claims, characterized in thatthe predefined selection procedure comprises an evaluation of quality characteristics (40) of the media, wherein the media are selected as a function of the evaluated quality characteristics (40).

7. The method according to any one of the preceding claims, characterized in thatthe media comprise images of a cabin of the vehicle (10) and / or images of an environment of the vehicle (10).

8. A vehicle (10), characterized in that - the vehicle (10) comprises a sensor device (12) configured to record sensor data (14) at respective record times over a time period, and to provide a sensor data history of the sensor data (14) to a perception device (28) of the vehicle (10),wherein the sensor data (14) comprise media recorded by recording units of the sensor device (12);- the perception device (28) is configured to generate a classification history of a classification at respective times over the time period according to a predefined classification procedure as a function of the sensor data history;- the perception device (28) is configured to select media as a function of the classification of the respective record times of the images according to a predefined selection procedure; and- the perception device (28) is configured to generate a media file (30) based on the selected media according to a predefined media generation procedure and to save the media file (30) on a memory device (32).

9. A perception device (28), characterized in that- the perception device (28) is configured to receive a sensor data history of sensor data (14) recorded at respective record times over a time period by a sensor device (12) of the vehicle (10), the sensor data (14) comprising media recorded by recording units of the sensor device (12);- the perception device (28) is configured to generate a classification history of a classification at respective times over the time period according to a predefined classification procedure as a function of the sensor data history;- the perception device (28) is configured to select media as a function of the classification of the respective record times of the images according to a predefined selection procedure; and- the perception device (28) is configured to generate a media file (30) based on the selected media according to a predefined media generation procedure and to save the media file (30) on a memory device (32).

10. A computer program product comprising program code for performing the method according to any one of claims 1 to 7.

11. A non-transitory computer-readable storage medium comprising at least the computer program product according to claim 10.

Images