Vehicle information display device

Abstract

The present invention provides a vehicle information display device that makes it easier for vehicle occupants to identify the portion of recorded video footage that captures an incident that occurred in the vehicle. [Solution] The vehicle information display device 100 records video information from a drive recorder camera 1a that captures images of the outside of the vehicle and displays the video on the vehicle's HMI 3. The vehicle information display device 100 includes a display control unit 15 that displays thumbnail images on the HMI 3 for selecting which video to display on the HMI 3 from the recorded video information, and an estimation unit 14 that estimates the occurrence of a predetermined incident to the vehicle based on the detection results of an on-board external sensor 1 or an internal sensor 2. When the occurrence of an incident is estimated, the display control unit 15 adds marker information to the thumbnail image corresponding to the video containing information captured at the estimated time of the incident and displays it on the HMI 3.

Description

Technical Field

[0001] The present disclosure relates to an information display device for a vehicle.

Background Art

[0002] Conventionally, in a thumbnail display screen where a first thumbnail array shifted left and right by an input device and a second thumbnail array shifted in联动 with the shift of the first thumbnail array are displayed, when any frame is selected, a technique for playing the video of the selected frame or the video from the most recent scene transition point onwards is known (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In an information display device for a vehicle that records information on an image of an in-vehicle camera that images the outside of the vehicle and displays the image on a display unit of the vehicle, a thumbnail image for selecting and playing the image to be displayed on the display unit among the recorded image information may be displayed on the display unit of the vehicle. Thumbnail images are generated at predetermined intervals in the recorded video, and a plurality of thumbnail images may be displayed on the display unit. When such a plurality of thumbnail images are displayed on the display unit, it may be difficult for the vehicle occupants to identify the portion of the recorded video information where an incident that occurred to the vehicle was imaged just by looking at the plurality of thumbnail images.

Means for Solving the Problems

[0005] One aspect of the present disclosure is a vehicle information display device that records video information from an in-vehicle camera that captures the outside of a vehicle and displays the video on a display unit of the vehicle, comprising: a display control unit that displays thumbnail images on the display unit of the vehicle for selecting the video to be displayed on the display unit from the recorded video information; and an estimation unit that estimates the occurrence of a predetermined incident to the vehicle based on the detection results of an in-vehicle external or internal sensor, wherein, when the occurrence of an incident is estimated, the display control unit adds marker information to the thumbnail image corresponding to the video containing information captured at the estimated time of the incident and displays it on the display unit.

[0006] In a vehicle information display device according to one aspect of this disclosure, an estimation unit estimates the occurrence of a predetermined incident to the vehicle based on the detection results of an on-board external or internal sensor. When the occurrence of an incident is estimated, a display control unit adds marker information to a thumbnail image corresponding to a video containing information captured at the estimated time of the incident, and the thumbnail image is displayed on the display unit. This makes it easier for the vehicle occupant to visually distinguish the thumbnail image with the marker information from a thumbnail image without the marker information. As a result, it becomes easier for the vehicle occupant to identify the portion of the recorded video information in which the incident that occurred to the vehicle was captured.

[0007] In one embodiment, the internal sensor includes a sound sensor that acquires the sound inside the vehicle, and the estimation unit may estimate the occurrence of an incident when the time rate of change of at least one of the amplitude, frequency, and sound pressure of the sound inside the vehicle is greater than or equal to a predetermined rate of change threshold. In this case, the estimation unit can estimate the occurrence of a predetermined incident to the vehicle based on the sound inside the vehicle acquired by the sound sensor.

[0008] Another aspect of the present disclosure is a vehicle information display device that records video information from an in-vehicle camera that captures the outside of a vehicle and displays the video on a display unit of the vehicle, comprising: a display control unit that displays thumbnail images on the display unit of the vehicle for selecting the video to be displayed on the display unit from the recorded video information; and an estimation unit that estimates the occurrence of a predetermined incident to the vehicle based on the detection result of a sound sensor that acquires the sound inside the vehicle, when the time rate of change of at least one of the amplitude, frequency, and sound pressure of the sound inside the vehicle is greater than or equal to a predetermined rate of change threshold, wherein when the occurrence of an incident is estimated, the display control unit adds marker information related to the estimated occurrence time to the thumbnail image corresponding to the video containing information captured at the estimated time of the incident and displays it on the display unit.

[0009] In a vehicle information display device according to another aspect of this disclosure, an estimation unit estimates the occurrence of a predetermined incident to the vehicle based on in-vehicle sounds acquired by a sound sensor. When the occurrence of an incident is estimated, a display control unit adds marker information related to the estimated occurrence time to a thumbnail image corresponding to a video containing information captured at the estimated occurrence time of the incident, and the thumbnail image is displayed on the display unit. Vehicle occupants can more easily visually distinguish the thumbnail image with the marker information related to the estimated occurrence time from a thumbnail image without the marker information. This makes it easier for vehicle occupants to identify the portion of the recorded video information in which the incident that occurred to the vehicle was captured.

[0010] In one embodiment, the display control unit may display the video on the display unit so that playback starts from the estimated time of the incident when a selection operation is performed by a vehicle occupant to select a thumbnail image to which marker information has been added. In this case, the vehicle occupant can easily play the video from the estimated time of the incident by viewing the thumbnail image to which marker information has been added and performing a selection operation on the thumbnail image.

[0011] In one embodiment, when the occurrence of multiple incidents is estimated, the display control unit may, when it starts playing video from the estimated time of an incident, skip to the next estimated time of the multiple incidents and play the video in response to a predetermined forward operation performed by the vehicle occupant, and skip to the previous estimated time of the multiple incidents and play the video in response to a predetermined backward operation performed by the vehicle occupant. In this case, the vehicle occupant can easily play video from any of the estimated time of the multiple incidents by performing a forward or backward operation. [Effects of the Invention]

[0012] According to some aspects of the present disclosure, the vehicle information display device makes it easier to identify the portion of the recorded video information in which an incident that occurred to the vehicle was captured. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram showing a vehicle information display device according to one embodiment. [Figure 2] This figure shows an example of a vehicle's display unit and thumbnail image. [Figure 3] (a) is a diagram showing an example of recorded video. (b) is a diagram showing an example of in-vehicle sound data corresponding to the recorded video. (c) is a diagram showing an example of the estimated time of an incident estimated based on the in-vehicle sound. [Figure 4] This figure shows an example of a thumbnail image with added marker information. [Figure 5] This figure shows an example of skipping video playback. [Figure 6] Figure 1 is a flowchart showing an example of ECU processing. [Figure 7] Figure 6 is a flowchart showing an example of the estimation process. [Figure 8](a) is a diagram showing an example of a recorded video. (b) is a diagram showing an example of impact data corresponding to the recorded video. (c) is a diagram showing an example of the estimated occurrence time of an incident estimated based on the impact data. [Figure 9] It is a flowchart showing another example of the estimation process of FIG. 6.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0015] FIG. 1 is a block diagram showing an information display device for a vehicle according to an embodiment. As shown in FIG. 1, the vehicle information display device 100 is mounted on a vehicle such as a passenger car. The vehicle information display device 100 is a device that records information of an image of a drive recorder camera 1a (in-vehicle camera) that captures the outside of the vehicle and displays the image on a display 20 (display unit) of the vehicle. Note that some functions of the vehicle information display device 100 may be executed on a server capable of communicating with the vehicle.

[0016] [Configuration of Vehicle Information Display Device] Hereinafter, the configuration of the vehicle information display device 100 according to the present embodiment will be described with reference to FIG. 1. As shown in FIG. 1, the vehicle information display device 100 includes an ECU [Electronic Control Unit] 10.

[0017] The ECU 10 is an electronic control unit having a CPU [Central Processing Unit] and a storage unit. The storage unit is composed of, for example, a ROM [Read Only Memory], a RAM [Random Access Memory], an EEPROM [Electrically Erasable Programmable Read-Only Memory], etc. In the ECU 10, various functions are realized by, for example, the CPU executing a program stored in the storage unit. The ECU 10 may be composed of a plurality of electronic units.

[0018] The ECU 10 is connected to an external sensor 1, an internal sensor 2, an HMI [Human Machine Interface] 3, a retractor 4, and an airbag ECU 5.

[0019] The external sensor 1 includes, for example, a drive recorder camera 1a that images the outside of the vehicle and a sonar sensor 1b.

[0020] The drive recorder camera 1a is a camera for imaging the outside of the vehicle. The drive recorder camera 1a is provided, for example, inside the front windshield of the vehicle or the like, and images the video in front of the vehicle. The drive recorder camera 1a intermittently or continuously records the video during parking or driving, and is used to record the situation in case of an accident or trouble. The drive recorder camera 1a may image the video behind the vehicle. The drive recorder camera 1a may image the video on the side of the vehicle. The drive recorder camera 1a may continuously image the video around the vehicle by rotating around an axis along the vertical direction of the vehicle. The drive recorder camera 1a transmits the information of the imaged video to the ECU 10.

[0021] The sonar sensor 1b is an ultrasonic sensor for detecting the approach of the vehicle to obstacles around the vehicle. The sonar sensor 1b is provided, for example, on the bumper of the vehicle or the like. The sonar sensor 1b emits ultrasonic waves and detects obstacles by receiving the reflected waves. The sonar sensor 1b transmits the detected obstacle information to the ECU 10. Note that the sonar sensor 1b is not essential.

[0022] The internal sensor 2 includes, for example, a vehicle speed sensor 2a, an acceleration sensor 2b, and a microphone 2c (sound sensor).

[0023] The vehicle speed sensor 2a is a detector for detecting the speed of the vehicle. The vehicle speed sensor 2a transmits the detected vehicle speed information to the ECU 10.

[0024] The acceleration sensor 2b is a detector that detects the acceleration of the vehicle. The acceleration sensor 2b may detect acceleration in the longitudinal direction and acceleration in the lateral direction of the vehicle. When an external impact is applied to the vehicle, the acceleration sensor 2b acquires acceleration information as impact data. The acceleration sensor 2b transmits the detected acceleration information to the ECU 10.

[0025] Microphone 2c is a sound sensor that acquires sounds inside the vehicle. Microphone 2c is installed, for example, on the ceiling or dashboard inside the vehicle. Microphone 2c acquires the voices of the occupants and sounds heard inside the vehicle. In-vehicle sounds are not limited to sounds that originate inside the vehicle, but may also include sounds that originate outside the vehicle but are heard inside the vehicle. Microphone 2c transmits the acquired recorded data of in-vehicle sounds to the ECU 10.

[0026] HMI3 is an interface for inputting and outputting information between ECU10 and the occupant. HMI3 includes, for example, a display 20 and speakers located inside the vehicle cabin. HMI3 outputs images from the display and audio from the speakers in response to control signals from ECU10. The display 20 may also function as a touch panel.

[0027] The display 20 may be a center display, a navigation display, or a HUD (Head-Up Display). The HUD presents information to the occupants by projecting images onto the vehicle's windshield.

[0028] HMI3 accepts selection operations from the occupant to select thumbnail images. This selection operation may involve, for example, touching a thumbnail image displayed on a touch panel.

[0029] The retractor 4 is a device for retracting the vehicle's seat belt. Normally, the retractor 4 retracts the seat belt with a predetermined force to achieve the appropriate tension. The retractor 4 may also control the operation of a mechanism that locks the seat belt extension in response to a signal from the retractor control unit 13. Note that the retractor 4 is not mandatory.

[0030] The airbag ECU 5 controls the operation of the airbag based on acceleration information detected by the acceleration sensor 2b. The airbag ECU 5 is an electronic control unit for controlling the vehicle's airbags. The airbag ECU 5 controls the operation of the airbag based on acceleration information detected by the acceleration sensor 2b. The airbag ECU 5 may also transmit a signal to the ECU 10 to deploy the airbag. Note that the airbag ECU 5 is not mandatory.

[0031] Next, the functional configuration of the ECU 10 will be described. The ECU 10 includes a recognition unit 11, a storage unit 12, a retractor control unit 13, an estimation unit 14, and a display control unit 15.

[0032] The recognition unit 11 recognizes the video captured by the drive recorder camera 1a. Based on the video information transmitted from the drive recorder camera 1a, the recognition unit 11 recognizes the recorded video and the time the recorded video was captured, etc., to be displayed on the display 20.

[0033] The recognition unit 11 generates thumbnail images based on the recognized recorded video and the time of capture. The thumbnail images are used to select and play back the video to be displayed on the display 20. The recognition unit 11 may generate one thumbnail image for each predetermined time interval of the recorded video. The recognition unit 11 may associate one thumbnail image with the playback start time of the corresponding recorded video. The recognition unit 11 may generate multiple thumbnail images for recorded video captured in the past period up to the present.

[0034] The recognition unit 11 may recognize the vehicle's approach to an obstacle around the vehicle based on obstacle information from the sonar sensor 1b. The recognition unit 11 may recognize the vehicle speed based on vehicle speed information from the vehicle speed sensor 2a. The recognition unit 11 may recognize the vehicle's acceleration based on acceleration information from the acceleration sensor 2b.

[0035] The recognition unit 11 may recognize (acquire) the in-vehicle sound based on the recorded in-vehicle sound data from the microphone 2c. The recognition unit 11 may also extract an envelope representing the change in amplitude of the in-vehicle sound from the recorded in-vehicle sound data from the microphone 2c as recording data for voice analysis described later.

[0036] The storage unit 12 stores information about the video recognized by the recognition unit 11 and information about the thumbnail images generated by the recognition unit 11. The storage unit 12, for example, stores the recorded video as time-series image data and stores the recorded audio data in a time-series correspondence with the image data. The storage unit 12 may also store the playback start time of the recorded video corresponding to each thumbnail image in association with that thumbnail image. The acquisition and storage of in-vehicle sound may be performed by the estimation unit 14, which will be described later.

[0037] The retractor control unit 13 controls the retractor 4 to activate the ELR (Emergency Locking Retractor) function based on acceleration information detected by the acceleration sensor 2b. For example, if the detected acceleration exceeds a predetermined acceleration threshold, the retractor control unit 13 locks the seat belt extension by the retractor 4. The acceleration threshold is the threshold for acceleration equivalent to that during sudden braking or a collision. Note that the retractor 4 is not mandatory.

[0038] The estimation unit 14 estimates the occurrence of a predetermined incident to the vehicle based on the detection results of the on-board external sensor 1 or internal sensor 2. A predetermined incident refers to an event inside or outside the vehicle that requires confirmation by playing back recorded video footage captured in the past period up to the present. A predetermined incident here includes abnormalities that appear in recorded data, such as a collision between the vehicle and another vehicle or obstacle, or the occurrence of loud noises such as loud shouts from occupants or collision sounds.

[0039] The display control unit 15 displays thumbnail images 21 on the display 20 for selecting which video to display on the display 20 from the recorded video information.

[0040] Figure 2 shows an example of a vehicle display unit and thumbnail images. As shown in Figure 2, the display 20 is, for example, a touch panel center display located on the vehicle's dashboard. Multiple thumbnail images 21 are displayed on the display 20. The thumbnail images 21 are representative still images extracted at predetermined time intervals from the video captured by the drive recorder camera 1a.

[0041] At the top of the display 20 is a clock display showing the current time, with "00:00" displayed in the upper right corner. This indicates the video playback time and is used to display the elapsed time of the video being played. For example, in the center of the display 20, multiple thumbnail images 21 are displayed in a grid arrangement. Each thumbnail image 21 displays the date and time of capture in a format such as "2022 / 05 / 24 10:34 AM". At the top of the thumbnail images 21, tabs such as "Front", "Rear", and "Around" are displayed. By touching these tabs, the video of the front, rear, and surroundings of the vehicle may be switched.

[0042] The estimation of incident occurrence will be explained in more detail. The estimation unit 14 estimates the occurrence of an incident, for example, based on the recorded data acquired by microphone 2c, if the time rate of change of at least one of the amplitude, frequency, and sound pressure of the in-vehicle sound (time rate of change of the recorded data) is greater than or equal to a predetermined rate of change threshold. The rate of change threshold is the threshold of the time rate of change of the recorded data used to determine whether or not an incident is estimated to have occurred.

[0043] The estimation unit 14 here calculates, for example, the rate of change of sound pressure over time. The estimation unit 14 may, for example, calculate the rate of change of sound pressure over time by applying a predetermined noise filter to the sound pressure data as one of the recording data, and then using the following difference equation to calculate the difference between adjacent samples in a time series and differentiate it. x'(t) = {x(t+Δt) / x(t)} × Δt However, t represents time, Δt represents the sample time of the recorded data, x(t) represents the filtered recorded data at time t, and x'(t) represents the rate of change of the recorded data at time t.

[0044] The estimation unit 14 estimates that an incident occurred at time t when the rate of change of the recorded data exceeds a predetermined rate of change threshold, and may also estimate this time t as the estimated time of incident occurrence. The estimated time of incident occurrence is the time at which the incident is estimated to have occurred. The estimated time of incident occurrence includes a time corresponding to the time on the clock and the relative elapsed time from the start time of playback of the recorded video corresponding to the thumbnail image containing that time.

[0045] The estimation unit 14 may also estimate the occurrence of an incident if the envelope value of the recorded data 23 is equal to or greater than a predetermined amplitude threshold. The estimation unit 14 may also estimate the time t at which the envelope value of the recorded data 23 becomes equal to or greater than a predetermined amplitude threshold as the estimated time of incident occurrence.

[0046] Figure 3(a) shows an example of recorded video. Figure 3(b) shows an example of in-vehicle sound data corresponding to the recorded video. Figure 3(c) shows an example of the estimated time of incident occurrence estimated based on in-vehicle sound.

[0047] Figure 3(a) shows the recorded video 22, which is footage captured by the drive recorder camera 1a. The recorded video 22 consists of a series of still images recorded along the time axis, and Figure 3(a) shows footage from a specific time period.

[0048] Figure 3(b) shows the recorded sound data 23 from inside the vehicle. The recorded data 23 was acquired by microphone 2c and is mapped to the time of the recorded video 22 along the time axis. The recorded data 23 shows instantaneous changes in sound pressure along the time axis. The envelope of the recorded data 23 may be a curve connecting the peak values ​​of the recorded data 23.

[0049] Figure 3(b) shows the sound pressure maxima 23a, 23b, and 23c. The sound pressure maxima 23a, 23b, and 23c correspond to the parts where the rate of change over time of the recorded data 23 exceeds a predetermined rate of change threshold. The sound pressure maxima 23a, 23b, and 23c represent the parts where the sound pressure of the recorded data 23 is higher than that of the recorded data 23 under normal conditions, and where the sound pressure can reach a maximum value. In the example in Figure 3(b), the sound pressure maxima 23a shows the peak of the largest sound pressure, the sound pressure maxima 23b shows the peak of the second largest sound pressure, and the sound pressure maxima 23c shows the peak of the third largest sound pressure. The estimation unit 14 estimates the occurrence of an incident in each of the sound pressure maxima 23a, 23b, and 23c.

[0050] Figure 3(c) shows the elapsed time 24. The elapsed time 24 is the relative elapsed time with the start time of a specific time period in the recorded video 22 as the origin (00m00s). The elapsed time 24 may also be the relative elapsed time with the start time of playback of the recorded video 22 when, for example, a thumbnail image corresponding to the recorded video 22 is selected as the origin. The end point of the elapsed time 24 (22m01s) may be the end time of the recorded video 22.

[0051] Figure 3(c) shows the estimated incident occurrence times 24a, 24b, and 24c. The estimated occurrence times 24a, 24b, and 24c correspond to the sound pressure peaks 23a, 23b, and 23c, respectively. The estimated occurrence times 24a, 24b, and 24c represent the times when the occurrence of the incident was estimated in each of the sound pressure peaks 23a, 23b, and 23c. Estimated occurrence time 24a indicates the estimated occurrence time of the incident that was estimated to have occurred in the sound pressure peak 23a (18m13s). Estimated occurrence time 24b indicates the estimated occurrence time of the incident that was estimated to have occurred in the sound pressure peak 23b (2m14s). Estimated occurrence time 24c indicates the estimated occurrence time of the incident that was estimated to have occurred in the sound pressure peak 23c (5m02s).

[0052] When an incident is estimated to have occurred, the display control unit 15 adds marker information related to the estimated time of occurrence to the thumbnail image 21 corresponding to the video containing information captured at the estimated time of the incident, and displays it on the display 20.

[0053] Figure 4 shows an example of a thumbnail image with added marker information. Figure 4 shows a thumbnail image 21 and marker information 25. Thumbnail image 21 is one of several thumbnail images 21 displayed on the display 20 that includes the estimated time of the incident.

[0054] The marker information 25 is visual information regarding the estimated time of occurrence. For example, the marker information 25 can be a string of characters indicating the estimated time of occurrence of the incident. The marker information 25 may be displayed superimposed on the thumbnail image 21. In Figure 4, the marker information 25 is displayed superimposed on the thumbnail image 21 in the format "1) 18m13s -> 2) 2m47s -> 3) 5m02s" at the top of the thumbnail image 21. Note that "->" means a right-pointing arrow.

[0055] The marker information 25 may include visual information regarding the scale of an incident among multiple incidents. In the example in Figure 4, the marker information 25 lists the estimated occurrence times of multiple incidents in order of the magnitude of the sound pressure peaks at the sound pressure maximums 23a, 23b, and 23c, as an example of the scale of an incident.

[0056] The display control unit 15 may display the video on the display 20 so that playback starts from the estimated time of the incident if a selection operation is performed by a vehicle occupant to select a thumbnail image 21 to which the marker information 25 has been added. The display control unit 15 may recognize that a selection operation has been performed, for example, if a vehicle occupant taps the thumbnail image 21 on the display 20 that displays the thumbnail image 21 to which the marker information 25 has been added.

[0057] When the display control unit 15 recognizes that a selection operation has been performed, it may read the recorded video 22 corresponding to the thumbnail image 21 with the marker information 25 attached from the storage unit 12 and start playing the recorded video 22 from the estimated time of the incident. In this case, if there are multiple estimated times of incident, the display control unit 15 may start playing the recorded video 22 from the estimated time of the largest incident 24a.

[0058] When the display control unit 15 estimates the occurrence of multiple incidents, if it starts playback of the recorded video 22 from the estimated incident occurrence time 24a, it may skip to another estimated occurrence time 24b or estimated occurrence time 24c and play back the recorded video 22.

[0059] The display control unit 15 may skip to the next of the multiple estimated occurrence times and play the video in response to a predetermined forward operation performed by the vehicle occupant. The display control unit 15 may also skip to the previous of the multiple estimated occurrence times and play the video in response to a predetermined return operation performed by the vehicle occupant.

[0060] Figure 5 shows an example of skip playback of a video. In Figure 5, the playback screen of the recorded video 22 is displayed on the display 20. The predetermined forward and backward operations may be operations performed by the occupant on a pre-set selection area 26 on the display 20.

[0061] A predetermined scrolling operation can be, for example, an operation in which the occupant taps the right-hand selection area 26a in the display area of ​​the display 20. Alternatively, a predetermined scrolling operation can be an operation in which the occupant swipes from the left side of the selection area 26a toward the right-hand selection area 26a in the display area of ​​the display 20.

[0062] The selected area 26a may include the right edge of the display 20 and, when the time from the start to the end of the elapsed time 24 corresponds to the entire width of the display 20, it may be the area from the right edge of the display 20 up to the horizontal position of the display 20 corresponding to the estimated occurrence time 24a closest to the end time. The selected area 26a is not limited to this example and may be an area of ​​a fixed width including the right edge of the display 20.

[0063] In the example in Figure 5, if the estimated occurrence time 24b is the playback start time and a predetermined forward operation is performed, playback can begin from the estimated occurrence time 24c. If the estimated occurrence time 24c is the playback start time and a predetermined forward operation is performed, playback can begin from the estimated occurrence time 24a.

[0064] A predetermined return operation can be, for example, an operation in which the occupant taps the left selection area 26b in the display area of ​​the display 20. Alternatively, a predetermined return operation may be an operation in which the occupant swipes from the right side of the selection area 26a toward the left selection area 26b in the display area of ​​the display 20.

[0065] The selected area 26b may include the left edge of the display 20 and, when the time from the start to the end of the elapsed time 24 corresponds to the entire width of the display 20, it may be the area from the left edge of the display 20 up to the horizontal position of the display 20 corresponding to the estimated occurrence time 24b closest to the start time. The selected area 26b is not limited to this example and may be an area of ​​a fixed width including the left edge of the display 20.

[0066] In the example in Figure 5, if the estimated occurrence time 24a is the playback start time and a predetermined return operation is performed, playback can begin from the estimated occurrence time 24c. If the estimated occurrence time 24c is the playback start time and a predetermined return operation is performed, playback can begin from the estimated occurrence time 24b.

[0067] Multiple selection areas 26 may be set on at least one of the right and left halves of the display 20. For example, selection area 26c is the area from the horizontal position of the display 20 corresponding to the estimated occurrence time 24b closest to the start time, to the horizontal position of the display 20 corresponding to the second closest estimated occurrence time 24c, when the time from the start to the end of the elapsed time 24 corresponds to the entire width of the display 20.

[0068] If the estimated occurrence time 24b is the playback start time, tapping the selected area 26c is treated as a predetermined forward operation, and playback can begin from the estimated occurrence time 24c. If the estimated occurrence time 24a is the playback start time, tapping the selected area 26c is treated as a predetermined back operation, and playback can begin from the estimated occurrence time 24c.

[0069] [Operation of the vehicle's information display device] Next, the operation of the vehicle information display device 100 will be explained with reference to the drawings. Figure 6 is a flowchart showing an example of the processing of the ECU in Figure 1. The processing in Figure 6 is executed, for example, when power is supplied to the ECU 10 and recorded video has been captured during a predetermined period in the past up to the present.

[0070] As shown in Figure 6, the ECU 10 of the vehicle information display device 100 performs the following steps: Step S01, the recognition unit 11 acquires and records video information. The recognition unit 11 acquires video information transmitted from the drive recorder camera 1a and recognizes the recorded video and the time the recorded video was captured. The recognition unit 11 stores the recorded video as time-series image data in the storage unit 12. The recognition unit 11 also stores the recorded audio data in the storage unit 12, corresponding it to the image data in time.

[0071] In step S02, the ECU 10 displays thumbnail images on the vehicle's display unit using the recognition unit 11 and the display control unit 15. The recognition unit 11 generates thumbnail images based on the recognized recorded video and the time of capture. The display control unit 15 displays thumbnail images 21 on the display 20 for selecting the video to be displayed on the display 20 from the recorded video information.

[0072] In step S03, the ECU 10 uses the recognition unit 11 and estimation unit 14 to estimate the occurrence of a predetermined incident. The recognition unit 11 and estimation unit 14 estimate the occurrence of an incident based on the detection results of the in-vehicle external sensor 1 or internal sensor 2 by performing the processing shown in Figure 7, for example.

[0073] Figure 7 is a flowchart showing an example of the estimation process in Figure 6. The process in Figure 7 is executed for each sample time of the audio data, for example, when video footage has been captured during a predetermined period in the past up to the present.

[0074] In step S11, the ECU 10 acquires the in-vehicle sound using the recognition unit 11. The recognition unit 11 acquires the in-vehicle sound of the vehicle based on the recorded in-vehicle sound data transmitted from the microphone 2c. Alternatively, the acquisition of the in-vehicle sound may be performed by the estimation unit 14.

[0075] In step S12, the ECU 10 calculates the rate of change of the in-vehicle sound using the estimation unit 14. The estimation unit 14 calculates the rate of change of at least one of the amplitude, frequency, and sound pressure of the in-vehicle sound based on the recording data acquired by the microphone 2c, for example. The estimation unit 14 may also calculate the rate of change of the sound pressure of the in-vehicle sound using the difference equation described above.

[0076] In step S13, the ECU 10 uses the estimation unit 14 to determine whether the rate of change of the in-vehicle sound exceeds a predetermined rate of change threshold. If the ECU 10 determines that the rate of change of the in-vehicle sound exceeds the predetermined rate of change threshold (step S13: YES), it proceeds to step S14. If the ECU 10 determines that the rate of change of the in-vehicle sound is less than the predetermined rate of change threshold (step S13: NO), it proceeds to step S16.

[0077] In step S14, the ECU 10 estimates that an incident has occurred using the estimation unit 14. The estimation unit 14 estimates the occurrence of an incident based on its determination that the rate of change of the in-vehicle sound exceeds a predetermined rate of change threshold.

[0078] In step S15, the ECU 10 uses the estimation unit 14 to identify the sound pressure maximum as the estimated time of incident occurrence. The estimation unit 14 identifies the portion of the recorded data at the sample time of the current process as the sound pressure maximum, and estimates the sample time of the current process as the estimated time of incident occurrence. After that, the process shown in Figure 7 is completed, and the process returns to S04 in Figure 6.

[0079] In step S16, the ECU 10, using the estimation unit 14, estimates that no incident has occurred. The estimation unit 14 determines that the rate of change of the in-vehicle noise is less than a predetermined rate of change threshold, and therefore does not estimate the occurrence of an incident. After that, the process shown in Figure 7 is terminated, and the process returns to S04 in Figure 6.

[0080] Returning to Figure 6, in step S04, the display control unit 15 determines whether or not an incident has been estimated to have occurred. If the ECU 10 estimates that an incident has occurred (step S04: YES), it proceeds to step S05. If the ECU 10 does not estimate that an incident has occurred (step S04: NO), it terminates the process shown in Figure 6.

[0081] In step S05, the ECU 10, using the display control unit 15, adds marker information to the thumbnail image corresponding to the video containing information captured at the estimated time of the incident. For example, the display control unit 15 adds a string indicating the estimated time of the incident as marker information 25 to the thumbnail image 21 corresponding to the video containing information captured at the estimated time of the incident.

[0082] In step S06, the ECU 10 causes the display control unit 15 to display a thumbnail image with added marker information on the display unit. The display control unit 15 then displays the thumbnail image 21 with added marker information 25 on the display 20. After that, the process shown in Figure 6 is terminated.

[0083] According to the vehicle information display device 100 described above, the estimation unit estimates the occurrence of a predetermined incident to the vehicle based on the detection results of an on-board external or internal sensor. When the occurrence of an incident is estimated, the display control unit adds marker information to a thumbnail image corresponding to the video containing information captured at the estimated time of the incident, and the thumbnail image is displayed on the display unit. This makes it easier for the vehicle occupant to visually distinguish the thumbnail image with the marker information from the thumbnail image without the marker information. As a result, it becomes easier for the vehicle occupant to identify the portion of the recorded video information in which the incident that occurred to the vehicle was captured.

[0084] In the information display device 100, the internal sensors include a sound sensor that acquires the sound inside the vehicle, and the estimation unit may estimate the occurrence of an incident when the time rate of change of at least one of the amplitude, frequency, and sound pressure of the sound inside the vehicle is greater than or equal to a predetermined rate of change threshold. In this case, the estimation unit can estimate the occurrence of a predetermined incident to the vehicle based on the sound inside the vehicle acquired by the sound sensor.

[0085] In the information display device 100, the display control unit may display the video on the display unit so that playback starts from the estimated time of the incident when a selection operation is performed by a vehicle occupant to select a thumbnail image to which marker information has been added. In this case, the vehicle occupant can easily play back the video from the estimated time of the incident by viewing the thumbnail image to which marker information has been added and performing a selection operation on the thumbnail image.

[0086] In the information display device 100, when the occurrence of multiple incidents is estimated, the display control unit may, when playback of the video starts from the estimated time of an incident, skip to the next estimated time of the multiple incidents in response to a predetermined forward operation performed by the vehicle occupant, and skip to the previous estimated time of the multiple incidents in response to a predetermined backward operation performed by the vehicle occupant. In this case, the vehicle occupant can easily play the video from any of the estimated time of the multiple incidents by performing a forward or backward operation.

[0087] While embodiments of this disclosure have been described above, this disclosure is not limited to the embodiments described above. This disclosure can be implemented in various forms, including the embodiments described above, with various modifications and improvements based on the knowledge of those skilled in the art.

[0088] In the above embodiment, an anomaly appearing in the recorded data was given as an example of a predetermined incident, but the invention is not limited to this example. The predetermined incident may also be an anomaly appearing in the acceleration data or impact data. Anomalies appearing in the acceleration data or impact data may include, for example, sudden acceleration, sudden braking, or sudden steering, to the extent that they correspond to an emergency avoidance situation. Anomalies appearing in the acceleration data or impact data may also include the activation of the seat belt's ELR or the deployment of an airbag.

[0089] The estimation unit 14 may estimate the occurrence of a predetermined incident to the vehicle based on acceleration information detected by the acceleration sensor 2b as impact data. The estimation unit 14 may also estimate the occurrence of a predetermined incident as impact data, for example, when the retractor 4 is controlled by the retractor control unit 13 to activate the ELR function. The estimation unit 14 may also estimate the occurrence of a predetermined incident as impact data, for example, in response to a signal sent from the airbag ECU 5 to the ECU 10 to deploy the airbag.

[0090] Figure 8(a) shows an example of recorded video. Figure 8(b) shows an example of impact data corresponding to the recorded video. Figure 8(c) shows an example of the estimated time of incident occurrence estimated based on the impact data.

[0091] As shown in Figure 8(b), the estimation unit 14 may estimate the occurrence of a predetermined incident to the vehicle based on impact data 27 instead of the audio data 23 in Figure 3(b). The impact peaks 27a, 27b, and 27c are the portions of the impact data 27 in which the occurrence of the incident is estimated by the estimation unit 14, and correspond to the portions where the impact is at its maximum. The estimated occurrence times 24a, 24b, and 24c correspond to the impact peaks 27a, 27b, and 27c, respectively.

[0092] Figure 9 is a flowchart showing another example of the estimation process in Figure 6 (a modified version of Figure 7). In the flowchart of Figure 9, the processes in steps S21 to S23 and S25 differ from the processes in steps S11 to S13 and S15.

[0093] In step S21, the ECU 10 acquires impact data using the recognition unit 11. The recognition unit 11 acquires impact data based on acceleration information detected by the acceleration sensor 2b. In step S22, the ECU 10 calculates the rate of change of the impact data using the estimation unit 14. The estimation unit 14 may calculate the rate of change of the impact data using the difference equation described above. The estimation unit 14 may calculate that the rate of change of the impact data exceeds a predetermined value when the retractor 4 is controlled by the retractor control unit 13 to activate the ELR function, or when it receives a signal from the airbag ECU 5 to deploy the airbag.

[0094] In step S23, if the ECU 10 determines that the impact data exceeds a predetermined rate of change threshold (step S23: YES), it performs the same processing as in step S14 as in step S24, and in step S25, the estimation unit 14 identifies the impact maximum as the estimated time of incident occurrence. The estimation unit 14 identifies the portion of the impact data at the sample time of the current processing as the impact maximum, and estimates the sample time of the current processing as the estimated time of incident occurrence. After that, the processing in Figure 9 is terminated, and the process returns to S04 in Figure 6.

[0095] Alternatively, a predetermined incident may be an anomaly appearing in the obstacle information detected by the sonar sensor 1b. As an anomaly appearing in the obstacle information, a situation in which obstacle information is detected by the sonar sensor 1b when the vehicle is moving slowly is not considered an anomaly, while a situation in which obstacle information is detected by the sonar sensor 1b when the vehicle is not moving slowly is considered an anomaly.

[0096] The estimation unit 14 may estimate the occurrence of a predetermined incident as an anomaly appearing in the obstacle information, for example, when the vehicle speed is above a predetermined vehicle speed threshold and an obstacle is detected by the sonar sensor 1b. The predetermined vehicle speed threshold is a vehicle speed threshold that corresponds to the point at which the vehicle is not slowing down to a certain extent or more, so that if an obstacle is detected by the sonar sensor 1b while the vehicle is traveling at that speed, there is a certain or greater possibility that the vehicle will not be able to stop without hitting the obstacle.

[0097] In the above embodiment, the marker information 25 is exemplified as a string of characters indicating the estimated time of occurrence of an incident as visual information relating to the estimated time of occurrence, but it is not limited to marker information relating to the estimated time of occurrence. The marker information may be a simple marker image such as a graphic. Also, although the marker information is displayed superimposed on the top of the thumbnail image 21, it may be displayed at the corners of the thumbnail image 21, or it may be displayed in the shape of a frame surrounding the thumbnail image 21.

[0098] In the above embodiment, the selection operation for selecting a thumbnail image was exemplified by touching the thumbnail image displayed on the touch panel, but the example is not limited to this. For example, the selection operation for selecting a thumbnail image may be the operation of pressing an operation button provided on the vehicle's steering wheel.

[0099] In the above embodiment, when the occurrence of multiple incidents was estimated, the display control unit 15 started playback of the recorded video 22 from the estimated occurrence time of one incident, then skipped to the other estimated occurrence times and played back the recorded video 22. However, this example is not essential.

[0100] In the above embodiment, the recognition unit 11 acquired the in-vehicle sound based on the recorded data of the in-vehicle sound transmitted from the microphone 2c. However, if the drive recorder camera 1a is equipped with a microphone, the in-vehicle sound may be acquired using the microphone on the drive recorder camera 1a. The sound sensor may also be located outside the vehicle. [Explanation of symbols]

[0101] 1...External sensor, 1a...Drive recorder camera (in-vehicle camera), 2...Internal sensor, 3...HMI (display unit), 2c...Microphone (sound sensor), 14...Estimation unit, 15...Display control unit, 20...Display (display unit), 21...Thumbnail image, 24a, 24b, 24c...Estimated occurrence time, 25...Landmark information, 100...Vehicle information display device.

Claims

1. A vehicle information display device that records video information from an onboard camera that captures images of the outside of a vehicle and displays the video on the vehicle's display unit, A display control unit that displays thumbnail images on the vehicle's display unit for selecting the video to be displayed on the display unit from the recorded video information, The system includes an estimation unit that estimates the occurrence of a predetermined incident to the vehicle based on the detection results of an external or internal sensor on board the vehicle, The display control unit, when the occurrence of the incident is estimated, adds marker information to the thumbnail image corresponding to the video containing information captured at the estimated time of the incident, and displays it on the display unit, and displays it on the display unit.

2. The internal sensor includes a sound sensor that acquires the sound inside the vehicle. The vehicle information display device according to claim 1, wherein the estimation unit estimates the occurrence of the incident when the time rate of change of at least one of the amplitude, frequency, and sound pressure of the in-vehicle sound is greater than or equal to a predetermined rate of change threshold.

3. A vehicle information display device that records video information from an onboard camera that captures images of the outside of a vehicle and displays the video on the vehicle's display unit, A display control unit that displays thumbnail images on the vehicle's display unit for selecting the video to be displayed on the display unit from the recorded video information, The system includes an estimation unit that estimates the occurrence of a predetermined incident to the vehicle based on the detection results of a sound sensor that acquires the in-vehicle sound, when the time rate of change of at least one of the amplitude, frequency, and sound pressure of the in-vehicle sound is greater than or equal to a predetermined rate of change threshold. The display control unit, when the occurrence of the incident is estimated, adds marker information related to the estimated time of occurrence to the thumbnail image corresponding to the video containing information captured at the estimated time of occurrence of the incident, and displays it on the display unit, a vehicle information display device.

4. The vehicle information display device according to claim 1 or 3, wherein the display control unit displays the video on the display unit so that playback starts from the estimated time of the incident when a selection operation to select the thumbnail image to which the marker information has been added is performed by an occupant of the vehicle.

5. The display control unit, When multiple incidents are estimated to have occurred, and playback of the video begins from the estimated time of occurrence of the incident, In response to a predetermined forwarding operation being performed by the occupant of the vehicle, the video is played back by skipping to the next of the multiple estimated occurrence times. The vehicle information display device according to claim 1 or 3, which skips to the previous estimated occurrence time among a plurality of estimated occurrence times and plays the video when a predetermined return operation is performed by the occupant of the vehicle.

Images