In-vehicle device and recording method

Abstract

To record information related to a remembered fact when a driver remembers something while driving.SOLUTION: An on-vehicle device 100 comprises: a visual line detection unit 120 for detecting a visual line direction of a driver of a vehicle; an estimation unit 130 for estimating a specific object by determining whether the driver is looking at the specific object in the visual line direction based upon at least a speed of a visual line movement of the driver; an information processing unit 160 that records information related to the specific object as memorandum information; and an output unit 190 that outputs the memorandum information recorded during driving when driving of the vehicle is completed.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an in-vehicle device and a recording method.

Background Art

[0002] Techniques for extracting matters of concern from the behavior of vehicle occupants and presenting information regarding the matters of concern have been proposed. Patent Document 1 discloses a technique for detecting an abnormal driving pattern and then determining whether the driver shows interest based on behavioral changes such as the movement of the driver's line of sight, and presenting information regarding facilities that the driver shows interest in.

[0003] Patent Document 2 discloses a technique for extracting the objects of interest of bus passengers and presenting related information. Here, the interests of the passengers are extracted from dynamic information such as line of sight and conversation, and customer information pre-registered in the passengers' accounts.

[0004] With the techniques according to Patent Documents 1 and 2, it is possible to present information desired by a driver or the like triggered by a behavioral change. On the other hand, it is difficult to record something that a driver suddenly remembers while the vehicle is in motion or to extract matters of concern that the driver himself / herself is not aware of using these techniques.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] As mentioned above, if the driver remembers something while driving, they cannot take their hands off the wheel while driving. There was a problem in that it was difficult to record information related to the facts that were recalled. Furthermore, while driving, drivers must perform vehicle maneuvers such as maneuvering at traffic lights and on curves, avoiding close proximity to other vehicles, and selecting their driving route. Because there are many things to think about for such reasons, by the time the driving is over, I have forgotten what I remembered. There was a problem with this. So, if the driver remembers something while driving, The present invention provides an in-vehicle device and recording method capable of recording factual information. [Means for solving the problem]

[0007] The present invention provides an in-vehicle device comprising: a gaze detection unit that detects the direction of the vehicle driver's gaze; an estimation unit that determines whether the driver is looking at a specific object in the direction of their gaze, based at least on the speed of the driver's gaze movement, and estimates the object; an information processing unit that records information related to the object as memo information; and an output unit that outputs the memo information recorded during driving when the vehicle driving is finished.

[0008] The present invention provides a recording method comprising: a gaze detection step for detecting the direction of a vehicle driver's gaze; an estimation step for determining whether the driver is looking at a specific object in the direction of their gaze, based at least on the speed of the driver's gaze movement, and estimating the object; an information processing step for recording information related to the object as memo information; and an output step for outputting the memo information recorded during driving when the vehicle driving is finished. [Effects of the Invention]

[0009] According to the present invention, when a driver remembers something while driving, the information related to the fact that was remembered... We can provide an in-vehicle device and recording method that can record information. [Brief explanation of the drawing]

[0010] [Figure 1] It is a block diagram showing the configuration of the in-vehicle device according to Embodiment 1. [Figure 2] It is a flowchart showing the operation of the in-vehicle device according to Embodiment 1. [Figure 3] It is a block diagram showing the configuration of the in-vehicle device according to Embodiment 2.

Mode for Carrying Out the Invention

[0011] (Embodiment 1) First, the configuration of the in-vehicle device 100 according to Embodiment 1 will be described using FIG. 1. The in-vehicle device 100 is, for example, a drive recorder. The in-vehicle device 100 includes an image acquisition unit 1 10, a gaze detection unit 120, an estimation unit 130, a storage unit 140, a determination unit 150, an information [[ID=XX]]processing unit 160, and an information presentation unit 170.

[0012] The image acquisition unit 110 acquires images of the driver of the vehicle and the outside of the vehicle from the camera 10. The image acquisition unit 110 may acquire an image obtained by simultaneously photographing the driver and the outside of the vehicle using an ultra-wide-angle lens (fisheye, 360 lens), etc., or may acquire a plurality of images photographed by separate cameras for the driver and the outside of the vehicle. Note that the camera 10 may be an infrared camera. The acquisition of an image of the outside of the vehicle in the image acquisition unit 110 is not necessarily an essential configuration and may be omitted.

[0013] The gaze detection unit 120 detects the gaze direction from the face image of the driver. The gaze direction is detected, for example, as one of the 9 quadrants of upper middle lower, left middle right. It is preferable that the gaze direction is detected in a more specific manner. Further, the gaze detection unit 120 measures the period during which the driver is staring in a certain direction. It should be noted that there seems to be an error in the original text where "XX" is used in line 24. It should probably be "processing unit 160" as per the context. The translation has been made accordingly.It may be detected, and the line-of-sight movement associated with the running of the vehicle may be detected. Also, the line-of-sight detection unit 120 may detect the line-of-sight movement due to looking at a moving object. The line-of-sight detection unit 120 outputs the detected line-of-sight direction to the estimation unit 130.

[0014] The line-of-sight detection unit 120 can detect the line of sight based on the positional relationship between the inner corner of the eye and the iris, and the positional relationship between the corneal reflection and the pupil. The line-of-sight detection unit 120 may detect the direction in which the driver is looking based on the orientation of the face and use it as the line-of-sight direction. As the line-of-sight detection technology, known technology may be used.

[0015] The estimation unit 130 estimates the object that the driver is looking at, at least based on the line-of-sight direction. For example, when it is detected that the driver's line-of-sight direction is slightly below the center in the left-right direction, the estimation unit 130 estimates that the object that the driver is looking at is the road or another vehicle running in front of the vehicle. The object that the driver is looking at may be estimated based on the line-of-sight direction and the situation around the vehicle. The situation around the vehicle is determined from an image of the outside of the vehicle acquired by the image acquisition unit 110, or map information acquired from a car navigation device (not shown) or the Internet via a communication unit (not shown). The estimation unit 130 outputs the estimation result to the determination unit 150.

[0016] Here, the estimation unit 130 first determines whether the driver is looking at the entire line-of-sight direction or looking at (gazing at) a specific object in the line-of-sight direction, from the movement of the driver's pupil, the change in the orientation of the face, the period during which the driver is looking in a certain direction, etc. For example, if the driver often looks at the center part in the left-right direction and the movement of the line of sight is fine, the estimation unit 130 determines that the driver is looking at the center part in the left-right direction and the movement of the line of sight is fine. ​​​​​​​​​​​It is assumed that the driver is looking at the entire direction of their line of sight. The speed and direction of the driver's eye movement, It determines whether there is a correlation with the vehicle's speed and whether the vehicle is looking at a specific object in its line of sight. It may be determined that the state is such. If the driver is looking at a specific object, the estimation unit 130 will The system estimates what the driver is looking at. The target is estimated to be a single facility, such as "○○ Post Office". It may be defined as a specific name, or it may be presumed to be a collective term for multiple stores, like "○○ Burger". It can be considered a genre, like "ramen shop." The scope is limited to facilities. It can also be rice paddies, trees, etc.

[0017] The estimation unit 130 uses image recognition technology to estimate the target based on the line of sight and the vehicle's external image. It may be determined. For example, if the driver is looking towards the upper left, and the image of the vehicle's exterior shows that If the image is recognized as a post office sign, the estimation unit 130 determines what the driver is looking at. The target can be presumed to be a post office. The estimation unit 130 recognizes roads, other vehicles, traffic lights, signs, etc. from images. It can be assumed that the object is consciously observed by the driver.

[0018] Furthermore, the estimation unit 130 estimates the object the driver is looking at based on the direction of gaze and map information. It may be fixed. For example, if the driver is looking slightly to the left in front, the estimation unit 130 will determine the current state of the vehicle. The map information indicates that the facility located to the left and slightly ahead of the driver's current position is a post office, and the driver It can be assumed that the object being viewed is a post office. The estimation unit 130 indicates the presence of signals, signs, etc. It may be possible to obtain this information from map data and estimate that it represents the object the driver is looking at.

[0019] Furthermore, the estimation unit 130 estimates the driver's gaze direction and vehicle driving information. The target may be estimated. For example, if the vehicle turns left after the driver looks directly to the left. The estimation unit 130 may estimate that the object the driver is looking at is the left side mirror.

[0020] Furthermore, the estimation unit 130, based on the direction of the driver's gaze and information indicating the location of the accident site, determines that the driver It can be assumed that the object being viewed is the accident scene. Information indicating the location of the accident scene is illustrated. It may also be obtained from the internet or other sources via a communication device.

[0021] The storage unit 140 is a storage device such as a hard disk, flash memory, or SD card. It is a storage medium. Furthermore, the storage unit 140 is a storage area for temporarily holding information. It may include volatile storage devices such as RAM (Random Access Memory). The storage unit 140 is The history 141 and memo information 142 are stored. The memory unit 140 stores the video footage captured by the camera 10. You may also memorize this further.

[0022] History 141 is a database that manages the history of the estimation results of the estimation unit 130. History 1 41 may only store items unrelated to vehicle operation. Also, history 141 is estimated The object that was detected and the time the driver saw the object may be stored in association. Furthermore, History 1 41 may remember the length of time the driver observed the object. Memo information 142 will be described later. This is information recorded by the information processing unit 160. Note that this includes history 141 and memo information 142. This data may be stored in a server outside the in-vehicle device 100.

[0023] The determination unit 150 determines if the object estimated by the estimation unit 130 is not related to the operation of the vehicle. In addition, the frequency with which the driver visually observes the object in question, that is, whether or not they normally see it, is determined based on the history 141. The determination is made by the determination unit 150. The determination result is displayed to the information processing unit 160 or the information presentation unit 150. Output to 70.

[0024] Here, the determination unit 150 first determines whether the object the driver is looking at is related to the operation of the vehicle. To identify. Things related to vehicle operation include roads, other vehicles, traffic lights, signs, mirrors, etc. This may include accident scenes as it relates to the operation of a vehicle.

[0025] Next, the determination unit 150 determines whether the driver normally looks at the object. The frequency of visually inspecting the target is 1 or more, depending on whether the target is included in history 141. The determination may be made depending on whether the number is less than or equal to the number of items in history 141. Depending on the circumstances, a determination is made based on whether the frequency of visually inspecting the target is above or below a predetermined number. This is also good. In addition, the determination unit 150 determines whether the subject has been observed more than three times in a week, and other recent factors. The frequency of visual inspection of the target can be determined by referring to the history 141 for the period, This is used to compare driving history, such as whether or not an elephant was seen at least once when passing on a road adjacent to an elephant twice. You may also want to determine how often the target is visually inspected.

[0026] Here, the determination unit 150 makes a determination considering the time the driver is looking at the object (gaze time). It is permissible to do so. For example, the determination unit 150 determines if the number of times the driver has looked at the target for 1 second or more is a predetermined number. In the above case, it may be determined that the driver regularly sees the object in question.

[0027] Here, the determination unit 150 may refer to the history 141, taking into account conditions such as time of day and day of the week. For example, the determination unit 150 considers that restaurants tend to be seen at night and on holidays. It may also be possible to determine whether the driver normally sees the object in question.

[0028] The information processing unit 160, when the driver does not normally look at the object, provides information related to the object. It is recorded as memo information 142. Therefore, the information processing unit 160 determines that the subject is unrelated to driving. Furthermore, if the information is something the driver does not normally see, memo information 142 will be recorded. The information processing unit 160 records the name of the target estimated by the estimation unit 130 as memo information 142. It is also possible to obtain an image of the subject from the image acquisition unit 110 and record it as memo information 142. It may be recorded. The information processing unit 160 processes the map information of the target estimated by the estimation unit 130, for example, the figure Capture the screen of the car navigation system that is not being displayed and record it as memo information 142. That's fine.

[0029] Furthermore, the information processing unit 160 creates a message that includes the name of the target as part of it, and memo information 1 It may be recorded as 42. The message is, for example, that the object the driver is looking at is a post office. If so, the message will be something like, "Do you have any business at the post office?" The information processing unit will then process the message. Create it as an audio memo, or as an image memo, and record it as memo information 142. That's good too.

[0030] Furthermore, memo information 142 may be voice recordings made by the driver. 160 is a message that includes the name of the target (for example, "Do you need anything at the post office?") After notifying the driver, the driver's voice is recorded by microphone 20 and memo information 142 is recorded. The information processing unit 160 may output the message from the speaker 30. It's often a good idea to display it on a monitor. This will remind the driver that they had business at the post office. It is possible to record information such as a voice message (e.g., "I had business at the post office").

[0031] Here, the information processing unit 160 includes both the time when the subject was photographed and the period during which the driver spoke. The video of the period may be recorded as memo information 142. In addition, the information processing unit 160 will operate The audio spoken by the person and the external video footage of the vehicle during the period in which the audio was spoken are combined with memo information 142. It may be recorded as follows: That is, the information processing unit 160 detects the driver's voice and records its end. External video footage may be recorded during the period leading up to the date.

[0032] Additionally, the name of the estimated target (for example, "○○ Post Office") is linked to the created file. It is preferable to do so. Therefore, the information processing unit 160 stores the memo information 142 and the name of the target. The names may be recorded in association with each other. For example, the information processing unit 160 may record "○○ Post Office" The name may also be used as the file name for memo information 142, and a voice memo created by speech synthesis using that name, It may be combined with memo information 142, and the name may be superimposed on the image memo as subtitle data. Often, the name in question can be embedded as metadata in memo information 142.

[0033] The information display unit 170 displays information related to the object when the driver is normally looking at the object. It is presented to the driver. Therefore, the information presentation unit 170 is for information related to driving. Information will be presented only if it is not something the driver normally sees. The information display unit 170 may output the displayed information to the speaker 30 or to the monitor. .

[0034] The information display unit 170 displays, for example, the name of the subject, such as "You often look at XX Ramen." Related information may be output. This allows the driver to see what they are potentially interested in. It is possible to recognize that.

[0035] Furthermore, the information display unit 170 transmits information related to the name of the target via a communication unit (not shown). The information may be obtained from the internet or other sources and output. The information display unit 170 is, for example, For example, whether or not "○○ Ramen" is running a campaign, or how good or bad its reputation is. It can be shown to the driver.

[0036] Furthermore, the information display unit 170 displays information related to the name of the target on a car navigation system that is not shown in the diagram. The data may be transmitted to a device so that it can be searched and registered on the car navigation system.

[0037] Furthermore, the information display unit 170 is intended for information related to driving, and which the driver normally sees. If something is missing, information may be provided. For example, if the driver does not normally see something. If a driver sees a "No Parking" sign, it can be assumed that they are looking for a parking space, so information can be provided. Section 170 may present the driver with parking information obtained from the internet or other sources. For example, if a driver is looking in their rearview mirror more than usual, it can be interpreted that they are concerned about the car behind them. Because it can be turned off, the information display unit 170 displays information to encourage the user to focus their attention forward. It is possible.

[0038] Next, we will explain the operation flow of the in-vehicle device 100 using Figure 2. First, the gaze detection unit 120 uses the driver's face image in the image acquired by the image acquisition unit 110 to determine the driver's perspective The direction of the line is detected (step S101). Next, the estimation unit 130 determines the entire direction of the driver's line of sight. This refers to a state of looking at something, or a state of looking at (gazing at) a specific object in the direction of one's gaze. Determine if (step S102). If the driver is not looking at a specific object (step S No. 102), the in-vehicle device 100 returns to processing step S101. The driver is in a specific position. If the driver is looking at an elephant (Yes in step S102), the estimation unit 130 will move in the direction of the driver's line of sight. Estimate a certain object (Step S103).

[0039] Next, the determination unit 150 determines whether the object estimated in step S103 is directly related to driving. Determine whether or not the object is related to the operation of a vehicle (step S104). If this is the case (Yes in step S104), the in-vehicle device 100 processes in step S101. Return to the previous page.

[0040] If the subject is not related to the operation of a vehicle (No. in step S104), the determination unit 15 0 determines whether the driver normally sees the object (step S105). If the driver does not normally observe the target (No. in step S105), the information processing unit 160 will determine the driver It is determined that the person has remembered something about the subject (step S106), and there is an image memo. Next, the voice memo is recorded as memo information 142 (step S107). Here, information processing Unit 160 may display messages such as, "Do you need anything at the post office?"

[0041] If the driver is normally looking at the object (Yes in step S105), the information display unit 170 will The system determines that the driver is interested in the subject (step S108), and then determines that the subject is of interest to the driver. The information is presented to the driver (step S109). Finally, the history of the estimated results 141 is updated. (Step S110).

[0042] The in-vehicle device 100 according to Embodiment 1 displays the object that the driver is looking at, as the driver normally looks at It distinguishes between things that are normally seen and things that are not normally seen. Therefore, the in-vehicle device 1 according to Embodiment 1 00 can display information that the driver is interested in. Also, information that the driver remembers. This information can be automatically recorded as memo information while driving, where you cannot take your hands off the wheel. can.

[0043] (Embodiment 2) According to Embodiment 2, the in-vehicle device 100a, when the vehicle operation is completed, the information processing unit 16 0 outputs the memo information recorded. Figure 3 is a block diagram showing an example configuration of the in-vehicle device 100a. The differences from Embodiment 1 will be explained below.

[0044] The in-vehicle device 100a connects with the driver's communication terminal 40 via the network 200. It continues. Network 200 includes the internet, intranet, and mobile phone network. This refers to a communication network such as a LAN (Local Area Network). Communication terminal 40 This includes mobile phones, smartphones, tablet devices, and PCs (Personal Computers). This includes the power supply (puter), etc. The in-vehicle device 100a receives the memo information 142 via the communication terminal 40. Alternatively, the output may be displayed on speaker 30, a monitor screen (not shown), or the like.

[0045] In addition to the configuration shown in Figure 1, the in-vehicle device 100a also includes a communication unit 180 and an output unit 190. It is present. The communication unit 180 is the communication interface with the network 200. In-vehicle device 1 When 00a outputs memo information 142 to speaker 30, etc., the in-vehicle device 100a is the communication unit It is not necessary to have 180.

[0046] The output unit 190 determines whether or not the operation of the vehicle on which the vehicle-mounted device 100a is installed has ended. The output unit 190 works in conjunction with a car navigation system (not shown) installed in the vehicle. When the vehicle arrives near its destination or when it is a certain distance away from its destination (for example, 100m) The driving may be considered complete when this happens. The destination is set in the car navigation system. In addition to the registered destination, your registered home or current location can also be determined if it is a parking lot. Furthermore, the output unit 190 acquires the vehicle's gear information and detects when the vehicle enters a parking operation, When the driving information determines that the parking operation is complete, it may be determined that the driving has ended. The output unit 190 detects that the engine has been turned off, and the operation has ended. It may be determined that this is the case.

[0047] The output unit 190 outputs memo information 142 when the vehicle operation is completed. 90 may output voice memos from speaker 30, and monitors that do not display image memos. It may also be displayed on the output unit 190. The output unit 190 sends the information to the communication terminal 40 which has been pre-registered as the destination. The memo information 142 may be output via the signal unit 180. The output unit 190 may be set as the destination. The registered email address, phone number, SNS (social networking service), etc. The memo information 142 may be output via the communication unit 180.

[0048] The in-vehicle device 100a according to Embodiment 2 outputs memo information recorded when the operation is completed. Therefore, the driver checks the memo information after the drive is finished and recalls any tasks. This can be done. In addition, the driver can confirm the memo information output from the speaker, monitor, etc. You can acknowledge it and take notes in a notebook or similar.

[0049] Furthermore, the aforementioned program is not compatible with various types of non-temporary computer-readable media (no (using n-transitory computer-readable medium) It can be stored and supplied to a computer. Non-temporary computer-readable media. This refers to various types of tangible storage media. Includes (um). Non-temporary computer-readable media include, for example, magnetic recording media and magneto-optical recording media. Media, CD-ROM (Read Only Memory), CD-R, CD-R / W, Includes semiconductor memory. Magnetic recording media include, for example, flexible disks, magnetic tapes, etc. Examples include hard disk drives. Magneto-optical recording media include, for example, magneto-optical disks. Semiconductor memory includes, for example, mask ROM and PROM (Programmable ROM). M), EPROM (Erasable PROM), Flash ROM, RAM (Ran These include DOM access (memory), etc. Also, programs are of various types. temporary computer-readable media It may be supplied to the computer by a temporary computer medium. Examples of readable media include electrical signals, optical signals, and electromagnetic waves. The medium transmits the program via wired communication channels such as electric wires and optical fibers, or via wireless communication channels. It can be supplied to computers.

[0050] The in-vehicle device according to the above embodiment is applicable to the field of driver monitoring systems. It is possible. Furthermore, a device other than the in-vehicle device can detect a person's gaze and determine what the person is looking at. It is also possible to estimate the target. For example, a glasses-type device could have a gaze detection device and the surrounding scenery. A similar system can be realized by attaching a camera to take pictures. This allows, This makes it possible to investigate a person's potential interests not only while they are driving, but also in their everyday life. [Explanation of symbols]

[0051] 10 Cameras 20 microphones 30 speakers 40 Communication terminals 100, 100a In-vehicle equipment 110 Image acquisition unit 120 Eye-line detection unit 130 Estimation part 140 Storage section 141 History 142 Memo Information 150 Judgment section 160 Information Processing Section 170 Information Presentation Department 180 Communications Department 190 Output section 200 Networks

Claims

1. A gaze detection unit that detects the direction of the vehicle driver's gaze, An estimation unit that determines, at least based on the speed of the driver's eye movement, that the driver is looking at a specific object in the direction of their gaze, and estimates the object; A database for managing the history of the estimation results of the estimation unit, If the number of times the driver has viewed the object for a predetermined period of time or longer is not included in the history at a predetermined number or more, an information processing unit records information related to the object as memo information. An output unit outputs the memo information recorded during operation when the operation of the vehicle ends, A vehicle-mounted device equipped with this technology.

2. A gaze detection step that detects the direction of the vehicle driver's gaze, An estimation step of determining, at least based on the speed of the driver's eye movement, that the driver is looking at a specific object in the direction of their gaze, and estimating the object; A database for managing the history of estimation results in the estimation step, If the number of times the driver has viewed the object for a predetermined period of time or longer is not included in the history at a predetermined number or more, the information processing step includes recording information related to the object as memo information. An output step in which, when the operation of the vehicle ends, the memo information recorded during operation is output, A recording method equipped with [a specific feature / feature].

Images