Attention reminding system and attention reminding method

Abstract

The present invention provides an attention reminding system and an attention reminding method to notify a driver of a sign of a state of distraction in a manner that is easy for the driver to accept. The attention reminding system has: a notification unit that, when the driver's line of sight dwell time of a vehicle exceeds a predetermined threshold, makes a notification of a predetermined level of prominence to the driver via an HMI device; a decision unit that decides a notification parameter set including parameters of a notification timing and a level of prominence; a behavior recording unit that records whether or not the driver has a reaction behavior to the notification using the notification parameter set; and a reaction information recording unit that, for each of different notification parameter sets, stores reaction information associated with a ratio of the number of notifications with the reaction behavior to the number of notifications in a storage device, and the decision unit calculates and decides an optimal parameter set that is estimated to have the largest reaction ratio based on the reaction information.

Description

Technical Field

[0001] This invention relates to a attention reminder system and method for reminding a vehicle driver to pay attention. Background Technology

[0002] In recent years, research and development has become active in providing sustainable transportation systems that take into account the disadvantaged among transportation participants. To achieve this, research and development related to preventative safety technologies is being conducted to further improve the safety and convenience of transportation.

[0003] Patent document 1 discloses a side-view notification device that provides attention reminders to the driver when the driver of a vehicle in autonomous driving mode has looked to the side for more than a specified time during a lane change.

[0004] [Existing Technical Documents]

[0005] [Patent Literature]

[0006] Patent Document 1: Japanese Patent Application Publication No. 2020-35043 Summary of the Invention

[0007] [The problem the invention aims to solve]

[0008] However, in preventive safety technology, there is a challenge: at intersections and other locations where drivers' attention to the traffic environment, such as the actions of other road users, becomes more important, it is crucial to communicate signs of driver inattention to the driver as early as possible, so as to encourage the driver to focus on driving behavior.

[0009] When informing the driver in advance of their inattention state, since the notification may not be able to make a definite judgment about the inattention state, it is also important that the stimuli (visual, auditory, and / or tactile) given to the driver for the purpose of the notification suppress the annoyance to the driver and are easily identifiable and acceptable to the driver.

[0010] To address this trade-off, one could consider altering the presentation of stimuli in light of the driver's attentional state and driving behavior. However, individual perceptions of stimuli (e.g., ease of acceptance, degree of discomfort) vary, and it cannot be argued that considering only the driver's attentional state and driving behavior adequately addresses the acceptability of notifications to the driver.

[0011] To address the aforementioned issues, the purpose of this application is to notify drivers of signs of inattention in a manner easily acceptable to them. Furthermore, this contributes to the development of sustainable transportation systems.

[0012] [Methods used to solve problems]

[0013] One aspect of the present invention is an attention reminder system comprising: a gaze measurement unit that measures the gaze dwell time of a driver of a vehicle; and a notification unit that, when the measured gaze dwell time exceeds a predetermined threshold, notifies the driver via an HMI device of a predetermined salience level. The attention reminder system further comprises: a decision unit that determines a notification parameter set, the notification parameter set including parameters specifying the timing of the notification and parameters specifying the salience level of the notification issued by the HMI device; and a behavior recording unit that records the behavior based on the timing of the notification issued by the notification unit according to the determined notification parameter set and the salience level. When the notification is issued, the behavior recording unit records whether the driver has a reaction to the notification; and the reaction information recording unit calculates the reaction ratio for different notification parameter sets and stores the reaction information formed by associating the notification parameter sets with the reaction ratios in a storage device, wherein the reaction ratio is the ratio of the number of times the driver has reacted to the notification to the number of times the notification is issued; and the decision unit calculates and determines the optimal parameter set based on the relationship between the notification parameter sets represented by the reaction information and the reaction ratios, wherein the optimal parameter set is the notification parameter set that is presumed to have the largest reaction ratio.

[0014] According to another aspect of the invention, the decision unit further determines an update period and at least one set of data collection parameters, the update period being the period up to the optimal set of parameters determined in the next update, the at least one set of data collection parameters being the notification parameter set specifically used for data collection, and during the update period, the decision unit, in cooperation with the notification unit, the behavior recording unit, and the reaction information recording unit, selectively notifies the use of the determined optimal set of parameters and the set of data collection parameters, and obtains the reaction information regarding the optimal set of parameters and the set of data collection parameters used.

[0015] According to another aspect of the invention, the update period is determined to be a period of predetermined length or a period during which the notification is made a predetermined number of times by the notification unit.

[0016] According to another aspect of the invention, the decision unit generates execution order information that specifies the order in which the notification unit executes the notification using the optimal parameter set and the notification using the data collection parameter set among multiple notifications during the update period, and the notification unit selects and uses the optimal parameter set and the data collection parameter set to perform the notification according to the execution order information.

[0017] According to another aspect of the invention, the attention alert system has a learning unit that, based on the reaction information, enables a driver reaction model to learn the relationship between the notification parameter set and the reaction ratio through machine learning, and the decision unit uses the driver reaction model to determine the optimal parameter set presumed to maximize the reaction ratio.

[0018] According to another aspect of the invention, the line-of-sight measurement unit measures the driver's line-of-sight dwell time when the vehicle enters and / or passes through the intersection.

[0019] According to another aspect of the invention, the gaze measurement unit measures the driver's gaze dwell time by tracking the driver's gaze based on a facial image from a camera located inside the vehicle.

[0020] According to another aspect of the invention, the presence or absence of the reaction behavior is determined based on whether the amount of eye movement between two temporally consecutive fixation points in the driver's eye movement is greater than a predetermined amount, and / or whether the size of the driver's eye movement range is greater than a predetermined size.

[0021] According to other aspects of the invention, the HMI device includes a tactile HMI device, an auditory HMI device, and / or a visual HMI device.

[0022] According to another aspect of the invention, a server device is included that is communicatively connected to the vehicle, the server device comprising the decision unit and the reaction information recording unit, and the vehicle comprising the line-of-sight measurement unit and the notification unit.

[0023] According to another aspect of the present invention, the decision unit, the reaction information recording unit, the line-of-sight measurement unit, and the notification unit are disposed in the vehicle.

[0024] Another aspect of the present invention is a attention reminder method executed by a computer of an attention reminder system that notifies the driver of a vehicle via an HMI device. This attention reminder method includes: a gaze measurement step, measuring the driver's gaze dwell time; and a notification step, notifying the driver via the HMI device when the measured gaze dwell time exceeds a predetermined threshold. The attention reminder method further includes: a decision step, determining a notification parameter set, the notification parameter set including parameters specifying the timing of the notification and parameters specifying the salience level of the notification issued by the HMI device; and a behavior recording step, recording the behavior based on the determined notification parameter set. When the notification is issued in the notification step according to the indicated timing and significance level, it is recorded whether the driver has a reaction to the notification; and in the reaction information recording step, a reaction ratio is calculated for different notification parameter sets, and the reaction information formed by associating the notification parameter sets with the reaction ratio is stored in a storage device, wherein the reaction ratio is the ratio of the number of times the driver has reacted to the notification to the number of notifications; and in the decision step, an optimal parameter set is calculated and determined based on the relationship between various notification parameter sets and the reaction ratio, wherein the optimal parameter set is the notification parameter set presumed to have the largest reaction ratio.

[0025] [Invention Effects]

[0026] According to the present invention, it is possible to notify the driver of signs of inattention in a manner that is easily acceptable to the driver. Attached Figure Description

[0027] Figure 1 This is a diagram illustrating the structure of a attention reminder system according to one embodiment of the present invention.

[0028] Figure 2 This is a diagram showing the structure of a vehicle equipped with a attention reminder device that constitutes an attention reminder system.

[0029] Figure 3 This diagram illustrates the structure of the interior of a vehicle equipped with a attention alert device.

[0030] Figure 4 This is an example diagram illustrating a scenario where an attention reminder system performs an action.

[0031] Figure 5 This is a diagram showing the structure of the attention reminder device that constitutes the attention reminder system.

[0032] Figure 6 This is a diagram showing the structure of the optimized server that constitutes the attention reminder system.

[0033] Figure 7 This is a flowchart illustrating the steps of the attention reminder method performed by the attention reminder system.

[0034] Explanation of reference numerals in the attached figures

[0035] 1…Attention alert system, 2…Vehicle, 3…Attention alert device, 4…Optimization server, 5…Communication network, 6…HMI device, 6a…Haptic HMI device, 6b…Audioscopic HMI device, 6c…Visual HMI device, 7…Object detection device, 8…Exterior camera, 8a…Front-facing camera, 8b…Left-side camera, 8c…Right-side camera, 8d…Rear camera, 9…GPS device, 10…Vehicle control device, 11…Interior camera, 12…Interior mirror, 13a, 13b…Door rearview mirrors, 14…Intersection, 1 5…Driver’s vehicle, 20…Driver’s seat, 21…Seat belt, 22…Steering wheel, 23…Instrument panel, 24…Display device, 25…Windshield, 30…Processor, 31…Memory, 32…First communicator, 33…Sight measurement unit, 34…Notification unit, 35…Judgment unit, 36…Behavior recording unit, 37…Reaction log, 38, 48…Program, 40…Processor, 41…Memory, 42…Second communicator, 43…Reaction information recording unit, 44…Learning unit, 45…Decision unit, 46…Reaction information, 47…Driver reaction model, 60…Line. Detailed Implementation

[0036] The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0037] Figure 1 This is a diagram illustrating the structure of a attention reminder system 1 according to an embodiment of the present invention. Additionally... Figure 2 This is a diagram showing the structure of a vehicle 2 equipped with a attention alert system 1. Figure 3 This is a diagram showing the structure of the interior of vehicle 2.

[0038] [Note the structure of the reminder system]

[0039] Attention alert system 1 includes attention alert device 3 installed on vehicle 2 and optimization server 4, which is a server device located outside vehicle 2. Attention alert device 3 and optimization server 4 are connected to each other in a manner that enables communication, for example, via a communication network 5 such as the Internet.

[0040] [Vehicle Structure]

[0041] Vehicle 2 is equipped with an HMI device 6 ( Figure 5In this embodiment, the HMI device 6 includes a tactile HMI device 6a that provides tactile stimulation to the driver, an auditory HMI device 6b that outputs sound as auditory stimulation to the driver, and a visual HMI device 6c that conveys visual information to the driver. However, this is just an example; the HMI device 6 can be any one of the tactile, auditory, and visual HMI devices, or it can include any two of them.

[0042] The tactile HMI device 6a, for example, is installed in the driver's seat 20 of vehicle 2, and is an electrically powered seatbelt that applies tactile stimulation to the driver by changing the tension of the seatbelt 21 worn by the driver. However, the electrically powered seatbelt is one example, and the tactile HMI device 6a can be any device capable of applying tactile stimulation to the driver. For example, the tactile HMI device 6a could also be a vibration actuator that applies vibration to the driver's hands via the steering wheel 22, or a massage chair installed in the driver's seat 20, which applies stimulation such as tapping, rubbing, pressing, and vibration to the driver's back and other body parts.

[0043] In this embodiment, the auditory HMI device 6b is a single loudspeaker. Alternatively, the auditory HMI device 6b may also be a loudspeaker system consisting of multiple loudspeakers.

[0044] In this embodiment, the visual HMI device 6c is an instrument display device installed on the instrument panel 23 of the vehicle 2. However, the instrument display device is one example, and the visual HMI device 6c could also be a general display device 24 that displays content other than the instrument panel, or a head-up display (not shown) that displays images on the windshield 25, etc.

[0045] Vehicle 2 also has an object detection device 7 for detecting objects in the surrounding environment of vehicle 2, and an external camera 8 for acquiring images of the surrounding environment. Figure 5 The vehicle 2 also includes a GPS device 9. The object detection device 7 can be, for example, a radar, lidar, and / or sonar located at the front of the vehicle 2 to detect objects (hereinafter also referred to as "front objects") in the environment in front of the vehicle 2. Alternatively, the object detection device 7 may also include multiple cameras, radar, lidar, and / or sonar distributed around the vehicle 2 to detect objects in the overall surrounding environment, including the left and right sides and rear of the vehicle 2.

[0046] The external camera 8 includes: a front camera 8a, which is located at the front of the vehicle 2 to capture the environment in front of the vehicle 2; a left-side camera 8b and a right-side camera 8c, which are located on the left and right side mirrors 13a and 13b of the vehicle door to capture the environment to the left and right of the vehicle 2; and a rear camera 8d, which is located at the rear of the vehicle to capture the environment behind the vehicle 2.

[0047] The vehicle 2 also has a vehicle control device 10, which detects the operating status of the accelerator pedal, brake pedal, turn indicator lights, steering wheel and other driving controls of the vehicle 2, and detects the vehicle speed, acceleration, yaw rate and other motion status of the vehicle 2.

[0048] Vehicle 2 also includes an interior camera 11 for capturing images of the interior, including the driver's, and an interior mirror 12 for the driver's visual confirmation of the rear of vehicle 2. The interior mirror 12 may also be a rearview mirror that displays images captured by a rear camera 8d of the rear of vehicle 2.

[0049] [Note: Summary of the system's actions]

[0050] In the attention alert system 1, the attention alert device 3 and the optimization server 4 cooperate to notify the driver of vehicle 2 that they may be inattentive. For example, as Figure 4 As shown, when the vehicle 2 passes through the intersection 14, the attention reminder system 1 notifies the driver by outputting tactile stimuli, auditory stimuli (e.g., sound), and / or visual information (e.g., light) from the HMI device 6 when it detects that the driver may be in a state of distraction.

[0051] Specifically, in accordance with the notification parameter set (described later) received from the optimization server 4, when the likelihood of a distracted state reaches a predetermined level, the attention reminder device 3 notifies the driver of the vehicle 2 via the HMI device 6. The HMI device 6 used for this notification can be any one or a combination of two or more of a tactile HMI device 6a, an auditory HMI device 6b, and a visual HMI device 6c.

[0052] Server 4 optimizes the aforementioned notification parameter set to improve the acceptability of notifications executed by the attention reminder device 3 using the HMI device 6. Here, in this embodiment, the degree of acceptability is the ratio of the number of times the driver reacts to these notifications to the number of times notifications are issued by the HMI device 6, i.e., the reaction ratio.

[0053] Optimize server 4 to perform data collection actions for optimizing the above notification parameters and to apply the notification using the optimal set of notification parameters.

[0054] During the data collection process, optimization server 4 generates a set of notification parameters specifically for data collection, i.e., a set of parameters for data collection. Optimization server 4 generates various sets of parameters for data collection with different values ​​and obtains reaction information representing the relationship between each of these sets of parameters and the reaction ratio.

[0055] Optimization server 4 uses the acquired reaction information to enable the driver reaction model to learn the relationship between the notification parameter set and the reaction ratio through machine learning. For example, whenever reaction information is acquired through data collection actions, optimization server 4 uses the acquired reaction information to further learn the driver reaction model.

[0056] In the application of the action, the optimization server 4 uses a driver reaction model that has been further learned based on the obtained reaction information to determine the set of notification parameters that is presumed to have the highest driver reaction rate, i.e., the optimal parameter set. The optimization server 4 sends the above-determined optimal parameter set to the attention reminder device 3, and the attention reminder device 3 uses the above-determined optimal parameter set to make a notification through the HMI device 6.

[0057] Optimization server 4 also uses reaction information obtained from actions to represent the relationship between the optimal parameter set and the reaction ratio. Optimization server 4 further uses this obtained reaction information to perform additional learning on the driver reaction model.

[0058] Thus, the optimization server 4 can determine the optimal set of parameters in a way that allows it to make notifications with higher response rates (i.e., notifications that are more acceptable to the driver) as the driver's response model is repeatedly learned.

[0059] [Note the structure of the reminder device]

[0060] Figure 5 This is a diagram showing the structure of the attention reminder device 3 provided by vehicle 2.

[0061] The attention reminder device 3 includes a processor 30, a memory 31, and a first communicator 32. The first communicator 32 is a transceiver used for communication between the attention reminder device 3 and the optimization server 4. The memory 31 is, for example, composed of volatile and / or non-volatile semiconductor memory and / or a hard disk device. In the memory 31, a reaction log 37 is stored via a behavior recording unit 36, described later.

[0062] The processor 30 is, for example, a computer equipped with a CPU. The processor 30 may also have a structure that includes a ROM on which programs are written, RAM for temporary data storage, etc. Furthermore, the processor 30 includes a gaze measurement unit 33, a notification unit 34, a judgment unit 35, and a behavior recording unit 36 ​​as functional elements or functional units.

[0063] These functional elements of the processor 30 are implemented, for example, by the processor 30, which is a computer, executing a program 38 stored in memory 31. Furthermore, the program 38 can be pre-stored in any computer-readable storage medium. Alternatively, all or part of the aforementioned functional elements of the processor 30 can be constituted by hardware comprising one or more electronic circuit components.

[0064] The gaze measurement unit 33 continuously detects the driver's gaze persistence in vehicle 2 and measures the gaze persistence time. For example, according to the prior art, the gaze measurement unit 33 tracks the driver's gaze based on a facial image obtained from the indoor camera 11, thereby measuring the driver's gaze persistence time. In this embodiment, the gaze measurement unit 33 measures the driver's gaze persistence time when vehicle 2 enters an intersection where the driver's attention to the traffic environment becomes more important, and / or when vehicle 2 passes through an intersection.

[0065] The notification unit 34, based on the notification parameter set received from the optimization server 4 (i.e., the aforementioned data collection parameter set or optimal parameter set), notifies the driver of vehicle 2 via the HMI device 6 that the driver may be in a state of inattention when the gaze dwell time calculated by the gaze measurement unit 33 exceeds a predetermined threshold. Furthermore, this notification is executed according to the salience level specified by the parameters of the notification parameter set.

[0066] In this embodiment, as shown in equation (1), the notification parameter set includes the values ​​of two parameters: a threshold a, which is a parameter for specifying the timing of the notification, and a constant b, which is a parameter for specifying the significance level of the notification.

[0067] Notification parameter set = (a, b)(1)

[0068] The notification unit 34, based on the threshold a included in the notification parameter set, initiates the notification through the HMI device 6 when the line-of-sight dwell time Ts calculated by the line-of-sight measurement unit 33 exceeds the threshold a, i.e., when equation (2) is satisfied. The notification is executed using a constant b included in the aforementioned notification parameter set as a parameter specifying the salience level of the notification, and the salience level SL given by equation (3).

[0069] Ts>a (2)

[0070] SL=b (3)

[0071] Here, salience level refers to the degree to which the stimulus output by HMI device 6 attracts or arouses human attention.

[0072] For example, when tactile stimulation is output as the tension of an electric seatbelt, the salience level of the tactile stimulus corresponds to the magnitude of the aforementioned tension. The greater the tension, the higher the salience level of the tactile stimulus.

[0073] Furthermore, the salience level of an auditory stimulus corresponds to the intensity, frequency, repetition period, and / or the period of intensity or frequency variation of the sound. The greater the intensity, the higher the frequency, the shorter the repetition period, and / or the shorter the period of intensity or frequency variation, the higher the salience level of the auditory stimulus.

[0074] Alternatively, when the tactile stimulus is a vibration delivered to the driver's hand from a vibration actuator located on the steering wheel 22 as described above, the salience level of this tactile stimulus corresponds to the intensity, frequency, repetition period, and / or the duration of the intensity or frequency variation period of the vibration. The greater the intensity, the higher the frequency, the shorter the repetition period, and / or the shorter the intensity or frequency variation period, the higher the salience level of the tactile stimulus imparted by the vibration.

[0075] Alternatively, when the visual information is output as graphic elements such as text or graphics displayed on a display device, the salience level of the visual information can be determined based on the brightness, brightness variation period, flicker period, or hue of the displayed graphic elements. For example, the higher the brightness, the shorter the brightness variation or flicker period, or the closer the hue is from cool to warm, the higher the salience level of the visual information.

[0076] Furthermore, the details of the tactile stimulation, auditory stimulation, and / or visual information (such as the intensity, frequency, tension, brightness, brightness variation, etc. of the various stimuli or visual information) and the correspondence between the salience level SL can be predetermined and stored in memory 31.

[0077] The notification unit 34 can refer to the information of the above correspondence and notify the driver from the HMI device 6 by providing the above tactile stimulation, auditory stimulation and / or visual information in a manner corresponding to the magnitude of the above-determined salience level SL.

[0078] As described above, in this embodiment, the notification parameter set includes both a data collection parameter set and an optimal parameter set. The notification unit 34, following instructions from the optimization server 4 (e.g., according to the execution order information described later), uses either the data collection parameter set or the optimal parameter set to notify the driver via the HMI device 6.

[0079] When the notification unit 34 uses the HMI device 6 to issue a notification based on the timing and significance level indicated by the notification parameter set determined by the optimization server 4, the judgment unit 35 determines whether there is a driver's reaction behavior to the notification.

[0080] Specifically, the determination unit 35 acquires changes in driving-related behaviors of the vehicle 2 from the time the notification unit 34 sends a notification via the HMI device 6 until a predetermined time has elapsed (e.g., 2 seconds from the start of the notification) as the driver's reaction behavior to the notification. In this embodiment, for example, the determination unit 35 determines the presence or absence of the reaction behavior based on whether the amount of eye movement between two temporally consecutive gaze points in the driver's eye movement is greater than or equal to a predetermined amount, and / or whether the size of the driver's eye movement range is greater than or equal to a predetermined size. The determination unit 35 determines that the reaction behavior exists when the amount of eye movement between the gaze points is greater than or equal to a predetermined amount, and / or when the size of the driver's eye movement range is greater than or equal to a predetermined size.

[0081] When the notification unit 34 uses the HMI device 6 to issue a notification based on the timing and significance level indicated by the notification parameter set determined by the optimization server 4, the behavior recording unit 36 ​​records whether the driver has any reaction behavior to the notification.

[0082] Specifically, the behavior recording unit 36, based on the judgment result of the judgment unit 35 regarding whether there is a reaction behavior, saves the reaction record that associates the notification parameter set used by the notification with the judgment result regarding whether there is a reaction behavior in response to the notification in the reaction log 37 stored in the memory 31.

[0083] Furthermore, at the end of the update period described later, the behavior recording unit 36 ​​sends the reaction records stored in the reaction log 37 to the optimization server 4.

[0084] [Optimize server structure]

[0085] Figure 6 This is a diagram showing the structure of optimized server 4.

[0086] The optimization server 4 includes a processor 40, a memory 41, and a second communicator 42. The second communicator 42 is a transceiver used by the optimization server 4 to communicate with the attention reminder device 3. The memory 41 is, for example, composed of volatile and / or non-volatile semiconductor memory and / or a hard disk device. The memory 41 stores reaction information 46 generated by the reaction information recording unit 43 (described later) and a driver reaction model 47 generated by machine learning through the learning unit 44.

[0087] The processor 40 is, for example, a computer equipped with a CPU. The processor 40 may also have a structure that includes a ROM on which programs are written, RAM for temporary data storage, etc. Furthermore, the processor 40 includes a response information recording unit 43, a learning unit 44, and a decision unit 45 as functional elements or functional units.

[0088] These functional elements of the processor 40 are implemented, for example, by the processor 40, which is a computer, executing a program 48 stored in memory 41. Furthermore, the program 48 can be pre-stored in any computer-readable storage medium. Alternatively, all or part of the aforementioned functional elements of the processor 40 can be constituted by hardware comprising one or more electronic circuit components.

[0089] Based on the reaction records sent from the attention alert device 3 of vehicle 2, the reaction information recording unit 43 calculates the ratio of the number of times the driver reacted to a notification to the number of times a notification using that parameter set was used, i.e., the reaction ratio, for each different notification parameter set. Furthermore, the reaction information recording unit 43 generates reaction information 46 that associates each notification parameter set with the reaction ratio for notifications using that parameter set. Then, the reaction information recording unit 43 stores the generated reaction information 46 in the memory 41.

[0090] The learning unit 44 uses the reaction information 46 stored in the memory 41 by the reaction information recording unit 43 to enable the driver reaction model 47 stored in the memory 41 to learn the relationship between the notification parameter set and the reaction ratio through machine learning. The driver reaction model 47, according to existing technology, can be constructed from a neural network, for example.

[0091] Whenever the reaction information recording unit 43 re-stores the reaction information 46 based on the reaction record received from the attention reminder device 3 of the vehicle 2 (for example, whenever the update period described later ends), the learning unit 44 uses the re-stored reaction information 46 to perform additional learning on the driver reaction model 47. Thus, whenever new reaction information is generated based on new reaction records received from the attention reminder device 3, the driver reaction model 47 stored in the memory 41 is updated to be able to make more accurate predictions.

[0092] The decision unit 45 determines a set of notification parameters for the notification action performed by the notification unit 34 of the attention reminder device 3 equipped in the vehicle 2 through the HMI device 6. As shown in equation (1) above, in this embodiment, the notification parameter set includes a threshold a, which is a parameter for the timing of issuing the notification, and a constant b, which is a parameter for the prominence level of the notification.

[0093] In this embodiment, the decision unit 45 determines the optimal set of notification parameters for data collection and the set of notification parameters for application, and uses them as the notification parameter set.

[0094] The data collection parameter set is a set of notification parameters specifically used for data collection operations in the attention alert device 3 equipped in vehicle 2. During the data collection operation, various data collection parameter sets with different values ​​are used, for example, randomly to collect response information.

[0095] For example, the decision unit 45 randomly determines each set of parameters for data collection. That is, the decision unit 45 randomly determines the value of the threshold a and the value of the constant b that constitute the notification parameter set from a predetermined value range, and generates a set of parameters for data collection consisting of the threshold a and the constant b determined above.

[0096] The optimal parameter set is a set of notification parameters used to enable the attention reminder device 3 of vehicle 2 to perform the following operation: providing notification in a manner easily acceptable to the driver. Specifically, the decision unit 45 calculates the notification parameter set that is presumed to have the highest response rate based on the relationship between various notification parameter sets and response rates, and determines the calculated notification parameter set as the optimal parameter set.

[0097] In this embodiment, a driver response model 47 that has learned the relationship between various notification parameter sets and response ratios is used to calculate the notification parameter set that is presumed to have the largest response ratio, and the calculated notification parameter set is determined as the optimal parameter set.

[0098] The decision unit 45 determines a predetermined period and, in cooperation with the attention reminder device 3, selectively performs data collection operations using the determined data collection parameter set and application operations using the determined optimal parameter set during the predetermined period. In this embodiment, the predetermined period is the update period until the next update of the determined optimal parameter set (i.e., the period until the optimal parameter set is re-determined).

[0099] Furthermore, during the aforementioned update period, the decision unit 45, in cooperation with the notification unit 34, the behavior recording unit 36, and the reaction information recording unit 43, selectively uses the optimal parameter set of the aforementioned decision and the aforementioned multiple data collection parameter sets, and obtains reaction information regarding the optimal parameter set and data collection parameter sets used in these notifications.

[0100] Specifically, the decision unit 45 first uses the driver reaction model 47 to determine the optimal parameter set, and then determines the period until the next update of the optimal parameter set, i.e., the update period. Additionally, the decision unit 45 generates multiple data collection parameter sets to be used during the update period. Here, the update period can be defined as a predetermined time period or a period during which the notification unit 34 sends a predetermined number of notifications.

[0101] The decision unit 45 also generates execution order information, which determines the order in which the notification unit 34 of the attention reminder device 3 executes notifications using the optimal parameter set and notifications using any parameter set from the multiple data collection parameter sets during the update period.

[0102] For example, the decision unit 45 determines the number of notifications using the data collection parameter set (hereinafter referred to as data collection notifications) and the number of notifications using the optimal parameter set (hereinafter referred to as application notifications) that should be executed during the aforementioned update period. Then, the decision unit 45 generates execution order information that randomly determines the order of data collection notifications and application notifications. The execution order information can, for example, specify whether the nth notification (n>1) from the beginning of the aforementioned update period is a data collection notification or an application notification, such as the first notification from the beginning of the aforementioned update period being a data collection notification, the second and third notifications being application notifications, the fourth being an application notification, and so on.

[0103] Here, the number of data collection notifications and the number of application notifications determined when generating the aforementioned execution order information can be determined, for example, based on the learning status of the driver reaction model 47 (e.g., the amount of reaction information used in previous learning). More specifically, the number of data collection notifications and the number of application notifications can be determined such that the ratio of the number of data collection notifications to the number of application notifications decreases as the driver reaction model 47 progresses in its learning (e.g., as the amount of reaction information used in previous learning increases).

[0104] The decision unit 45 sends the aforementioned decision or generated optimal parameter set, data collection parameter set, execution order information, and update period information to the attention reminder device 3. Thus, the update period for the aforementioned decision begins, and the notification unit 34 of the attention reminder device 3 executes data collection notifications or application notifications according to the execution order information. Furthermore, the behavior recording unit 36 ​​of the attention reminder device 3 saves a reaction record indicating whether the driver has reacted in the reaction log 37 when data collection notifications or application notifications are issued. Moreover, if the number of notifications occurring during the update period exceeds the number of notifications included in the execution order information, the notification unit 34 can return to the beginning of the execution order information and reuse that execution order information repeatedly.

[0105] At the end of the aforementioned update period, the behavior recording unit 36 ​​sends the reaction records stored in the reaction log 37 to the optimization server 4. The reaction information recording unit 43 of the optimization server 4 receives the reaction records and generates reaction information. In addition, the learning unit 44 of the optimization server 4 uses the generated reaction information to perform additional learning on the driver reaction model 47.

[0106] Next, the decision unit 45 uses the driver reaction model 47 after the additional learning described above to determine a new optimal parameter set (i.e., update the optimal parameter set), and determines or generates the next update period, the parameter set for data collection, and the execution order information. The decision unit 45 sends these new optimal parameter sets, the parameter set for data collection, the execution order information, and the update period information to the attention reminder device 3, and begins the operation of the new update period.

[0107] Subsequently, similarly as described above, during repeated update periods, data collection notifications and application notifications are executed in combination within each update period (i.e., data collection actions and application actions are repeated within each update period).

[0108] [Note the action flow in the reminder system]

[0109] Next, the action steps of the attention reminder system 1 will be explained. Figure 7 This is a flowchart illustrating the steps of the attention reminder method executed by the computer of attention reminder system 1 (i.e., the processor 30 of attention reminder device 3 and the processor 40 of optimization server 4). Repeated execution. Figure 7 The processing is shown.

[0110] When processing begins, firstly, the decision unit 45 of the optimization server 4 uses the driver reaction model 47 to calculate and determine the set of notification parameters that is presumed to maximize the driver's reaction ratio, i.e., the optimal parameter set (S100). Additionally, the decision unit 45 determines the update period until the next update of the aforementioned optimal parameter set and the notification parameter set specifically used for data collection during this update period, i.e., multiple data collection parameter sets (S102). The decision unit 45 sends the aforementioned determined or generated optimal parameter set, data collection parameter sets, and update period information to the attention reminder device 3. Thus, the update period begins in the attention reminder device 3. The decision unit 45 may also generate execution order information that determines the usage order of the optimal parameter set and data collection parameter sets during the update period when sending the aforementioned optimal parameter set, and send this information to the attention reminder device 3.

[0111] Next, note that the notification unit 34 of the reminder device 3 executes, for example, an arbitrary notification (S104) that uses the data collection parameter set (i.e., a data collection notification) and an application notification that uses the optimal parameter set (i.e., an application notification) according to the execution order information described above. According to each notification parameter set, when the line-of-sight dwell time Ts satisfies the above equation (2), the HMI device 6 issues a notification according to the salience level SL given by equation (3).

[0112] Then, whenever a notification is output from the HMI device 6, the behavior recording unit 36 ​​of the attention reminder device 3 saves and stores the record of whether the driver has reacted, i.e., the reaction record, in the memory 31 as the reaction log 37 (S104) based on the judgment result of the judgment unit 35 regarding whether there is a reaction behavior. Here, whether there is a reaction behavior in response to the notification is determined based on whether the driver has performed a reaction behavior within a first predetermined time period (e.g., 2 seconds) from the start of the notification.

[0113] The notification unit 34 can also repeatedly send notifications while changing the notification parameter set if there is no response within the first predetermined time, until the driver takes a response. For example, the notification unit 34 determines whether the driver has taken a response within a second predetermined time (e.g., 10 seconds) from the date of the notification (S108). Furthermore, if the driver does not take a response within the second predetermined time (S108, no), the notification unit 34 can return to step S104 and repeat the process.

[0114] On the other hand, when the driver reacts within the second specified time (S108, Yes), the notification unit 34 determines whether the update period has ended (S110). If the update period has not ended (S110, No), the notification unit 34 returns to step S104 and repeats the process.

[0115] On the other hand, at the end of the update period (S110, Yes), the behavior recording unit 36 ​​sends the reaction records collected and stored in the reaction log 37 during the update period to the optimization server 4. Based on the received reaction records, the reaction information recording unit 43 of the optimization server 4 generates reaction information 46 and stores it in the memory 41 (S112). The learning unit 44 uses the reaction information 46 stored in the memory 41 to perform additional learning on the driver reaction model 47, and this process ends.

[0116] After this process is completed, remind System 1 to restart the process from step S100 and repeat the above data collection and application actions.

[0117] [Other Implementation Methods]

[0118] In the above implementation, the set of parameters for data collection is determined during each update period, but a portion or all of the set of parameters for data collection can also be used across multiple update periods.

[0119] Furthermore, in the above embodiment, the decision unit 45 randomly determines the order in which the data collection parameter set and the optimal parameter set are used to generate execution order information, and the notification unit 34 uses any one of the multiple data collection parameter sets and the optimal parameter set to send a notification according to the execution order information. Alternatively, the decision unit 45 may not generate execution order information, and the notification unit 34 may randomly determine which of the multiple data collection parameter sets and the optimal parameter set to use for notification.

[0120] In addition, in the above-described embodiment, the decision unit 45 determines multiple sets of data collection parameters during each update, but it may also determine at least one set of data collection parameters during each update.

[0121] Furthermore, in the above embodiment, the decision unit 45 of the optimization server 4 selectively executes data collection operations using a data collection parameter set and application operations using an optimal parameter set during the update period. However, the data collection operations and application operations do not necessarily need to be selectively executed by mixing them during the update period; they can also be executed separately in divided periods. The aforementioned period can be set to a predetermined time length or a period during which the notification unit 34 sends a predetermined number of notifications.

[0122] Furthermore, in the above embodiment, the decision unit 45 uses a driver reaction model 47, which is based on reaction information generated by the reaction information recording unit 43 and has undergone machine learning, to determine the optimal parameter set. However, while using the driver reaction model 47 is one method for determining the optimal parameter set, the decision unit 45 can also use any other method that does not use the driver reaction model 47 to determine the optimal parameter set based on the relationship between the notification parameter set represented by the reaction information and the reaction ratio.

[0123] Other methods could include multivariate analysis of the parameters that make up the set of notification parameters (e.g., threshold a and constants b and c) and the response ratio (e.g., analysis of covariance).

[0124] Furthermore, in the above embodiment, the decision unit 45 randomly determines the values ​​of each parameter constituting the notification parameter set to determine the data collection parameter set. However, the method for determining the data collection parameter set is not limited to this method (random value determination). For example, the decision unit 45 may also use a genetic algorithm to perform operations such as selection, crossover, and / or mutation on the value set of each parameter constituting a notification parameter set to generate a new version of the value set, and determine the notification parameter set with the generated version of the value set as the data collection parameter set.

[0125] Alternatively, the decision unit 45 may, based on the response information obtained so far, use a probabilistic method such as optimal arm identification to determine the set of notification parameters that presumably result in a larger response ratio as a reward as the set of parameters for data collection and / or the optimal set of parameters.

[0126] In addition, in the above embodiment, there is one driver reaction model 47, but in the case where multiple drivers take turns driving the vehicle 2, the driver reaction model 47 can also be made for each driver.

[0127] Furthermore, in the above-described embodiment, the attention reminder system 1 comprises an attention reminder device 3 mounted on the vehicle 2 and an optimization server 4 placed externally on the vehicle 2, but is not limited to this structure. All functional elements of the optimization server 4 and the attention reminder device 3 may also be located within the vehicle 2. For example, the reaction information recording unit 43, learning unit 44, and decision unit 45 of the optimization server 4 may be located within the processor 30 of the attention reminder device 3, while the driver reaction model 47 is stored in the memory 31, thus implementing the attention reminder system 1 solely through the attention reminder device 3.

[0128] Alternatively, all the functional elements of the attention reminder device 3 and the optimization server 4 in the above embodiments can be installed on a server device external to the vehicle 2. For example, the gaze measurement unit 33, notification unit 34, judgment unit 35, and behavior recording unit 36 ​​of the attention reminder device 3 can also be installed in the processor 40 of the optimization server 4. In this case, the optimization server 4 and various devices of the vehicle 2 (HMI device 6, object detection device 7, external camera 8, etc.) can be connected communicatively.

[0129] This invention is not limited to the structure of the above embodiments and can be implemented in various ways without departing from its spirit.

[0130] [Structure supported by the above embodiments]

[0131] The above implementation supports the following structure.

[0132] (Structure 1) An attention reminder system comprising: a gaze measurement unit that measures the gaze dwell time of a driver of a vehicle; and a notification unit that, when the measured gaze dwell time exceeds a predetermined threshold, notifies the driver via an HMI device of a predetermined salience level, wherein the attention reminder system further comprises: a decision unit that determines a notification parameter set, the notification parameter set including parameters specifying the timing of the notification and parameters specifying the salience level of the notification issued by the HMI device; and a behavior recording unit that records the notification based on the timing of the notification and the salience level indicated by the determined notification parameter set. When the notification is issued, the behavior recording unit records whether the driver has a reaction to the notification; and the reaction information recording unit calculates the reaction ratio for different notification parameter sets and stores the reaction information formed by associating the notification parameter sets with the reaction ratios in a storage device. The reaction ratio is the ratio of the number of times the driver has reacted to the notification to the number of times the notification is issued. The decision unit calculates and determines the optimal parameter set based on the relationship between the notification parameter sets represented by the reaction information and the reaction ratios. The optimal parameter set is the notification parameter set that is presumed to have the largest reaction ratio.

[0133] According to the attention reminder system of Structure 1, since the degree of acceptability of the driver to the notification related to the distraction state is evaluated by setting the driver's response rate to the notification, and the notification parameter set for the manner of specifying the notification is calculated in a way that maximizes the response rate, it is possible to notify the driver of the possibility of a distraction state in a manner that is easily acceptable to the driver of the vehicle.

[0134] (Structure 2) According to the attention reminder system of Structure 1, the decision unit further determines an update period and at least one set of data collection parameters, the update period being the period up to the optimal set of parameters determined in the next update, and the at least one set of data collection parameters being the notification parameter set specifically used for data collection.

[0135] During the update, the decision-making unit, in cooperation with the notification unit, the behavior recording unit, and the reaction information recording unit, selectively notifies the use of the determined optimal parameter set and data collection parameter set, and obtains reaction information regarding the used optimal parameter set and data collection parameter set.

[0136] According to the attention reminder system of Structure 2, data collection and application actions are performed separately during the same update period, thus suppressing the driver's discomfort with notifications of the use of the data collection parameter set.

[0137] (Structure 3) According to the attention reminder system of Structure 2, the update period is determined to be a period of a predetermined length of time or a period during which the notification is made a predetermined number of times by the notification unit.

[0138] According to the attention reminder system of Structure 3, the amount of reaction record data collected during the update period can be controlled by the length of the update period or the number of notifications.

[0139] (Structure 4) According to the reminder system of Structure 2 or 3, wherein the decision unit generates execution order information, which specifies the order in which the notification unit executes the notification using the optimal parameter set and the notification using the data collection parameter set among the multiple notifications during the update period, and the notification unit selects and uses the optimal parameter set and the data collection parameter set to perform the notification according to the execution order information.

[0140] According to the attention reminder system in Structure 4, the notification department can select the notification parameter set according to the execution order information, thus simplifying the process.

[0141] (Structure 5) An attention reminder system according to any one of Structures 1-4, wherein the attention reminder system has a learning unit that, based on the reaction information, enables a driver reaction model to learn the relationship between the notification parameter set and the reaction ratio through machine learning, and the decision unit uses the driver reaction model to determine the optimal parameter set that is presumed to maximize the reaction ratio.

[0142] Based on the attention reminder system of Structure 5, the optimal set of parameters presumed to maximize the response ratio can be easily determined.

[0143] (Structure 6) A attention reminder system according to any one of Structures 1-5, wherein the gaze measurement unit measures the driver's gaze dwell time when the vehicle enters and / or passes through the intersection.

[0144] According to the attention reminder system in Structure 6, at intersections where the driver's attention becomes more important, notifications related to the driver's distracted state can prompt the driver to focus on driving behavior.

[0145] (Structure 7) An attention reminder system according to any one of Structures 1 to 6, wherein the gaze measurement unit measures the driver’s gaze dwell time by tracking the driver’s gaze based on a facial image from a camera located in the vehicle’s interior.

[0146] According to the attention reminder system of structure 7, the duration of gaze can be easily measured by eye tracking based on camera images without the use of special equipment.

[0147] (Structure 8) An attention reminder system according to any one of Structures 1-7, wherein the presence or absence of the reaction behavior is determined based on whether the amount of eye movement between two temporally consecutive fixation points in the driver's eye movement is greater than a predetermined amount, and / or whether the size of the driver's eye movement range is greater than a predetermined size.

[0148] According to the attention reminder system of structure 8, it is possible to easily determine whether the driver responds to notifications related to a distracted state based on the way the driver's gaze moves in the vehicle.

[0149] (Structure 9) An attention reminder system according to any one of Structures 1-8, wherein the HMI device includes a tactile HMI device, an auditory HMI device and / or a visual HMI device.

[0150] According to the attention reminder system of structure 9, various HMI devices can be used to notify the driver in various ways.

[0151] (Structure 10) A attention reminder system according to any one of Structures 1-9, wherein the attention reminder system includes a server device that is communicatively connected to the vehicle, the server device having the decision unit and the reaction information recording unit, and the vehicle having the line-of-sight measurement unit and the notification unit.

[0152] The attention reminder system according to structure 10 can be distributed to server devices and vehicle-mounted devices for processing, thus reducing the load on vehicle-mounted devices and lowering vehicle costs, for example.

[0153] (Structure 11) A attention reminder system according to any one of Structures 1-9, wherein the decision unit, the reaction information recording unit, the line of sight measurement unit and the notification unit are disposed in the vehicle.

[0154] According to the attention reminder system of structure 11, the entire attention reminder system can be installed in the vehicle, thus enabling more stable processing that is not affected by the tension of communication lines outside the vehicle.

[0155] (Structure 12) A attention reminder method, executed by a computer of an attention reminder system that notifies a driver of a vehicle via an HMI device, the attention reminder method comprising: a gaze measurement step, measuring the gaze dwell time of the driver of the vehicle; and a notification step, notifying the driver via the HMI device when the measured gaze dwell time exceeds a predetermined threshold, wherein the attention reminder method further comprises: a decision step, determining a notification parameter set, the notification parameter set including parameters specifying the timing of the notification and parameters specifying the salience level of the notification issued by the HMI device; and a behavior recording step, recording the behavior according to the determined notification parameter set. When the notification is issued in the notification step according to the timing and significance level shown, it is recorded whether the driver has a reaction to the notification; and in the reaction information recording step, a reaction ratio is calculated for different notification parameter sets, and the reaction information formed by associating the notification parameter sets with the reaction ratio is stored in a storage device, wherein the reaction ratio is the ratio of the number of times the driver has reacted to the notification to the number of notifications; and in the decision step, an optimal parameter set is calculated and determined according to the relationship between various notification parameter sets and the reaction ratio, wherein the optimal parameter set is the notification parameter set that is presumed to have the largest reaction ratio.

[0156] According to the attention reminder method of structure 12, since the degree of acceptability of the driver to the notification related to the distraction state is set as the driver's response rate to the notification, and the notification parameter set for the manner of specifying the notification is calculated in a way that maximizes the response rate, it is possible to notify the driver of the possibility of a distraction state in a manner that is easily acceptable to the driver of the vehicle.

Claims

1. A attention reminder system, the attention reminder system having: The line-of-sight measurement unit measures the driver's line-of-sight persistence time in a vehicle; and The notification department, when the measured gaze persistence time exceeds a predetermined threshold, notifies the driver via the HMI device of a predetermined salience level. in, This attention reminder system also has: The decision unit determines the notification parameter set, which includes parameters specifying the timing of the notification and parameters specifying the salience level of the notification issued by the HMI device. When the notification unit issues the notification based on the timing and salience level indicated by the determined notification parameter set, the behavior recording unit records whether the driver has a reaction to the notification. as well as The reaction information recording unit calculates a reaction ratio for each of the various notification parameter sets, and stores the reaction information, which is the association between the notification parameter sets and the reaction ratio, in a storage device. The reaction ratio is the ratio of the number of times the driver reacts to the notification to the total number of notifications. The decision unit calculates and determines the optimal parameter set based on the relationship between the notification parameter set represented by the reaction information and the reaction ratio. The optimal parameter set is the notification parameter set that is presumed to have the largest reaction ratio.

2. The attention reminder system according to claim 1, wherein, The decision-making unit also determines the update period and at least one set of data collection parameters, the update period being the period up to the optimal parameter set determined in the next update, and the at least one set of data collection parameters being the notification parameter set specifically used for data collection. During the update, the decision-making unit, in cooperation with the notification unit, the behavior recording unit, and the reaction information recording unit, selectively notifies the use of the determined optimal parameter set and data collection parameter set, and obtains reaction information regarding the used optimal parameter set and data collection parameter set.

3. The attention reminder system according to claim 2, wherein, The update period is determined to be either a predetermined time period or a period during which the notification department sends a predetermined number of notifications.

4. The attention reminder system according to claim 2, wherein, The decision unit generates execution order information, which specifies the order in which the notification unit executes the notifications using the optimal parameter set and the notifications using the data collection parameter set among multiple notifications during the update period. The notification unit selects and uses the optimal parameter set and the data collection parameter set according to the execution order information to perform the notification.

5. The attention reminder system according to claim 1, wherein, The attention alert system includes a learning unit that, based on the reaction information, enables the driver's reaction model to learn the relationship between the notification parameter set and the reaction ratio through machine learning. The decision unit uses the driver reaction model to determine the optimal set of parameters that is presumed to maximize the reaction ratio.

6. The attention reminder system according to claim 1, wherein, The line-of-sight measurement unit measures the driver's line-of-sight dwell time when the vehicle enters and / or passes through the intersection.

7. The attention reminder system according to claim 1, wherein, The gaze measurement unit measures the driver's gaze dwell time by tracking the driver's gaze based on a facial image obtained from a camera located inside the vehicle.

8. The attention reminder system according to claim 1, wherein, The presence or absence of the reaction behavior is determined based on whether the amount of eye movement between two temporally consecutive fixation points in the driver's eye movement is greater than a predetermined amount, and / or whether the size of the driver's eye movement range is greater than a predetermined size.

9. The attention reminder system according to claim 1, wherein, The HMI device includes a tactile HMI device, an auditory HMI device, and / or a visual HMI device.

10. The attention reminder system according to any one of claims 1 to 9, wherein, The attention alert system includes a server device that is communicatively connected to the vehicle. The server device includes the decision-making unit and the reaction information recording unit. The vehicle is equipped with the sight measurement unit and the notification unit.

11. The attention reminder system according to any one of claims 1 to 9, wherein, The decision unit, the reaction information recording unit, the line-of-sight measurement unit, and the notification unit are disposed in the vehicle.

12. A method for reminding attention, the method being executed by a computer of an attention reminder system that notifies the driver of a vehicle via an HMI device, the method comprising: The line-of-sight measurement step measures the time the driver's line of sight remains in the vehicle. as well as The notification step involves notifying the driver via the HMI device when the measured gaze dwell time exceeds a predetermined threshold. This attention reminder method also has the following features: The decision-making process involves determining a set of notification parameters, which includes parameters specifying the timing of the notification and parameters specifying the salience level of the notification issued by the HMI device. The behavior recording step, when the notification is made in the notification step according to the timing and salience level represented by the determined set of notification parameters, records whether the driver's reaction behavior to the notification exists; and The reaction information recording step involves calculating a reaction ratio for each of the different notification parameter sets, and storing the reaction information, which is the association between the notification parameter set and the reaction ratio, in a storage device. The reaction ratio is the ratio of the number of times the driver reacted to the notification to the total number of notifications. In the decision step, an optimal set of parameters is calculated and determined based on the relationship between various sets of notification parameters and the response ratio. This optimal set of parameters is the set of notification parameters presumed to have the largest response ratio.

Images