Reflection prediction device and reflection prediction method

Abstract

This reflection prediction device (100, 200, 300, 400) comprises: an operator information acquisition unit (101) that acquires operator information which includes information relating to the actions of an operator of a vehicle; and a prediction unit (104) that uses a learning model trained with, as training data, a data set including operator information acquired by the operator information acquisition unit (101) and information indicating whether or not a reflection relating to an oral cavity has occurred, to predict the occurrence of a reflection relating to the oral cavity of the operator on the basis of input, to the learning model, of the operator information acquired by the operator information acquisition unit (101).

Description

Reflection prediction device and reflection prediction method

[0001] The present disclosure relates to a reflection prediction device and a reflection prediction method.

[0002] A driving assistance device that determines the driver's level of alertness based on a facial image of the driver has been disclosed (see, for example, Patent Document 1). This driving assistance device determines the driver's level of alertness based on the result of detecting the driver's yawn based on the facial image of the driver.

[0003] JP 2013-156707 A

[0004] However, oral reflexes such as yawning can themselves affect the driving operations of the vehicle driver, so if driving assistance is initiated based on the occurrence of an oral reflex, there is a problem that the driving assistance may not be initiated until after the driving operations have been affected by the oral reflex.

[0005] The present disclosure was made in response to the recognition of the above-mentioned problem, and aims to provide a reflex prediction device and a reflex prediction method that can predict a driver's oral cavity-related reflexes.

[0006] The reflex prediction device of the present disclosure is characterized by comprising: a driver information acquisition unit that acquires driver information including information regarding the vehicle driver's actions; and a prediction unit that predicts the occurrence of a driver's oral reflex based on input of the driver information acquired by the driver information acquisition unit into a learning model that has been trained using as training data a dataset including the driver information acquired by the driver information acquisition unit and information indicating whether an oral reflex has occurred.

[0007] The reflex prediction device according to the present disclosure can predict the occurrence of a reflex related to the driver's oral cavity by using a learning model that is trained based on information about the driver's actions.

[0008] 5A is a diagram showing a feature point group of an image of a driver used as driver information by the reflection prediction device 100 according to the first embodiment, and FIG. 5B is a diagram showing image information of the face of the driver used as driver information by the reflection prediction device 100 according to the first embodiment. A graph showing an example of a prediction result by the reflection prediction device according to the first embodiment. A block diagram showing a schematic configuration of a reflection prediction system according to a second embodiment. A flowchart showing an example of a process performed by the reflection prediction device according to the second embodiment. A block diagram showing a schematic configuration of a reflection prediction system according to a third embodiment. A flowchart showing an example of a process performed by the reflection prediction device according to the third embodiment. A graph showing an example of a prediction result by the reflection prediction device according to the third embodiment and a timing of driving assistance by the driving assistance device. 12A is a plan view showing the positional relationship between a host vehicle and another vehicle before driving assistance is performed by a driving assistance device according to embodiment 3, and FIG. 12B is a plan view showing the positional relationship between a host vehicle and another vehicle after driving assistance is performed by the driving assistance device according to embodiment 3. A block diagram showing a schematic configuration of a reflection prediction system according to embodiment 4. A flowchart showing an example of processing performed by a reflection prediction device according to embodiment 4.

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1. First, referring to FIG. 1, a schematic configuration of a reflex prediction system 1 according to embodiment 1 will be described. FIG. 1 is a block diagram showing a schematic configuration of the reflex prediction system 1 according to embodiment 1. The reflex prediction system 1 is a system for predicting the occurrence of a driver's oral reflex while driving a vehicle. As shown in FIG. 1, the reflex prediction system 1 includes a driver information detection device 10, a reflex prediction device 100, and an external device 70, which are connected wirelessly or by wire so as to be able to communicate with each other. Note that in embodiment 1, a "vehicle" simply refers to a specific vehicle equipped with the reflex prediction system 1, and a "driver" simply refers to a driver who is in the vehicle and drives the vehicle. Furthermore, in embodiment 1, a "reflex" of a driver refers to a short-term involuntary movement that occurs unconsciously or independently of the driver's will.

[0010] The driver information detection device 10 detects information about the driver while driving a vehicle. For example, the driver information detection device 10 is configured with an imaging device (not shown) that captures image information of the driver sitting in the driver's seat by capturing images of the interior of the vehicle, and an image recognition unit (not shown) that detects various information about the driver through image recognition based on the image information captured by the imaging device. For example, the imaging device of the driver information detection device 10 captures images of the interior of the vehicle at a preset frame rate to capture image information about the driver as video information. Furthermore, for example, the driver information detection device 10 detects information about the driver, such as the driver's posture, the degree of eyelid opening, the driver's gaze direction, the driver's facial expression, and the degree of mouth opening, through image recognition of the video image information. Furthermore, for example, the driver information detection device 10 detects information about the driver's behavior, such as changes in the driver's posture, the degree of eyelid opening, the driver's gaze direction, the driver's facial expression, and the degree of mouth opening, through image recognition of the video image information.

[0011] In addition to or instead of the above configuration, the driver information detection device may be configured to detect driver information using various sensors. For example, the driver information detection device may be configured to detect driver information using physiological sensors such as a body temperature sensor, a sweat sensor, a heart rate sensor, and a pulse sensor that detect information indicating the physiological state of the driver, a microphone for acquiring audio information indicating what the driver is saying, an infrared sensor for detecting the driver, an ultrasonic sensor for detecting the driver, a millimeter-wave sensor for detecting the driver, a LiDAR (Light Detection and Ranging) sensor for detecting the driver, a sensor for detecting driving operations by the driver, etc. For example, sensors that detect driving operations by the driver include sensors that detect the operation of the steering wheel, shift lever, accelerator pedal, brake pedal, etc.

[0012] In addition to the above configuration, the driver information detection device may also have an identification unit that identifies each driver when there are multiple drivers driving the same vehicle. For example, the identification unit identifies each driver based on differences in the detected driver information between the drivers. Specifically, the identification unit identifies each driver by image recognition based on image information of the driver. The driver information detection device 10 outputs the detected information to the reflection prediction device 100. Note that the driver information detection device 10 may be configured to output information indicating the identification result of each driver to the reflection prediction device 100 in addition to the detected driver information. Furthermore, the driver information detection device 10 may be configured to output image information of the driver acquired by photographing the driver directly to the reflection prediction device 100.

[0013] The external device 70 acquires information from the reflection prediction device 100. For example, the external device 70 is configured by a storage device that stores the information acquired from the reflection prediction device 100. Furthermore, for example, the external device 70 is configured by a control device that controls various devices of the vehicle based on the information acquired from the reflection prediction device 100. Furthermore, for example, the external device 70 is configured by a display device that displays the information acquired from the reflection prediction device 100.

[0014] As shown in FIG. 1 , the reflex prediction device 100 includes a driver information acquisition unit 101 and a prediction unit 104. The driver information acquisition unit 101 acquires driver information, which is information about the driver's actions while driving a vehicle, based on information from the driver information detection device 10. For example, the driver information acquisition unit 101 acquires information indicating the movement of the driver's upper body, including the head, as the driver information. For example, the driver information acquisition unit 101 acquires information indicating the movement of the driver's upper body, including the head, by quantifying the movement of the driver's upper body, including the face direction, gaze, eyelids, eyebrows, nose, mouth, shoulders, arms, chest, etc., movement associated with breathing, movement due to heartbeat, movement due to pulse, etc. As the driver information, the driver information acquisition unit 101 also acquires information indicating the movement of the driver, including changes in the driver's facial expression. For example, the driver information acquisition unit 101 acquires information indicating the change in the driver's facial expression by quantifying the characteristics of the change in the driver's facial expression.

[0015] Furthermore, for example, the driver information acquisition unit 101 acquires, as the driver information, information indicating the driver's movements, including specific precursory actions of the driver's oral reflexes. For example, the driver's oral reflexes include sneezing, yawning, coughing, hiccups, etc. Note that hiccups can be considered an oral reflex because they often involve contraction of the oral muscles. Generally, these oral reflexes are preceded by precursory actions before the characteristic actions of the reflex occur.

[0016] For example, premonitory actions for a sneeze include opening the mouth wide as if taking a deep breath, breathing in several times with the mouth open, closing the eyes, bringing a hand close to the mouth, and expanding the chest. Premonitory actions for a yawn include a sleepy expression, slowly opening the mouth wide, and slowly closing the eyes. Premonitory actions for a cough include slightly opening the mouth before breathing in, squinting the eyes, changing the direction of the face, and a pained expression. Premonitory actions for a hiccup include slight eye movements and up-and-down shoulder movements. For example, the driver information acquisition unit 101 acquires information indicating premonitory actions by quantifying the characteristics of these premonitory actions for oral cavity-related reflexes.

[0017] Further, for example, the driver information acquisition unit 101 acquires, as the driver information, information indicating the driver's movements, including information regarding the voice uttered by the driver. Note that the information regarding the voice uttered by the driver is uttered by the driver moving his / her lungs, vocal cords, and oral cavity, and therefore can be considered information regarding the driver's actions. The driver information acquisition unit 101 may acquire, as the driver information, sound information indicating the voice uttered by the driver, or may acquire information other than sound information obtained by speech recognition based on the voice uttered by the driver, or may acquire phoneme information. Because oral reflexes are often accompanied by characteristic vocalizations, the prediction accuracy by the prediction unit 104 can be improved by including information regarding the voice uttered by the driver in the driver information.

[0018] Furthermore, the driver information acquisition unit 101 acquires, as the driver information, information indicating the movement of the driver, including information indicating the voice uttered by the driver. Note that the driver information acquired by the driver information acquisition unit 101 may include all of the information indicating the movement of the driver's upper body, the information indicating changes in facial expression, the information indicating predictive actions, and the information indicating the voice, or may be composed of only some of these pieces of information. Furthermore, when there are multiple drivers driving the same vehicle, the driver information acquisition unit may be configured to identify the multiple drivers based on the information from the driver information detection device 10 and then acquire the driver information of each driver in an identifiable state, or may be configured to acquire the driver information without identifying the multiple drivers.

[0019] The prediction unit 104 uses a learning model trained using as training data a dataset including the driver information acquired by the driver information acquisition unit 101 and information indicating whether or not an oral reflex has occurred, to predict whether or not the driver's oral reflex will occur, based on the input of the driver information acquired by the driver information acquisition unit 101 to the learning model. In other words, the prediction unit 104 learns as training data a dataset including the driver information acquired by the driver information acquisition unit 101 and information indicating whether or not an oral reflex has occurred, and predicts whether or not the driver's oral reflex will occur, using a learning model in which the driver information is an input variable and whether or not the driver's oral reflex will occur is an output variable.

[0020] For example, the prediction unit 104 learns, as teacher data, a dataset including driver information for a period from the time the driver's oral reflex occurred until a specific time ago and driver information for a period from the time the driver's oral reflex did not occur until the specific time ago, and predicts the occurrence of the driver's oral reflex using a learning model that infers the probability of the driver's oral reflex occurring within a specific time based on the input of the driver information acquired by the driver information acquisition unit 101. For example, the prediction unit 104 learns, as teacher data, a dataset including driver information for a period from the time the driver's oral reflex occurred until the specific time ago and driver information for a period from the time the driver's oral reflex did not occur until the specific time ago, and predicts the occurrence of the driver's oral reflex using a learning model that infers the probability of the driver's oral reflex occurring within 15 seconds based on the input of the driver information acquired by the driver information acquisition unit 101.

[0021] Furthermore, for example, the prediction unit 104 predicts the time until the occurrence of an oral reflex in the driver. Specifically, the prediction unit 104 learns, as training data, a data set including driver information for a period from the time when the oral reflex in the driver occurred until a specific time ago and driver information for a period from the time when the oral reflex in the driver did not occur until the specific time ago, and predicts the time until the occurrence of an oral reflex in the driver using a learning model that infers the time until the occurrence of an oral reflex in the driver based on input of the driver information acquired by the driver information acquisition unit 101.

[0022] Furthermore, for example, the prediction unit 104 predicts the time until the occurrence of the driver's oral reflex. Specifically, the prediction unit 104 learns, as training data, a data set including driver information for a period from the time the driver's oral reflex occurred until a specific time ago, the time the driver's oral reflex ended, and driver information for a period from the time the driver's oral reflex did not occur until the specific time ago, and predicts the time until the driver's oral reflex occurs using a learning model that infers the time until the driver's oral reflex ends based on input of the driver information acquired by the driver information acquisition unit 101.

[0023] Further, for example, the prediction unit 104 predicts whether or not a driver's oral reflex will occur and the type of the driver's oral reflex that will occur, using a learning model trained using as training data a dataset including the driver information acquired by the driver information acquisition unit 101 and information indicating which or none of a plurality of preset types of oral reflexes has occurred. In other words, the prediction unit 104 predicts whether or not a driver's oral reflex will occur and which of the preset multiple reflexes the driver's oral reflex that will occur is, using a learning model trained using as training data a dataset including the driver information acquired by the driver information acquisition unit 101 and information indicating which or none of a plurality of preset oral reflexes has occurred. Specifically, the prediction unit 104 predicts which of the preset multiple reflexes is a sneeze, yawn, cough, or hiccup. The prediction unit 104 may be configured to predict the occurrence of only one of the sneeze, yawn, cough, and hiccup reflexes. The prediction unit 104 outputs the prediction result to the external device 70 .

[0024] In addition, since oral reflexes may occur continuously over a relatively short period of time, the driver information input in the learning and inference stages of the learning model may include actions in which the driver's oral reflexes are occurring.

[0025] Next, the hardware configuration of the reflection prediction device 100 will be described with reference to Figures 2 and 3. Figure 2 is a diagram showing an example of the hardware configuration of the reflection prediction device 100, and Figure 3 is a diagram showing an example of the hardware configuration of the reflection prediction device 100 that is different from that shown in Figure 2. For example, as shown in Figure 2, the reflection prediction device 100 is a computer having a processor 100a, a memory 100b, and an I / O port 100c, and is configured so that the processor 100a reads and executes a program stored in the memory 100b.

[0026] 3, the reflection prediction device 100 is a computer that has a processing circuit 100d, which is dedicated hardware, and an I / O port 100c, and executes a program. The processing circuit 100d is configured, for example, by a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. Each function of the reflection prediction device 100 is realized by the processor 100a or the processing circuit 100d, which is dedicated hardware, executing a program. Note that the reflection prediction device 100 may also include hardware other than those described above, such as a hardware timer.

[0027] Next, with reference to FIGS. 1 and 4 to 6, the details of the processing performed by the reflection prediction device 100 will be described. FIG. 4 is a flowchart showing an example of the processing performed by the reflection prediction device 100 according to the first embodiment. As shown in FIG. 4, when the reflection prediction device 100 starts the processing, it first acquires driver information (step ST02). In this processing, the reflection prediction device 100 acquires driver information related to the driver's actions using the driver information acquisition unit 101 based on information from the driver information detection device 10. For example, the reflection prediction device 100 acquires information from the driver information detection device 10 at specific time intervals. Note that the driver information acquisition unit 101 may be configured to acquire, from the driver information detection device 10, information that quantifies the presence or absence of a driver's action, the type of action, the number of actions, the magnitude of the action, the time the action was performed, and the like, as the driver information, or may be configured to acquire image information acquired by photographing the driver as is. When the driver information acquisition unit 101 is configured to acquire image information from the driver information detection device 10, the driver information acquisition unit 101 extracts characteristics of the driver's movements from the acquired image information and acquires information indicating the driver's movements based on the extracted characteristics.

[0028] FIG. 5A is a diagram showing a feature point group of an image of a driver that the reflex prediction device 100 according to embodiment 1 uses as driver information, and FIG. 5B is a diagram showing image information of the driver's face that the reflex prediction device 100 according to embodiment 1 uses as driver information. For example, as shown in FIG. 5A , the reflex prediction device 100 generates a wireframe model from the image information of the driver acquired from the driver information detection device 10 and extracts a feature point group based on the generated wireframe model, thereby acquiring feature point group information as driver information. Also, for example, as shown in FIG. 5B , the reflex prediction device 100 acquires, as driver information, information indicating the shape and position of specific parts of the driver's face from the image information of the driver. Specifically, the reflex prediction device 100 acquires, as driver information, information indicating the degree of opening of the driver's mouth and eyelids from the image information of the driver.

[0029] The information acquired by the reflex prediction device 100 as driver information may be information indicating the shape and position of the driver's upper and lower eyelids, as well as the shape and position of the mouth, information indicating the shape and position of the driver's hands, information indicating the amount of movement of each part of the driver, information indicating the speed at which each part of the driver moves, or may include multiple of these pieces of information.

[0030] After performing the process of step ST02, the reflex prediction device 100 predicts the occurrence of a reflex related to the driver's oral cavity based on the driver information acquired in the process of step ST01 (step ST03). In this process, the reflex prediction device 100 inputs the driver information acquired in the process of step ST02 into a learning model, and predicts whether a reflex related to the driver's oral cavity will occur based on the output result of the learning model. For example, the reflex prediction device 100 makes a prediction by inputting the driver information acquired by the driver information acquisition unit 101 into the learning model each time the driver information acquisition unit 101 acquires the driver information based on image information of the driver at a specific time, into the learning model each time the driver information acquisition unit 101 acquires the driver information. When the reflex prediction device 100 is configured in this way, the learning model makes inferences according to the multiple inputs of driver information based on the multiple inputs of driver information.

[0031] The reflex prediction device 100 may be configured to collectively input driver information for a specific period acquired multiple times from the driver information detection device 10 by the driver information acquisition unit 101 into a learning model and perform prediction based on the driver information for the specific period. For example, the reflex prediction device 100 may be configured to extract a specific period of time series data of the driver information acquired by the driver information acquisition unit 101 from the driver information detection device 10 as partial time series data, input the extracted partial time series data into a learning model to perform prediction, and set a range of partial time series data to be input to the learning model next by sliding the range of the partial time series data. For example, the partial time series data may be composed of two pieces of information: driver information acquired based on image information acquired at a specific time and driver information acquired based on image information acquired at a time earlier than the specific time (e.g., 0.5 seconds earlier), or may be composed of three or more pieces of driver information.

[0032] FIG. 6 is a graph showing an example of a prediction result by the reflex prediction device 100 according to the first embodiment. In FIG. 6 , the upper graph shows a change in the driver's mouth opening, the middle graph shows a change in the driver's eyelid opening, and the lower graph shows a graph of time-series data of the prediction result of the driver's oral reflex. For example, in the graph shown in FIG. 6 , the driver's mouth opening increases slightly at time t=t1. This is the result of the driver talking to himself, causing his mouth to open slightly. Note that when the driver is talking, the mouth opening increases further than the opening at time t=t1. In addition, in the graph shown in FIG. 6 , the period from time t=t4 to t5 is the early period of a yawn, and the driver's mouth is slightly opening. In addition, in the graph shown in FIG. 6 , the period from time t=t5 to t7 is the peak period of the yawn, and time t=t8 is the time when the yawning ends.

[0033] The learning model according to the first embodiment does not predict the occurrence of a yawn at time t=t1, even when the driver's eyelid opening degree is not decreased and the driver's mouth opening degree increases. Thus, the learning model according to the first embodiment is configured, as a result of machine learning, not to predict the occurrence of a yawn even when the driver's mouth opening degree increases and the driver's level of alertness is not decreased. Furthermore, the learning model according to the first embodiment predicts the occurrence of a yawn at time t=t4, when the driver's eyelid opening degree decreases and the driver's mouth opening degree increases slightly after time t=t2. Thus, the learning model according to the first embodiment is configured, as a result of machine learning, to predict the occurrence of a yawn when the driver's mouth opening degree increases and the driver's level of alertness is decreased. After performing the process of step ST03, the reflex prediction device 100 outputs the prediction result from the prediction unit 104 to an external device (step ST04).

[0034] After the processing of step ST04, the reflection prediction device 100 determines whether the driver has finished driving (step ST10). In this processing, the reflection prediction device 100 may determine the end of driving based on, for example, a signal from a vehicle control device (not shown) indicating that the vehicle has been parked, or based on the stop of power supply to the reflection prediction device 100. If the driving has not finished in the processing of step ST10 (NO in step ST10), the reflection prediction device 100 returns the processing to step ST02 and performs the processing from step ST02 to step ST10 again. If the driving has finished in the processing of step ST10 (YES in step ST10), the reflection prediction device 100 ends the processing.

[0035] As described above, the reflex prediction device 100 according to the first embodiment includes a driver information acquisition unit that acquires driver information including information about the behavior of the vehicle driver, and a prediction unit that predicts the occurrence of an oral reflex of the driver based on input of the driver information acquired by the driver information acquisition unit into the learning model, using a learning model trained as training data that includes the driver information acquired by the driver information acquisition unit and information indicating whether an oral reflex has occurred. With this configuration, the reflex prediction device 100 can predict the occurrence of an oral reflex of the driver by using the learning model trained based on information about the behavior of the vehicle driver.

[0036] Embodiment 2 Next, a reflex prediction system 2 according to embodiment 2 will be described with reference to Figures 6 to 8. The reflex prediction system 2 according to embodiment 2 differs from the reflex prediction system 1 according to embodiment 1 in the configuration for the reflex prediction device to detect reflexes and the configuration for the reflex prediction device to generate a learning model, but the other configurations are similar, and the same configurations as those in embodiment 1 will be assigned the same names and symbols as those in embodiment 1 and will not be described again.

[0037] FIG. 7 is a block diagram showing a schematic configuration of a reflection prediction system 2 according to the second embodiment. As shown in FIG. 7 , the reflection prediction system 2 includes a driver information detection device 10, a reflection prediction device 200, and a storage device 80. The storage device 80 stores information acquired from the reflection prediction device 200. For example, the storage device 80 is configured as part of a drive recorder provided in the vehicle and includes a hard disk, a solid state drive (SSD), a read only memory (ROM), a random access memory (RAM), etc., and accumulates and stores information acquired from the reflection prediction device 200. Note that the storage device 80 may be provided outside the vehicle.

[0038] 7 , the reflex prediction device 200 includes a driver information acquisition unit 101, a reflex detection unit 202, a learning unit 203, a prediction unit 104, and a time information acquisition unit 211. The reflex detection unit 202 detects the occurrence of a reflex related to the driver's oral cavity based on the driver information acquired by the driver information acquisition unit 101. For example, the reflex detection unit 202 detects the occurrence of a reflex related to the driver's oral cavity by detecting a characteristic movement of the driver's reflex related to the oral cavity based on the driver information acquired by the driver information acquisition unit 101. For example, the reflex detection unit 202 detects the occurrence of a sneeze by detecting one or more of the following characteristic movements of a sneeze: a deep exhalation, a movement of closing the eyelids, and a vocalization accompanying a sneeze.

[0039] Furthermore, for example, the reflex detection unit 202 detects the occurrence of a yawn by detecting one or more of the following characteristic movements of a yawn: opening the mouth wide, slowly exhaling with the mouth wide open, lowering the eyelid opening, turning the face up, and vocalization accompanying a yawn. Furthermore, for example, the reflex detection unit 202 detects the occurrence of a cough by detecting one or more of the following characteristic movements of a cough: opening the mouth slightly, exhaling forcefully in a short time, turning the face up and down significantly, placing a hand over the mouth, closing the eyelids, and vocalization accompanying a cough. Furthermore, for example, the reflex detection unit 202 detects the occurrence of a cough by detecting one or more of the following characteristic movements of a hiccup: momentarily inhaling air with the mouth, moving the shoulders up and down, and vocalization accompanying a hiccup. Furthermore, for example, the reflex detection unit 202 detects the occurrence of a yawn by the driver during the period from time t5 to t8 based on the driver information indicating the mouth opening and eyelid opening shown in FIG. 6. The reflex detection unit 202 may be configured to learn driver information on a state in which an oral reflex is being performed and driver information on a state in which an oral reflex is being performed as training data, and to detect an oral reflex using a learning model that detects an oral reflex based on the input of driver information. For example, in FIG. 6, the mouth is wide open and the eyelid opening is small from t=t5 to t7, so this period may be clearly considered a period in which a yawn occurred and used as training data for reflex detection. The prediction unit predicts the reflex using information at least prior to t=t5.

[0040] The learning unit 203 generates a learning model trained using as training data a dataset including the driver information acquired by the driver information acquisition unit 101 and information indicating whether an oral cavity reflex has occurred, detected by the reflex detection unit 202. The learning model generated by the learning unit 203 in the second embodiment is similar to the learning model according to the first embodiment, and therefore a description thereof will be omitted. When there are multiple drivers driving the same vehicle, the learning unit 203 may be configured to identify each driver and generate a learning model for each driver, or may be configured to generate a learning model common to multiple drivers. When configured to generate a learning model common to multiple drivers, a learning model corresponding to an average driver is generated based on these multiple drivers.

[0041] The time information acquisition unit 211 acquires information indicating the time. For example, the time information acquisition unit 211 acquires the information indicating the time by measuring the elapsed time from a specific time using a hardware timer or a software timer. Furthermore, for example, the time information acquisition unit 211 acquires the information indicating the time by receiving a signal from a timing device (not shown) that is communicatively connected to the reflection prediction device 100 and measures time, such as a GNSS (Global Navigation Satellite System) receiver equipped in a vehicle. The information indicating the time acquired by the time information acquisition unit 211 is used to synchronize the various processes performed by the reflection prediction device 200 and for storing information in the storage device 80.

[0042] The hardware configuration of the reflection prediction device 200 according to the second embodiment is similar to the hardware configuration of the reflection prediction device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0043] Next, details of the processing performed by the reflection prediction device 200 will be described with reference to Figures 7 and 8. Figure 8 is a flowchart showing an example of processing performed by the reflection prediction device 200 according to embodiment 2. Note that some of the processing performed by the reflection prediction device 200 according to embodiment 2 is similar to the processing performed by the reflection prediction device 100 according to embodiment 1, and therefore processing similar to that performed by the reflection prediction device 100 according to embodiment 1 will be assigned the same reference numerals and will not be described again.

[0044] As shown in FIG. 8 , when the reflex prediction device 200 starts processing, it first loads a learning model (step ST01). In this process, the reflex prediction device 200, for example, references information stored in the storage device 80 and loads information constituting the learning model. For example, in an initial state in which no learning model has been performed on the host vehicle, the reflex prediction device 300 loads a pre-set initial learning model. For example, the initial learning model is a learning model that provides results previously learned using an average person. Note that, for example, the storage device 80 may be configured to store multiple initial learning models based on age, body type, oral shape, etc., and the prediction unit may select one of these multiple initial learning models based on the driver's selection or the results of image recognition, etc., and use it to predict oral reflexes and train the learning model. In addition, in this process, if learning of a learning model has already been performed in the vehicle, the reflex prediction device 300 loads the currently trained learning model, for example, a learning model that reflects the results of the previous training. The reflection prediction device 200 may be configured to read the learning model from a storage unit (not shown) that is provided in the reflection prediction device and stores information. After performing the process of step ST01, the reflection prediction device 200 performs the process of step ST02.

[0045] After the processing of step ST03, the reflex prediction device 200 attempts to detect a reflex related to the driver's oral cavity using the reflex detection unit 202 (step ST21). In this processing, the reflex prediction device 200 attempts to detect a reflex related to the driver's oral cavity, for example, based on driver information acquired during a specific period that is a preset time after the period during which the driver information used by the learning model when predicting the occurrence of a reflex related to the driver's oral cavity in the processing of step ST03 was acquired.

[0046] After the process of step ST21, the reflex prediction device 200 determines whether the prediction result of the process of step ST03 was correct (step ST22). In this process, the reflex prediction device 200 compares the prediction result of the occurrence of a reflex related to the driver's oral cavity in the process of step ST03 with the detection result of the reflex related to the driver's oral cavity in the process of step ST21, and determines whether the prediction result of the process of step ST03 was correct.

[0047] In the process of step ST22, if the prediction result is correct (YES in step ST22), the reflex prediction device 200 performs additional learning of the learning model using the prediction result as the correct answer (step ST23). In this process, the reflex prediction device 200 performs additional learning of the learning model using, as training data, a data set including the driver information used when the prediction unit 104 made the prediction and information indicating the occurrence of the reflex related to the driver's oral cavity predicted by the prediction unit 104.

[0048] If the prediction result is incorrect in the process of step ST22 (NO in step ST22), the reflex prediction device 200 performs additional learning of the learning model, regarding the prediction result as incorrect (step ST24). In this process, the reflex prediction device 200 performs additional learning of the learning model using as training data a data set including the driver information used when the prediction unit 104 made the prediction and information indicating that the reflex related to the driver's oral cavity predicted by the prediction unit 104 did not occur. After performing either the process of step ST23 or the process of step ST24, the reflex prediction device 200 performs the process of step ST10.

[0049] If driving has been completed in the process of step ST10 (YES in step ST10), the reflex prediction device 100 outputs the learning result to the storage device 80 (step ST25). In this process, the reflex prediction device 200 outputs the learning model updated by the additional learning to the storage device 80, thereby storing a new learning model based on the learning result in the storage device 80. Note that in this process, the reflex prediction device 200 may be configured to output the driver information acquired by the driver information acquisition unit 101, the prediction result by the prediction unit 104, and the detection result by the reflex detection unit 202, together with time information, to the storage device 80, and store the output information in the storage device 80. For example, the driver information acquisition unit 101 may be configured to extract driver information for a period from the time when the driver's oral reflex occurred to a specific time before from the acquired driver information, and store the extracted information in the storage device 80.

[0050] Furthermore, for example, the driver information acquisition unit 101 may be configured to extract, from the acquired driver information, information regarding a change in the driver's field of vision when the driver's oral reflex occurs, and store the information in the storage device 80. Specifically, the driver information acquisition unit 101 may be configured to extract, from the acquired driver information, information indicating a change in the driver's line of sight, information indicating a change in the driver's facial orientation, and information indicating a change in the degree of eyelid opening before and after the driver's oral reflex occurs, and store the information in the storage device 80. After performing the process of step ST25, the reflex prediction device 200 updates the learning model stored in the reflex prediction device 200, makes the new learning model available to the prediction unit 104 (step ST26), and ends the process.

[0051] As described above, the reflex prediction device 200 according to the second embodiment is configured to generate a learning model trained using as training data a dataset including the driver information acquired by the driver information acquisition unit 101 and information indicating whether an oral reflex has occurred, detected by the reflex detection unit 202. Generally, there are large individual differences in predictive actions of oral reflexes, and it is difficult to improve the prediction accuracy when predicting the occurrence of an oral reflex using a common learning model for multiple drivers. The reflex prediction device 200 according to the second embodiment can update the learning model through additional learning, thereby improving the prediction accuracy when predicting the occurrence of an oral reflex using the learning model.

[0052] Note that, although the reflection prediction device 200 according to the second embodiment is assumed to perform learning of the learning model by a learning unit 203 provided in the vehicle, the reflection prediction device may also be configured to perform learning of the learning model by a server provided outside the vehicle and communicatively connected to the vehicle. Generally, a server provided outside the vehicle has higher CPU resources than a processing device provided in the vehicle. Therefore, when the learning model is performed by a learning unit provided in the server, faster and more efficient learning can be achieved than when the learning model is performed by a learning unit provided in the vehicle. Furthermore, when the learning unit is provided outside the vehicle, the learning unit may be configured to perform learning of the learning model by inputting driver information acquired from drivers of multiple vehicles. When the learning model is performed by inputting driver information acquired from multiple drivers, a learning model corresponding to an average driver based on these multiple drivers is generated.

[0053] Furthermore, the reflex prediction device may include both a learning unit provided in the vehicle and a learning unit provided outside the vehicle, and may be configured to perform prediction by using a combined learning model generated by the learning unit provided in the vehicle and a learning model generated by a learning unit provided outside the vehicle when making a prediction by the prediction unit. For example, if the learning unit provided outside the vehicle has a learning model corresponding to an average driver learned using driver information acquired from a plurality of drivers, the occurrence of the driver's oral reflex may be predicted based on an inference result using a learning model with a higher reliability between the learning model provided in the learning unit provided in the vehicle and the learning model provided in the learning unit provided outside the vehicle, or the occurrence of the driver's oral reflex may be determined when at least one of the learning models provided in the learning unit provided in the vehicle and the learning model provided in the learning unit provided outside the vehicle predicts the occurrence of the driver's oral reflex.

[0054] Furthermore, when the prediction unit makes a prediction using a learning model stored in a server located outside the vehicle, the number of learning models stored in the server located outside the vehicle is not limited to one. For example, when multiple learning models corresponding to the characteristics of the driver are stored in the server, the reflex prediction device may be configured to select a learning model corresponding to the driver's characteristics from among the multiple learning models based on the driver's selection or the results of image recognition, etc., and use the selected learning model to predict the occurrence of an oral reflex. For example, possible driver characteristics include age, body type, gender, oral cavity shape, disease history, etc.

[0055] 9 to 11, a reflex prediction system 3 according to a third embodiment will be described. The reflex prediction system 3 according to the third embodiment differs from the reflex prediction system 1 according to the first embodiment in the configuration for providing driving assistance to a vehicle based on the prediction results of a learning model, but the other configurations are the same. Therefore, the same names and symbols as those in the first embodiment will be used and descriptions thereof will be omitted.

[0056] FIG. 9 is a block diagram showing a schematic configuration of a reflection prediction system 3 according to a third embodiment. As shown in FIG. 9 , the reflection prediction system 3 according to the third embodiment includes a driver information detection device 10, a driving assistance device 20, a notification device 30, and a reflection prediction device 300. The driving assistance device 20 controls at least one of the speed and direction of travel of the vehicle to provide driving assistance, which is assistance for the driver when driving the vehicle. For example, the driving assistance device 20 includes an image sensor, a LiDAR sensor, a millimeter-wave sensor, etc. (not shown) that detect the positions of surrounding objects such as other vehicles, pedestrians, road structures, and lane markings located around the vehicle, and obtains position information indicating the positions of the surrounding objects based on information from these sensors. Based on the information indicating the positions of the surrounding objects, the driving assistance device 20 performs driving assistance for the vehicle, etc.

[0057] For example, the driving assistance device 20 provides driving assistance such as controlling the vehicle speed so that the distance between the vehicle and other vehicles in front and behind the vehicle becomes a specific distance, controlling the vehicle speed so that the vehicle speed becomes a preset speed, steering using an LKAS (Lane Keeping Assist System) to prevent the vehicle from leaving its lane, controlling the vehicle speed so that the vehicle does not run parallel to other adjacent vehicles, and other driving assistance using automated driving.

[0058] The notification device 30 notifies the driver of the information acquired from the reflection prediction device 300. For example, the notification device 30 is arranged in a position such as a dashboard where the driver and passengers can recognize the content of the notification, and is configured with a device capable of notifying information such as a liquid crystal display device, an LED light-emitting device, a speaker, etc., and outputs at least one of a video and a sound indicating the information acquired from the reflection prediction device 300, thereby visually and audibly notifying the driver of the information acquired from the reflection prediction device 300. The notification device 30 may also be provided outside the vehicle.

[0059] 9 , the reflex prediction device 300 includes a driver information acquisition unit 101, a prediction unit 104, a driving assistance information generation unit 305, a notification information generation unit 306, and a time information acquisition unit 211. The driving assistance information generation unit 305 generates driving assistance information for the driving assistance device 20 to set the content of vehicle control, based on the prediction by the prediction unit 104 of the occurrence of a reflex related to the oral cavity of the driver of the vehicle. The driving assistance information generation unit 305 sets the control content of the driving assistance device 20 by outputting the generated driving assistance information to the driving assistance device 20.

[0060] For example, the driving assistance information generation unit 305 generates driving assistance information so that driving assistance by the driving assistance device 20 is started or the content of driving assistance is changed from a time before the time when the driver's oral cavity-related reflex is predicted to occur, based on the prediction result by the prediction unit 104. Furthermore, for example, the driving assistance information generation unit 305 generates driving assistance information so that driving assistance by the driving assistance device 20 is ended or the content of driving assistance is restored to its original state at a time after the time when the driver's oral cavity-related reflex is predicted to end, based on the prediction result by the prediction unit 104.

[0061] Generally, oral reflexes involve involuntary movements and may affect the driving operation of the driver. When the occurrence of the oral reflex of the driver is predicted, the driving assistance information generation unit 305 outputs driving assistance information to the driving assistance device 20, thereby starting driving assistance by the driving assistance device 20 or changing the content of driving assistance, thereby suppressing unintended driving operations by the driver due to the occurrence of the oral reflex of the driver and contact between the vehicle and surrounding objects due to unintended driving operations by the driver.

[0062] The notification information generation unit 306 generates notification information for causing the notification device 30 to notify information related to the driver's reflexes. For example, the notification information generation unit 306 generates notification information for causing the notification device 30 to notify information related to the control of the driving assistance device 20 set by the driving assistance information generation unit 305. Specifically, the notification information generation unit 306 generates notification information for causing the notification device 30 to notify information indicating that control by the driving assistance device 20 set by the driving assistance information generation unit 305 will be started. Furthermore, specifically, the notification information generation unit 306 generates notification information for causing the notification device 30 to notify information indicating that control by the driving assistance device 20 set by the driving assistance information generation unit 305 is being performed. Furthermore, specifically, the notification information generation unit 306 generates notification information for causing the notification device 30 to notify information indicating the content of control by the driving assistance device 20 set by the driving assistance information generation unit 305.

[0063] Furthermore, for example, the notification information generation unit 306 generates notification information for notifying the notification device 30 of information relating to the detection result by the reflection detection unit 202. Specifically, the notification information generation unit 306 generates notification information for notifying the notification device 30 of information indicating that the reflex related to the driver's oral cavity has ended, based on the detection result by the reflection detection unit 202. Furthermore, for example, the notification information generation unit 306 generates notification information for notifying the notification device 30 that the driving assistance by the driving assistance device 20 has ended or that the content of driving assistance has returned to its original state, based on the information set by the driving assistance information generation unit 305.

[0064] The hardware configuration of the reflection prediction device 300 according to the third embodiment is similar to the hardware configuration of the reflection prediction device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0065] 10 is a flowchart showing an example of processing performed by the reflection prediction apparatus 300 according to embodiment 3. Note that some of the processing performed by the reflection prediction apparatus 300 according to embodiment 3 is similar to the processing performed by the reflection prediction apparatus 100 according to embodiment 1, and therefore, processing similar to that performed by the reflection prediction apparatus 100 according to embodiment 1 is denoted by the same reference numerals and description thereof will be omitted.

[0066] After the processing of step ST03, the reflex prediction device 300 determines whether or not a reflex related to the driver's oral cavity is predicted to occur (step ST31). If the processing of step ST31 does not predict that a reflex related to the driver's oral cavity will occur (NO in step ST31), in other words, if the processing of step ST31 predicts that a reflex related to the driver's oral cavity will not occur, the reflex prediction device 300 performs the processing of step ST10. If the processing of step ST31 predicts that a reflex related to the driver's oral cavity will occur (YES in step ST31), the reflex prediction device 300 generates driving assistance information (step ST32).

[0067] 11 is a graph showing an example of a prediction result by the reflex prediction device 300 according to the third embodiment and the timing of driving assistance by the driving assistance device. For example, the reflex prediction device 300 generates driving assistance information by the driving assistance information generation unit 305 so that the start time of driving assistance by the driving assistance device 20 coincides with time t4, which is the time predicted by the prediction unit 104 that the driver's oral cavity reflex will start. Furthermore, for example, the reflex prediction device 300 generates driving assistance information by the driving assistance information generation unit 305 so that the end time of driving assistance by the driving assistance device 20 is later than time t8, which is the time predicted by the prediction unit 104 that the driver's oral cavity reflex will end. Furthermore, for example, the reflex prediction device 300 generates driving assistance information by the driving assistance information generation unit 305 so that driving assistance by the driving assistance device 20 is being performed at time t6, when the degree of eyelid opening is lowest due to the driver's oral cavity reflex predicted by the prediction unit 104. Time t6, when the degree of eyelid opening is lowest, is the time when the driver's driving aptitude is lowest. In order to ensure that driving assistance is being provided by the driving assistance device 20 at such a time, it is possible to ensure that driving assistance is being provided by the driving assistance device 20 during a specific period that includes the intermediate time between the start and end of the driver's oral reflex.

[0068] For example, in the processing of step ST32, the reflex prediction device 300 generates driving assistance information for changing the control of the driving assistance device 20 so as to increase the degree of vehicle autonomy by the driving assistance device 20, based on a prediction that the driver will experience an oral reflex. In other words, in this processing, the driving assistance information generation unit 305 generates driving assistance information for changing the control of the driving assistance device 20 so as to reduce the degree of driver intervention when driving the vehicle, based on a prediction that the driver will experience an oral reflex. In other words, in this processing, the driving assistance information generation unit 305 generates driving assistance information for changing the control of the driving assistance device 20 so as to strengthen driving assistance by the driving assistance device 20, based on a prediction that the driver will experience an oral reflex. In addition, for example, in this processing, the driving assistance information generation unit 305 generates driving assistance information for changing the control of the driving assistance device 20 so as to reduce the possibility of contact between the vehicle and a surrounding object, based on a prediction that the driver will experience an oral reflex.

[0069] Furthermore, specifically, in the processing of step ST32, the driving assistance information generation unit 305 generates driving assistance information for causing the driving assistance device 20 to control the speed of the vehicle so as to increase the inter-vehicle distance between the vehicle and other vehicles in front and behind. Furthermore, specifically, in the processing of step ST32, the driving assistance information generation unit 305 generates driving assistance information for causing the driving assistance device 20 to reduce the speed of the vehicle. Furthermore, specifically, in the processing of step ST32, the driving assistance information generation unit 305 generates driving assistance information for increasing the autonomous driving level performed by the driving assistance device 20. Furthermore, specifically, in the processing of step ST32, the driving assistance information generation unit 305 generates driving assistance information for causing the driving assistance device 20 to control the speed of the vehicle so as to avoid running side by side with other adjacent vehicles.

[0070] More specifically, the driving assistance information generation unit 305 generates driving assistance information for switching the LKAS, ACC (Adaptive Cruise Control), and AEB (Autonomous Emergency Braking) controls by the driving assistance device 20 that are currently off to on, based on a prediction that the driver will have a mouth reflex. Furthermore, for example, the driving assistance information generation unit 305 changes parameters for the LKAS, ACC, and AEB controls by the driving assistance device 20 that are currently on, based on a prediction that the driver will have a mouth reflex, so as to reduce the possibility of contact between the vehicle and a surrounding object. Furthermore, for example, the driving assistance information generation unit 305 generates driving assistance information for controlling the speed of the vehicle by the driving assistance device 20 so as to increase the inter-vehicle distance, based on a prediction that the driver will have a mouth reflex.

[0071] Furthermore, for example, the driving assistance information generation unit 305 generates driving assistance information for strengthening the LKAS by the driving assistance device 20 so as to move the vehicle to the center of the lane earlier when the vehicle approaches a lane boundary line, based on a prediction that the driver will have a mouth reflex. Furthermore, for example, the driving assistance information generation unit 305 may be configured to generate driving assistance information for temporarily increasing the operating force required to operate the steering wheel, or for temporarily increasing the operating force required to operate one or both of the accelerator pedal and the brake pedal, or for temporarily making the vehicle fully autonomous, based on a prediction that the driver will have a mouth reflex.

[0072] Furthermore, for example, the driving assistance information generation unit 305 generates driving assistance information so that the driving assistance device 20 performs driving assistance according to the type of reflex related to the driver's oral cavity predicted by the prediction unit 104. In other words, the driving assistance information generation unit 305 generates driving assistance information for the driving assistance device 20 to set the content of vehicle control, based on which of a plurality of preset reflexes the reflex related to the driver's oral cavity predicted to occur by the prediction unit 104 is. For example, if the reflex predicted by the prediction unit 104 is yawning or coughing, it is possible that the driver will close their eyelids or their attention will decrease, and therefore the driving assistance information generation unit 305 generates driving assistance information for the driving assistance device 20 to perform control such as adjusting the vehicle speed so that the inter-vehicle distance from another vehicle is wider, turning on the LKAS, or changing parameters to strengthen LKAS control.

[0073] Furthermore, for example, if the reflex predicted by the prediction unit 104 is a sneeze or hiccup, it is conceivable that the driver may unintentionally operate the steering wheel, accelerator pedal, or brake pedal due to the reflex, and therefore driving assistance information is generated by the driving assistance device 20 to perform control such as adjusting the vehicle speed to avoid being adjacent to another vehicle traveling in an adjacent lane, or switching to fully automated driving. After performing the processing of step ST32, the reflex prediction device 300 outputs the driving assistance information generated by the driving assistance information generation unit 305 to the driving assistance device 20 (step ST33).

[0074] 12A is a plan view showing the positional relationship between the host vehicle M1 and the other vehicles M2 and M3 before driving assistance is provided by the driving assistance device 20 according to embodiment 3, and FIG. 12B is a plan view showing the positional relationship between the host vehicle M1 and the other vehicles M2 and M3 after driving assistance is provided by the driving assistance device 20 according to embodiment 3. For example, as shown in FIG. 12A , before driving assistance is provided by the driving assistance device 20, the distance between the host vehicle M1 and the adjacent other vehicle M2 in the front-to-rear direction (the up-and-down direction in FIG. 12A ) is small, and the vehicles are traveling side by side. Thereafter, driving assistance is provided by the driving assistance device 20, and the speed of the vehicle M1 is controlled. As shown in FIG. 12B , after driving assistance is provided by the driving assistance device 20, the distance between the host vehicle M1 and the other vehicles M2 and M3 in the front-to-rear direction is secured.

[0075] After the processing of step ST33, the reflection prediction device 300 generates notification information for notifying the notification device 30 that driving assistance by the driving assistance device 20 will be started, based on the driving assistance information generated by the driving assistance information generation unit 305 being output to the driving assistance device 20 (step ST34), and outputs the generated notification information to the notification device 30 (step ST35). By the processing of step ST35, information such as "Driving assistance will start soon" or "Driving assistance will start in x seconds" is notified by the notification device 30. At this time, the driving assistance information generation unit 305 may be configured to generate driving assistance information that causes the notification device 30 to notify information indicating the content of the driving assistance.

[0076] Furthermore, the notification information generation unit 306 may be configured to output notification information to the notification device 30 to notify the notification device 30 that driving assistance has been performed after driving assistance has been performed by the driving assistance device 20 or after the driver's oral reflex has ended. When the notification information generation unit 306 is configured in this manner, the notification device 30 may be triggered to notify the notification device 30 when a specific operation has been performed by a vehicle-related person on the reflex prediction device 400 or the notification device 30, when a specific time has passed since the end of the driver's oral reflex, or when driving has ended. After performing the processing of step ST35, the reflex prediction device 300 performs the processing of step ST10, and if driving has ended in the processing of step ST10 (YES in step ST10), the reflex prediction device 100 ends the processing.

[0077] As described above, the reflex prediction device 300 according to the third embodiment includes a driving assistance information generation unit 305 that generates driving assistance information for setting the content of vehicle control by the driving assistance device 20, which provides driving assistance by controlling at least one of the speed and traveling direction of the vehicle, based on the prediction result by the prediction unit 104. With this configuration, the reflex prediction device 300 according to the third embodiment prevents the driver from performing an unintended driving operation due to the occurrence of a reflex related to the driver's mouth, and prevents the vehicle from coming into contact with surrounding objects due to the driver's unintended driving operation.

[0078] In addition, the reflection prediction device 300 according to the third embodiment may be configured to output the prediction result by the prediction unit 104, the driving assistance information generated by the driving assistance information generation unit 305, and the notification information generated by the notification information generation unit 306 to an external storage device together with time information relating to these pieces of information.

[0079] The driving assistance information generation unit may be configured to acquire position information of surrounding objects from the driving assistance device 20, and generate driving assistance information for controlling the driving assistance device based on the acquired position information of the surrounding objects so as to reduce the possibility of contact between the vehicle and the surrounding objects. Such a driving assistance information generation unit constitutes a surrounding object information acquisition unit that acquires the position information of the surrounding objects in the third embodiment.

[0080] 13 and 14, a reflection prediction system 4 according to a fourth embodiment will be described. The reflection prediction system 4 according to the fourth embodiment differs in part from the reflection prediction system 1 according to the first embodiment, but the same components as those in the first embodiment are given the same names and symbols as those in the first embodiment, and description thereof will be omitted.

[0081] FIG. 13 is a block diagram showing a schematic configuration of a reflection prediction system 4 according to the fourth embodiment. As shown in FIG. 13 , the reflection prediction system 4 according to the fourth embodiment includes a driver information detection device 10, a position information acquisition device 40, a peripheral object detection device 50, a vehicle information acquisition device 60, a reflection prediction device 400, a storage device 80, and a display device 90. The position information acquisition device 40 acquires position information indicating the position where the vehicle is located. For example, the position information acquisition device 40 is configured by a GNSS receiver provided in the vehicle, and acquires position information indicating the coordinates where the vehicle is located. The position information acquisition device 40 outputs the acquired position information to the reflection prediction device 400.

[0082] The peripheral object detection device 50 detects the positions of objects around the vehicle. For example, the peripheral object detection device 50 is configured with an image sensor, a LiDAR sensor, a millimeter-wave sensor, etc., and detects the positions of peripheral objects located around the vehicle, such as other vehicles, pedestrians, road structures, and lane markings. The vehicle information acquisition device 60 detects information related to the vehicle's traveling. For example, the vehicle information acquisition device 60 is configured with a speed sensor that detects the vehicle's speed, an acceleration sensor that detects the vehicle's acceleration, a sensor that detects the vehicle's direction of movement, a sensor that detects the steering wheel operation angle, a sensor that detects the accelerator pedal operation amount, and a sensor that detects the brake pedal operation amount. The display device 90, which serves as an alarm device, displays an image based on information from the reflection prediction device 400. For example, the display device 90 is disposed in a position, such as on the dashboard, where the driver and other passengers can recognize the displayed content, and is configured with a liquid crystal display panel, an organic or inorganic electroluminescence (EL) panel, a dot matrix display, or other display device. Note that the display device 90 may be provided outside the vehicle.

[0083] 13 , the reflex prediction device 400 includes a driver information acquisition unit 101, a driver state detection unit 110, a prediction unit 104, a position information acquisition unit 405, a surrounding object information acquisition unit 406, a vehicle information acquisition unit 407, a contact degree calculation unit 408, a memory information control unit 409, a display control unit 410, and a time information acquisition unit 211. The driver state detection unit 110 includes a reflex detection unit 202 and a motion detection unit 403, and detects information about the driver's state related to the occurrence of a reflex. The motion detection unit 403 detects information about the driver's motion related to the occurrence of a reflex. For example, the motion detection unit 403 acquires information indicating a driving operation performed by the driver on the vehicle based on the driver information acquired by the driver information acquisition unit 101, and extracts operation information indicating a predetermined specific driving operation from the acquired information. Specifically, based on the driver information acquired by the driver information acquisition unit 101, the action detection unit 403 extracts information indicating the driving operation performed by the driver on the vehicle when a reflex related to the driver's oral cavity occurs from the acquired driver operation information.

[0084] The position information acquisition unit 405 acquires position information indicating the position where the vehicle is located in association with time, based on information from the position information acquisition device 40. The peripheral object information acquisition unit 406 acquires peripheral object information indicating the positions, movement speeds, and movement directions of objects around the vehicle, based on information from the peripheral object detection device 50. The vehicle information acquisition unit 407 acquires vehicle information, based on information from the vehicle information acquisition device 60, including, as information about the traveling of the vehicle, one or more of the following: information about the vehicle's motion state, indicating the vehicle's speed, vehicle acceleration, and movement direction, and information indicating the forces applied by the driver to the steering wheel, accelerator pedal, brake pedal, etc., and information indicating the amounts of operation of these pedals.

[0085] The contact degree calculation unit 408 calculates a contact degree indicating the degree of possibility of contact between the vehicle and a peripheral object based on the information acquired by the peripheral object information acquisition unit 406 and the vehicle information acquisition unit 407. For example, the contact degree calculation unit 408 calculates a contact degree as a larger value the shorter the distance between the vehicle and the peripheral object. Furthermore, for example, if the vehicle does not perform an evasive action to avoid the peripheral object, the contact degree calculation unit 408 calculates a contact degree as a larger value the shorter the time until contact. Furthermore, for example, if the vehicle is violating a law or regulation, such as by deviating from its lane or speeding, the contact degree calculation unit 408 calculates a contact degree as a larger value than if the vehicle is not violating a law or regulation. Furthermore, for example, the contact degree calculation unit 408 calculates a contact degree as a larger value the greater the impact of contact between the vehicle and the peripheral object. For example, if the peripheral object is a pedestrian, the contact degree calculation unit 408 calculates a contact degree as a larger value than if the peripheral object is a roadside tree. For example, the contact degree calculation unit 408 calculates the contact degree as a value ranging from 0, which is the smallest contact degree, to 1, which is the largest contact degree. The contact degree may be expressed as a binary value, a multi-value, or a continuous value. Any method may be used to calculate the contact degree.

[0086] The stored information control unit 409 outputs information acquired by the reflex prediction device 400 from other devices and information indicating the results of processing performed by the reflex prediction device 400 to the storage device 80 and stores the information in the storage device 80. For example, the stored information control unit 409 stores in the storage device 80 images of the interior of the vehicle acquired by the driver information acquisition unit 101 for a predetermined time (for example, one minute) before and after the point in time when it predicts that a reflex related to the driver's oral cavity will occur. Furthermore, for example, the storage information control unit 409 may be configured to, when it predicts that the driver's oral reflex will occur, store in the storage device 80, in addition to the video of the interior of the vehicle acquired by the driver information acquisition unit 101, one or more pieces of information indicating the sound inside the vehicle, the prediction result by the prediction unit 104, the type of predicted reflex, the position information acquired by the position information acquisition unit 405, time information, the magnitude of the force applied by the driver to the steering wheel, brake pedal, and accelerator pedal and the amount of operation of these during the period when the driver's oral reflex occurred, the driver's actions during the period when the driver's oral reflex occurred (e.g., looking aside, closing eyes, etc.), the degree of contact during the period when the driver's oral reflex occurred, etc.

[0087] The display control unit 410 outputs information acquired by the reflex prediction device 400 from other devices and information indicating the results of processing performed by the reflex prediction device 400 to the display device 90, and causes the information to be displayed on the display device 90. For example, the display control unit 410 causes the display device 90 to display some or all of the information stored in the storage device 80 by the stored information control unit 409. Furthermore, for example, after the driver's oral reflex has ended, when driving has ended, or when the operator has performed a specific operation on the reflex prediction device 400 or the display device 90, the display control unit 410 causes the display device 90 to display information indicating the magnitude of the force applied by the driver to the steering wheel, brake pedal, and accelerator pedal and changes in the amount of operation of these pedals during the period when the driver's oral reflex occurred, the driver's actions (e.g., looking aside, closing eyes, etc.) during the period when the driver's oral reflex occurred, changes in the degree of contact during the period when the driver's oral reflex occurred, etc. This makes it possible to notify the driver, other occupants, the family members of the driver and occupants, the vehicle manager, the supervisor of the work performed using the vehicle, and other related parties of the vehicle of the influence of the driver's oral cavity-related reflex on driving, for example, an increase in the degree of contact, etc. Note that the storage device 80 and the display device 90 may be configured integrally, or the display device 90 may be provided with a speaker that outputs audio and configured to output part or all of the information from the reflex prediction device 400 as audio.

[0088] The hardware configuration of the reflection prediction device 400 according to the fourth embodiment is similar to the hardware configuration of the reflection prediction device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0089] 14 is a flowchart showing an example of processing performed by the reflection prediction apparatus 400 according to embodiment 4. Note that some of the processing performed by the reflection prediction apparatus 400 according to embodiment 4 is similar to the processing performed by the reflection prediction apparatus 100 according to embodiment 1, and therefore, processing similar to that performed by the reflection prediction apparatus 100 according to embodiment 1 is denoted by the same reference numerals and description thereof will be omitted.

[0090] After performing the process of step ST03, the reflection prediction device 400 detects the driver's oral cavity-related reflexes and movements using the driver state detection unit 110 (step ST41). After performing the process of step ST41, the reflection prediction device 400 acquires vehicle position information using the position information acquisition unit 405 (step ST42). After performing the process of step ST42, the reflection prediction device 400 acquires peripheral object information using the peripheral object information acquisition unit 406 (step ST43). After performing the process of step ST43, the reflection prediction device 400 acquires vehicle information using the vehicle information acquisition unit 407 (step ST44). After performing the process of step ST, the reflection prediction device 400 calculates the contact degree using the contact degree calculation unit 408 (step ST45). After performing the process of step ST45, the reflection prediction device 400 outputs the information acquired through each process to the storage device 80 (step ST46). After performing the process of step ST46, the reflection prediction device 400 displays the information acquired through each process on the display device 90 (step ST47). After performing the process of step ST47, the reflection prediction device 400 performs the process of step ST10. If the driving has not ended in the process of step ST10 (NO in step ST10), the reflection prediction device 400 returns the process to step ST02 and performs the processes from step ST02 to step ST10 again. If the driving has ended in the process of step ST10 (YES in step ST10), the reflection prediction device 400 ends the process.

[0091] In any of the above-described embodiments, the reflection prediction device may include some or all of the other components of the reflection prediction system, or some of the components of the reflection prediction device may be provided in an external device that is communicatively connected to the reflection prediction device.

[0092] In addition, the present disclosure allows for free combination of the respective embodiments, modification of any of the components of the respective embodiments, or omission of any of the components of the respective embodiments.

[0093] The reflex prediction device according to the present disclosure can predict the occurrence of a reflex related to the driver's oral cavity and can be used, for example, to provide driving assistance for a vehicle based on the prediction results.

[0094] 1 Reflection prediction system, 2 Reflection prediction system, 3 Reflection prediction system, 4 Reflection prediction system, 10 Driver information detection device, 20 Driving assistance device, 30 Notification device, 40 Position information acquisition device, 50 Peripheral object detection device, 60 Vehicle information acquisition device, 70 External device, 80 Storage device, 90 Display device, 100 Reflection prediction device, 101 Driver information acquisition unit, 104 Prediction unit, 110 Driver state detection unit, 200 Reflection prediction device, 202 Reflection detection unit, 203 Learning unit, 211 Time information acquisition unit, 300 Reflection prediction device, 305 Driving assistance information generation unit, 306 Notification information generation unit, 400 Reflection prediction device, 403 Action detection unit, 405 Position information acquisition unit, 406 Peripheral object information acquisition unit, 407 Vehicle information acquisition unit, 408 Contact degree calculation unit, 409 Memory information control unit, 410 display control unit, 600 reflection prediction device, M1 vehicle, M2 other vehicle, M3 other vehicle.

Claims

1. A reflex prediction device comprising: a driver information acquisition unit that acquires driver information including information regarding the actions of a vehicle driver; and a prediction unit that uses a learning model trained using a data set including the driver information acquired by the driver information acquisition unit and information indicating whether an oral reflex has occurred as training data to predict the occurrence of the driver's oral reflex based on the input of the driver information acquired by the driver information acquisition unit into the learning model.

2. The reflex prediction device described in claim 1, characterized in that the learning model used by the prediction unit is a learning model learned using a dataset including the driver information for the period from the time when the driver's oral cavity reflex occurred to a specific time ago, and the driver information for the period from the time when the driver's oral cavity reflex did not occur to a specific time ago, as training data.

3. The reflex prediction device according to claim 1, characterized in that the driver information acquired by the driver information acquisition unit includes information indicating changes in the driver's facial expression.

4. The reflex prediction device according to claim 1, characterized in that the driver information acquired by the driver information acquisition unit includes information indicating a specific precursory action of the driver's oral cavity-related reflex.

5. The reflection prediction device according to claim 1, characterized in that the driver information acquired by the driver information acquisition unit includes information indicating the voice uttered by the driver.

6. The reflex prediction device according to claim 1, characterized in that the prediction unit predicts the time until the driver's oral reflex occurs based on the input of driver information acquired by the driver information acquisition unit.

7. The reflex prediction device according to claim 1, characterized in that the prediction unit predicts which of a plurality of preset reflexes the reflex occurring in the driver's oral cavity will be based on the input of driver information acquired by the driver information acquisition unit.

8. The reflection prediction device according to claim 1, further comprising a learning unit for generating the learning model.

9. A reflection prediction device according to any one of claims 1 to 8, characterized in that the driving assistance device, which provides driving assistance by controlling at least one of the speed and direction of travel of the vehicle based on the prediction results by the prediction unit, is provided with a driving assistance information generation unit which generates driving assistance information for setting the content of control of the vehicle.

10. The reflex prediction device according to claim 9, characterized in that the prediction unit predicts which of a plurality of preset reflexes the reflex occurring in the driver's oral cavity will be, based on the input of driver information acquired by the driver information acquisition unit, and the driving assistance information generation unit generates driving assistance information for the driving assistance device to set the content of control of the vehicle, based on which of a plurality of preset reflexes the reflex occurring in the driver's oral cavity predicted by the prediction unit is.

11. The reflection prediction device according to claim 9, further comprising a peripheral object information acquisition unit that acquires information regarding the positions of objects surrounding the vehicle, and the driving assistance information generation unit generates driving assistance information based on the information acquired by the peripheral object information acquisition unit.

12. The reflex prediction device according to claim 1, characterized in that the driver information acquisition unit extracts driver information from the acquired driver information for the period from the time when the driver's oral reflex occurred to a specific time before, and stores the information in a storage device.

13. The reflection prediction device according to claim 12, characterized in that the driver information acquisition unit acquires information indicating driving operations by the driver, extracts information indicating pre-set driving operations from the acquired information, and stores the extracted information in the storage device.

14. The reflex prediction device according to claim 12, characterized in that the driver information acquisition unit extracts information regarding changes in the driver's field of vision when a reflex related to the driver's oral cavity occurs from the acquired driver information and stores the information in the storage device.

15. A reflex prediction device as described in claim 12, characterized in that it comprises: a surrounding object information acquisition unit that acquires information regarding the positions of surrounding objects of the vehicle; and a contact degree calculation unit that calculates a contact degree indicating the degree of possibility of contact between the vehicle and the surrounding object based on the information acquired by the surrounding object information acquisition unit, and stores in the storage device a change in the contact degree when a reflex related to the driver's oral cavity occurs.

16. A reflex prediction method performed by a device having a driver information acquisition unit and a prediction unit, comprising: a step in which the driver information acquisition unit acquires driver information including information regarding the actions of the vehicle driver; and a step in which the prediction unit predicts the occurrence of an oral reflex of the driver based on input of the driver information acquired by the driver information acquisition unit into the learning model, using a data set including the driver information acquired by the driver information acquisition unit and information indicating whether an oral reflex has occurred as training data.

Images