Information processing device and information processing method

Abstract

An information processing device (100, 200, 300, 400, 500, 600, 700, 800) comprises: a driver information acquisition unit (101) that acquires information about a driver of a vehicle, such information pertaining to a change in driving suitability with respect to the driving of the driver during driving of the vehicle; a driving suitability calculation unit (102) that calculates the driving suitability of the driver of the vehicle on the basis of the information acquired by the driver information acquisition unit (101); a passenger behavior information acquisition unit (103) that acquires information pertaining to the behavior of a passenger other than the driver of the vehicle; and a related information generation unit (114, 314, 414, 514) that, on the basis of the driving suitability calculated by the driving suitability calculation unit and the information acquired by the passenger behavior information acquisition unit (103), generates related information in which the information indicating a change in the driving suitability of the driver of the vehicle and the information pertaining to behavior by the passenger when the driving suitability of the driver of the vehicle has changed are associated with each other.

Description

Information processing device and information processing method

[0001] The present disclosure relates to an information processing device and an information processing method.

[0002] A driving assistance device that controls a vehicle based on speech from a passenger in the vehicle has been disclosed (see, for example, Patent Document 1). This driving assistance device aims to improve the driver's internal concentration on driving by controlling the vehicle to emphasize the running sounds of the vehicle heard in the vehicle cabin when the passenger speaks in a state determined to be an ideal state in which the driver's internal concentration on driving is high.

[0003] Japanese Patent Application Laid-Open No. 2017-182556

[0004] However, depending on the content, the words and actions of occupants other than the vehicle driver may not affect changes in driving aptitude, which indicates the driver's degree of driving aptitude while driving.However, the invention described in Patent Document 1 has the problem that it is not possible to know how the words and actions of the above occupants affect changes in driving aptitude.

[0005] The present disclosure was made in response to the recognition of the above-mentioned problem, and aims to provide an information processing device and an information processing method that can acquire information indicating how the occupant's words and actions affect changes in driving aptitude while driving.

[0006] The information processing device of the present disclosure is characterized by comprising: a driver information acquisition unit that acquires information about the vehicle driver regarding changes in the driver's driving aptitude while driving the vehicle; a driving aptitude calculation unit that calculates the vehicle driver's driving aptitude based on the information acquired by the driver information acquisition unit; an occupant behavior information acquisition unit that acquires information about the behavior of occupants other than the vehicle driver; and an associated information generation unit that generates associated information that correlates information indicating changes in the vehicle driver's driving aptitude and information about the behavior of occupants when the vehicle driver's driving aptitude changes, based on the driving aptitude calculated by the driving aptitude calculation unit and the information acquired by the occupant behavior information acquisition unit.

[0007] The information processing device of the present disclosure generates related information that associates information indicating changes in the driving aptitude of a vehicle driver with information indicating the words and actions of occupants when the driving aptitude of the vehicle driver changes, thereby making it possible to obtain information indicating how the words and actions of occupants affect changes in driving aptitude while driving.

[0008] 9A is a graph showing an example of related information notified by the information processing device according to the second embodiment, and FIG. 9B is a diagram showing classification names of icons added to the related information. 9B is a diagram showing map information included in related information notified by the information processing device according to the third embodiment. 9C is a block diagram showing a schematic configuration of an information processing system according to a fourth embodiment. 9D is a flowchart showing an example of processing performed by the information processing device according to the fourth embodiment. 9E is a flowchart showing an example of processing performed by the information processing device according to the fourth embodiment. 9F is a graph showing an example of related information generated by the information processing device according to the fourth embodiment. 9G is a diagram showing an example of related information generated by the information processing device according to the fourth embodiment. 9H is a block diagram showing a schematic configuration of an information processing system according to a fifth embodiment. 9H is a flowchart showing an example of processing performed by the information processing device according to the fifth embodiment. A block diagram showing a schematic configuration of an information processing system according to embodiment 6. A flowchart showing an example of processing performed by an information processing device according to embodiment 6. A block diagram showing a schematic configuration of an information processing system according to embodiment 7. A flowchart showing an example of processing performed by an information processing device according to embodiment 7. A block diagram showing a schematic configuration of an information processing system according to embodiment 8. A flowchart showing an example of processing performed by an information processing device according to embodiment 8.

[0009] Embodiments of the present disclosure will be described in detail below with reference to the drawings. Embodiment 1. First, referring to FIG. 1, a schematic configuration of an information processing system 1 according to embodiment 1 will be described. FIG. 1 is a block diagram showing a schematic configuration of the information processing system 1 according to embodiment 1. The information processing system 1 is a system provided in a vehicle for generating associated information that correlates information indicating changes in a driver's driving aptitude with information indicating the behavior of passengers other than the driver while the vehicle is being driven. As shown in FIG. 1, the information processing system 1 includes an in-vehicle information acquisition device 10, an information processing device 100, and an external device 90, which are connected wirelessly or via a wire to enable communication with each other. Note that in embodiment 1, a "vehicle" simply refers to a specific vehicle equipped with the information processing system 1, a "driver" simply refers to a driver who drives the vehicle, and a "passenger" simply refers to a passenger other than the driver who is in the vehicle. In embodiment 1, the term "passenger" is not limited to humans but also includes pets, livestock, and other living creatures.

[0010] The in-vehicle information acquisition device 10 acquires information about the interior of the vehicle while the driver is driving the vehicle. For example, the in-vehicle information acquisition device 10 includes a driver information detection unit 11 that detects information about the driver and an occupant information detection unit 12 that detects information about the occupants. For example, the driver information detection unit 11 is configured with an imaging device (not shown) that captures image information about the driver sitting in the driver's seat by photographing the interior of the vehicle, and an image recognition unit (not shown) that detects various information about the driver through image recognition based on the image information acquired by the imaging device. For example, the driver information detection unit 11 detects information about the driver through image recognition, such as information indicating the driver's behavior, information indicating the driver's line of sight, information indicating the driver's blinking frequency, information indicating the driver's facial orientation, information indicating the driver's psychological state, and information indicating the driver's physiological state, such as the driver's body temperature, sweat rate, heart rate, pulse rate, and alertness level. In addition to or instead of the above configuration, the driver information detection unit may have sensors such as a body temperature sensor, a sweat sensor, a heart rate sensor, a pulse sensor, etc. that detect information indicating the physiological state of the driver, a microphone for acquiring audio information indicating what the driver is saying, an infrared sensor for detecting the driver's movements, an ultrasonic sensor for detecting the driver's movements, a LiDAR (Light Detection and Ranging) sensor for detecting the driver's movements, etc.

[0011] The occupant information detection unit 12 acquires information regarding the behavior of occupants while the driver is driving the vehicle. For example, the occupant information detection unit 12 is configured with an imaging device (not shown) that captures images of the interior of the vehicle to acquire image information of occupants sitting in seats other than the driver's seat, and an image recognition unit (not shown) that detects information regarding the behavior of occupants through image recognition based on the image information acquired by the imaging device. For example, the occupant information detection unit 12 detects information regarding the behavior of occupants, such as information indicating the behavior of occupants and information indicating the occupants' interference (incitement) with the driver, through image recognition. Note that, in addition to or instead of the above configuration, the occupant information detection unit may include sensors such as a microphone for acquiring audio information indicating the occupants' speech, an infrared sensor for detecting the occupants' behavior, an ultrasonic sensor for detecting the occupants' behavior, and a LiDAR (Light Detection and Ranging) sensor for detecting the occupants' behavior. In addition, when multiple occupants are riding in the vehicle, the occupant information detection unit 12 may be configured to acquire information regarding the behavior and speech of these multiple occupants.For example, the occupant information detection unit 12 may be configured to identify the multiple occupants and then acquire information regarding the behavior and speech of each occupant, or it may be configured to acquire information regarding the behavior and speech of each occupant without identifying the multiple occupants.

[0012] In addition, the in-vehicle information acquisition device may have the occupant information detection unit and the driver information detection unit formed integrally, or may be configured so that the functions of the driver information detection unit and the occupant information detection unit are achieved by a single device. For example, the in-vehicle information acquisition device may have a single imaging device that can simultaneously capture images of the driver and occupants, and may be configured to acquire driver information and occupant information based on image information acquired by the imaging device. The in-vehicle information acquisition device 10 outputs the acquired information to the information processing device 100.

[0013] The external device 90 acquires information from the information processing device 100. For example, the external device 90 is configured by a storage device that stores the information acquired from the information processing device 100. Furthermore, for example, the external device 90 is configured by a control device that controls various devices of the vehicle based on the information acquired from the information processing device 100. Furthermore, for example, the external device 90 is configured by a display device that displays the information acquired from the information processing device 100.

[0014] The information processing device 100 is a device for generating related information in which information indicating a change in a driver's driving aptitude with respect to driving while the driver is driving a vehicle and information indicating the speech and behavior of passengers other than the driver of the vehicle are mutually associated. As shown in Fig. 1, the information processing device 100 includes a driver information acquisition unit 101, a driving aptitude calculation unit 102, a passenger speech and behavior information acquisition unit 103, a time information acquisition unit 107, and a related information generation unit 114.

[0015] The driver information acquisition unit 101 acquires driver information, which is information about the driver regarding changes in the driver's driving aptitude while driving the vehicle, based on information from the in-vehicle information acquisition device 10. Generally, the driver's driving aptitude, which indicates the degree of suitability for driving, changes while driving in accordance with changes in the driver's psychological state and physiological state while driving. The driver information acquisition unit 101 acquires driver information that changes in association with such changes in driving aptitude, based on information from the in-vehicle information acquisition device 10.

[0016] For example, the driver information acquisition unit 101 acquires driver information indicating the psychological state of the driver based on information from the in-vehicle information acquisition device 10. Specifically, the driver information acquisition unit 101 acquires driver information indicating whether the driver's psychological state is calm or a state other than a calm state, such as a restless state, an angry state, an excited state, a stressed state, an anxious state, an impatient state, or a dissatisfied state, based on the information from the in-vehicle information acquisition device 10. For example, the driver information acquisition unit 101 acquires driver information indicating the driver's psychological state based on facial expression characteristics acquired from the image information of the driver, head movements of the driver acquired from the image information of the driver, movements of the driver touching his or her face acquired from the image information of the driver, the frequency of the driver's blinking acquired from the image information of the driver, the content of the driver's remarks acquired from the voice information of the driver, the volume and frequency of the driver's voice acquired from the voice information of the driver, and the like, which are acquired based on the information from the in-vehicle information acquisition device 10.

[0017] Furthermore, for example, the driver information acquisition unit 101 acquires driver information indicating the physiological state of the driver based on information from the in-vehicle information acquisition device 10. Specifically, the driver information acquisition unit 101 acquires driver information indicating whether the physiological state of the driver is in a normal mental and physical state or an abnormal mental and physical state, such as a state of feeling unwell or fatigued, based on the information from the in-vehicle information acquisition device 10. Specifically, the driver information acquisition unit 101 acquires driver information indicating whether the physiological state of the driver is in a state of high alertness or a state of low alertness, based on the information from the in-vehicle information acquisition device 10. For example, the driver information acquisition unit 101 acquires driver information indicating the physiological state of the driver based on information acquired based on the in-vehicle information acquisition device 10, such as facial expression characteristics acquired from the driver's image information, the driver's head movements acquired from the driver's image information, the driver's face touching movements acquired from the driver's image information, the driver's blinking frequency acquired from the driver's image information, the degree of eyelid opening of the driver acquired from the driver's image information, and the driver's facial direction acquired from the driver's image information.

[0018] Furthermore, for example, the driver information acquisition unit 101 acquires driver information indicating the driver's level of concentration on driving based on information from the in-vehicle information acquisition device 10. Specifically, the driver information acquisition unit 101 acquires driver information indicating whether the driver is in a state of high concentration on driving or a state of low concentration on driving, for example, a state of absentminded driving, based on information from the in-vehicle information acquisition device 10. For example, the driver information acquisition unit 101 acquires driver information indicating the driver's level of concentration based on information acquired based on the in-vehicle information acquisition device 10, such as a change in the driver's line of sight, the frequency of the driver's blinking acquired from the driver's image information, the degree of eyelid opening acquired from the driver's image information, the driver's facial direction acquired from the driver's image information, the content of the driver's remarks acquired from the driver's voice information, etc.

[0019] In addition, when there are multiple drivers driving the same vehicle, the driver information acquisition unit 101 may be configured to identify these multiple drivers based on information from the in-vehicle information acquisition device 10 and then acquire the driver information of each driver in an identifiable state, or may be configured to acquire driving situation information without identifying the multiple drivers.

[0020] The driving aptitude calculation unit 102 calculates the driving aptitude of the driver based on the driver information acquired by the driver information acquisition unit 101. For example, the driving aptitude calculation unit 102 calculates the driving aptitude by quantifying the psychological state of the driver based on the driver information indicating the psychological state of the driver acquired by the driver information acquisition unit 101. Furthermore, for example, the driving aptitude calculation unit 102 calculates the driving aptitude by quantifying the physiological state of the driver based on the driver information indicating the physiological state of the driver acquired by the driver information acquisition unit 101. Furthermore, for example, the driving aptitude calculation unit 102 calculates the driving aptitude by quantifying the driver's concentration level based on the driver information indicating the driver's concentration level on driving acquired by the driver information acquisition unit 101. Furthermore, for example, the driving aptitude calculation unit 102 calculates the driving aptitude based on values ​​obtained by quantifying multiple of the driver's psychological state, physiological state, and concentration level based on the driver information acquired by the driver information acquisition unit 101. In addition, when the driver information acquired by the driver information acquisition unit 101 is information that can identify multiple drivers, the driving aptitude calculation unit 102 may be configured to calculate the driving aptitude for each of the multiple drivers. Furthermore, the driving aptitude calculation unit 102 that identifies multiple drivers based on the information from the driver information acquisition unit 101 and then calculates the driving aptitude of each driver in this way constitutes an identification unit that identifies the driver of the vehicle in Embodiment 1. Details of the method for calculating the driving aptitude will be described later.

[0021] The occupant behavior information acquisition unit 103 acquires occupant behavior information, which is information regarding the behavior of occupants, based on information from the in-vehicle information acquisition device 10. For example, the occupant behavior information acquisition unit 103 acquires occupant behavior information, which is information indicating whether or not occupants have performed any of their behaviors, based on information from the in-vehicle information acquisition device 10. Furthermore, for example, the occupant behavior information acquisition unit 103 acquires occupant behavior information, which is information regarding the content of occupants' behavior, based on information from the in-vehicle information acquisition device 10. Specifically, the occupant behavior information acquisition unit 103 acquires occupant behavior information regarding the content of occupant behavior that may affect the driver's psychological state, physiological state, or concentration level, based on information from the in-vehicle information acquisition device 10. For example, the occupant behavior information acquisition unit 103 acquires occupant behavior information regarding the content of occupant speech and the content of occupant actions, based on information from the in-vehicle information acquisition device 10. For example, the occupant behavior information acquisition unit 103 acquires occupant behavior information indicating what the occupant said to the driver as the occupant behavior information regarding the content of utterances by the occupant. Furthermore, for example, the occupant behavior information acquisition unit 103 acquires occupant behavior information indicating the content of conversations between multiple occupants as the occupant behavior information regarding the content of utterances by the occupant. Furthermore, for example, the occupant behavior information acquisition unit 103 acquires occupant behavior information indicating the content of monologue by the occupant as the occupant behavior information regarding the content of utterances by the occupant.

[0022] Furthermore, for example, the occupant behavior information acquisition unit 103 acquires occupant behavior information indicating the content of occupant speech, such as crying, yelling, laughter, etc., as occupant behavior information regarding the content of occupant speech. Furthermore, for example, the occupant behavior information acquisition unit 103 acquires occupant behavior information indicating the length of time the occupant remained silent after finishing speaking, as occupant behavior information regarding the content of occupant speech. In other words, the occupant behavior information regarding the content of occupant speech includes information indicating that the occupant was silent.

[0023] Furthermore, for example, the occupant behavior information acquisition unit 103 acquires occupant behavior information indicating how the occupant touched the driver as occupant behavior information regarding the content of the occupant's actions. The content of how the occupant touched the driver may include the part of the driver's body that was touched, whether the occupant hit the driver, whether the occupant grabbed the driver, whether the occupant grabbed the driver's clothes, and whether the occupant pulled or pushed the driver. Furthermore, for example, the occupant behavior information acquisition unit 103 acquires, as occupant behavior information regarding the content of the occupant's actions, information indicating whether the occupant hit any part inside the vehicle, such as the driver's seat, window glass, door, ceiling, dashboard, or floor. Furthermore, for example, the occupant behavior information acquisition unit 103 acquires, as occupant behavior information regarding the content of the occupant's actions, occupant behavior information indicating whether the occupant shook the vehicle body. Furthermore, for example, the occupant behavior information acquisition unit 103 acquires, as occupant behavior information relating to the content of the occupant's behavior, occupant behavior information indicating the length of time during which the occupant's behavior is not detected after the occupant has finished the behavior. That is, the occupant behavior information relating to the content of the occupant's behavior includes information indicating that the occupant's behavior is not detected.

[0024] The occupant behavior information acquisition unit 103 may be configured to acquire occupant behavior information regarding these occupant behaviors as any one or a combination of information such as a binary signal indicating the presence or absence of the behavior, character string information indicating the content of the behavior, symbol information such as an icon indicating the content of the behavior, audio information indicating the content of the behavior, and video information indicating the content of the behavior, or may be configured to acquire occupant behavior information indicating the result of classifying information indicating the content of utterances according to a plurality of preset categories. For example, the occupant behavior information acquisition unit 103 may be configured to acquire, based on information from the in-vehicle information acquisition device 10, as occupant behavior information regarding the content of the occupant's behavior, the occupant behavior information indicating the result of classifying the content of the occupant's behavior into any one of behaviors that cause the driver's psychological state to be calm, the driver's physiological state to be normal, and the driver's concentration level to be high, and behaviors that cause the driver's psychological state to be uncalm, the driver's physiological state to be abnormal, and the driver's concentration level to be low,

[0025] In addition, when there are multiple occupants in the same vehicle, the occupant behavior information acquisition unit 103 may be configured to identify these multiple occupants and acquire the occupant behavior information of each occupant in an identifiable state based on information from the in-vehicle information acquisition device 10, or may be configured to acquire the occupant behavior information without identifying the multiple occupants. Furthermore, the occupant behavior information acquisition unit 103, which identifies the multiple occupants and acquires the occupant behavior information of each occupant based on information from the in-vehicle information acquisition device 10 in this way, constitutes an identification unit that identifies the occupants of the vehicle in the first embodiment.

[0026] The time information acquisition unit 107 acquires information indicating the time. For example, the time information acquisition unit 107 acquires the information indicating the time by measuring the elapsed time from a specific time using a hardware timer or a software timer. Furthermore, for example, the time information acquisition unit 107 acquires the information indicating the time by receiving a signal from a timing device (not shown) that is communicatively connected to the information processing device 100 and measures time, such as a GNSS (Global Navigation Satellite System) receiver provided in a vehicle. The information indicating the time acquired by the time information acquisition unit 107 is used to synchronize the various processes performed by the information processing device 100 and for the related information generated by the related information generation unit 114.

[0027] The related information generation unit 114 generates related information in which information indicating a change in the driving aptitude of the vehicle driver and occupant behavior information indicating the behavior of the occupant when the driving aptitude of the vehicle driver changes are associated with each other, based on the driving aptitude calculated by the driving aptitude calculation unit 102 and the occupant behavior information acquired by the occupant behavior information acquisition unit 103. By referring to the related information generated in this manner, it becomes possible to know how the driver's driving aptitude changed when the occupant performed a behavior. Details of the related information will be described later. The related information generation unit 114 outputs the generated related information to the external device 90.

[0028] Next, the hardware configuration of the information processing device 100 will be described with reference to Figures 2 and 3. Figure 2 is a diagram showing an example of the hardware configuration of the information processing device 100, and Figure 3 is a diagram showing an example of the hardware configuration of the information processing device 100 that is different from that shown in Figure 2. For example, as shown in Figure 2, the information processing device 100 is a computer having a processor 100a, a memory 100b, and an I / O port 100c, and is configured so that the processor 100a reads and executes a program stored in the memory 100b.

[0029] 3, the information processing device 100 is a computer that has a processing circuit 100d, which is dedicated hardware, and an I / O port 100c, and executes a program. The processing circuit 100d is configured, for example, by a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. Each function of the information processing device 100 is realized by the processor 100a or the processing circuit 100d, which is dedicated hardware, executing a program. Note that the information processing device 100 may also have hardware other than those described above, such as a hardware timer.

[0030] Next, details of the processing performed by the information processing device 100 will be described with reference to FIGS. 1 and 4 to 6. FIG. 4 is a flowchart showing an example of the processing performed by the information processing device 100 according to the first embodiment. As shown in FIG. 4, when the information processing device 100 starts the processing, it first acquires driver information (step ST01). In this processing, the information processing device 100 acquires driver information related to changes in driving aptitude based on information from the in-vehicle information acquisition device 10. For example, in this processing, the information processing device 100 acquires driver information indicating the driver's psychological state, physiological state, and concentration level based on information from the in-vehicle information acquisition device 10.

[0031] After performing the process of step ST01, the information processing device 100 calculates the driving aptitude level based on the driver information acquired in the process of step ST01 (step ST02). For example, in this process, the information processing device 100 quantifies the driver's psychological state, physiological state, and concentration level based on the driver information acquired by the driver information acquisition unit 101, and calculates the driving aptitude level based on each value obtained by the quantification. Specifically, the information processing device 100 determines whether the driver's psychological state, physiological state, and concentration level are each suitable for driving based on the driver information acquired by the driver information acquisition unit 101, and quantifies the driver's psychological state, physiological state, and concentration level by assigning a value of 1 if the driver's psychological state is suitable for driving and a value of 0 if the driver's physiological state is unsuitable for driving, assigning a value of 1 if the driver's physiological state is suitable for driving and a value of 0 if the driver's concentration level is unsuitable for driving, and assigning a value of 1 if the driver's concentration level is suitable for driving and a value of 0 if the driver's concentration level is unsuitable for driving. Furthermore, if the quantified psychological state, physiological state, and concentration level of the driver are all 1, the information processing device 100 calculates the driving aptitude as 1, and if any one of these is 0, the information processing device 100 calculates the driving aptitude as 0.

[0032] After the processing of step ST02, the information processing device 100 acquires occupant behavior information (step ST03). In this processing, the information processing device 100 acquires occupant behavior information related to the behavior of occupants based on information from the occupant behavior information acquisition unit 103. For example, in this processing, the information processing device 100 acquires, as the occupant behavior information, a binary signal indicating the presence or absence of behavior by the occupant based on the information from the occupant behavior information acquisition unit 103.

[0033] After performing the processing of step ST03, the information processing device 100 generates related information (step ST09). In this processing, the information processing device 100 generates related information in which information indicating a change in the driving aptitude of the vehicle driver and occupant behavior information regarding the behavior of the occupant when the driving aptitude of the vehicle driver changes are associated with each other, based on the driving aptitude calculated by the driving aptitude calculation unit 102 and the occupant behavior information acquired by the occupant behavior information acquisition unit 103. For example, the information processing device 100 generates related information in which information indicating a change in the driving aptitude of the vehicle driver and occupant behavior information indicating whether or not the occupant behaved in a manner that caused the change in the driving aptitude of the vehicle driver were associated with each other at the time of acquisition, based on the driving aptitude calculated by the driving aptitude calculation unit 102, the occupant behavior information acquired by the occupant behavior information acquisition unit 103, and time information acquired by the time information acquisition unit 107.

[0034] FIG. 5 is a graph showing an example of related information generated by the information processing device 100 according to the first embodiment. FIG. 5 illustrates related information obtained when the driving aptitude calculation unit 102 calculates a driving aptitude, which is a binary value of 1 or 0. As shown in FIG. 5 , for example, the driving aptitude indicated by the upper graph and the occupant behavior information indicated by the lower graph are binary time-series data that discretely change between 1 and 0 over time, and constitute related information associated with each other at the time of acquisition. FIG. 5 illustrates that the driving aptitude S(t) is S(t)=1 during the period from t=t0 when driving started to t2, but S(t)=0 during the periods from t=t2 to t3 and from t4 to t6. Meanwhile, the occupant's behavior is illustrated as having been detected during the periods from t=t1 to t3 and from t5 to t6.

[0035] For example, if the driver's driving aptitude declines within a predetermined fourth hour after the occupant's speech or behavior begins, it can be assumed that the occupant's speech or behavior has caused the decline in the driver's driving aptitude. Furthermore, if the driver's driving aptitude declines while the occupant is still speaking or behaving, it can be assumed that the occupant's speech or behavior has caused the decline in the driver's driving aptitude. Furthermore, if the occupant's speech or behavior is of a type that changes the driver's psychological state to a state other than a calm state, and the driver's driving aptitude declines after the occupant's speech or behavior begins, it can be assumed that the occupant's speech or behavior has caused the decline in the driver's driving aptitude.

[0036] Specifically, the decline in driving aptitude during the period from time t=t2 to t3 occurred because the occupant used gestures to talk to the driver during the period from time t=t1 to t3, causing the driver's concentration on driving to decrease, and the driver was in a state of looking away during the period from time t=t2 to t3. The occupant's conversation with the driver ended at time t=t3, causing the driver's concentration on driving to increase, and the driving aptitude recovered. Furthermore, for example, the driver's driving aptitude declined due to a decline in alertness at time t=t4, and the passenger, noticing the driver's decline in alertness, spoke to the driver, tapped him on the shoulder, opened the window, and so on during the period from time t=t5 to t6, causing the driver's driving aptitude to recover at time t=t6. In this way, by referring to the related information, it is possible to know how the driver's driving aptitude changed when the occupant made a comment or action.

[0037] FIG. 6 is a graph showing an example of related information different from that shown in FIG. 5 , generated by the information processing device 100 according to the first embodiment. FIG. 6 shows related information obtained when the driving aptitude calculation unit 102 calculates driving aptitude, which is a continuous value. As shown in FIG. 6 , for example, the driving aptitude shown in the upper graph is time-series data that changes continuously over time, and the occupant behavior information shown in the lower graph is binary time-series data that changes discretely between 1 and 0 over time, constituting related information associated with each other at the time of acquisition. For example, the driving aptitude, which is a continuous value, can be calculated using the following formula (1): driving aptitude S(t)=a×Ps(t)+b×Ph(t)+c×C(t) (1) In formula (1), Ps(t) is a value indicating the psychological state, Ph(t) is a value indicating the physiological state, C(t) is a value indicating the degree of concentration on driving, and a, b, and c are weighting coefficients, respectively.

[0038] In this way, by calculating the driving aptitude as time-series data that continuously changes over time, it is possible to know more detailed changes in the driving aptitude than when it is calculated as binary time-series data. For example, when the driving aptitude is composed of multiple parameters such as psychological state, physiological state, and concentration level, it may be easier to express the driving aptitude as a multi-value or continuous value than as a binary value.

[0039] In this way, the driving aptitude may be a value represented discretely by two or more values, or may be a value represented analogically (continuously) by a continuous value. The related information is generated by accumulating the related information over a certain period of time to include multiple changes in the driving aptitude of the vehicle driver and information on the words and actions of each occupant when the driving aptitude changed multiple times. Note that the driving aptitude may be calculated by the driving aptitude calculation unit 102 as a continuous value as shown in FIG. 6 , and then converted to 1 when the driving aptitude is equal to or greater than a predetermined threshold value of 0.5, or to 1 when the driving aptitude is less than 0.5, thereby constituting the related information.

[0040] Furthermore, when the occupant behavior information is acquired as information indicating the content of the behavior, the related information may be configured, for example, by associating the time-series data shown in Figures 5 and 6 with the occupant behavior information indicating the content of the behavior. For example, the related information may be configured such that, when the related information shown in Figure 5 is displayed on the display device, an operator of the display device selects a portion corresponding to a time in the time-series data of the occupant behavior information, thereby displaying occupant behavior information indicating the content of the occupant's behavior corresponding to the selected time, or such that an operator of the display device selects a portion corresponding to a time in the time-series data of driving aptitude, thereby displaying driver information corresponding to the selected time.

[0041] In general, the occupant's behavior may increase the driver's driving aptitude, decrease the driver's driving aptitude, or have no effect on the driver's driving aptitude. For example, if the occupant's behavior is related to an argument with the driver, the driver's driving aptitude may decrease. Also, if the occupant's behavior is intended to calm the driver when the driver is feeling angry or excited while driving, the driver's driving aptitude may increase. As such, depending on the content of the occupant's behavior, the driver's driving aptitude may increase or decrease. Therefore, it is desirable that the related information include information indicating the content of the occupant's behavior.

[0042] Specifically, the related information may include information indicating the type (attribute) of the occupant's behavior and information indicating the type (attribute) of the reason for the change in driving aptitude. For example, the display device may be configured so that an operator of the display device selects a portion of the time-series data corresponding to a time, thereby displaying information indicating the type (attribute) of the occupant's behavior and information indicating the type (attribute) of the reason for the decline in driving aptitude. Specifically, the related information may be configured to display a character string such as "talking" as an attribute of the occupant's behavior and / or, or to display character strings such as "looking away" or "decreased alertness" as attributes of the reason for the decline in the driver's driving aptitude. By referring to such related information, the operator of the display device can know what behaviors of the occupant influenced the change in the driver's driving aptitude.

[0043] In addition, the related information has color information of the time series data, and may be configured to display in different colors a state in which the driver's driving aptitude is relatively high and a state in which it is relatively low, for example, so that a state in which the driver's driving aptitude is relatively high is displayed in blue and a state in which it is low is displayed in yellow, or may be configured to display in different colors a state in which the occupant's behavior that increased the driver's driving aptitude and a state in which it decreased the driver's driving aptitude are displayed in blue and yellow, respectively.

[0044] After performing the process of step ST09, the information processing device 100 outputs the related information generated by the related information generation unit 114 from the related information generation unit 114 to the external device 90 (step ST15). In this process, the information processing device 100 may be configured to output the generated related information to the external device 90 every short time, for example, every 30 ms when the cycle of the process shown in FIG. 4 is 30 ms, or may be configured to collectively output related information for a certain period of time, for example, several minutes or several hours, to the external device 90, or may be configured to collectively output related information from the time the driver starts driving the vehicle to the time the driver finishes driving the vehicle to the external device 90.

[0045] Furthermore, the information processing device 100 may be configured to collectively output to the external device 90 related information generated up to that point in response to a specific operation of the information processing device 100 by the driver or a passenger, or may be configured to output to the external device 90 at a frequency set based on one or both of the frequency of changes in driving aptitude and the frequency of speech and behavior by the passenger, or one or both of the presence or absence of changes in driving aptitude and the presence or absence of speech and behavior by the passenger. For example, the information processing device 100 may be configured to output related information to the external device 90 every 30 ms when the passenger is speaking or behaving, and to output or not output related information to the external device 90 every 1 s when the passenger is not speaking or behaving. Furthermore, for example, the information processing device 100 may be configured to set the frequency of outputting related information to the external device 90 based on the status of surrounding objects, such as the relative position between the vehicle and surrounding objects, based on information from a surrounding object detection device (not shown) that detects surrounding objects of the vehicle.

[0046] Furthermore, for example, the information processing device 100 extracts, from the generated related information, related information generated during a specific period including the time when the driver's driving aptitude changed, and outputs the extracted related information to the external device 90. Specifically, the information processing device 100 extracts, from the generated related information, related information generated during a period including the time when the driver's driving aptitude fell below a predetermined threshold, from the time up to a predetermined specific first hour, and related information generated during a period from the time up to a predetermined specific second hour, and outputs the extracted related information to the external device 90. Furthermore, for example, the information processing device 100 excludes some or all of the related information generated during a specific period from the generated related information and outputs the extracted related information to the external device 90. Specifically, the information processing device 100 excludes all of the related information generated during a period when the driver's driving aptitude remained high and did not change from the generated related information and outputs the extracted related information to the external device 90. Furthermore, specifically, the information processing device 100 excludes from the generated related information video information included in related information generated during a period in which the driver's driving aptitude remained unchanged at a high value, and outputs the generated related information to the external device 90. Furthermore, specifically, when the occupant behavior information included in the generated related information includes multiple types of information, such as binary information indicating the presence or absence of behavior by the occupant, audio information indicating the content of the behavior by the occupant, and video information indicating the content of the behavior by the occupant, the information processing device 100 excludes from these multiple types of information a specific type of information included in the related information generated during a period in which the driver's driving aptitude remained unchanged at a high value, and outputs the resulting information to the external device 90. This configuration enables the information processing device 100 to reduce the amount of information stored in the external device 90 and the amount of information to be processed by the external device 90.

[0047] After the processing of step ST15, the information processing device 100 determines whether the driver has finished driving (step ST16). In this processing, the information processing device 100 may determine the end of driving based on, for example, a signal indicating that the vehicle has been parked from a vehicle control device (not shown), or based on the stop of power supply to the information processing device 100. If the driving has not finished in the processing of step ST16 (NO in step ST16), the information processing device 100 returns the processing to step ST01 and performs the processing from step ST01 to step ST16 again. If the driving has finished in the processing of step ST16 (YES in step ST16), the information processing device 100 ends the processing.

[0048] As described above, the information processing device 100 according to embodiment 1 includes a driver information acquisition unit that acquires information about the vehicle driver regarding changes in the driver's driving aptitude while driving the vehicle, a driving aptitude calculation unit that calculates the driving aptitude of the vehicle driver based on the information acquired by the driver information acquisition unit, an occupant behavior information acquisition unit that acquires information about the behavior of occupants other than the vehicle driver, and an associated information generation unit that generates associated information that correlates information indicating changes in the vehicle driver's driving aptitude and information about the behavior of occupants when the vehicle driver's driving aptitude changes, based on the driving aptitude calculated by the driving aptitude calculation unit and the information acquired by the occupant behavior information acquisition unit.

[0049] With this configuration, the information processing device 100 generates associated information that correlates information indicating changes in the vehicle driver's driving aptitude with information indicating the occupant's verbal behavior when the vehicle driver's driving aptitude changes, thereby making it possible to obtain information indicating how the occupant's verbal behavior affects changes in driving aptitude during driving. Furthermore, by referring to the generated associated information after driving, vehicle stakeholders, such as the driver, occupant, the driver and occupant's family members, the vehicle manager, and the supervisor of work performed using the vehicle, can learn how the occupant's verbal behavior affects changes in driving aptitude during driving, which can be useful for managing the driver's driving aptitude. Furthermore, by having a vehicle control device refer to the generated associated information, it is possible to provide driving assistance to the driver that takes changes in driving aptitude into consideration when the driver drives the vehicle.

[0050] 7 to 9, an information processing system 2 according to the second embodiment will be described. The information processing system 2 according to the second embodiment differs from the information processing system 1 according to the first embodiment in the configuration for performing notification based on related information, but the other configurations are the same. The same names and symbols as those in the first embodiment are used for the same configurations as those in the first embodiment, and the description thereof will be omitted.

[0051] FIG. 7 is a block diagram showing a schematic configuration of an information processing system 2 according to the second embodiment. As shown in FIG. 7 , the information processing system 2 includes an in-vehicle information acquisition device 10, an information processing device 200, a storage device 60, and an alarm device 70. The storage device 60 stores information acquired from the information processing device 200. For example, the storage device 60 is configured as part of a drive recorder provided in the vehicle and includes a hard disk, a solid state drive (SSD), a read only memory (ROM), a random access memory (RAM), etc., and accumulates and stores information acquired from the information processing device 200. Note that the storage device 60 may be provided outside the vehicle.

[0052] The notification device 70 notifies the driver of the information acquired from the information processing device 200. For example, the notification device 70 is disposed in a position such as a dashboard where the driver and passengers can recognize the content of the notification, and is configured by a device capable of notifying information such as a liquid crystal display device, an LED light-emitting device, a speaker, etc., and outputs at least one of a video and a sound indicating the information acquired from the information processing device 200, thereby visually and audibly notifying the driver of the information acquired from the information processing device 200. The notification device 70 may be provided outside the vehicle.

[0053] As shown in FIG. 7 , the information processing device 200 includes a driver information acquisition unit 101, a driving aptitude calculation unit 102, an occupant speech and behavior information acquisition unit 103, a time information acquisition unit 107, and a control unit 210. The control unit 210 includes a related information generation unit 114, a notification determination unit 215, and a notification information generation unit 216. The notification determination unit 215 determines whether to notify the driver of the related information based on the driving aptitude calculated by the driving aptitude calculation unit 102. For example, the notification determination unit 215 determines whether to notify the driver of the related information based on whether the driving aptitude is less than a predetermined threshold value. Furthermore, for example, in a case where the driving aptitude is composed of multiple parameters such as a psychological state, a physiological state, and a concentration level, the notification determination unit 215 determines whether to notify the driver of the related information based on whether at least one of the multiple parameters is less than a predetermined threshold value.

[0054] The notification information generation unit 216 generates notification information for causing the notification device 70 to notify that the driving aptitude of the vehicle driver has changed in association with the behavior of the occupant, based on the related information generated by the related information generation unit 114. In other words, the notification information generation unit 216 generates notification information indicating the content of the related information to be notified by the notification device 70, based on the related information generated by the related information generation unit 114. For example, the notification information generation unit 216 generates notification information indicating the content of the related information to be notified by extracting some information from the related information generated by the related information generation unit 114. Specifically, the notification information generation unit 216 generates notification information indicating the content of the related information to be notified by extracting, from the related information generated by the related information generation unit 114, related information that was generated during a period from the time the driver's driving aptitude decreased to a predetermined specific third hour before that. More specifically, the notification information generation unit 216 extracts, from the related information generated by the related information generation unit 114, related information generated during a period from the time the driver's driving aptitude declined to a predetermined specific third hour before, and generates notification information including information indicating the content of the occupant behavior information included in the extracted related information and information indicating the decline in driving aptitude. The notification information generation unit 216 processes and modifies the related information generated by the related information generation unit 114 to generate notification information indicating the content of the related information used for notification. The notification information generation unit 216 may generate notification information for causing the notification device 70 to notify that the vehicle driver's driving aptitude has increased (returned to the level before the decline) in relation to the behavior of the occupant, based on the related information generated by the related information generation unit 114.

[0055] The hardware configuration of the information processing device 200 according to the second embodiment is similar to the hardware configuration of the information processing device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0056] Next, details of the processing performed by the information processing device 200 will be described with reference to Figures 7 to 9. Figure 8 is a flowchart showing an example of processing performed by the information processing device 200 according to embodiment 2. Note that some of the processing performed by the information processing device 200 according to embodiment 2 is similar to the processing performed by the information processing device 100 according to embodiment 1, and therefore, processing similar to that performed by the information processing device 100 according to embodiment 1 will be assigned the same reference numerals and description thereof will be omitted.

[0057] 8 , after performing the process of step ST09, the information processing device 200 determines whether the driving aptitude is less than a preset threshold value (step ST11). In this process, the information processing device 200 compares the driving aptitude calculated by the driving aptitude calculation unit 102 with a preset threshold value, and determines whether the driving aptitude is less than the threshold value, thereby determining whether to notify the related information generated by the related information generation unit 114.

[0058] If the driving aptitude is less than the threshold value in the processing of step ST11 (step ST11), the information processing device 200 generates notification information (step ST13) and outputs the notification information from the notification information generation unit 216 to the notification device 70 (step ST14).

[0059] FIG. 9A is a graph showing an example of related information notified by the information processing device 200 according to the second embodiment, and FIG. 9B is a diagram showing the classification names of icons attached to the related information. As shown in FIG. 9A , for example, the information processing device 200 generates related information in which information on multiple types of icons is added as information indicating the classification of the content of the occupant's speech and behavior to the related information generated by the information processing device 100 according to the first embodiment, and notifies the related information by displaying the generated related information on the notification device 70 as a display device. By displaying the related information including information indicating the content of the occupant's speech and behavior in this manner, the information processing device 200 allows a viewer of the notification device 70 to know what type of speech and behavior of the occupant affects changes in driving aptitude. Note that the information processing device 200 may be configured to cause the notification device 70 to display the classification names of the icons shown in FIG. 9B as character strings instead of the icons shown in FIGS. 9A and 9B .

[0060] As described above, the information processing device 200 according to the second embodiment includes the notification information generation unit 216 that generates notification information for causing the notification device 70 to notify that the driving aptitude of the driver of the vehicle has changed in association with the behavior of the occupant, based on the related information generated by the related information generation unit 114. As configured in this way, the information processing device 200 according to the second embodiment can notify related parties of the vehicle, such as the driver, occupants, families of the driver and occupants, a vehicle manager, and a supervisor of work performed using the vehicle, of information indicating how the behavior of the occupant affects changes in driving aptitude while driving, by notifying such parties.

[0061] In the second embodiment, the information processing device 200 is configured to notify the driver of the relevant information when the driving aptitude is less than a threshold value, but is not limited to this. The information processing device may be configured to make a notification based on any trigger. For example, the information processing device may be configured to notify the driver of all the relevant information generated up to that point in response to a specific operation on the notification device 70, or may be configured to notify the driver of the relevant information when the vehicle is parked at home or at a destination, or may be configured to automatically notify the driver of the relevant information when the driver finishes driving if the driver's driving aptitude is estimated to have decreased due to the driver's behavior while driving.

[0062] Third Embodiment Next, an information processing system 3 according to a third embodiment will be described with reference to Figures 10 and 11. The information processing system 3 according to the third embodiment differs from the information processing system 2 according to the second embodiment in the configuration for generating related information including vehicle position information, but the other configurations are the same, and the same names and symbols as those in the second embodiment will be used to omit the description thereof.

[0063] Fig. 10 is a block diagram showing a schematic configuration of an information processing system 3 according to embodiment 3. As shown in Fig. 10, the information processing system 3 according to embodiment 3 includes an in-vehicle information acquisition device 10, a position information acquisition device 20, an information processing device 300, a storage device 60, and a notification device 70. The position information acquisition device 20 is configured, for example, by a GNSS receiver provided in the vehicle, and acquires information indicating the location of the vehicle. The position information acquisition device 20 outputs the acquired position information to the information processing device 300.

[0064] 10 , the information processing device 300 includes a driver information acquisition unit 101, a driving aptitude calculation unit 102, an occupant behavior information acquisition unit 103, a position information acquisition unit 304, a time information acquisition unit 107, and a control unit 310. The position information acquisition unit 304 acquires information indicating the position where the vehicle is located in association with the time, based on information from the position information acquisition device 20.

[0065] The control unit 310 includes a related information generation unit 314, a notification determination unit 215, and a notification information generation unit 316. The related information generation unit 314 generates related information in which information indicating a change in the driving aptitude of the vehicle driver, information regarding the behavior of the occupant when the driving aptitude of the vehicle driver changed, and vehicle location information when the driving aptitude of the vehicle driver changed are associated with each other at the time of acquisition, based on the driving aptitude calculated by the driving aptitude calculation unit 102 and the location information acquired by the location information acquisition unit 304. For example, the related information generation unit 314 generates related information including information indicating a location on the map where the behavior of the occupant occurred and information indicating a location on the map where the driver's driving aptitude changed, based on map information acquired in advance and the location information acquired by the location information acquisition unit 304.

[0066] The notification information generation unit 316 generates notification information including vehicle location information based on the related information generated by the related information generation unit 314, in order to notify the notification device 70 that the driving aptitude of the vehicle driver has changed in relation to the words and actions of the occupants.

[0067] 11 is a diagram showing map information included in related information notified by the information processing device 300 according to the third embodiment. As shown in FIG. 11 , the information processing device 300 displays information indicating the route traveled by the vehicle on the map, the time when the vehicle was located at a specific position on the map, the driving aptitude at the specific position on the map, and occupant behavior information at the specific position on the map on the notification device 70, which is a display device. For example, FIG. 11 shows that S(t)=0 during the periods from time t=t2 to t3 and from t4 to t6. FIG. 11 also shows that the driver's psychological state was "irritated" during the period from time t=t2 to t3, and that the occupant engaged in "teasing" behavior when the driver was in that psychological state. 11. It also shows that the driver's physiological state was in a "decreased alertness" state during the period from time t=t4 to t6, and that the occupant's "assistance" behavior while the driver was in that physiological state caused the driver's physiological state to return to a normal state from the "decreased alertness" state. Note that the information processing device may be configured to generate both the information shown in FIG. 11 and the information generated by the related information generation unit 114 of the second embodiment, or may be configured to generate only one of them.

[0068] The hardware configuration of the information processing device 300 according to the third embodiment is similar to the hardware configuration of the information processing device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0069] As described above, the information processing device 300 according to the third embodiment includes a position information acquisition unit 304 that acquires vehicle position information, and is configured to generate associated information that correlates information indicating a change in the driving aptitude of the vehicle driver with the vehicle position information when the driving aptitude of the vehicle driver changes, based on the driving aptitude calculated by the driving aptitude calculation unit 102 and the position information acquired by the position information acquisition unit 304. With this configuration, the information processing device 300 according to the third embodiment can acquire information indicating how the occupant's behavior and the location on the map affect changes in driving aptitude during driving. For example, the associated information generated by the information processing device 300 makes it possible to know changes in driving aptitude that take location on the map into consideration, such as whether the driver's driving aptitude is likely to decrease at a specific location on the map.

[0070] 12 to 14, an information processing system 4 according to a fourth embodiment will be described. The information processing system 4 according to the fourth embodiment differs from the information processing system 3 according to the third embodiment in the configuration related to the degree of contact indicating the possibility of contact between the vehicle and a surrounding object, but the other configurations are the same. The same names and symbols as those in the third embodiment will be used for the same configurations as those in the third embodiment, and the description thereof will be omitted.

[0071] 12 is a block diagram showing a schematic configuration of an information processing system 4 according to embodiment 4. As shown in FIG. 12 , the information processing system 4 according to embodiment 4 includes an in-vehicle information acquisition device 10, a position information acquisition device 20, a peripheral object detection device 30, a vehicle information acquisition device 40, an information processing device 400, a storage device 60, and an alarm device 70. The peripheral object detection device 30 detects the positions of objects around the vehicle. For example, the peripheral object detection device 30 is configured with an image sensor, a LiDAR sensor, a millimeter-wave sensor, etc., and detects the positions of other vehicles, pedestrians, road structures, lane markings, etc. as peripheral objects located around the vehicle.

[0072] The vehicle information acquisition device 40 detects information related to the traveling of the vehicle. For example, the vehicle information acquisition device 40 is configured with a speed sensor that detects the speed of the vehicle, an acceleration sensor that detects the acceleration of the vehicle, a sensor that detects the direction of movement of the vehicle, a sensor that detects the operation angle of the steering wheel, a sensor that detects the operation amount of the accelerator pedal, a sensor that detects the operation amount of the brake pedal, etc.

[0073] 12 , the information processing device 400 includes a driver information acquisition unit 101, a driving aptitude calculation unit 102, an occupant speech and behavior information acquisition unit 103, a position information acquisition unit 304, a peripheral object information acquisition unit 405, a vehicle information acquisition unit 406, a time information acquisition unit 107, and a control unit 410. The peripheral object information acquisition unit 405 acquires peripheral object information indicating the positions, moving speeds, and moving directions of objects around the vehicle based on information from the peripheral object detection device 30. The vehicle information acquisition unit 406 acquires vehicle information relating to the vehicle's motion state, indicating the vehicle speed, vehicle acceleration, moving direction, etc., as information relating to the vehicle's traveling, based on information from the vehicle information acquisition device 40.

[0074] The control unit 410 includes a contact degree calculation unit 411, a related information generation unit 414, a notification determination unit 415, and a notification information generation unit 416. The contact degree calculation unit 411 calculates a contact degree indicating the degree of possibility of contact between the vehicle and a peripheral object based on information acquired by the peripheral object information acquisition unit 405 and the vehicle information acquisition unit 406. For example, the contact degree calculation unit 411 calculates a contact degree value that is greater the shorter the distance between the vehicle and the peripheral object. Furthermore, for example, if the vehicle does not perform an evasive action to avoid the peripheral object, the contact degree calculation unit 411 calculates a contact degree value that is greater the shorter the time until contact. Furthermore, for example, if the vehicle violates laws and regulations, such as lane departure or speeding, the contact degree calculation unit 411 calculates a contact degree value that is greater than the value when the vehicle does not violate laws and regulations. Furthermore, for example, the contact degree calculation unit 411 calculates a contact degree value that is greater the impact of contact between the vehicle and the peripheral object. For example, if the surrounding object is a pedestrian, the contact degree is calculated to be a larger value than if the surrounding object is a roadside tree. For example, the contact degree calculation unit 411 calculates the contact degree as a numerical value ranging from 0, which is the smallest contact degree, to 1, which is the largest contact degree. Note that the contact degree may be expressed as a binary value, a multi-value value, or a continuous value. Furthermore, any method may be adopted as a specific method for calculating the contact degree.

[0075] The related information generation unit 414 generates related information including the degree of contact calculated by the contact degree calculation unit 411. The notification determination unit 415 determines whether to issue a notification based on the magnitude of the degree of contact calculated by the contact degree calculation unit 411. The notification information generation unit 416 generates information to be notified by the notification device 70 based on the related information generated by the related information generation unit 414.

[0076] The hardware configuration of the information processing device 400 according to the fourth embodiment is similar to the hardware configuration of the information processing device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0077] 13 is a flowchart showing an example of processing performed by the information processing device 400 according to embodiment 4. The information processing device 400 calculates the contact degree in step ST06 based on the surrounding object information acquired in step ST04 and the vehicle information acquired in step ST05. If the driving aptitude is less than a preset threshold and the contact degree is equal to or greater than a preset threshold (YES in step ST12), notification information is generated (step ST33) and the notification device 70 issues a notification. Note that the trigger for the notification is not limited to this, and the notification may be triggered by a specific operation by a person related to the vehicle on the information processing device 400 or the notification device 70, or by the end of driving. Furthermore, in the processing of step ST15, the information processing device 400 outputs related information including information indicating the contact degree in addition to the information output by the information processing devices according to embodiments 1 to 3. In addition, the information processing device 400 may be further configured to output related information including information indicating the type of contact object related to the calculated contact degree, the distance until contact, the time until contact, etc.

[0078] FIG. 14 is a graph showing an example of related information generated by the information processing device 400 according to the fourth embodiment. For example, FIG. 14 shows that the degree of contact increases at each of times t=tx1, tx2, tx3, tx4, and tx5. FIG. 14 also shows that the degree of contact increases due to the occupant's behavior during the period from time t1 to t3. FIG. 14 also shows that the degree of contact increases at times t=tx4 and tx5 during the period from time t=t4 to t6 due to the occupant's behavior during the period from time t5 to t6, but that no increase in the degree of contact occurs thereafter. The information processing device 400 may be configured to notify the type of the degree of contact using the notification device 70, which is a display device, or may be configured to display a video or image of the surroundings of the vehicle when a specific position on the graph of the degree of contact is selected by an operator of the display device.

[0079] 15 and 16, an information processing system 5 according to the fifth embodiment will be described. The information processing system 5 according to the fifth embodiment differs from the information processing system 4 according to the fourth embodiment in the configuration of a part of the control unit, but the other configurations are the same. The same components as those in the fourth embodiment are given the same names and symbols as those in the fourth embodiment, and the description thereof will be omitted.

[0080] Fig. 15 is a block diagram showing a schematic configuration of an information processing system 5 according to embodiment 5. As shown in Fig. 15, the information processing system 5 according to embodiment 5 includes an in-vehicle information acquisition device 10, a position information acquisition device 20, a peripheral object detection device 30, a vehicle information acquisition device 40, an information processing device 500, a storage device 60, and an alarm device 70. Also, as shown in Fig. 15, the information processing device 500 includes a driver information acquisition unit 101, a driving aptitude calculation unit 102, an occupant speech and behavior information acquisition unit 103, a position information acquisition unit 304, a peripheral object information acquisition unit 405, a vehicle information acquisition unit 406, a time information acquisition unit 107, and a control unit 510.

[0081] The control unit 510 includes a contact degree calculation unit 411, a recognition detection unit 512, a recognition time calculation unit 513, a related information generation unit 514, a notification determination unit 415, and a notification information generation unit 516. The recognition detection unit 512 detects, based on information acquired by the driver information acquisition unit 101, that the driver of the vehicle has recognized a factor that will increase the degree of contact with the vehicle. For example, the recognition detection unit 512 detects, based on driver information indicating the driver's line of sight acquired by the driver information acquisition unit 101, that the driver of the vehicle has recognized a factor that will increase the degree of contact with the vehicle. Specifically, the recognition detection unit 512 detects, based on the driver's line of sight being directed in the direction of a surrounding object that has the greatest impact on an increase in the degree of contact when the contact degree calculation unit 411 calculates the degree of contact, that the driver of the vehicle has recognized a factor that will increase the degree of contact with the vehicle.

[0082] The recognition time calculation unit 513 calculates a recognition time from when the vehicle driver's contact level with the vehicle exceeds a preset threshold to when the vehicle driver recognizes a factor that increases the contact level with the vehicle, based on the detection result by the recognition detection unit 512. The related information generation unit 514 generates related information such that the recognition time calculated by the recognition time calculation unit 513 is included in the related information. The notification information generation unit 516 generates notification information for causing the notification device 70 to notify that the vehicle driver's driving aptitude has changed in relation to the behavior of the occupant, based on the related information generated by the related information generation unit 514.

[0083] The hardware configuration of the information processing device 500 according to the fifth embodiment is similar to the hardware configuration of the information processing device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0084] 16 is a flowchart showing an example of processing performed by the information processing device 500 according to embodiment 5. In the processing of step ST07, the information processing device 500 detects that the vehicle driver has recognized a factor that increases the degree of vehicle contact, and in the processing of step ST08, calculates a recognition time from when the degree of vehicle contact exceeds a preset threshold until the vehicle driver recognizes the factor that increases the degree of vehicle contact. Furthermore, in the processing of step ST39, the information processing device 500 generates related information so that the recognition time calculated by the recognition time calculation unit 513 is included in the related information, and in the processing of step ST43, generates notification information for causing the notification device 70 to notify that the vehicle driver's driving aptitude has decreased in relation to the behavior of the occupant.

[0085] Sixth Embodiment Next, an information processing system 6 according to a sixth embodiment will be described with reference to Figures 17 and 18. The information processing system 6 according to the sixth embodiment differs from the information processing system 1 according to the first embodiment in the configuration related to vehicle driving assistance, but the other configurations are the same, and the same names and symbols as those in the first embodiment will be used and descriptions thereof will be omitted.

[0086] FIG. 17 is a block diagram showing a schematic configuration of an information processing system 6 according to embodiment 6. As shown in FIG. 17 , the information processing system 6 according to embodiment 6 includes an in-vehicle information acquisition device 10, a driving assistance device 50, an information processing device 600, and a storage device 60. The driving assistance device 50 controls at least one of the speed and traveling direction of the vehicle to provide driving assistance for assisting the driver when driving the vehicle. For example, the driving assistance device 50 provides driving assistance such as controlling the speed of the vehicle so that a specific inter-vehicle distance is maintained between the vehicle and other vehicles in front and behind the vehicle, controlling the speed of the vehicle so that the speed of the vehicle becomes a preset speed, steering to prevent the vehicle from deviating from its lane using a Lane Keeping Assist System (LKAS), controlling the speed of the vehicle so that the vehicle does not run parallel to adjacent vehicles, and other driving assistance using automated driving.

[0087] 17 , the information processing device 600 includes a driver information acquisition unit 101, a driving aptitude calculation unit 102, an occupant speech and behavior information acquisition unit 103, a time information acquisition unit 107, and a control unit 610. The control unit 610 includes a related information generation unit 114 and a driving assistance information generation unit 616. The driving assistance information generation unit 616 generates driving assistance information for the driving assistance device 50 to set the content of vehicle control based on a change in the driving aptitude of the vehicle driver. The driving assistance information generation unit 616 sets the control content of the driving assistance device 50 by outputting the generated driving assistance information to the driving assistance device 50.

[0088] The hardware configuration of the information processing device 600 according to the sixth embodiment is similar to the hardware configuration of the information processing device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0089] 18 is a flowchart showing an example of processing performed by the information processing device 600 according to the sixth embodiment. In the processing of step ST11, if the driving aptitude is less than a preset threshold (YES in step ST11), the information processing device 600 generates driving assistance information that changes the control of the driving assistance device 50 so as to increase the degree of vehicle autonomy provided by the driving assistance device 50 (step ST52). In other words, in this processing, the driving assistance information generation unit 616 generates driving assistance information that changes the control of the driving assistance device 50 so as to reduce the degree of driver intervention when driving the vehicle, based on the fact that the driving aptitude is less than the preset threshold. In other words, in this processing, the driving assistance information generation unit 616 generates driving assistance information that changes the control of the driving assistance device 50 so as to increase the driving assistance provided by the driving assistance device 50, based on the fact that the driving aptitude is less than the preset threshold. Also, for example, in this process, the driving assistance information generation unit 616 generates driving assistance information that changes the control of the driving assistance device 50 so as to reduce the possibility of contact between the vehicle and surrounding objects, based on the fact that the driving suitability is below a predetermined threshold value.

[0090] Specifically, in the process of step ST52, the driving assistance information generation unit 616 generates driving assistance information for controlling the vehicle speed by the driving assistance device 50 so as to increase the inter-vehicle distance between the vehicle and other vehicles in front and behind. Also, specifically, in the process of step ST52, the driving assistance information generation unit 616 generates driving assistance information for causing the driving assistance device 50 to reduce the vehicle speed. Also, specifically, in the process of step ST52, the driving assistance information generation unit 616 generates driving assistance information for increasing the level of autonomous driving by the driving assistance device 50. Also, specifically, in the process of step ST52, the driving assistance information generation unit 616 generates driving assistance information for causing the driving assistance device 50 to control the vehicle speed so as to avoid running parallel to other adjacent vehicles. Note that the phrase "generating driving assistance information with a normal degree of autonomy" in ST53 may be interpreted as setting the driving assistance information to driving assistance information stored in advance before the vehicle started traveling, or to driving assistance information manually set by the driver, or may be interpreted as restoring the original setting.

[0091] Seventh Embodiment Next, an information processing system 7 according to a seventh embodiment will be described with reference to Fig. 19 and Fig. 20. The information processing system 7 according to the seventh embodiment differs from the information processing system 2 according to the second embodiment in the configuration of a part of the control unit, but the other configurations are the same, and the same names and symbols as those in the second embodiment are used for the same configurations as those in the second embodiment, and the description thereof will be omitted.

[0092] Fig. 19 is a block diagram showing a schematic configuration of an information processing system 7 according to embodiment 7. As shown in Fig. 19, the information processing system 7 according to embodiment 7 includes an in-vehicle information acquisition device 10, an information processing device 700, a storage device 60, and an alarm device 70. Also, as shown in Fig. 19, the information processing device 700 includes a driver information acquisition unit 101, a driving aptitude calculation unit 102, an occupant speech and behavior information acquisition unit 103, a time information acquisition unit 107, and a control unit 710. The control unit 710 includes a related information generation unit 114, a similarity determination unit 715, and an alarm information generation unit 216.

[0093] The similarity determination unit 715 determines, based on the related information generated by the related information generation unit, whether information regarding the occupant's speech and behavior when the vehicle driver's driving aptitude deteriorated at a specific time and information regarding the occupant's speech and behavior at a time after the specific time are similar. In other words, the similarity determination unit 715 determines, based on the related information generated by the related information generation unit, whether information regarding the occupant's speech and behavior when the vehicle driver's driving aptitude deteriorated at a specific time in the past and newly acquired information regarding the occupant's speech and behavior are similar. For example, the similarity determination unit 715 extracts features from the content of the occupant's speech and behavior when the vehicle driver's driving aptitude deteriorated at a specific time in the past and the newly acquired content of the occupant's speech and behavior, and compares the extracted features to determine whether they are similar.

[0094] The hardware configuration of the information processing device 700 according to the seventh embodiment is similar to the hardware configuration of the information processing device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0095] 20 is a flowchart showing an example of processing performed by the information processing device 700 according to the seventh embodiment. In the processing of step ST60, the information processing device 700 determines whether the newly acquired information on the occupant's behavior is similar to information on the occupant's behavior when the driving aptitude of the vehicle driver deteriorated at a specific time in the past. If it is determined that the information is similar in the processing of step ST60 (YES in step ST60), the information processing device 700 generates notification information for causing the notification device 70 to notify that the vehicle driver's driving aptitude has deteriorated in relation to the occupant's behavior, based on the related information generated by the related information generation unit 114 (step ST63).

[0096] In addition, when the information processing device determines that newly acquired information regarding the behavior and speech of the occupant is similar to information regarding the behavior and speech of the occupant when the driving aptitude of the vehicle driver declined at a specific time in the past, the information processing device may be configured to generate driving assistance information that changes the control of the driving assistance device to increase the degree of autonomy of the vehicle by the driving assistance device, or may be configured to generate driving assistance information that changes the control of the driving assistance device to reduce the possibility of contact between the vehicle and surrounding objects.

[0097] 21 and 22, an information processing system 8 according to an eighth embodiment will be described. The information processing system 8 according to the eighth embodiment differs from the information processing system 2 according to the second embodiment in the configuration of a part of the control unit, but the other configurations are the same. The same components as those in the second embodiment are given the same names and symbols as those in the second embodiment, and the description thereof will be omitted.

[0098] Fig. 21 is a block diagram showing a schematic configuration of an information processing system 8 according to embodiment 8. As shown in Fig. 21 , the information processing system 8 according to embodiment 8 includes an in-vehicle information acquisition device 10, an information processing device 800, a storage device 60, and an alarm device 70. Also, as shown in Fig. 21 , the information processing device 800 includes a driver information acquisition unit 101, a driving aptitude calculation unit 102, an occupant speech and behavior information acquisition unit 103, a time information acquisition unit 107, and a control unit 810. The control unit 810 includes a learning model generation unit 814, an alarm determination unit 815, and an alarm information generation unit 816.

[0099] The learning model generation unit 814 uses the driving aptitude calculated by the driving aptitude calculation unit 102 and the information acquired by the occupant behavior information acquisition unit 103 as training data, and generates a learning model that outputs the driving aptitude of the vehicle driver based on the input of the information acquired by the occupant behavior information acquisition unit 103.

[0100] The hardware configuration of the information processing device 800 according to the eighth embodiment is similar to the hardware configuration of the information processing device 100 according to the first embodiment, and therefore description thereof will be omitted.

[0101] FIG. 22 is a flowchart showing an example of processing performed by the information processing device 800 according to the eighth embodiment. The processing shown in FIG. 22 infers driving aptitude using a learning model generated by the learning model generation unit 814 and performs notification based on the inference result. In the processing of step ST70, the information processing device 800 inputs information acquired by the occupant behavior information acquisition unit 103 into the learning model generated by the learning model generation unit 814, and infers the driver's driving aptitude. After performing the processing of step ST70, the information processing device 800 causes the notification determination unit 815 to determine whether the driving aptitude output (inferred) by the learning model is less than a preset threshold (step ST71). In the processing of step ST71, if the driving aptitude is less than the threshold (YES in step ST71), the notification information generation unit 816 generates notification information to cause the notification device 70 to notify that the vehicle driver's driving aptitude has deteriorated (step ST72).

[0102] In any of the above-described embodiments, the information processing device may include some or all of the other components of the information processing system, or some of the components of the information processing device may be provided in an external device that is communicatively connected to the information processing device.

[0103] In any of the above-described embodiments, the information processing device may include some or all of the other components of the information processing system, or some of the components of the information processing device may be provided in an external device that is communicatively connected to the information processing device.

[0104] In addition, the present disclosure allows for free combination of the respective embodiments, modification of any of the components of the respective embodiments, or omission of any of the components of the respective embodiments.

[0105] The information processing device according to the present disclosure can determine how the occupant's words and actions affect changes in driving aptitude while driving, and can therefore be used, for example, to provide driving assistance to the driver in accordance with the occupant's words and actions.

[0106] REFERENCE SIGNS LIST 1 Information processing system, 2 Information processing system, 3 Information processing system, 4 Information processing system, 5 Information processing system, 6 Information processing system, 7 Information processing system, 8 Information processing system, 10 In-vehicle information acquisition device, 11 Driver information detection unit, 12 Occupant information detection unit, 20 Position information acquisition device, 30 Peripheral object detection device, 40 Vehicle information acquisition device, 50 Driving assistance device, 60 Storage device, 70 Notification device, 90 External device, 100 Information processing device, 101 Driver information acquisition unit, 102 Driving aptitude calculation unit, 103 Occupant speech and behavior information acquisition unit, 107 Time information acquisition unit, 114 Related information generation unit, 200 Information processing device, 210 Control unit, 215 Notification determination unit, 216 Notification information generation unit, 300 Information processing device, 304 Position information acquisition unit, 310 Control unit, 314 Related information generation unit, 316 Notification information generation unit, 400 Information processing device, 405 Peripheral object information acquisition unit, 406 Vehicle information acquisition unit, 410 Control unit, 411 Contact degree calculation unit, 414 Related information generation unit, 415 Notification determination unit, 416 Notification information generation unit, 500 Information processing device, 510 Control unit, 512 Recognition detection unit, 513 Recognition time calculation unit, 514 Related information generation unit, 516 Notification information generation unit, 600 Information processing device, 610 Control unit, 616 Driving assistance information generation unit, 700 Information processing device, 710 Control unit, 715 Similarity determination unit, 800 Information processing device, 810 Control unit, 814 Learning model generation unit, 815 Notification determination unit, 816 Notification information generation unit.

Claims

1. A driver information acquisition unit acquires information about the driver of the vehicle regarding changes in the driver's driving aptitude while the vehicle is in operation, A driving aptitude calculation unit calculates the driving aptitude of the vehicle's driver based on the information acquired by the driver information acquisition unit, A crew member behavior information acquisition unit acquires information regarding the actions and statements of crew members other than the driver of the vehicle, The system comprises a related information generation unit that generates related information, which is a combination of the driving aptitude calculated by the driving aptitude calculation unit and the information acquired by the occupant behavior information acquisition unit, and which generates related information that is a combination of information indicating a change in the driving aptitude of the vehicle's driver and information regarding the occupant's behavior when the driving aptitude of the vehicle's driver changes. An information processing device characterized by the following:

2. The related information generation unit outputs related information to the storage device so that the related information generated during a specific period while the vehicle is in operation is stored in the storage device. The information processing apparatus according to claim 1, characterized in that it is a product of the present invention.

3. The vehicle is equipped with a notification information generation unit that generates notification information based on the related information generated by the related information generation unit, to notify the notification device that the driver's suitability for driving the vehicle has changed in relation to the words and actions of the occupants. The information processing apparatus according to claim 1, characterized in that it is a product of the present invention.

4. The aforementioned driving aptitude calculation unit calculates the driving aptitude of the vehicle driver based on values ​​relating to at least one of the psychological state and physiological state of the vehicle driver, which are calculated based on the information acquired by the driver information acquisition unit. The information processing apparatus according to claim 1, characterized in that it is a product of the present invention.

5. The driving aptitude calculation unit calculates the driving aptitude of the vehicle driver based on the degree of concentration the vehicle driver has on driving, which is calculated based on the information acquired by the driver information acquisition unit. The information processing apparatus according to claim 1, characterized in that it is a product of the present invention.

6. The vehicle is equipped with a contact degree calculation unit that calculates the degree of possibility of contact with surrounding objects, The contact degree calculation unit calculates the degree of contact of the vehicle based on the position of the surrounding objects relative to the vehicle. The related information generation unit generates related information based on the information acquired by the occupant behavior information acquisition unit and the contact degree calculated by the contact degree calculation unit, which includes information indicating a change in the driver's suitability of the vehicle's driver, information regarding the occupant's behavior when the driver's suitability of the vehicle's driver changes, and information regarding the occupant's behavior when the vehicle's contact degree changes, all of which are linked to each other. The information processing apparatus according to claim 1, characterized in that it is a product of the present invention.

7. A recognition detection unit detects, based on the information acquired by the driver information acquisition unit, that the driver of the vehicle has recognized a factor that increases the vehicle's contact rate. The system includes a recognition time calculation unit that calculates the time from when the vehicle's contact level exceeds a preset threshold until the driver recognizes the factor, based on the detection results from the recognition detection unit. The related information generation unit generates related information based on the information acquired by the occupant behavior information acquisition unit, the contact degree calculated by the contact degree calculation unit, and the calculation result by the cognitive time calculation unit, which is a relationship between information indicating a change in the vehicle's contact degree, information regarding the occupant's behavior when the vehicle's contact degree changes, and the calculation result by the cognitive time calculation unit. The information processing apparatus according to claim 6, characterized in that it is a product of the same name.

8. The recognition detection unit detects that the vehicle driver has recognized a factor that increases the degree of contact with the vehicle, based on the direction of the vehicle driver's gaze calculated based on the information acquired by the driver information acquisition unit. The information processing apparatus according to claim 7, characterized in that it is a product of the same name.

9. The aforementioned driving aptitude calculation unit calculates the driving aptitude of the vehicle driver based on the vehicle driver's gaze direction, which is calculated based on the information acquired by the driver information acquisition unit. The information processing apparatus according to claim 1, characterized in that it is a product of the present invention.

10. A driver assistance device that provides driver assistance by controlling at least one of the vehicle's speed and direction of travel based on changes in the driver's driving aptitude of the vehicle includes a driver assistance information generation unit that generates driver assistance information for setting the content of the vehicle's control. The information processing apparatus according to claim 1, characterized in that it is a product of the present invention.

11. Based on the related information generated by the related information generation unit, the system includes a similarity determination unit that determines whether information regarding the behavior of the occupants when the driver's driving ability of the vehicle deteriorated at a specific time is similar to information regarding the behavior of the occupants at a time after the specific time. The information processing apparatus according to claim 1, characterized in that it is a product of the present invention.

12. The vehicle is equipped with a notification information generation unit that generates notification information to notify the notification device that the driver's driving aptitude of the vehicle's driver may be decreasing, based on the determination result by the similarity determination unit. The information processing apparatus according to claim 11, characterized in that it is a feature of the present invention.

13. A driver assistance device that provides driving assistance by controlling at least one of the vehicle's speed and direction of travel based on the determination result by the similarity determination unit includes a driver assistance information generation unit that generates driver assistance information for setting the content of the vehicle's control. The information processing apparatus according to claim 11, characterized in that it is a feature of the present invention.

14. The vehicle is equipped with a driver identification unit that identifies the driver of the vehicle based on the information acquired by the driver information acquisition unit, The aforementioned driving aptitude calculation unit calculates the driving aptitude for each driver of the vehicle. The information processing apparatus according to claim 1, characterized in that it is a product of the present invention.

15. The aforementioned occupant behavior information acquisition unit identifies occupants other than the driver of the vehicle and acquires information regarding the behavior of each occupant other than the driver of the vehicle. The information processing apparatus according to claim 1, characterized in that it is a product of the present invention.

16. It is equipped with a time information acquisition unit that obtains time information indicating the time, Based on the driving aptitude calculated by the driving aptitude calculation unit and the time information acquired by the time information acquisition unit, related information is generated in which information indicating the change in the driving aptitude of the vehicle's driver and the time information when the driving aptitude of the vehicle's driver changed are linked together. The information processing apparatus according to claim 1, characterized in that it is a product of the present invention.

17. The vehicle is equipped with a location information acquisition unit that acquires location information of the vehicle, Based on the driving aptitude calculated by the driving aptitude calculation unit and the location information acquired by the location information acquisition unit, related information is generated in which information indicating the change in the driving aptitude of the vehicle's driver and the location information of the vehicle at the time the driving aptitude of the vehicle's driver changed are linked together. The information processing apparatus according to claim 1, characterized in that it is a product of the present invention.

18. A driver information acquisition unit acquires information about the driver of the vehicle regarding changes in the driver's driving aptitude while the vehicle is in operation, A driving aptitude calculation unit calculates the driving aptitude of the vehicle's driver based on the information acquired by the driver information acquisition unit, A crew member behavior information acquisition unit acquires information regarding the actions and statements of crew members other than the driver of the vehicle, The system includes a learning model generation unit that uses the driving aptitude calculated by the driving aptitude calculation unit and the information acquired by the occupant behavior information acquisition unit as training data, and generates a learning model that outputs the driving aptitude of the vehicle's driver based on the input of the information acquired by the occupant behavior information acquisition unit. An information processing device characterized by the following:

19. An autonomous control device that autonomously controls at least one of the vehicle's speed and direction of travel based on the driver aptitude of the vehicle's driver output by the learning model includes a control setting information generation unit that generates control setting information for setting the content of the vehicle's autonomous control. The information processing apparatus according to claim 18, characterized in that it is a product of the present invention.

20. An information processing method performed by a device comprising a driver information acquisition unit, a driver aptitude calculation unit, a passenger behavior information acquisition unit, and a related information generation unit, The driver information acquisition unit acquires information about the driver of the vehicle regarding changes in the driver's driving aptitude while the vehicle is in operation. The driving aptitude calculation unit calculates the driving aptitude of the vehicle's driver based on the information acquired by the driver information acquisition unit. The occupant behavior information acquisition unit acquires information regarding the behavior of occupants other than the driver of the vehicle. The related information generation unit includes the step of generating related information that links together information indicating a change in the driver's driving aptitude of the vehicle's driver and information regarding the behavior of the occupants when the driver's driving aptitude of the vehicle's driver changes, based on the driving aptitude calculated by the driving aptitude calculation unit and the information acquired by the occupant behavior information acquisition unit. An information processing method characterized by the following: