Internal state estimation device and internal state estimation method

Abstract

An internal state estimation device (10) comprises: a characteristic change extraction unit (13) that, on the basis of biometric information acquired chronologically from a passenger of a vehicle, extracts a first timing, which is the timing when a characteristic change occurs in the biometric information; an environmental information change extraction unit (14) that, on the basis of environmental information which is acquired chronologically and which, when changed, impacts the internal state of the passenger, extracts a second timing, which is timing when a change occurs in the environmental information; a timing determination unit (15) that determines which of the first timing extracted by the characteristic change extraction unit and the second timing extracted by the environmental information change extraction unit precedes or follows the other of the first timing and the second timing; and an internal state estimation unit (16) that estimates the internal state of the passenger on the basis of at least one of the biometric information and the environmental information. The internal state estimation unit determines the validity of the internal state estimation according to the determination result by the timing determination unit.

Description

[Technical Field]

[0001] The present disclosure relates to an internal state estimation device and an internal state estimation method. [Background technology]

[0002] Conventionally, there is known a technique for estimating the internal state, such as the emotions, of a vehicle occupant by using the occupant's biological information. For example, Patent Document 1 discloses a technique for measuring two types of biological fluctuations, heart rate fluctuation and chaotic fluctuation, from the driver's pulse wave, and estimating four states that may impede the driver's driving, namely, "depressed state of adaptability," "drowsiness and fatigue state," "tension and elation state," and "stress state," based on these two types of biological fluctuations. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-74805 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 (hereinafter also referred to simply as "the prior art") estimates the driver's internal state, such as emotions, by using pulse waves as biometric information of the driver. However, biometric information such as pulse waves does not necessarily change in association with changes in the driver's internal state, and may change due to actions that do not involve changes in the driver's internal state, such as breathing and talking. In such cases, the prior art may erroneously estimate the driver's internal state.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an internal state estimation device that can suppress erroneous estimation of the internal state of a vehicle occupant. [Means for solving the problem]

[0006] The internal state estimation device of the present disclosure comprises a first extraction unit that extracts a first timing, which is the timing at which a characteristic change occurs in biometric information acquired in time series from a vehicle occupant, based on the biometric information; a second extraction unit that extracts a second timing, which is the timing at which a change occurs in environmental information, which is information acquired in time series and is information whose change affects the internal state of the occupant; a determination unit that determines whether the first timing extracted by the first extraction unit and the second timing extracted by the second extraction unit occur before or after the first timing; and an estimation unit that estimates the internal state of the occupant based on at least one of the biometric information and the environmental information, and is characterized in that the estimation unit determines whether or not the internal state can be estimated depending on the determination result by the determination unit. [Effects of the Invention]

[0007] According to the present disclosure, with the above-described configuration, it is possible to suppress erroneous estimation of the internal state of a vehicle occupant. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of an internal state estimating device according to a first embodiment. [Figure 2] 4 is a flowchart showing an example of operation of the internal state estimating device according to the first embodiment. [Figure 3] 5 is a diagram showing specific examples of bio-information (heart rate) and environmental information (gear position) according to the first embodiment. FIG. [Figure 4] FIG. 4A is a diagram showing an example of the configuration of a first database in embodiment 1, FIG. 4B is a diagram showing an example of the configuration of a second database in embodiment 1, and FIG. 4C is a diagram showing an example of the configuration of a third database in embodiment 1. [Figure 5] 4A to 4C are diagrams illustrating the effects of the internal state estimating device according to the first embodiment. [Figure 6]6A and 6B are diagrams illustrating an example of a hardware configuration of an internal state estimating device according to the first embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of the configuration of an internal state estimating device according to a second embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of movement of an estimation target section in the second embodiment. [Figure 9] 10 is a diagram showing specific examples of bio-information (heart rate) and environmental information (gear position) in the second embodiment. FIG. [Figure 10] FIG. 10 is a diagram illustrating an example of the configuration of an internal state estimating device according to a third embodiment. [Figure 11] FIG. 11A is a diagram showing an example of the order of the first timing and the second timing in embodiment 1, and FIG. 11B is a diagram showing an example of the order of the first timing and the second timing in embodiment 3. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the drawings. Embodiment 1 1 is a diagram illustrating an example of the configuration of an internal state estimating device 10 according to embodiment 1. The internal state estimating device 10 estimates the internal state of a vehicle occupant. The vehicle occupant may be the driver of the vehicle or a passenger.

[0010] As shown in FIG. 1, the internal state estimation device 10 includes a biometric information acquisition unit 11, an environmental information acquisition unit 12, a feature change extraction unit (first extraction unit) 13, an environmental information change extraction unit (second extraction unit) 14, a timing determination unit (determination unit) 15, and an internal state estimation unit (estimation unit) 16.

[0011] The biometric information acquisition unit 11 is connected to a biometric information detection unit (not shown) that detects biometric information P(k) from a vehicle occupant. The biometric information acquisition unit 11 acquires the biometric information P(k) detected in time series by the biometric information detection unit from the biometric information detection unit. The biometric information acquisition unit 11 outputs the acquired biometric information P(k) to the feature change extraction unit 13.

[0012] Here, the biometric information P(k) is, for example, at least one of information indicating the occupant's heartbeat, information indicating a pulse wave, a pulse, and information indicating a facial image. The biometric information detection unit that detects the biometric information P(k) is configured with, for example, a heartbeat sensor, a pulse wave sensor, a pulse sensor, a camera, etc.

[0013] The environmental information acquisition unit 12 is connected to an environmental information detection unit (not shown) that detects environmental information E(i). The environmental information acquisition unit 12 acquires the environmental information E(i) detected in time series by the environmental information detection unit from the environmental information detection unit. The environmental information acquisition unit 12 outputs the acquired environmental information E(i) to the environmental information change extraction unit 14.

[0014] Here, environmental information E(i) is information that affects the internal state of the occupant when the information changes, more specifically, information that causes a change in the internal state of the occupant when the information changes, and is mainly information regarding external factors other than the occupant.

[0015] Examples of the environmental information E(i) include internal vehicle information (in-vehicle information), external vehicle information (external vehicle information), and vehicle control information. The vehicle control information includes, for example, information related to gear position, navigation information, steering wheel operation, and pedal operation information such as an accelerator pedal and a brake pedal. The environmental information detection unit that detects the environmental information E(i) includes, for example, a control device, a navigation system, a sensor, and a camera mounted on the vehicle. The internal state of the occupant includes, for example, the occupant's emotions, as well as the occupant's stress, fatigue, or depression.

[0016] Note that "k" in the biometric information P(k) is an input number assigned to the biometric information, and "i" in the environmental information E(i) is an input number assigned to the environmental information, both of which are integers equal to or greater than 1. Note that when looking at the input numbers k and i at any given time, the values ​​of k and i are not necessarily the same. For example, one second after the internal state estimation device 10 starts the internal state estimation process, k may be "30" and i may be "15." This corresponds to the internal state estimation device 10 having acquired 30 pieces of biometric information P(k) and 15 pieces of environmental information E(i) one second after the internal state estimation device 10 starts the internal state estimation process.

[0017] The characteristic change extraction unit 13 acquires biometric information P(k) from the biometric information acquisition unit 11. Based on the biometric information P(k) acquired from the biometric information acquisition unit 11, the characteristic change extraction unit 13 extracts a first timing kF, which is the timing at which a characteristic change occurs in the biometric information P(k).

[0018] For example, when a specific time change occurs in the biometric information P(k) acquired from the biometric information acquisition unit 11, the characteristic change extraction unit 13 determines that a characteristic change has occurred in the biometric information P(k). Here, the specific time change in the biometric information P(k) refers to a change in the biometric information P(k) that occurs mainly due to a change in the internal state of the occupant, or a change in the biometric information P(k) that may have occurred due to a change in the internal state of the occupant. For example, if the biometric information P(k) is a heart rate, a specific time change in the heart rate refers to the heart rate acquired over time exceeding a specific upper threshold or falling below a specific lower threshold.

[0019] The feature change extraction unit 13 outputs the extracted first timing kF to the timing determination unit 15. In addition, the feature change extraction unit 13 sets the biometric information P(k) used to extract the first timing kF as the feature change extraction result F(kF), and outputs the feature change extraction result F(kF) to the internal state estimation unit 16.

[0020] The environmental information change extraction unit 14 acquires environmental information E(i) from the environmental information acquisition unit 12. Based on the environmental information E(i) acquired from the environmental information acquisition unit 12, the environmental information change extraction unit 14 extracts a second timing iZ, which is the timing at which a change occurs in the environmental information E(i).

[0021] For example, the environmental information change extraction unit 14 determines that a change has occurred in the environmental information E(i) when a specific time change has occurred in the environmental information E(i) acquired from the environmental information acquisition unit 12. A specific time change in the environmental information E(i) means, for example, if the environmental information E(i) is a gear position, that the gear position has changed from a drive gear to a reverse gear, or, if the environmental information E(i) is navigation information, that the vehicle's driving scene has changed (for example, the vehicle has entered a highway from an ordinary road), etc.

[0022] The environmental information change extraction unit 14 outputs the extracted second timing iZ to the timing determination unit 15. In addition, the environmental information change extraction unit 14 sets the environmental information E(i) used to extract the second timing iZ as the environmental information change extraction result Z(iZ), and outputs the environmental information change extraction result Z(iZ) to the internal state estimation unit 16.

[0023] The timing determination unit 15 acquires the first timing kF output from the feature change extraction unit 13 and the second timing iZ output from the environmental information change extraction unit 14. Then, the timing determination unit 15 determines which of the acquired first timing kF and second timing iZ comes first and which of the acquired second timings iZ comes later, and outputs the determination result RT(kF, iZ) to the internal state estimation unit 16.

[0024] The internal state estimation unit 16 acquires the feature change extraction result F(kF) output from the feature change extraction unit 13 and the environmental information change extraction result Z(iZ) output from the environmental information change extraction unit 14. The internal state estimation unit 16 also acquires the determination result RT(kF, iZ) output from the timing determination unit 15.

[0025] The internal state estimation unit 16 estimates the internal state of the occupant based on at least one of the feature change extraction result F(kF) (i.e., the biological information P(k)) obtained from the feature change extraction unit 13 and the environmental information change extraction result Z(iZ) (i.e., the environmental information E(i)) obtained from the environmental information change extraction unit 14. At this time, the internal state estimation unit 16 determines whether or not to estimate the internal state of the occupant, depending on the determination result RT(kF, iZ) obtained from the timing determination unit 15.

[0026] Specifically, the internal state estimation unit 16 allows the execution of internal state estimation when the timing determination unit 15 determines that the first timing kF precedes the second timing iZ, or when the timing determination unit 15 determines that the first timing kF and the second timing iZ are simultaneous. That is, in either of the above cases, the internal state estimation unit 16 estimates the internal state of the occupant based on at least one of the feature change extraction result F(kF) (i.e., the biological information P(k)) obtained from the feature change extraction unit 13 and the environmental information change extraction result Z(iZ) (i.e., the environmental information E(i)) obtained from the environmental information change extraction unit 14. Then, the internal state estimation unit 16 outputs the estimation result R(t).

[0027] On the other hand, when the timing determination unit 15 determines that the first timing kF is later than the second timing iZ, the internal state estimation unit 16 rejects the execution of internal state estimation. In other words, when the timing determination unit 15 determines that the first timing kF is later than the second timing iZ, the internal state estimation unit 16 does not estimate the internal state of the vehicle occupant.

[0028] Next, an example of the operation of the internal state estimating device 10 according to the first embodiment will be described with reference to the flowchart shown in FIG.

[0029] First, the biometric information acquiring unit 11 acquires biometric information P(k) detected in time series by a biometric information detecting unit (not shown) from the biometric information detecting unit (step ST1). The biometric information acquiring unit 11 outputs the acquired biometric information P(k) to the feature change extracting unit 13.

[0030] Next, the characteristic change extraction unit 13 extracts a first timing kF, which is the timing at which a characteristic change occurs in the biometric information P(k), based on the biometric information P(k) acquired from the biometric information acquisition unit 11 (step ST2). Furthermore, the characteristic change extraction unit 13 outputs the extracted first timing kF to the timing determination unit 15, and outputs the biometric information P(k) used to extract the first timing kF to the internal state estimation unit 16 as a characteristic change extraction result F(kF).

[0031] Meanwhile, the environmental information acquisition unit 12 acquires environmental information E(i) detected in time series by an environmental information detection unit (not shown) from the environmental information detection unit (step ST3). The environmental information acquisition unit 12 outputs the acquired environmental information E(i) to the environmental information change extraction unit 14.

[0032] Next, the environmental information change extraction unit 14 extracts a second timing iZ, which is the timing at which a change occurs in the environmental information E(i), based on the environmental information E(i) acquired from the environmental information acquisition unit 12 (step ST4). Furthermore, the environmental information change extraction unit 14 outputs the extracted second timing iZ to the timing determination unit 15, and outputs the environmental information E(i) used to extract the second timing iZ to the internal state estimation unit 16 as an environmental information change extraction result Z(iZ).

[0033] The above steps ST1 and ST2 and steps ST3 and ST4 may be performed in parallel.

[0034] Next, the timing determination unit 15 determines which of the first timing kF and the second timing iZ occurs first (step ST5). The timing determination unit 15 also outputs the determination result RT(kF, iZ) to the internal state estimation unit 16.

[0035] For example, if the timing determination unit 15 determines that the first timing kF is earlier than the second timing iZ or that the first timing kF and the second timing iZ are simultaneous, the timing determination unit 15 sets the determination result RT(kF, iZ) to "1." On the other hand, if the timing determination unit 15 determines that the first timing kF is later than the second timing iZ, the timing determination unit 15 sets the determination result RT(kF, iZ) to "0."

[0036] Next, the internal state estimation unit 16 checks whether the determination result RT(kF, iZ) acquired from the timing determination unit 15 is "1", that is, whether it has been determined that the first timing kF precedes the second timing iZ, or whether the first timing kF and the second timing iZ are simultaneous (step ST6). As a result, if the determination result RT(kF, iZ) is not "1" (step ST6; NO), the internal state estimation unit 16 determines that the internal state estimation is not performed and ends the process. On the other hand, if the determination result RT(kF, iZ) is "1" (step ST6; YES), the process proceeds to step ST7.

[0037] In step ST7, the internal state estimation unit 16 enables the execution of internal state estimation, and estimates the internal state of the vehicle occupant based on at least one of the feature change extraction result F(kF) (i.e., the biometric information P(k)) obtained from the feature change extraction unit 13 and the environmental information change extraction result Z(iZ) (i.e., the environmental information E(i)) obtained from the environmental information change extraction unit 14 (step ST7). Then, the internal state estimation unit 16 outputs the estimation result R(t).

[0038] Next, an example of the operation of the internal state estimation device 10, including the method of estimating the internal state by the internal state estimation unit 16, will be described with a specific example.

[0039] For ease of understanding, it is assumed here that the biometric information P(k) is the occupant's heart rate and the environmental information E(i) is the vehicle's gear position. In the following description, it is assumed that, starting from the time when the internal state estimation device 10 starts operating, the biometric information acquisition unit 11 acquires the occupant's heart rate one per second, for a total of 15 times, and the environmental information acquisition unit 12 acquires the vehicle's gear position one per second, for a total of 15 times. Specific examples of the heart rate and gear position in the above case are shown in FIG. 3.

[0040] Furthermore, here, the specific time change in heart rate that is the criterion for determining that a characteristic change has occurred in the heart rate is defined as the heart rate exceeding the upper threshold value of "1.3."

[0041] First, feature change extraction unit 13 extracts "5 seconds (t=5)", the timing at which the heart rate exceeded the upper threshold of 1.3, as first timing kF (corresponding to step ST2 described above), and outputs the first timing kF to timing determination unit 15. Furthermore, feature change extraction unit 13 outputs the 15 heart rates (the 15 from "1.0" at t=1 to "1.4" at t=15) used to extract first timing kF as feature change extraction result F(kF) to internal state estimation unit 16.

[0042] Meanwhile, the environmental information change extraction unit 14 extracts "9 seconds (t=9)", which is the timing when the gear position changes from drive gear (shown as D in FIG. 3) to reverse gear (shown as R in FIG. 3), as the second timing iZ (corresponding to step ST4 described above), and outputs the second timing iZ to the timing determination unit 15. Furthermore, the environmental information change extraction unit 14 outputs the 15 gear positions (the 15 from "D" at t=1 to "R" at t=15) used to extract the second timing iZ to the internal state estimation unit 16 as the environmental information change extraction result Z(iZ).

[0043] Next, the timing determination unit 15 determines which of the first timing kF, "5 seconds (t=5)," comes first, and the second timing iZ, "9 seconds (t=9)," and determines that the first timing kF comes first (corresponding to step ST5 described above). Furthermore, since the timing determination unit 15 has determined that the first timing kF comes first, it sets the determination result RT(kF, iZ) to "1" and outputs the determination result RT(kF, iZ) to the internal state estimation unit 16.

[0044] Next, the internal state estimation unit 16 checks whether the determination result RT(kF, iZ) obtained from the timing determination unit 15 is "1" (corresponding to step ST6 described above). As a result, since the determination result RT(kF, iZ) is "1", the internal state estimation unit 16 determines that it is possible to estimate the internal state, and estimates the internal state of the occupant based on at least one of the 15 heart rates and the 15 gear positions (corresponding to step ST7 described above).

[0045] Here, we will explain an example of a method for estimating the internal state by the internal state estimation unit 16. The internal state estimation unit 16 estimates the internal state of an occupant, for example, using a database for internal state estimation that has been created in advance. Note that this database is recorded in advance in a recording unit (not shown) provided in the internal state estimation device 10.

[0046] An example of the database is shown in Fig. 4A. As shown in Fig. 4A, this database (hereinafter also referred to as "first database") is configured by associating information (hereinafter also referred to as "first change information") that defines the manner of change (transition) of 15 pieces of biometric information P(k) derived from the feature change extraction result F(kF) with information that indicates the internal state of the occupant (hereinafter also referred to as "internal state information").

[0047] For example, in the example of Fig. 4A, the first change information is set to "the heart rate has risen to 1.5 (exceeding the upper threshold of 1.3)," and "tension" is set as the internal state information associated with this first change information. Similarly, in the example of Fig. 4A, the first change information is set to "the heart rate has risen to 1.7 (exceeding the upper threshold of 1.3)," and "nervous" is set as the internal state information associated with this first change information. Although not explicitly shown in Fig. 4A, the change in biological information P(k) indicated in the first change information also includes the characteristic change (exceeding the upper threshold of 1.3) that occurred at the above-mentioned first timing kF.

[0048] Similarly, in the example of Fig. 4A, the first change information is set to "heart rate has risen to 1.9 (exceeding the upper threshold of 1.3)," and "pressure" is set as the internal state information associated with this first change information. Also, in the example of Fig. 4A, the first change information is set to "heart rate has risen to 2.1 (exceeding the upper threshold of 1.3)," and "anxiety" is set as the internal state information associated with this first change information.

[0049] For example, in the example of Fig. 3, it can be seen from the 15 heart rate values ​​that are the feature change extraction result F(kF) that the heart rate has exceeded the upper threshold of 1.3 and has risen to 1.9. Therefore, the internal state estimation unit 16 searches the first database using the fact that the heart rate has risen to 1.9 (i.e., the manner of change in the heart rate) as a key, and identifies "pressure" as the internal state information corresponding to that key. Then, based on the identified internal state information, the internal state estimation unit 16 outputs the estimation result of the occupant's internal state as "pressure."

[0050] Although the above description deals with an example of operation in which the heart rate rises above the upper threshold of 1.3, the same applies to a case in which the heart rate falls below the lower threshold (e.g., 0.7). In this case, in the first database, the first change information may be set to, for example, "the heart rate has fallen to 0.6 (below the lower threshold of 0.7)," and "fatigue" may be set as the internal state information associated with this first change information.

[0051] Next, another example of the database is shown in Fig. 4B. As shown in Fig. 4B, this database (hereinafter also referred to as the "second database") is configured by associating information (hereinafter also referred to as the "second change information") that defines the manner of change (transition) of environmental information E(i) derived from the environmental information change extraction result Z(iZ) with information that indicates the internal state of the occupant (internal state information). Note that the change in environmental information E(i) indicated in the second change information also includes the change in environmental information E(i) that occurred at the above-mentioned second timing iZ.

[0052] For example, in the example of Fig. 4B, the second change information is set to "the gear position has changed from drive gear to reverse gear," and the internal state information associated with this second change information is set to "tense." Also, in the example of Fig. 4B, the second change information is set to "the gear position has changed from drive gear to parking gear (P)," and the internal state information associated with this second change information is set to "calm." Also, in the example of Fig. 4B, the second change information is set to "the gear position has changed from drive gear to neutral gear (N)," and the internal state information associated with this second change information is set to "caution."

[0053] For example, in the example of Fig. 3, it can be seen from the 15 gear positions that are the environmental information change extraction result Z(iZ) that the gear position has been changed from drive gear to reverse gear. Therefore, the internal state estimation unit 16 searches the second database using the fact that the gear position has been changed from drive gear to reverse gear (i.e., the manner in which the gear position has changed) as a key, and identifies "tension," which is the internal state information corresponding to that key. Then, based on the identified internal state information, the internal state estimation unit 16 outputs the estimation result of the occupant's internal state as "tension."

[0054] Note that, although an example of operation when the environmental information E(i) is the gear position has been described here, the same applies when the environmental information E(i) is something other than the gear position. For example, if the environmental information E(i) is brake pedal operation information, then in the second database, the second change information may be set to, for example, "the number of times the brake pedal is depressed in a predetermined period of time has increased to 8 times (exceeding the upper threshold of 5 times)," and the internal state information associated with this second change information may be set to, for example, "impatience."

[0055] Next, another example of a database is shown in Figure 4C. As shown in Figure 4C, this database (hereinafter also referred to as the "third database") is formally configured as a combination of the first and second databases described above. Specifically, this third database is configured by associating a combination of the first change information and second change information described above with internal state information.

[0056] For example, in the example of Fig. 4C, the combination of change information is set to "the gear position has been changed from drive gear to reverse gear, and the heart rate has increased to 1.5 (exceeding the upper threshold of 1.3)", and "tension" is set as the internal state information associated with this combination. Also, in the example of Fig. 4C, the combination of change information is set to "the gear position has been changed from drive gear to reverse gear, and the heart rate has increased to 1.7 (exceeding the upper threshold of 1.3)", and "nervous" is set as the internal state information associated with this combination.

[0057] Similarly, in the example of Fig. 4C, the combination of change information is set to "the gear position has been changed from drive gear to reverse gear, and the heart rate has risen to 1.9 (exceeding the upper threshold of 1.3)", and the internal state information associated with this combination is set to "pressure". Also, in the example of Fig. 4C, the combination of change information is set to "the gear position has been changed from drive gear to reverse gear, and the heart rate has risen to 2.1 (exceeding the upper threshold of 1.3)", and the internal state information associated with this combination is set to "anxiety".

[0058] For example, in the example of FIG. 3 , the 15 heart rates that are the feature change extraction result F(kF) and the 15 gear positions that are the environmental information change extraction result Z(iZ) indicate that the heart rate has exceeded the upper threshold of 1.3 and increased to 1.9, and that the gear position has been changed from drive gear to reverse gear. Therefore, the internal state estimation unit 16 searches the third database using the fact that the gear position has been changed from drive gear to reverse gear and that the heart rate has increased to 1.9 (i.e., the manner of change in both the gear position and the heart rate) as a key, and identifies "pressure," which is the internal state information corresponding to the key. Then, the internal state estimation unit 16 outputs the estimation result of the internal state of the vehicle occupant as "pressure," based on the identified internal state information.

[0059] The internal state estimation unit 16 can estimate the internal state of the occupant with high accuracy by estimating the internal state of the occupant using, for example, any one of the first to third databases. Note that if the internal state estimation unit 16 estimates the internal state of the occupant using the third database, it can estimate the internal state of the occupant based on the relationship between the heart rate, which is the biometric information P(k), and the gear position, which is the environmental information E(i). This improves the estimation accuracy compared to when the internal state of the occupant is estimated using the first or second database.

[0060] In addition, when the emotions of the vehicle occupants are used as the internal state indicated by the internal state information set in each of the above databases, an emotion model defined in, for example, Russell's emotional circle model can be used to construct each of the databases.

[0061] Russell's circumplex model of emotions is a model that shows that all emotions are arranged in a circular pattern on a plane expressed in two dimensions: "pleasant-unpleasant" and "arousal-unarousal." Many different emotion models have been proposed, but Russell's circumplex model of emotions is expressed in a simple structure and can be applied comprehensively to all emotions, making it suitable for use in building the above databases.

[0062] For example, in Russell's circumplex model of emotions, emotions classified as "unpleasant" and "arousal" are defined as "tension," "nervous," "pressure," and "worry," while emotions classified as "unpleasant" and "non-arousal" are defined as "sadness," "depression," "apathy," and "fatigue." The same model also defines emotions classified as "pleasant" and "arousal" as "attention," "excitement," "vigor," and "happiness," while emotions classified as "pleasant" and "non-arousal" are defined as "contentment," "calmness," "relaxation," and "calmness."

[0063] An administrator or the like of the internal state estimation device 10 can construct each of the above databases by using, for example, each of the above emotions as an internal state. Note that each internal state needs to be associated with at least one of the first change information and the second change information, as shown in Figures 4A to 4C.

[0064] It should be noted that the contents of the first change information, second change information, and internal state information in each database described above are merely examples, and other contents may be set. For example, in the above example, an example was described in which the occupant's emotion (such as tension) is used as the internal state information, but the internal state information is not limited to this, and for example, the occupant's stressed state, fatigued state, or depressed state may be used. Furthermore, in the above example, an example was described in which the name of the occupant's emotion (such as "tension") is set as the internal state information in each database, but the internal state information in each database is not limited to this, and for example, information that quantifies the occupant's emotion may be set.

[0065] Furthermore, each of the databases described above may be set with different information for each occupant, for example. For example, the occupant may appropriately select an emotion (e.g., impatience, tension, energy, etc.) that they wish to estimate as their own internal state and a type of environmental information E(i) (e.g., gear position, navigation information, etc.), and may generate each of the databases in advance based on the selected emotion and type of environmental information E(i). In this case, the occupant can use a database customized for estimating their own internal state, improving convenience.

[0066] In addition, in the above explanation, an example was described in which the internal state estimation unit 16 uses the judgment result RT(kF, iZ) between the first timing kF and the second timing iZ obtained from the timing judgment unit 15 when deciding whether to perform the internal state estimation process, but the internal state estimation unit 16 is not limited to this, and for example, the judgment result RT(kF, iZ) obtained from the timing judgment unit 15 may be used directly to estimate the internal state of the occupant.

[0067] For example, if the judgment result RT(kF, iZ) obtained from the timing judgment unit 15 is "1", that is, if the first timing kF precedes the second timing iZ, or if the first timing kF and the second timing iZ are simultaneous, the internal state estimation unit 16 may output an estimation result R(t) indicating that the internal state of the occupant is "tense".

[0068] In the above description, an example has been described in which one each of the first timing kF and the second timing iZ is extracted, but a case in which at least one of the first timing kF and the second timing iZ is extracted multiple times is also assumed. In this case, the timing determination unit 15 may set the determination result RT(kF, iZ) to "1" if there is at least one case in which the first timing kF precedes the second timing iZ or the first timing kF and the second timing iZ are simultaneous in the extracted timing group.

[0069] Furthermore, the internal state estimation unit 16 may estimate the internal state of the occupant using, for example, a machine learning model created for internal state estimation. In this case, the machine learning model may be, for example, any one of: (1) a model trained to output internal state information corresponding to first change information derived from the feature change extraction result F(kF) upon input of the first change information, (2) a model trained to output internal state information corresponding to second change information derived from the environmental information change extraction result Z(iZ) upon input of the second change information, or (3) a model trained to output internal state information corresponding to the combination of the first change information and the second change information upon input of a combination of the first change information and the second change information.

[0070] Furthermore, the internal state estimation unit 16 may estimate the internal state of the occupant using a method similar to the method in the conventional technology described above. For example, the internal state estimation unit 16 may measure two types of biological fluctuations, heart rate fluctuation and chaotic fluctuation, from the occupant's pulse wave acquired as biological information P(k), and estimate the occupant's internal state based on these two types of biological fluctuations.

[0071] Furthermore, the internal state estimation unit 16 may output the estimation result of the occupant's internal state in two stages, such as "anxious" or "anxious," or in multiple stages (e.g., 0-100 stages) such as "anxiety on a 90 / 100 scale."

[0072] As described above, in the internal state estimation device 10 according to the first embodiment, the internal state estimation unit 16 determines whether or not to estimate the internal state of the occupant, depending on the determination result by the timing determination unit 15. Specifically, the internal state estimation unit 16 allows the execution of internal state estimation when the timing determination unit 15 determines that the first timing kF precedes the second timing iZ, or when the first timing kF and the second timing iZ are simultaneous. On the other hand, the internal state estimation unit 16 denies the execution of internal state estimation when the timing determination unit 15 determines that the first timing kF is after the second timing iZ. This is based on the idea that when the internal state of the occupant changes (for example, the occupant becomes nervous), biological information changes (for example, the heart rate increases significantly) before environmental information changes (for example, the gear position is changed from the drive gear to the reverse gear). Therefore, the internal state estimating device 10 according to the first embodiment can prevent erroneous estimation of the internal state of the occupant based on a change in biological information that is not accompanied by a change in the internal state of the occupant.

[0073] In this regard, the above-mentioned conventional technology estimates the driver's internal state, such as emotions, by using the driver's pulse wave as biometric information. However, biometric information, such as the pulse wave, not only changes with changes in the driver's internal state when switching from normal driving to parking, as shown in W2 of FIG. 5, but also may change with actions that do not involve changes in the driver's internal state, such as breathing and talking during normal driving, as shown in W1 of FIG. 5. In such cases, the above-mentioned conventional technology may erroneously estimate the driver's internal state as "tension" or the like by detecting the change in W1. In contrast, the internal state estimation device 10 according to the first embodiment prevents such problems from occurring.

[0074] Next, an example of a hardware configuration of the internal state estimation device 10 according to the first embodiment will be described with reference to Fig. 6. The functions of the biometric information acquisition unit 11, the environmental information acquisition unit 12, the feature change extraction unit 13, the environmental information change extraction unit 14, the timing determination unit 15, and the internal state estimation unit 16 in the internal state estimation device 10 are realized by processing circuits. The processing circuit may be dedicated hardware as shown in Fig. 6A, or may be a CPU (also referred to as a central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 22 that executes a program stored in a memory 23 as shown in Fig. 6B.

[0075] When the processing circuit is dedicated hardware, the processing circuit 21 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of each of the biometric information acquisition unit 11, the environmental information acquisition unit 12, the feature change extraction unit 13, the environmental information change extraction unit 14, the timing determination unit 15, and the internal state estimation unit 16 may be realized by the processing circuit 21 individually, or the functions of each unit may be realized collectively by the processing circuit 21.

[0076] When the processing circuit is a CPU 22, the functions of the biometric information acquisition unit 11, the environmental information acquisition unit 12, the feature change extraction unit 13, the environmental information change extraction unit 14, the timing determination unit 15, and the internal state estimation unit 16 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 23. The processing circuit realizes the functions of each unit by reading and executing the programs stored in the memory 23. In other words, the internal state estimation device 10 includes a memory for storing programs that, when executed by the processing circuit, result in the execution of, for example, each step shown in FIG. 2 . Furthermore, these programs can also be said to cause a computer to execute the procedures and methods of the biometric information acquisition unit 11, the environmental information acquisition unit 12, the feature change extraction unit 13, the environmental information change extraction unit 14, the timing determination unit 15, and the internal state estimation unit 16. Here, examples of memory 23 include non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically EPROM), magnetic disk, flexible disk, optical disk, compact disk, mini disk, or DVD (Digital Versatile Disc).

[0077] It is also possible to realize some of the functions of the biometric information acquisition unit 11, the environmental information acquisition unit 12, the feature change extraction unit 13, the environmental information change extraction unit 14, the timing determination unit 15, and the internal state estimation unit 16 with dedicated hardware and some with software or firmware. For example, the function of the biometric information acquisition unit 11 can be realized by a processing circuit as dedicated hardware, and the functions of the environmental information acquisition unit 12, the feature change extraction unit 13, the environmental information change extraction unit 14, the timing determination unit 15, and the internal state estimation unit 16 can be realized by the processing circuit reading and executing a program stored in the memory 23.

[0078] Thus, the processing circuitry can implement each of the above-described functions by hardware, software, firmware, or a combination thereof.

[0079] As described above, according to the first embodiment, the internal state estimation device 10 includes a characteristic change extraction unit 13 that extracts, based on biometric information P(k) acquired over time from a vehicle occupant, a first timing kF at which a characteristic change occurs in the biometric information; an environmental information change extraction unit 14 that extracts, based on environmental information E(i), which is information acquired over time and whose change affects the internal state of the occupant, a second timing iZ at which a change occurs in the environmental information; a timing determination unit 15 that determines whether the first timing kF extracted by the characteristic change extraction unit 13 occurs before or after the second timing iZ extracted by the environmental information change extraction unit 14; and an internal state estimation unit 16 that estimates the internal state of the occupant based on at least one of the biometric information P(k) and the environmental information E(i). The internal state estimation unit 16 determines whether to estimate the internal state based on the determination result of the timing determination unit 15. This makes it possible for the internal state estimation device 10 according to the first embodiment to prevent erroneous estimation of the internal state of the occupant.

[0080] Furthermore, the internal state estimation unit 16 allows estimation of the internal state when the timing determination unit 15 determines that the first timing kF precedes the second timing iZ, or when the first timing kF and the second timing iZ are simultaneous. This allows the internal state estimation device 10 according to the first embodiment to accurately prevent erroneous estimation of the internal state of the occupant.

[0081] Furthermore, the internal state estimation unit 16 estimates the internal state of the occupant using a database in which at least one of first change information, which is information that defines the manner of change in the biological information P(k) including a characteristic change at a first timing kF, and second change information, which is information that defines the manner of change in the environmental information E(i) including a change at a second timing iZ, is associated with information indicating the occupant's internal state. This allows the internal state estimation device 10 according to the first embodiment to accurately estimate the occupant's internal state.

[0082] Furthermore, the biological information P(k) is at least one of information indicating the occupant's heart rate, information indicating the occupant's pulse, and information indicating an image of the occupant's face. This allows the internal state estimation device 10 according to the first embodiment to prevent erroneous estimation of the occupant's internal state by using information that can be easily acquired from the occupant.

[0083] Furthermore, the environmental information E(i) is at least one of internal information of the vehicle, external information of the vehicle, and control information of the vehicle. As a result, the internal state estimation device 10 according to the first embodiment can suppress erroneous estimation of the internal state of the occupant by using information that can be easily acquired from the vehicle.

[0084] Embodiment 2 In the first embodiment, an internal state estimation device was described that determines whether an occupant's internal state can be estimated based on the result of determining whether the first timing kF and the second timing iZ occur before or after the first timing kF. In the second embodiment, an internal state estimation device is described that sets an estimation target section having a predetermined time range on the time series in which the biological information P(k) and the environmental information E(i) are acquired, and determines whether the internal state can be estimated for each estimation target section.

[0085] Fig. 7 is a diagram showing an example of the configuration of an internal state estimating device 10b according to Embodiment 2. The internal state estimating device 10b according to Embodiment 2 is obtained by adding a target section setting unit 17 to the internal state estimating device 10 according to Embodiment 1 shown in Fig. 1. Since the other configuration of the internal state estimating device 10b according to Embodiment 2 is the same as that of the internal state estimating device 10 according to Embodiment 1 shown in Fig. 1, the same reference numerals are used and their description will be omitted.

[0086] The target section setting unit 17 sets an estimation target section L having a predetermined time range on the time series in which the biological information P(k) and the environmental information E(i) are acquired. The estimation target section L determines a section on the time series in which the biological information P(k) and the environmental information E(i) are acquired that is a target section for estimating the internal state of the occupant by the internal state estimation device 10b.

[0087] 8, the target interval setting unit 17 is capable of moving the estimation target interval L in the time direction (the direction in which time advances) every time a predetermined condition is satisfied. The predetermined condition is, for example, that the biometric information acquiring unit 11 acquires st pieces of biometric information P(k) (st is an integer equal to or greater than 1). In this case, the target interval setting unit 17 moves the estimation target interval L in the time direction every time the biometric information acquiring unit 11 acquires st pieces of biometric information P(k).

[0088] In the internal state estimation device 10b according to the second embodiment, when an estimation target section L is set by the target section setting unit 17, extraction of a first timing kF by the feature change extraction unit 13, extraction of a second timing iZ by the environmental information change extraction unit 14, precedence / sequence determination by the timing determination unit 15, and estimation of an internal state by the internal state estimation unit 16 are performed within the range of the set estimation target section L. Furthermore, in the internal state estimation device 10b, when the estimation target section L is moved by the target section setting unit 17, each of the above processes is performed for each estimation target section L after the move.

[0089] For example, the feature change extraction unit 13 extracts a first timing kF based on Tp pieces of biometric information P(k-Tp+1) to P(k) included in an estimation target section L set by the target section setting unit 17, out of the biometric information P(k) acquired from the occupant in time series. The feature change extraction unit 13 outputs the extracted first timing kF to the timing determination unit 15, and outputs the Tp pieces of biometric information P(k-Tp+1) to P(k) used to extract the first timing kF as a feature change extraction result F(kF) to the internal state estimation unit 16. Note that Tp is, for example, an integer equal to or greater than 2.

[0090] Furthermore, the environmental information change extraction unit 14 extracts the second timing iZ based on Tp pieces of environmental information E(i-Tp+1) to E(i) included in the estimation target section L set by the target section setting unit 17, out of the environmental information E(i) acquired in time series from the vehicle and its surroundings. Furthermore, the environmental information change extraction unit 14 outputs the extracted second timing iZ to the timing determination unit 15, and outputs the Tp pieces of environmental information E(i-Tp+1) to E(i) used to extract the second timing iZ to the internal state estimation unit 16 as the environmental information change extraction result Z(iZ).

[0091] In addition, the timing determination unit 15 determines the order of a first timing kF extracted by the feature change extraction unit 13 based on Tp pieces of biometric information P(k-Tp+1) to P(k) included in the estimation target section L, and a second timing iZ extracted by the environmental information change extraction unit 14 based on Tp pieces of environmental information E(i-Tp+1) to E(i) included in the estimation target section L.

[0092] Furthermore, when the timing determination unit 15 determines that the first timing kF is earlier than the second timing iZ, or when the first timing kF and the second timing iZ are simultaneous, the internal state estimation unit 16 enables the execution of internal state estimation, and estimates the internal state of the vehicle occupant based on at least one of the feature change extraction result F(kF) (i.e., Tp pieces of biometric information P(k-Tp+1) to P(k)) acquired from the feature change extraction unit 13 and the environmental information change extraction result Z(iZ) (i.e., Tp pieces of environmental information E(i-Tp+1) to E(i)) acquired from the environmental information change extraction unit 14. On the other hand, when the timing determination unit 15 determines that the first timing kF is later than the second timing iZ, the internal state estimation unit 16 does not estimate the internal state in the estimation target section L.

[0093] Next, an example of the operation of the internal state estimating device 10b, including the method of estimating the internal state by the internal state estimating unit 16, will be described with a specific example.

[0094] For ease of understanding, the following description will be given using a specific example similar to that of the first embodiment. Specifically, the biometric information P(k) is the occupant's heart rate, the environmental information E(i) is the vehicle's gear position, the biometric information acquisition unit 11 acquires the occupant's heart rate at one per second, a total of 15 pieces, and the environmental information acquisition unit 12 acquires the vehicle's gear position at one per second, a total of 15 pieces. Here, the specific time change in the heart rate, which is the criterion for determining that a characteristic change has occurred in the heart rate, is when the heart rate exceeds the upper threshold value of "1.3." A specific example of the heart rate and gear position in the above case is shown in FIG. 9.

[0095] First, the target interval setting unit 17 sets an estimation target interval L having a predetermined time range on the time series in which the biometric information P(k) and the environmental information E(i) are acquired. Here, if the predetermined time range is "10 seconds," the target interval setting unit 17 sets the interval from 1 second (t=1) to 10 seconds (t=10) as the first estimation target interval L1, for example.

[0096] In this case, the feature change extraction unit 13 refers to the 10 pieces of biometric information P(1) to P(10) included in the estimation target section L1, extracts "5 seconds (t=5)", the timing at which the heart rate exceeded the upper threshold of 1.3, from these 10 pieces of biometric information, as the first timing kF, and outputs the first timing kF to the timing determination unit 15. In addition, the feature change extraction unit 13 outputs the 10 heart rates (the 10 from "1.0" at t=1 to "1.7" at t=10) used to extract the first timing kF as the feature change extraction result F(kF) to the internal state estimation unit 16.

[0097] Meanwhile, the environmental information change extraction unit 14 refers to the ten pieces of environmental information E(1) to E(10) included in the estimation target section L1, and extracts, from these ten pieces of environmental information, "9 seconds (t=9)", which is the timing at which the gear position changes from drive gear to reverse gear, as the second timing iZ, and outputs the second timing iZ to the timing determination unit 15. Furthermore, the environmental information change extraction unit 14 outputs the ten gear positions (the ten from "D" at t=1 to "R" at t=10) used to extract the second timing iZ to the internal state estimation unit 16 as the environmental information change extraction result Z(iZ).

[0098] Next, the timing determination unit 15 determines which of the first timing kF, "5 seconds (t=5)," comes first, and the second timing iZ, "9 seconds (t=9)," and determines that the first timing kF comes first. Furthermore, since the timing determination unit 15 has determined that the first timing kF comes first, it sets the determination result RT(kF, iZ) to "1" and outputs the determination result RT(kF, iZ) to the internal state estimation unit 16.

[0099] Next, the internal state estimation unit 16 checks whether the determination result RT(kF, iZ) obtained from the timing determination unit 15 is "1". As a result, since the determination result RT(kF, iZ) is "1", the internal state estimation unit 16 determines that it is possible to estimate the internal state, and estimates the internal state of the occupant in the first estimation target section L1 based on at least one of the above 10 heart rates and the above 10 gear positions. The method of estimating the internal state by the internal state estimation unit 16 is the same as the method described in embodiment 1.

[0100] Here, the target interval setting unit 17 can move the estimation target interval L in the time direction each time a predetermined condition is satisfied, and the predetermined condition is, for example, that two pieces of biometric information P(k) are acquired by the biometric information acquisition unit 11. In this case, the target interval setting unit 17 moves the estimation target interval L1 in the time direction at 2 seconds (t=2), when two pieces of biometric information P(k) are acquired by the biometric information acquisition unit 11, and sets the next estimation target interval L2. The next estimation target interval L2 is 10 seconds long, from 3 seconds (t=3) to 12 seconds (t=12).

[0101] When the next estimation target section L2 is set, the feature change extraction section 13, the environmental information change extraction section 14, the timing determination section 15, and the internal state estimation section 16 perform the above-mentioned processes based on the 10 pieces of biometric information and environmental information included in the estimation target section L2. At this time, the internal state of the occupant estimated by the internal state estimation section 16 becomes the internal state of the occupant in the estimation target section L2.

[0102] Thereafter, the target section setting unit 17 moves the estimation target section L2 in the time direction and sets the next estimation target section L3 at 4 seconds (t=4), which is the timing when the next two pieces of biometric information P(k) are acquired by the biometric information acquisition unit 11. The next estimation target section L3 is 10 seconds from 5 seconds (t=5) to 14 seconds (t=14).

[0103] When the next estimation target section L3 is set, the feature change extraction unit 13, the environmental information change extraction unit 14, the timing determination unit 15, and the internal state estimation unit 16 perform the above-mentioned processes based on the 10 pieces of biometric information and environmental information included in the estimation target section L3. At this time, the internal state of the occupant estimated by the internal state estimation unit 16 becomes the internal state of the occupant in the estimation target section L3.

[0104] In the internal state estimation device 10b according to the second embodiment, the target section setting unit 17 moves the estimation target section L in the time direction, and estimates the internal state of the occupant for each shifted estimation target section L.

[0105] In this way, the internal state estimating device 10b according to the second embodiment can obtain an estimation result of the internal state of an occupant for a desired section on a time series by including the target section setting unit 17. Furthermore, the internal state estimating device 10b according to the second embodiment can estimate the internal state of the occupant in more detail and continuously by moving the estimation target section L in the time direction.

[0106] As described above, according to the second embodiment, the internal state estimating device 10b includes a target interval setting unit 17 that sets an estimation target interval L having a predetermined time range on the time series in which the biological information P(k) and the environmental information E(i) are acquired. The feature change extracting unit 13 extracts a first timing kF based on the biological information acquired in time series from the occupant, the biological information being included in the estimation target interval L set by the target interval setting unit 17. The environmental information change extracting unit 14 extracts a first timing kF based on the environmental information acquired in time series, the environmental information being included in the estimation target interval L set by the target interval setting unit 17. The timing determination unit 15 determines whether the first timing kF extracted by the feature change extraction unit 13 based on the biometric information included in the estimation target section L comes before or after the second timing iZ extracted by the environmental information change extraction unit 14 based on the environmental information included in the estimation target section L, and the internal state estimation unit 16 estimates the internal state of the occupant based on at least one of the biometric information and the environmental information included in the estimation target section L, and determines whether or not to estimate the internal state depending on the determination result by the timing determination unit 15. As a result, the internal state estimation device 10b according to the second embodiment can obtain an estimation result of the internal state of the occupant for a desired section on the time series in addition to the effects of the first embodiment.

[0107] Furthermore, the target section setting unit 17 shifts the estimation target section L in the time direction each time a predetermined condition is satisfied. This allows the internal state estimating device 10b according to the second embodiment to estimate the internal state of the occupant in more detail and continuously.

[0108] Furthermore, the predetermined condition is that st pieces of biometric information P(k) (st is an integer equal to or greater than 1) are acquired. This allows the internal state estimating device 10b according to the second embodiment to move the estimation target section L according to the number of acquired pieces of biometric information.

[0109] Embodiment 3 In the first embodiment, an internal state estimating device was described that determines whether to execute the process for estimating the internal state of an occupant depending on the result of determining whether the first timing occurs before or after the second timing. In the third embodiment, an internal state estimating device that can more precisely determine whether the first timing occurs before or after the second timing will be described.

[0110] Fig. 10 is a diagram illustrating an example configuration of an internal state estimating device 10c according to Embodiment 3. In the internal state estimating device 10c according to Embodiment 3, the feature change extracting unit 13 is replaced with a feature change extracting unit 13b, as compared to the internal state estimating device 10 according to Embodiment 1 shown in Fig. 1. The other configuration of the internal state estimating device 10c according to Embodiment 3 is the same as that of the internal state estimating device 10 according to Embodiment 1 shown in Fig. 1, and therefore the same reference numerals are used and description thereof will be omitted.

[0111] The characteristic change extraction unit 13b acquires biometric information P(k) from the biometric information acquisition unit 11. Based on the biometric information P(k) acquired from the biometric information acquisition unit 11, the characteristic change extraction unit 13b extracts, as a first timing kF, the timing at which a characteristic change in the biometric information P(k) starts.

[0112] For example, the characteristic change extraction unit 13b first extracts the timing at which a characteristic change occurs in the biometric information P(k) in the same manner as the characteristic change extraction unit 13 in Embodiment 1. Next, the characteristic change extraction unit 13b uses the extracted timing as a starting point and traces back in time from that timing to search for the timing at which the characteristic change started in the biometric information P(k).

[0113] For example, if the biometric information P(k) is a heart rate, the characteristic change extraction unit 13b starts from the timing when the heart rate exceeds an upper threshold and goes back in time from that timing to sequentially calculate a derivative value at each point of a function representing the biometric information P(k). Then, the characteristic change extraction unit 13b determines the timing at which the derivative value is 0 as the timing at which the characteristic change started. Alternatively, the characteristic change extraction unit 13b may determine the timing at which the characteristic change started by going back a predetermined time (for example, 2 seconds) from the extracted timing. Then, the characteristic change extraction unit 13b extracts the timing at which the characteristic change started as a first timing kF.

[0114] The feature change extraction unit 13b outputs the extracted first timing kF to the timing determination unit 15. In addition, the feature change extraction unit 13b sets the biometric information P(k) used to extract the first timing kF as the feature change extraction result F(kF), and outputs the feature change extraction result F(kF) to the internal state estimation unit 16.

[0115] Next, a specific example and effect of the processing by the feature change extraction unit 13b will be described with reference to FIG.

[0116] 11A, in the internal state estimating device 10 according to the first embodiment, the characteristic change extracting unit 13 extracts, as the first timing kF, the timing at which a characteristic change occurs in the biometric information P(k) (for example, the timing at which the heart rate exceeds an upper threshold), based on the biometric information P(k) acquired from the biometric information acquiring unit 11. In this case, depending on the content of the upper threshold, the first timing kF may occur after the second timing iZ, even though a characteristic change has already begun to occur in the biometric information P(k).

[0117] In this case, the timing determination unit 15 determines that the first timing kF is later than the second timing iZ, and the internal state estimation unit 16 rejects the estimation of the occupant's internal state, and the process of estimating the occupant's internal state may not be executed. Although such a case may be avoidable depending on the content of the upper limit threshold, resetting the upper limit threshold each time for this purpose is cumbersome and may not be practical from an operational standpoint.

[0118] 11B, in the internal state estimating device 10c according to the third embodiment, the characteristic change extractor 13b extracts the timing at which the characteristic change in the biological information P(k) starts as the first timing kF, instead of the timing at which the characteristic change occurs in the biological information P(k). This allows the internal state estimating device 10c according to the third embodiment to more precisely determine which of the first timing kF and the second timing iZ occurs first, and avoids the problem of not estimating the internal state of the occupant even though the characteristic change in the biological information P(k) starts to occur before the change in the environmental information E(i).

[0119] Here, an example has been described in which the internal state estimating device 10c according to the third embodiment is configured by applying the feature change extraction unit 13b to the internal state estimating device 10 according to the first embodiment, but the internal state estimating device 10c is not limited to this, and may also be configured by applying the feature change extraction unit 13b to the internal state estimating device 10b according to the second embodiment.

[0120] As described above, according to the third embodiment, the characteristic change extractor 13b extracts the timing at which a characteristic change in the biometric information P(k) starts as the first timing kF, instead of the timing at which a characteristic change occurs in the biometric information P(k). As a result, the internal state estimating device 10c according to the third embodiment can more precisely determine the order of the first timing kF and the second timing iZ, in addition to the effects of the first embodiment. Furthermore, as a result, the internal state estimating device 10c according to the third embodiment can avoid a problem in which the internal state of the occupant is not estimated even though a characteristic change in the biometric information P(k) starts to occur before a change in the environmental information E(i).

[0121] In addition, the present disclosure allows for free combination of the embodiments, modification of any of the components of the embodiments, or omission of any of the components of the embodiments. [Industrial Applicability]

[0122] The present disclosure is suitable for use in an internal state estimation device, as it can suppress erroneous estimation of the internal state of a vehicle occupant. [Explanation of symbols]

[0123] 10 internal state estimation device, 10b internal state estimation device, 10c internal state estimation device, 11 biometric information acquisition unit, 12 environmental information acquisition unit, 13 feature change extraction unit (first extraction unit), 13b feature change extraction unit, 14 environmental information change extraction unit (second extraction unit), 15 timing judgment unit (judgment unit), 16 internal state estimation unit (estimation unit), 17 target section setting unit, 21 processing circuit, 22 CPU, 23 memory, E environmental information, F feature change extraction result, i input number, iZ second timing, k input number, kF first timing, L estimation target section, L1 estimation target section, L2 estimation target section, L3 estimation target section, P biometric information, R estimation result, RT judgment result, Z environmental information change extraction result.

Claims

1. a first extraction unit that extracts, based on biometric information acquired in time series from a vehicle occupant, a first timing at which a characteristic change occurs in the biometric information; a second extraction unit that extracts a second timing at which a change occurs in environmental information, the environmental information being information acquired in a time series manner and having an effect on the internal state of the occupant when the change occurs; a determination unit that determines whether the first timing extracted by the first extraction unit and the second timing extracted by the second extraction unit occur before or after each other; an estimation unit that estimates an internal state of the occupant based on at least one of the biological information and the environmental information, The estimation unit An internal state estimation device, characterized in that it determines whether the internal state can be estimated depending on the determination result by the determination unit, or estimates the internal state based on the determination result by the determination unit.

2. The estimation unit When the determination unit determines that the first timing precedes the second timing, or when the determination unit determines that the first timing and the second timing are simultaneous, the internal state can be estimated.

2. The internal state estimation device according to claim 1.

3. a target interval setting unit that sets an estimation target interval having a predetermined time range on a time series in which the biological information and the environmental information are acquired; The first extraction unit extracting the first timing based on the biological information included in the estimation target section set by the target section setting unit from the biological information acquired in time series from the occupant; The second extraction unit extracting the second timing based on environmental information included in an estimation target section set by the target section setting unit from the environmental information acquired in time series; The determination unit determining whether a first timing extracted by the first extraction unit based on biological information included in the estimation target section and a second timing extracted by the second extraction unit based on environmental information included in the estimation target section are prior to or subsequent to each other; The estimation unit an internal state of the occupant is estimated based on at least one of biological information and environmental information included in the estimation target section, and whether or not the internal state can be estimated is determined according to a determination result by the determination unit; Alternatively, the internal state is estimated based on the determination result by the determination unit.

2. The internal state estimation device according to claim 1.

4. The target section setting unit 4. The internal state estimation device according to claim 3, wherein the estimation target section is shifted in the time direction each time a predetermined condition is satisfied.

5. The predetermined condition is:

5. The internal state estimation device according to claim 4, wherein st pieces of said biological information (st is an integer equal to or greater than 1) are acquired.

6. The estimation unit The internal state of the occupant is estimated using a database in which at least one of first change information, which is information defining a change in the biological information including a characteristic change at the first timing, and second change information, which is information defining a change in the environmental information including a change at the second timing, is associated with information indicating the internal state of the occupant.

6. The internal state estimation device according to claim 1, wherein:

7. The biological information is 6. The internal state estimation device according to claim 1, wherein the information is at least one of information indicating the heart rate of the occupant, information indicating the pulse rate of the occupant, and information indicating an image of the face of the occupant.

8. The environmental information is 6. The internal state estimation device according to claim 1, wherein the information is at least one of internal information of the vehicle, external information of the vehicle, and control information of the vehicle.

9. The first extraction unit Instead of the timing at which a characteristic change occurs in the biometric information, the timing at which the characteristic change in the biometric information starts is extracted as the first timing.

6. The internal state estimation device according to claim 1, wherein:

10. An internal state estimation method by an internal state estimation device, a first extraction unit extracting, based on biometric information acquired in time series from a vehicle occupant, a first timing at which a characteristic change occurs in the biometric information; a second extraction unit extracting, based on environmental information that is information acquired in a time series from the vehicle and its surroundings, a second timing that is a timing at which a change occurs in the environmental information; a determination unit determining whether the first timing extracted by the first extraction unit and the second timing extracted by the second extraction unit occur before or after the first timing; an estimation unit estimating an internal state of the occupant based on at least one of the biological information and the environmental information, The estimation unit An internal state estimation method, characterized in that it is determined whether the internal state can be estimated depending on a determination result by the determination unit, or the internal state is estimated based on the determination result by the determination unit.

Images