INTERNAL CONDITION ESTIMATION DEVICE AND INTERNAL CONDITION ESTIMATION METHOD

Abstract

An interior state estimation device (10) comprises: a characteristic change extraction unit (13) for extracting a first time point, which is a time point at which a characteristic change occurs in biometric information based on the biometric information acquired in time series from an occupant of a vehicle; an environmental information change extraction unit (14) for extracting a second time point, which is a time point at which a change in environmental information occurs based on the environmental information, which is information acquired in time series and is information at which a change in the information affects an internal state of the occupant;a timing determination unit (15) for determining an order of the first timing extracted by the characteristic change extraction unit and the second timing extracted by the environmental information change extraction unit; and an internal state estimation unit (16) for estimating an internal state of the occupant based on at least one of the biometric information and the environmental information, wherein the internal state estimation unit determines whether or not the internal state can be estimated depending on a result of the determination performed by the timing determination unit.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an internal state estimation apparatus and an internal state estimation method. BACKGROUND

[0002] Conventionally, a technique for estimating an internal state, such as the emotion of a vehicle occupant, using the occupant's biometric information is known. For example, Patent Literature 1 discloses a method for measuring two types of biological fluctuations, namely heart rate fluctuation and chaotic fluctuation, from a driver's pulse wave, and estimating four conditions that affect the driver's driving based on the two types of biological fluctuations: an "adaptive performance decline or depressive state," a "drowsy or fatigued state," a "tension or mood elevation state," and a "stressed state." CITATION LISTPATENT LITERATURE

[0003] Patent Literature 1: JP 2020-74805 A SUMMARY OF THE INVENTION OBJECT

[0004] In the method described in Patent Literature 1 (hereinafter referred to simply as the "prior art"), an internal state, such as the driver's emotion, is estimated using a pulse wave as the driver's biometric information. However, the biometric information including the pulse wave does not necessarily change in conjunction with a change in the driver's internal state (internal state), and may also change due to operations that do not involve a change in the driver's internal state, such as breathing and conversation. In this case, the prior art described above has the possibility of incorrectly estimating the driver's internal state.

[0005] The present disclosure has been made to solve the above problems, and an object of the present disclosure is to provide an interior state estimation device capable of suppressing an erroneous estimation of an interior state of an occupant of a vehicle. SOLUTION TO THE TASK

[0006] An internal state estimation device according to the present disclosure includes: a first extraction unit for extracting a first time point, which is a time point at which a characteristic change occurs in biometric information based on the biometric information acquired in time series from an occupant of a vehicle; a second extraction unit for extracting a second time point, which is a time point at which a change occurs in environmental information based on the environmental information, which is information acquired in time series and is information at which a change in the information affects an internal state of the occupant; a determination unit for determining an order of the first time point extracted by the first extraction unit and the second time point extracted by the second extraction unit;and an estimation unit for estimating an internal state of the occupant based on at least one of the biometric information and the environmental information, wherein the estimation unit determines whether or not the internal state can be estimated depending on a result of the determination performed by the determination unit; ADVANTAGEOUS EFFECTS OF THE INVENTION

[0007] According to the present disclosure, with the above configuration, it is possible to suppress an erroneous estimation of the internal state of the occupant of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a diagram illustrating a configuration example of an internal state estimation device according to a first embodiment. Fig. 2 is a flowchart illustrating an operation example of the internal state estimation device according to the first embodiment. Fig. 3 is a diagram illustrating a specific example of biometric information (heart rate) and environmental information (gait position) in the first embodiment. Fig. 4A is a diagram illustrating a configuration example of a first database according to the first embodiment, Fig. 4B is a diagram illustrating a configuration example of a second database according to the first embodiment, and Fig. 4C is a diagram illustrating a configuration example of a third database according to the first embodiment. Fig. 5 is a diagram for explaining an effect of the internal state estimation device according to the first embodiment. Fig. 6A and Fig. 6B are diagrams illustrating a hardware configuration example of the internal state estimation device according to the first embodiment. Fig. 7 is a diagram showing a configuration example of an internal state estimation device according to a second embodiment. Fig. 8 is a diagram for explaining an example of the movement of an estimation target portion in the second embodiment. Fig. 9 is a diagram illustrating a specific example of biometric information (heart rate) and environmental information (gait position) in the second embodiment. Fig. 10 is a diagram illustrating a configuration example of an internal state estimation device according to a third embodiment. Fig. 11A is a diagram showing an example of the order of a first time measurement and a second time measurement in the first embodiment, and Fig. 11B is a diagram showing an example of the order of the first timing and the second timing in the third embodiment. DESCRIPTION OF THE EMBODIMENTS

[0008] The embodiments will be described in detail below with reference to the drawings. First embodiment.

[0009] Fig. 1 is a diagram showing a configuration example of an interior state estimation device 10 according to a first embodiment. The interior state estimation device 10 estimates an interior state of a vehicle occupant.

[0010] The occupant of the vehicle may be a driver of the vehicle or another passenger.

[0011] For example, in Fig. 1, the indoor state estimation device 10 includes a biometric information acquisition unit 11, an environmental information acquisition unit 12, a property change extraction unit (first extraction unit) 13, an environmental information change extraction unit (second extraction unit) 14, a time point determination unit (determination unit) 15, and an indoor state estimation unit (estimation unit) 16.

[0012] The biometric information acquisition unit 11 is connected to a biometric information recognition unit (not shown) that recognizes biometric information P(k) from a vehicle occupant. The biometric information acquisition unit 11 acquires the biometric information P(k) recognized in time series by the biometric information acquisition unit from the biometric information acquisition unit. The biometric information acquisition unit 11 outputs the acquired biometric information P(k) to the characteristic change extraction unit 13.

[0013] Here, the biometric information P(k) includes, for example, at least one piece of information indicating a heartbeat of the occupant, one piece of information indicating a pulse wave, a pulse rate, and one piece of information indicating a facial image. Furthermore, the biometric information recognition unit that recognizes the biometric information P(k) includes, for example, a heart rate sensor, a pulse wave sensor, a pulse rate sensor, a camera, and the like.

[0014] The environmental information acquisition unit 12 is connected to an environmental information detection unit (not shown) that detects the environmental information E(i). The environmental information acquisition unit 12 acquires the environmental information E(i) detected in time series by the environmental information acquisition unit. The environmental information acquisition unit 12 outputs the acquired environmental information E(i) to the environmental information change extraction unit 14.

[0015] Here, the environmental information E(i) is information that affects the internal state of the occupant when the information changes, more precisely, information that causes a change in the internal state of the occupant when the information changes, and is mainly information that concerns external factors other than the occupant.

[0016] Examples of the environmental information E(i) include in-vehicle information (vehicle interior information), out-of-vehicle information (vehicle exterior information), and vehicle control information. Furthermore, the vehicle control information includes, for example, information about a gear position, navigation information, information about a steering wheel operation, information about a pedal operation such as an accelerator pedal and a brake pedal, and the like. The environmental information detection unit that detects the environmental information E(i) includes, for example, a control device, a navigation system, a sensor, a camera, and the like mounted on the vehicle. Furthermore, the internal state of the occupant includes, for example, a stress state, a fatigue state, a depression state, or the like of the occupant, in addition to the occupant's emotion.

[0017] Note that "k" in the biometric information P(k) is an input number assigned to the biometric information, and "i" in the environmental information E(i) is an input number assigned to the environmental information, and both are integers equal to or greater than 1. Note that when the input numbers k and i are considered at any time, the values ​​of k and i are not necessarily the same. For example, 1 second after the internal state estimation device 10 starts processing the internal state estimation, k may have the value "30" and i may have the value "15." That is, 30 pieces of the biometric information P(k) and 15 pieces of the environmental information E(i) are acquired by the internal state estimation device 10 at a time point of 1 second after the internal state estimation device 10 starts processing the internal state estimation.

[0018] The characteristic change extraction unit 13 acquires the biometric information P(k) from the biometric information acquisition unit 11. The characteristic change extraction unit 13 extracts a first time point kF, which is a time point at which a characteristic change occurs in the biometric information P(k), based on the biometric information P(k) acquired by the biometric information acquisition unit 11.

[0019] For example, when a specific temporal change occurs in the biometric information P(k) acquired by the biometric information acquisition unit 11, the characteristic change extraction unit 13 determines that a characteristic change occurs in the biometric information P(k). Here, the specific temporal change in the biometric information P(k) refers to a change mainly caused by a change in the internal state of the occupant among the changes in the biometric information P(k), or a change that may be caused by a change in the internal state of the occupant among the changes in the biometric information P(k).For example, if the biometric information P(k) is the heart rate, the specific temporal change of the heart rate means that the heart rate detected in the time series exceeds a certain upper limit, or that the heart rate is below a certain lower limit.

[0020] The characteristic change extraction unit 13 outputs the extracted first time point kF to the time point determination unit 15. In addition, the characteristic change extraction unit 13 sets the biometric information P(k) used for the extraction of the first time measurement kF as a characteristic change extraction result F(kF) and outputs the characteristic change extraction result F(kF) to the internal state determination unit 16.

[0021] The environmental information change extraction unit 14 acquires the environmental information E(i) from the environmental information acquisition unit 12. Based on the environmental information E(i) acquired by the environmental information acquisition unit 12, the environmental information change extraction unit 14 extracts a second time point iZ, which is a time point at which a change in the environmental information E(i) occurs.

[0022] For example, when a specific temporal change occurs in the environmental information E(i) acquired by the environmental information acquisition unit 12, the environmental information change extraction unit 14 determines that a change has occurred in the environmental information E(i). The specific temporal change in the environmental information E(i) refers, for example, to a change in the gear position from a drive gear to a reverse gear when the environmental information E(i) is a gear position, and to a change in a driving scene of a vehicle (for example, a vehicle traveling from a general road to a highway) when the environmental information E(i) is navigation information.

[0023] The environmental information change extraction unit 14 outputs the extracted second timing iZ to the timing determination unit 15. Further, the environmental information change extraction unit 14 sets the environmental information E(i) used to extract the second timing iZ as an environmental information change extraction result Z(iZ) and outputs the environmental information change extraction result Z(iZ) to the internal state estimation unit 16.

[0024] The timing determination unit 15 acquires the first time kF output from the property change extraction unit 13 and the second time iZ output from the environmental information change extraction unit 14. Then, the timing determination unit 15 determines the order of the acquired first time kF and the second time iZ, and outputs the determination result RT(kF, iZ) to the internal state estimation unit 16.

[0025] The internal state estimation unit 16 acquires the property change extraction result F(kF) acquired by the property change extraction unit 13 and the environmental information change extraction result Z(iZ) output from the environmental information change extraction unit 14. Furthermore, the internal state estimation unit 16 acquires the determination result RT(kF, iZ) output from the timing determination unit 15.

[0026] The interior state estimation unit 16 estimates the interior state of the occupant based on at least one of the characteristic change extraction results F(kF) (i.e., the biometric information P(k)) acquired by the characteristic change extraction unit 13 and the environmental information change extraction results Z(iZ) (i.e., the environmental information E(i)) acquired by the environmental information change extraction unit 14. At this time, the interior state estimation unit 16 determines whether or not it is capable of estimating the interior state of the occupant depending on the determination result RT(kF, iZ) acquired by the timing determination unit 15.

[0027] Specifically, the state estimation unit 16 enables the execution of the internal state estimation when the timing determination unit 15 determines that the first time point kF is earlier than the second time point iZ, or when the first time point kF and the second time point iZ are determined to be the same. That is, in each of the above cases, the internal state estimation unit 16 estimates the internal state of the occupant based on at least one of the characteristic change extraction results F(kF) (i.e., the biometric information P(k)) acquired by the characteristic change extraction unit 13 and the environmental information change extraction results Z(iZ) (i.e., the environmental information E(i)) acquired by the environmental information change extraction unit 14. Then, the internal state determination unit 16 outputs the estimation result R(t).

[0028] On the other hand, if the timing determination unit 15 determines that the first time point kF is later than the second time point iZ, the state estimation unit 16 does not perform the estimation of the internal state. That is, if the timing determination unit 15 determines that the first time point kF is later than the second time point iZ, the state estimation unit 16 does not perform the estimation of the internal state of the vehicle occupant.

[0029] Next, an operation example of the internal state estimation device 10 according to the first embodiment will be described with reference to a Fig. The flowchart shown in Figure 2 is described.

[0030] First, the biometric information acquisition unit 11 acquires biometric information P(k) recognized by a biometric information acquisition unit (not shown) in time series from the biometric information acquisition unit (step ST1). The biometric information acquisition unit 11 outputs the acquired biometric information P(k) to the characteristic change extraction unit 13.

[0031] Next, the characteristic change extraction unit 13 extracts a first time point kF, which is a time point at which a characteristic change occurs in the biometric information P(k), based on the biometric information P(k) acquired by the biometric information acquisition unit 11 (step ST2). Furthermore, the characteristic change extraction unit 13 outputs the extracted first time point kF to the time point determination unit 15 and outputs the biometric information P(k) used to extract the first time point kF to the internal state estimation unit 16 as a characteristic change extraction result F(kF).

[0032] On the other hand, the environmental information acquisition unit 12 acquires the environmental information E(i) acquired by the environmental information detection unit (not shown) in time series (step ST3). The environmental information acquisition unit 12 outputs the acquired environmental information E(i) to the environmental information change extraction unit 14.

[0033] Next, the environmental information change extraction unit 14 extracts a second time point iZ, which is a time point at which a change in the environmental information E(i) occurs, based on the environmental information E(i) acquired by the environmental information acquisition unit 12 (step ST4). Further, the environmental information change extraction unit 14 outputs the extracted second time iZ to the time point determination unit 15 and outputs the environmental information E(i) used to extract the second time iZ to the internal state estimation unit 16 as an environmental information change extraction result Z(iZ).

[0034] It should be noted that steps ST1 to ST2 and steps ST3 to ST4 described above can be performed in parallel.

[0035] Next, the timing determination unit 15 determines the order of the first time point kF and the second time point iZ (step ST5). Furthermore, the timing determination unit 15 outputs the determination result RT(kF, iZ) to the internal state estimation unit 16.

[0036] For example, the timing determination unit 15 sets the determination result RT(kF, iZ) to "1" when it determines, as a result of the determination, that the first time kF is earlier than the second time iZ, or that the first time kF and the second time iZ are the same. On the other hand, the timing determination unit 15 sets the determination result RT(kF, iZ) to "0" when it determines, as a result of the determination, that the first time kF is later than the second time iZ.

[0037] Next, the internal state estimation unit 16 checks whether the determination result RT(kF, iZ) acquired by the timing determination unit 15 is "1," that is, whether the first timing kF is earlier than the second timing iZ, or whether the first timing kF and the second timing iZ are determined to be the same (step ST6). If the determination result RT(kF, iZ) is not "1" (step ST6; NO), the state estimation unit 16 determines not to perform the internal state estimation and ends the processing. On the other hand, if the determination result RT(kF, iZ) is "1" (step ST6; YES), the process proceeds to step ST7.

[0038] In step ST7, the interior state estimation unit 16 enables the execution of the interior state estimation and estimates the interior state of the vehicle occupant based on at least one of the result F(kF) (i.e., the biometric information P(k)) acquired by the characteristic change extraction unit 13 and the result Z(iZ) (i.e., the environmental information change extraction unit 14) acquired by the environmental information change extraction unit 14 (step ST7). Then, the interior state determination unit 16 outputs the estimation result R(t).

[0039] Next, an operation example of the internal state estimation device 10 including an internal state estimation process by the internal state determination unit 16 will be described using a specific example.

[0040] To facilitate the description, it is assumed here that the biometric information P(k) is the occupant's heart rate, and the environmental information E(i) is the vehicle's gear position. Furthermore, in the following description, it is assumed that the biometric information acquisition unit 11 acquires a total of 15 occupant heart rates one by one per second, and the environmental information acquisition unit 12 acquires a total of 15 vehicle gear positions one by one per second, starting from the time the interior state estimation device 10 starts operating. Fig. Figure 3 illustrates a specific example of the heart rate and gear position in the above case.

[0041] In addition, here the specific temporal change of heart rate, which is a reference for determining that a characteristic change in heart rate has occurred, means that the heart rate has exceeded “1.3”, which is the upper limit threshold.

[0042] First, the characteristic change extraction unit 13 extracts "5 seconds (t=5)", which is the time at which the heart rate exceeds the upper limit threshold of 1.3, as the first timing determination unit kF (corresponding to the above-described step ST2), and outputs the first timing determination unit kF to the timing determination unit 15. Furthermore, the characteristic change extraction unit 13 outputs 15 heart rates (15 heart rates from "1.0" at t=1 to "1.4" at t=15) used to extract the first timing kF to the internal state estimation unit 16 as characteristic change extraction results F(kF).

[0043] On the other hand, the environment information change extraction unit 14 extracts “9 seconds (t=9)”, which is the time at which the gear position is received from the drive gear (in Fig. 3 belonging to D) to reverse gear (in Fig. 3 corresponding to R) changes as the second time point iZ (corresponding to the above-described step ST4), and outputs the second time point iZ to the time point determining unit 15. Further, the environmental information change extraction unit 14 outputs 15 gear positions (15 gear positions from "D" at t=1 to "R" at t=15) used to extract the second time point iZ to the interior state estimation unit 16 as the environmental information change extraction result Z(iZ).

[0044] Next, the timing determination unit 15 determines the order of "5 seconds (t=5)" corresponding to the first time point kF and "9 seconds (t=9)" corresponding to the second time point iZ, and determines that the first time point kF is earlier (corresponding to step ST5 described above). Furthermore, since determining that the first time point kF is earlier, the timing determination unit 15 sets the determination result RT(kF, iZ) to "1" and outputs the determination result RT(kF, iZ) to the internal state estimation unit 16.

[0045] Next, the internal state estimation unit 16 checks whether the determination result RT(kF, iZ) acquired by the timing determination unit 15 is "1" or not (corresponding to step ST6 described above). Since the determination result RT(kF, iZ) is "1", the internal state determination unit 16 enables the execution of internal state estimation and estimates the internal state of the occupant based on at least one of the 15 heart rates and the 15 gear positions (corresponding to step ST7 described above).

[0046] Here, an example of an interior state determination method by the interior state determination unit 16 will be described. The interior state determination unit 16 estimates the internal state of the occupant using, for example, a pre-created database for internal state estimation. Note that this database is pre-recorded in a recording unit (not shown) included in the interior state estimation device 10.

[0047] An example of such a database is in Fig. 4A. As shown in Fig. 4A, the database (hereinafter also referred to as “first database”) is configured by linking information (hereinafter also referred to as “first change information”) defining a change mode (transition) of 15 pieces of biometric information P(k) derived from the characteristic change extraction result F(kF) with information (hereinafter also referred to as “internal state information”) indicating an internal state of the occupant.

[0048] For example, in the example of Fig. 4A, "the heart rate has increased to 1.5 (over 1.3, which is the upper limit)" is set as the first change information, and "voltage" is set as the internal state information associated with the first change information. Similarly, in the example of Fig. 4A, "the heart rate has increased to 1.7 (over 1.3, the upper limit)" is defined as the first change information, and "nervous" as the internal state information linked to the first change information. It should be noted that, although in Fig. 4A, the change in the biometric information P(k) indicated by the first change information also includes the property change (exceeding 1.3, which is the upper limit threshold) that occurs at the first time point kF described above.

[0049] Similarly, in the example of Fig. 4A, "the heart rate has increased to 1.9 (over 1.3, which is the upper limit threshold)" is set as the first change information, and "heavy pressure" is set as the internal state information associated with the first change information. In addition, in the example of Fig. 4A “the heart rate has increased to 2.1 (over 1.3, the upper limit)” is set as the first change information, and “anxiety” is set as the internal state information associated with the first change information.

[0050] For example, in the example of Fig. 3, it can be seen that the heart rate has increased to 1.9 and exceeded 1.3, the upper limit threshold, by 15 heart rates, which are the characteristic change extraction results F(kF). Therefore, the internal state determination unit 16 searches the first database with the fact that the heart rate has increased to 1.9 (i.e., a change mode of the heart rate) as the key, and determines "severe pressure," which is the internal state information corresponding to the key. Then, the internal state determination unit 16 outputs an estimation result of the occupant's internal state as "severe pressure" based on the determined internal state information.

[0051] Note that an operation example is described here for a case where the heart rate rises above 1.3, which is the upper limit. However, the same applies to the case where the heart rate falls below the lower limit (for example, 0.7). In this case, in the first database, for example, "the heart rate has dropped to 0.6 (below 0.7, the lower limit threshold)" can be set as the first change information, and "fatigue," for example, can be set as the internal state information associated with the first change information.

[0052] Another example of the database is in Fig. 4B. As shown in Fig. 4B, the database (hereinafter also referred to as "second database") is configured by associating information (hereinafter also referred to as "second change information") defining a change mode (transition) of the environmental information E(i), which was derived from the result of extracting environmental information changes Z(iZ), with information (internal state information) indicating the internal state of the occupant. Note that the change in the environmental information E(i) indicated by the second change information also includes the change in the environmental information E(i) that occurs at the second time point iZ described above.

[0053] For example, in the example of Fig. 4B, "the gear position has been changed from the drive gear to the reverse gear" is set as the second change information, and "voltage" is set as the internal state information associated with the second change information. In the example of Fig. 4B, "the gear position has been changed from the drive gear to the parking gear (P)" is set as the second change information, and "rest" is set as the internal state information associated with the second change information. In the example of Fig. 4B, “the gear position has been changed from the drive gear to the neutral gear (N)” is set as the second change information, and “caution” is set as the internal state information associated with the second change information.

[0054] For example, in the example of Fig. 3, it can be seen that the gear position has been changed from the drive gear to the reverse gear by 15 gear positions, which are the result of extracting the environmental information Z(iZ). Therefore, the interior state determination unit 16 searches the second database with the fact that the gear position has been changed from the drive gear to the reverse gear (i.e., a gear position change mode) as the key and determines "voltage," which is the internal position information associated with the key. Then, the interior state determination unit 16 outputs an estimation result of the occupant's internal state as "voltage" based on the determined internal state information.

[0055] Note that although the operation example in a case where the environmental information E(i) indicates the gear position has been described here, the same applies to a case where the environmental information E(i) indicates something other than the gear position. For example, if the environmental information E(i) is operation information of a brake pedal, the fact that the number of times the brake pedal has been depressed during the predetermined time has increased to 8 (exceeding the upper limit of 5) may be set in the second database as second change information, and, for example, "impatient" may be set as the internal state information associated with the second change information.

[0056] Another example of a database is in Fig. 4C. As shown in Fig. As shown in Figure 4C, this database (hereinafter also referred to as the "third database") is formally configured by combining the first and second databases described above. Specifically, the third database is configured by linking the combination of the first change information and the second change information with the internal state information.

[0057] For example, in the example of Fig. 4C, the fact that "the gear position was changed from drive to reverse and the heart rate increased to 1.5 (exceeding the upper limit of 1.3)" is set as the combination of change information, and "voltage" is set as the internal state information associated with the combination. Furthermore, for example, in the example of Fig. 4C, the fact that “the gear position was changed from forward gear to reverse gear and the heart rate increased to 1.7 (which exceeds the upper limit of 1.3)” is set as the combination of the change information, and “nervous” is set as the internal state information associated with the combination.

[0058] Similarly, in the example of Fig. 4C, the fact that "the gear position has been changed from drive to reverse and the heart rate has increased to 1.9 (exceeding the upper limit of 1.3)" is set as the combination of change information, and "heavy pressure" is set as the internal state information associated with the combination. In addition, in the example of Fig. 4C, the fact that “the gear position was changed from forward gear to reverse gear and the heart rate increased to 2.1 (which exceeds the upper limit of 1.3)” is set as the combination of change information, and “anxiety” is set as the internal state information associated with the combination.

[0059] For example, in the example of Fig. 3 that the heart rate has risen to 1.9 and exceeded 1.3, the upper limit threshold, by 15 heart rates, which are the characteristic change extraction results F(kF), and 15 gear positions, which are the environmental information change extraction results Z(iZ), and that the gear position has changed from the drive gear to the reverse gear. Therefore, the internal state determination unit 16 searches the third database with the fact that the gear position has changed from the drive gear to the reverse gear and the heart rate has risen to 1.9 (that is, modes of change in both the gear position and the heart rate) as the key, and determines "heavy pressure," which is the internal state information associated with the key. Then, the internal state determination unit 16 outputs an estimation result of the internal state of the vehicle occupant as "heavy pressure" based on the determined internal state information.

[0060] The interior state estimation unit 16 can accurately estimate the interior state of the occupant by estimating the interior state of the occupant using, for example, any one of the first database to the third database. Note that when the interior state estimation unit 16 estimates the interior state of the occupant using the third database, the interior state estimation unit can estimate the interior state of the occupant based on the relationship between the heart rate, which is the biometric information P(k), and the gear position, which is the environmental information E(i). Therefore, the estimation accuracy can be improved compared to the case of estimating the interior state of the occupant using the first database or the second database.

[0061] It should be noted that in a case where the emotion of the vehicle occupant is used as the internal state specified by the internal state information in each of the databases described above, for example, an emotion model defined in Russell's circumplex model of emotion can be used to construct each of the databases.

[0062] Russell's circumplex model of emotions is a model that states that all emotions are arranged in a circular pattern on a plane represented two-dimensionally by "pleasant - unpleasant" and "awakening - non-awakening." Various emotion models have been proposed, but Russell's circumplex model of emotions is expressed in a simple structure and can be exhaustively applied to all emotions, making it suitable for constructing any of the databases described above.

[0063] For example, in Russell's circumplex model of emotions, "tension," "nervousness," "intense pressure," "anxiety," and the like are defined as emotions classified into "unpleasant" and "arousal," and "sadness," "depression," "lethargy," "fatigue," and the like are defined as emotions classified into "unpleasant" and "non-arousal." Furthermore, in the same model, "caution," "excitement," "energy," "joy," and the like are defined as emotions classified into "pleasant" and "arousal," and "contentment," "pleasant," "relaxing," "calm," and the like are defined as emotions classified into "pleasant" and "non-arousal."

[0064] For example, the manager or the like of the internal state estimation device 10 can construct each of the above-described databases by using each of the above-described emotions as an internal state. As shown in Fig. As shown in Figures 4A to 4C, at least one of the first change information and the second change information must be associated with each internal state.

[0065] It should be noted that the contents of the first change information, the second change information, and the internal state information in each database described above are merely examples, and contents other than those mentioned above may be set. For example, in the above description, the example of using the occupant's emotion (tension or the like) as internal state information was described, but the internal state information is not limited to this, and, for example, a stress state, a fatigue state, a depression state, or the like of the occupant may be used. In addition, in the above example, an example of setting the name ("tension" or the like) of the occupant's emotion as internal state information in each database was described.However, the internal state information in each database is not limited to this, and for example, information acquired by digitizing the occupant's emotion can be set.

[0066] Furthermore, in each database described above, for example, different information can be set for each occupant. For example, the occupant can select the emotion (e.g., impatient, tense, energetic, and the like) that the occupant wants to estimate as the occupant's internal state and the type (e.g., gear position, navigation information, and the like) of the environmental information E(i) accordingly, and generate the databases in advance based on the selected emotion and a type of environmental information E(i). In this case, the occupant can use a database adapted for estimating an internal state of the occupant, and comfort is improved.

[0067] Furthermore, in the above description, an example was described in which the interior state determination unit 16 uses the determination result RT(kF, iZ) of the order of the first timing kF and the second timing iZ acquired by the timing determination unit 15 when determining whether or not to perform the interior state determination processing. However, the interior state determination unit 16 is not limited to this, and, for example, the determination result RT(kF, iZ) acquired by the timing determination unit 15 may be used as used for estimating the interior state of the occupant.

[0068] For example, when the determination result RT(kF, iZ) acquired by the timing determination unit 15 is “1”, that is, when the first time point kF is earlier than the second time point iZ, or when the first time point kF and the second time point iZ are the same, the internal state estimation unit 16 may output an estimation result R(t) indicating that the internal state of the occupant is “tension”.

[0069] Moreover, in the above description, an example was described in which a first time point kF and a second time point iZ are extracted. However, it is also assumed that a plurality of at least one of the first time points kF and the second time points iZ are extracted. In this case, the time point determining unit 15 may set the determination result RT(kF, iZ) to "1" if there is at least one case where the first time point kF is earlier than the second time point iZ or the first time point kF and the second time point iZ are the same in the extracted time group.

[0070] In addition, the internal state determination unit 16 may estimate the internal state of the occupant by using, for example, a machine learning model created for internal state estimation.In this case, the machine learning model may be, for example, one of the following models: (1) a model learned to output internal state information corresponding to first change information in response to input of the first change information derived from the property change extraction result F(kF), (2) a model learned to output internal state information corresponding to second change information in response to input of the second change information derived from the environmental information change extraction result Z(iZ), and (3) a model learned to output internal state information corresponding to a combination of the first change information and the second change information in response to input of a combination of the first change information and the second change information.

[0071] Furthermore, the internal state determination unit 16 may estimate the internal state of the occupant by a method similar to the prior art method described above. For example, the internal state estimation unit 16 may measure two types of biological fluctuations, namely the heart rate fluctuation and the chaotic fluctuation from the occupant's pulse wave acquired as biometric information P(k), and estimate the internal state of the occupant based on the two types of biological fluctuations.

[0072] In addition, the internal state determination unit 16 may output the estimation result of the internal state of the occupant in two levels, such as “anxiety” or “impatience”, or output the estimation result in multiple levels (for example, 0-100 levels), such as “anxiety in 90 / 100 levels”.

[0073] As described above, in the interior state estimation device 10 according to the first embodiment, the interior state determination unit 16 determines whether or not it is capable of estimating the internal state of the occupant depending on the determination result of the timing determination unit 15. Specifically, the state estimation unit 16 enables the execution of the internal state estimation when the timing determination unit 15 determines that the first time point kF is before the second time point iZ, or when the first time point kF and the second time point iZ are determined to be the same. On the other hand, when the timing determination unit 15 determines that the first time point kF is after the second time point iZ, the state estimation unit 16 does not perform the internal state estimation.This is based on the idea that when the occupant's internal state changes (e.g., the occupant becomes nervous), the biometric information changes (e.g., the heart rate increases sharply) before the environmental information changes (e.g., the gear position changes from drive to reverse). Therefore, in the internal state estimation device 10 according to the first embodiment, it is possible to suppress an erroneous estimation of the occupant's internal state based on a change in the biometric information without changing the occupant's internal state.

[0074] In this context, in the prior art described above, an internal state, such as the driver's emotions, is estimated by using a pulse wave as the driver's biometric information. However, the biometric information including the pulse wave may not only change due to the change in the driver's internal state at the time of switching from normal driving to parking, as described in W2 of Fig. 5, but also by operating without changing the driver’s internal state, such as breathing and talking during normal driving, as in W1 of Fig. 5. In this case, according to the prior art described above, there is a possibility that a change in W1 is recorded and the driver's internal state is erroneously estimated as "tension" or the like. In the internal state estimation device 10 according to the first embodiment, however, the occurrence of such a problem is suppressed.

[0075] Next, an example of the hardware configuration of the internal state estimation device 10 according to the first embodiment will be described with reference to FIG. Fig. 6. The functions of the biometric information acquisition unit 11, the environmental information acquisition unit 12, the environmental information change extraction unit 13, the environmental information change extraction unit 14, the time determination unit 15, and the internal state estimator 16 in the internal state estimator 10 are implemented by a processing circuit. The processing circuit may be dedicated hardware, as shown in Fig. 6A, or a central processing unit (CPU, also referred to as central processor, processing unit, arithmetic unit, microprocessor, microcomputer, processor or digital signal processor (DSP))22 which executes a program stored in a memory 23, as in Fig. 6B.

[0076] In a case where the processing circuit is dedicated hardware, the processing circuit 21 belongs to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination thereof. The functions of the environmental information acquisition unit 11, the environmental information acquisition unit 12, the property change extraction unit 13, the environmental information change extraction unit 14, the timing determination unit 15, and the internal state determination unit 16 can be implemented by the processing circuit 21, or the functions of the respective units can be implemented jointly by the processing circuit 21.

[0077] When the processing circuit is the CPU 22, the functions of the environmental information acquisition unit 11, the environmental information acquisition unit 12, the property change extraction unit 13, the environmental information change extraction unit 14, the timing determination unit 15, and the internal state determination unit 16 are implemented by software, firmware, or a combination of software and firmware. The software and firmware are described as programs and stored in the memory 23. The processing circuit reads the program stored in the memory 23 and executes it to implement the function of each unit. That is, the internal state estimation device 10 includes a memory for storing a program that, when executed by the processing circuit, is used to execute each of the functions described in, for example, Fig. 2. These programs can also be said to cause a computer to execute procedures and methods performed by the biometric information acquisition unit 11, the environmental information acquisition unit 12, the characteristic change extraction unit 13, the environmental information change extraction unit 14, the timing determination unit 15, and the internal state determination unit 16. Examples of the memory 23 include a non-volatile or volatile semiconductor memory such as a random access memory (RAM), a read-only memory (ROM), a flash memory, an erasable programmable ROM (EPROM) or an electrical EPROM (EEPROM), a magnetic disk, a flexible disk, an optical disk, a compact disc, a mini-disc, a digital versatile disc (DVD), or the like.

[0078] Note that some of the functions of the environmental information acquisition unit 11, the environmental information acquisition unit 12, the environmental information change extraction unit 13, the environmental information change extraction unit 14, the timing determination unit 15, and the internal state determination unit 16 may be implemented by dedicated hardware, and others may be implemented by software or firmware. For example, the functions of the environmental information acquisition unit 11 may be implemented by a processing circuit as dedicated hardware, and the functions of the environmental information acquisition unit 12, the property change extraction unit 13, the environmental information change extraction unit 14, the timing determination unit 15, and the internal state estimation unit 16 may be implemented by the processing circuit that reads and executes programs stored in the memory 23.

[0079] As described above, the processing circuitry may implement the functions described above through hardware, software, firmware, or a combination thereof.

[0080] As described above, the internal state estimation device 10 according to the first embodiment includes the characteristic change extraction unit 13 that extracts the first time point kF, which is the time point at which the characteristic change occurs in the biometric information based on the biometric information P(k) acquired in time series from the occupant of the vehicle; the environmental information change extraction unit 14 that extracts the second time point iZ, which is the time point at which the change in the environmental information occurs based on the environmental information E(i), which is the information acquired in time series and is the information at which the change in the information affects the internal state of the occupant; the time point determining unit 15 that determines the order of the first time point kF extracted by the characteristic change extraction unit 13 and the second time point iZ;extracted by the environmental information change extraction unit 14, and the state estimation unit 16, which estimates the internal state of the occupant based on at least one of the biometric information P(k) and the environmental information E(i), wherein the state estimation unit 16 determines whether or not it is capable of estimating the internal state depending on a determination result by the timing determination unit 15. Thus, the internal state estimation device 10 according to the first embodiment can suppress an erroneous estimation of the internal state of the occupant.

[0081] Furthermore, the interior state determination unit 16 enables estimation of the interior state when the timing determination unit 15 determines that the first time point kF is earlier than the second time point iZ, or when the first time point kF and the second time point iZ are determined to be the same. As a result, the interior state estimation device 10 according to the first embodiment can accurately suppress an erroneous estimation of the occupant's interior state.

[0082] Furthermore, the interior state determination unit 16 estimates the interior state of the occupant using a database in which at least one of the first change information, which is information defining a type of change in the biometric information P(k) including a characteristic change at the first time kF, and the second change information, which is information defining a type of change in the environmental information E(i), including a change at the second time iZ, is associated with information indicating the interior state of the occupant. As a result, the interior state estimation device 10 according to the first embodiment can accurately estimate the interior state of the occupant.

[0083] The biometric information P(k) includes information indicating the occupant's heartbeat, information indicating the occupant's pulse rate, and information indicating the occupant's facial image. Therefore, the interior state estimation device 10 according to the first embodiment can suppress an erroneous estimation of the occupant's interior state by using information that can be easily acquired from the occupant.

[0084] The environmental information E(i) is at least one of in-vehicle information, out-of-vehicle information, and vehicle control information. Therefore, the interior state estimation device 10 according to the first embodiment can suppress an erroneous estimation of the occupant's interior state by using information that can be easily acquired by the vehicle.

[0085] Second Embodiment. In the first embodiment, the internal state estimation device was described, which determines whether or not it is capable of estimating the internal state of the occupant depending on the result of determining the order of the first time lapse kF and the second time lapse iZ. In a second embodiment, an internal state estimation device is described, which sets an estimation target section with a predetermined time range based on a time series in which biometric information P(k) and environmental information E(i) are acquired, and determines whether or not it is capable of estimating an internal state for each estimation target section.

[0086] Fig. Fig. 7 is a diagram showing a configuration example of an internal state estimation device 10b according to the second embodiment. The internal state estimation device 10b according to the second embodiment is acquired by the internal state estimation device 10 according to the first embodiment shown in Fig. 1, a target section setting unit 17 is added. Since the other configurations of the internal state estimation device 10b according to the second embodiment are the same as those of the internal state estimation device 10 according to the first embodiment shown in Fig. 1, the same reference numerals are given, and the description thereof is omitted.

[0087] The target section setting unit 17 sets an estimation target section L with a predetermined time range on the time series in which the biometric information P(k) and the environmental information E(i) are acquired. The estimation target section L defines a section in which the interior state estimator 10b estimates the interior state of the occupant in the time series in which the biometric information P(k) and the environmental information E(i) are acquired.

[0088] For example, as in Fig. 8, the target section setting unit 17 moves the estimation target section L in a time direction (the direction in which time advances) each time a predetermined condition is met. The predetermined condition is, for example, that the biometric information acquisition unit 11 acquires st pieces of biometric information P(k) (st is an integer equal to or greater than 1). In this case, the target section setting unit 17 moves the estimation target section L in the time direction each time st pieces of biometric information P(k) are acquired by the biometric information acquisition unit 11.

[0089] In the interior state estimation device 10b according to the second embodiment, when the target section setting unit 17 sets the estimation target section L, the property change extraction unit 13 extracts the first time determination kF, the environmental information change extraction unit 14 extracts the second time determination iZ, the timing determination unit 15 performs the order determination, and the interior state estimation unit 16 performs the estimation of the interior state within the set range of the estimation target section L. Moreover, in the interior state estimation device 10b, when the target section setting unit 17 moves the estimation target section L, each processing described above is performed for each estimation target section L after the movement.

[0090] For example, the characteristic change extraction unit 13 extracts the first timing kF based on Tp pieces of biometric information P(k-Tp+1) to P(k) included in the estimation target section L set by the target section setting unit 17 among the pieces of biometric information P(k) acquired in time series from the occupant. In addition, the characteristic change extraction unit 13 outputs the extracted first timing kF to the timing determination unit 15 and outputs Tp pieces of the biometric information P(k-Tp+1) to P(k) used to extract the first timing kF to the indoor state estimation unit 16 as characteristic change extraction results F(kF). Note that Tp is, for example, an integer equal to or greater than 2.

[0091] Further, the environmental information change extraction unit 14 extracts the second time point iZ based on Tp pieces of environmental information E(i-Tp+1) to E(i) included in the estimation target section L set by the target section setting unit 17 among pieces of environmental information E(i) acquired in time series from the vehicle and the surroundings of the vehicle. Further, the environmental information change extraction unit 14 outputs the extracted second time point iZ to the time point determination unit 15, and outputs Tp pieces of environmental information E(i-Tp+1) to E(i) used to extract the second time point iZ to the interior state estimation unit 16 as environmental information change extraction results Z(iZ).

[0092] In addition, the timing determination unit 15 determines the order of the first timing kF extracted by the characteristic change extraction unit 13 based on the Tp pieces of biometric information P(k-Tp+1) to P(k) included in the estimation target section L, and the second timing iZ extracted by the environmental information change extraction unit 14 based on the Tp pieces of environmental information E(i-Tp+1) to E(i) included in the estimation target section L.

[0093] When the timing determination unit 15 determines that the first timing change kF is before the second timing change iZ or that the first timing change kF and the second timing change iZ are the same, the interior state estimation unit 16 enables the execution of the interior state estimation and estimates the interior state of the occupant of the vehicle based on at least one of the characteristic change extraction results F(kF) (that is, the Tp pieces of the biometric information P(k-Tp+1) to P(k)) acquired by the characteristic change extraction unit 13 and the environmental information change extraction unit Z(iZ) (that is, the Tp pieces of the environmental information E(i-Tp+1) to E(i)) acquired by the environmental information change extraction unit 14.On the other hand, when the timing determining unit 15 determines that the first time point kF is later than the second time point iZ, the state estimating unit 16 does not estimate the internal state in the estimation target section L.

[0094] Next, an operation example of the internal state estimation device 10b including an internal state estimation process by the internal state determination unit 16 will be described using a specific example.

[0095] To make the description easy to understand, a specific example similar to that of the first embodiment will be described here. That is, it is assumed that the biometric information P(k) is the occupant's heart rate, the surrounding information E(i) is the gear position of the vehicle, the biometric information acquisition unit 11 acquires a total of 15 heart rates of the occupant individually in 1 second, and the surrounding information acquisition unit 12 acquires a total of 15 gear positions of the vehicle individually in 1 second. In addition, here, the specific temporal change of the heart rate, which is a reference for determining that a characteristic change of the heart rate has occurred, means that the heart rate has exceeded "1.3", which is the upper limit. A specific example of the heart rate and gear position in the above case is shown in Fig. 9 shown.

[0096] First, the target section setting unit 17 sets the estimation target section L with a predetermined time range to the time series in which the biometric information P(k) and the surrounding information E(i) are acquired. Here, the predetermined time range is assumed to be "10 seconds," and the target section setting unit 17 sets, for example, a range from 1 second (t=1) to 10 seconds (t=10) as the first estimation target section L1.

[0097] In this case, the characteristic change extraction unit 13 refers to the 10 pieces of biometric information P(1) to P(10) included in the estimation target section L1, extracts "5 seconds (t=5)", which is the time point when the heart rate exceeds 1.3, which is the upper limit threshold, as the first timing determination unit kF in the 10 pieces of biometric information, and outputs the first timing determination unit kF to the timing determination unit 15. Furthermore, the characteristic change extraction unit 13 outputs 10 heart rates (10 heart rates from "1.0" at t=1 to "1.7" at t=10) used to extract the first timing kF to the internal state estimation unit 16 as characteristic change extraction results F(kF).

[0098] On the other hand, the environmental information change extraction unit 14 refers to the 10 pieces of environmental information E(1) to E(10) included in the estimation target section L1, extracts "9 seconds (t=9)", which is the timing at which the gear position changes from the drive gear to the reverse gear in the 10 pieces of environmental information, as the second timing iZ, and outputs the second timing iZ to the timing determination unit 15. Further, the environmental information change extraction unit 14 outputs 10 gear positions (10 gear positions from "D" at t=1 to "R" at t=10) used to extract the second timing iZ to the interior state estimation unit 16 as the environmental information change extraction result Z(iZ).

[0099] Next, the timing determination unit 15 determines the order of "5 seconds (t=5)", which is the first time point kF, and "9 seconds (t=9)", which is the second time point iZ, and determines that the first time point kF is earlier. Furthermore, since determining that the first time point kF is earlier, the timing determination unit 15 sets the determination result RT(kF, iZ) to "1" and outputs the determination result RT(kF, iZ) to the internal state estimation unit 16.

[0100] Next, the internal state estimation unit 16 checks whether the determination result RT(kF, iZ) acquired by the timing determination unit 15 is "1" or not. Since the determination result RT(kF, iZ) is "1", the internal state estimation unit 16 enables the execution of the internal state estimation and estimates the internal state of the occupant in the first estimation target section L1 based on at least one of the 10 heart rates and the 10 gear positions. The internal state determination method by the internal state estimation unit 16 is similar to the method described in the first embodiment.

[0101] Here, the target section setting unit 17 is movable in the estimation target section L in the time direction every time a predetermined condition is met. The predetermined condition is, for example, that two pieces of biometric information P(k) are acquired by the biometric information acquisition unit 11. In this case, the target section setting unit 17 moves the estimation target section L1 in the time direction to a time point of 2 seconds (t=2), which is the time point at which two pieces of biometric information P(k) are acquired by the acquisition unit 11, and sets the next estimation target section L2. The next estimation target section L2 is 10 seconds from 3 seconds (t=3) to 12 seconds (t=12).

[0102] When the next estimation target section L2 is set, the characteristic change extraction unit 13, the environmental information change extraction unit 14, the timing determination unit 15, and the internal state estimation unit 16 perform the above-described processing based on the 10 pieces of biometric information and environmental information included in the estimation target section L2. At this time, the internal state of the occupant estimated by the internal state determination unit 16 becomes the internal state of the occupant in the estimation target section L2.

[0103] Thereafter, the target period setting unit 17 shifts the estimation target period L2 in the time direction to a time point of 4 seconds (t=4), which is the time point at which two pieces of biometric information P(k) are acquired by the next biometric information acquisition unit 11, and sets the next estimation target period L3. The next estimation target period L3 is 10 seconds from 5 seconds (t=5) to 14 seconds (t=14).

[0104] When the next estimation target section L3 is set, the characteristic change extraction unit 13, the environmental information change extraction unit 14, the timing determination unit 15, and the internal state estimation unit 16 perform the above-described processing based on the 10 pieces of biometric information and the environmental information included in the estimation target section L3. At this time, the internal state of the occupant estimated by the internal state determination unit 16 becomes the internal state of the occupant in the estimation target section L3.

[0105] The internal state estimation device 10b according to the second embodiment estimates the internal state of the occupant for each moved estimation target section L, while the target section setting unit 17 similarly moves the estimation target section L in the time direction.

[0106] As described above, by including the target section setting unit 17, the interior state estimation device 10b according to the second embodiment can acquire an estimation result of the occupant's interior state for a desired section of a time series. Furthermore, the interior state estimation device 10b according to the second embodiment can continuously estimate the occupant's interior state in more detail by moving the estimation target section L in the time direction.

[0107] As described above, the interior state estimation device 10b according to the second embodiment includes the target section setting unit 17 that sets the estimation target section L with the predetermined time range to the time series in which the biometric information P(k) and the environmental information E(i) are acquired; the characteristic change extraction unit 13 extracts the first time point kF based on the biometric information included in the estimation target section L set by the target section setting unit 17 among the pieces of biometric information acquired in time series from the occupant; the environmental information change extraction unit 14 extracts the second time point iZ based on the environmental information included in the estimation target section L set by the target section setting unit 17 among the pieces of environmental information acquired in time series;the time point determination unit 15 determines the order of the first time point kF extracted by the property change extraction unit 13 on the basis of the biometric information included in the estimation target section L and the second time point iZ extracted by the environmental information change extraction unit 14 on the basis of the environmental information included in the estimation target section L, and the interior state estimation unit 16 estimates the interior state of the occupant on the basis of at least one of the biometric information and the environmental information included in the estimation target section L, and determines whether it is capable ofto estimate the interior state depending on the determination result by the time-point determination unit 15. As a result, the interior state estimation device 10b according to the second embodiment can acquire an estimation result of the occupant's interior state for a desired portion of a time series, in addition to the effects of the first embodiment.

[0108] The target section setting unit 17 moves the estimation target section L in the time direction each time a predetermined condition is met. This allows the interior state estimation device 10b according to the second embodiment to continuously estimate the interior state of the occupant more accurately.

[0109] Furthermore, the predetermined condition is that st pieces of biometric information P(k) (st represents an integer equal to or greater than 1) are acquired. Consequently, the internal state estimation device 10b according to the second embodiment can move the estimation target portion L depending on the number of acquired pieces of biometric information.

[0110] Third Embodiment. In the first embodiment, the interior state estimation device was described, which determines whether or not the processing for estimating the occupant's interior state can be executed depending on the result of determining the order of the first and second timings. In a third embodiment, an interior state estimation device capable of more accurately determining the order of a first timing and a second timing is described.

[0111] Fig. Fig. 10 is a diagram showing a configuration example of an internal state estimation device 10c according to the third embodiment. The internal state estimation device 10c according to the third embodiment differs from the internal state estimation device 10 according to the first embodiment shown in Fig. 1, in that the property change extraction unit 13 is changed into a property change extraction unit 13b. Since the other configurations of the internal state estimation device 10c according to the third embodiment are the same as those of the internal state estimation device 10 according to the first embodiment shown in Fig. 1, the same reference numerals are given, and the description thereof is omitted.

[0112] The characteristic change extraction unit 13b acquires biometric information P(k) from the biometric information acquisition unit 11. The characteristic change extraction unit 13b extracts, based on the biometric information P(k) acquired by the biometric information acquisition unit 11, a time point at which a characteristic change of the biometric information P(k) begins as a first time point kF.

[0113] For example, the characteristic change extraction unit 13b first extracts a time point at which a characteristic change occurs in the biometric information P(k), similar to the characteristic change extraction unit 13 in the first embodiment. Next, the characteristic change extraction unit 13b searches for a time point at which a characteristic change begins in the biometric information P(k) by tracing the time point in the time direction with the extracted time point as the starting point.

[0114] For example, in a case where the biometric information P(k) is a heart rate, the characteristic change extraction unit 13b sequentially calculates a differential value at each point of the function indicating the biometric information P(k) while backtracking the time point in the time direction with the time point at which the heart rate exceeds the upper limit threshold as a starting point. Then, the characteristic change extraction unit 13b sets the time point at which the differential value is 0 as the time point at which the characteristic change starts. Alternatively, the characteristic change extraction unit 13b may set a time point acquired by backtracking the extracted time point in the time direction by a predetermined time (for example, 2 seconds) as the time point at which the characteristic change starts.Then, the property change extraction unit 13b extracts the time at which the property change starts as the first time kF.

[0115] The characteristic change extraction unit 13b outputs the extracted first time point kF to the time point determination unit 15. In addition, the characteristic change extraction unit 13b sets the biometric information P(k) used to extract the first time point kF as a characteristic change extraction result F(kF) and outputs the characteristic change extraction result F(kF) to the internal state estimation unit 16.

[0116] Next, a specific example and an effect of the processing by the property change extraction unit 13b will be described with reference to Fig. 11 described.

[0117] For example, the property change extraction unit 13 in the internal state estimation device 10 according to the first embodiment, as shown in Fig. 11A, the time at which the characteristic change in the biometric information P(k) occurs (for example, the time at which the heart rate exceeds the upper limit value) is defined as the first time point kF based on the biometric information P(k) acquired by the biometric information acquisition unit 11. In this case, the first time point kF may be later than the second time point iZ depending on the content of the upper limit threshold, even though a characteristic change in the biometric information P(k) has begun.

[0118] In this case, the timing determination unit 15 determines that the first time point kF is later than the second time point iZ, and the state estimation unit 16 determines that the estimation of the occupant's internal state is disabled, and the estimation processing of the occupant's internal state cannot be executed. Although there is a possibility that such a case can be avoided depending on the contents of the upper limit threshold, it is complicated to individually reset the upper limit threshold for this purpose and may not be realistic from an operational standpoint.

[0119] In this regard, in the internal state estimation device 10c according to the third embodiment, as shown in Fig.11B, the characteristic change extraction unit 13b determines a time point at which a characteristic change in the biometric information P(k) begins as the first time point kF instead of the time point at which the characteristic change in the biometric information P(k) occurs. As a result, the internal state estimation device 10c according to the third embodiment can more accurately determine the order of the first time point kF and the second time point iZ and avoid a problem that the internal state of the occupant is not estimated even though the characteristic change in the biometric information P(k) occurs before the change in the environmental information E(i).

[0120] Note that, here, an example of the configuration of the internal state estimation device 10c according to the third embodiment by applying the property change extraction unit 13b to the internal state estimation device 10b according to the first embodiment has been described. However, the internal state estimation device 10c is not limited to this and can also be configured by applying the property change extraction unit 13b to the internal state estimation device 10b according to the second embodiment.

[0121] As described above, the characteristic change extraction unit 13b according to the third embodiment extracts the time at which the characteristic change in the biometric information P(k) begins as the first time point kF instead of the time at which the characteristic change in the biometric information P(k) occurs. As a result, in addition to the effects of the first embodiment, the interior state estimation device 10c according to the third embodiment can more accurately determine the order of the first time point kF and the second time point iZ. As a result, in the interior state estimation device 10c according to the third embodiment, it is possible to avoid a problem in which the interior state of the occupant is not estimated even though the characteristic change in the biometric information P(k) occurs before the change in the environmental information E(i).

[0122] It should be noted that in the present disclosure, it is possible to freely combine the respective embodiments, modify any components of the respective embodiments, or omit any components in the respective embodiments. INDUSTRIAL APPLICABILITY

[0123] The present disclosure can suppress an erroneous estimation of an interior state of an occupant of a vehicle and is suitable for use in an interior state estimation device. REFERENCE SYMBOL LIST 10 Internal condition estimation device, 10b Internal state estimation device, 10c Internal condition estimation device, 11 Biometric information acquisition unit, 12 Environmental information acquisition unit, 13 Property change extraction unit (first extraction unit), 13b Property change extraction unit, 14 Environmental information change extraction unit (second extraction unit), 15 Time-determination unit (determination unit), 16 Internal state estimation unit (estimation unit), 17 Target section setting unit, 21 processing circuit, 22 CPUs, 23 storage, E Environmental information, F Property change extraction result, i Incoming number, iZ second point in time, k entry number, kF first time point, L Estimate target section, L1 estimation target section, L2 estimation target section, L3 estimation target section, P biometric information, R estimated result, RT determination result, Z Environmental information change extraction result QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2020-74805 A

[0003]

Claims

[1] An internal state estimation device comprising: a first extraction unit for extracting a first time point, which is a time point at which a characteristic change occurs in biometric information, based on the biometric information acquired in time series from an occupant of a vehicle; a second extraction unit for extracting a second time point, which is a time point at which a change in environmental information occurs, based on the environmental information, which is information acquired in time series and is information at which a change in the information affects an internal state of the occupant; a determination unit for performing a determination of an order of the first time point extracted by the first extraction unit and the second time point extracted by the second extraction unit; and an estimation unit for estimating an internal state of the occupant based on at least one of the biometric information and the environmental information, wherein the estimation unit determines, depending on a result of the determination performed by the determination unit, whether the internal state can be estimated or not. [2] The internal state estimation device according to claim 1, wherein the estimation unit determines that it is possible to estimate the internal state in a case where the determination unit determines that the first time point is earlier than the second time point, or in a case where the determination unit determines that the first time point and the second time point are the same. [3] The internal state estimation device according to claim 1 or 2, further comprising: a target section setting unit for setting an estimation target section with a predetermined time range on a time series in which the biometric information and the environmental information are acquired, wherein the first extraction unit extracts the first time point based on a part of the biometric information included in the estimation target section set by the target section setting unit from the biometric information acquired in time series from the occupant, the second extraction unit extracts the second time point based on a part of the environmental information included in the estimation target section set by the target section setting unit from the environmental information acquired in time series, the unit of destination an order of a first time point extracted by the first extraction unit based on the part of the biometric information included in the estimation target section and a second time point extracted by the second extraction unit based on the part of the environmental information included in the estimation target section, and the estimation unit estimates the internal state of the occupant based on at least part of the biometric information and the part of the environmental information included in the estimation target section, and determines whether or not the internal state can be estimated depending on the result of the determination performed by the determination unit. [4] The internal state estimation device according to claim 3, wherein the target section setting unit moves the estimation target section in a time direction every time a predetermined condition is satisfied. [5] The internal state estimation device according to claim 4, wherein the predetermined condition is that st pieces of the biometric information (st represents an integer equal to or greater than 1) are acquired. [6] Internal state estimation device according to one of claims 1 to 5, wherein the estimation unit estimates the internal state of the occupant using a database in which at least one of first change information defining a change mode of the biometric information including the characteristic change at the first time and second change information defining a change mode of the environmental information including the change at the second time is linked to information indicating the internal state of the occupant. [7] The interior state estimation device according to any one of claims 1 to 6, wherein the biometric information includes at least one of the following: information indicative of a heartbeat of the occupant, information indicative of a pulse rate of the occupant, and information indicative of a facial image of the occupant. [8] The internal state estimation device according to any one of claims 1 to 7, wherein the environmental information includes at least one of the following information: internal information of the vehicle, external information of the vehicle, and control information of the vehicle. [9] Internal state estimation device according to one of claims 1 to 8, wherein the first extraction unit as the first time point, extracts a time point at which a property change in the biometric information begins, instead of the time point at which the property change in the biometric information occurs. [10] An internal state estimation method performed by an internal state estimator, the method comprising the following steps: Extracting, by a first extraction unit, a first time point, which is a time point at which a characteristic change occurs in biometric information based on the biometric information acquired in time series from an occupant of a vehicle; Extracting, by a second extraction unit, a second time point, which is a time point at which a change in environmental information occurs, based on the environmental information, which is information acquired in time series from the vehicle and a surrounding area of ​​the vehicle; Performing, by a determining unit, a determination of an order of the first time point extracted by the first extraction unit and the second time point extracted by the second extraction unit; and Estimating, by an estimation unit, an internal state of the occupant based on at least one of the biometric information and the environmental information, wherein the estimation unit determines whether the internal state can be estimated or not depending on a result of the determination performed by the determination unit.

Images