Abnormal state estimation device and abnormal state estimation method

Abstract

The present invention is provided with: an information acquisition unit (11) that acquires information concerning an occupant; and a threshold setting unit (14, 14a, 14b) that sets an abnormal state determination threshold, for inferring an abnormal state of the occupant, on the basis of an abnormal state score obtained within a first threshold setting time by inputting, into a machine learning model (17), the information which concerns the occupant and which is acquired by the information acquisition unit (11).

Description

[Technical field]

[0001] The present invention relates to an abnormal state estimating device and an abnormal state estimating method for estimating an abnormal state of a vehicle occupant. [Background technology]

[0002] Conventionally, there is known a technique for estimating whether or not a vehicle occupant is in an abnormal state based on information about the occupant and a machine learning model (hereinafter referred to as a "machine learning model"). At that time, in order to prevent erroneous estimation of an abnormal state of an occupant, it is necessary to take into account individual differences in information about the occupant. For example, Patent Document 1 discloses a technology in which the difference between the average value of a biosignal for each time period of a user using a vehicle and the biosignal of the user being measured is calculated as a biovariation amount, and if the calculated biovariation amount exceeds a preset illness threshold, the user is determined to be in poor health. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-61580 A Summary of the Invention [Problem to be solved by the invention]

[0004] As described above, the conventional technology disclosed in Patent Document 1 determines that a user is in poor health when the user's biovariation amount exceeds the illness threshold. This illness threshold is a fixed range. However, the range of bioinformation that indicates that the user is in good health differs from user to user. Therefore, even if the user's biovariation amount exceeds the illness threshold, the user is not necessarily in poor health. Therefore, even if the conventional technology disclosed in Patent Document 1 is applied to a technology for estimating an abnormal state of an occupant based on information about the vehicle occupant and a machine learning model, there is still a problem that the abnormal state of the occupant may be erroneously estimated.

[0005] The present invention has been made to solve the above-mentioned problems, and aims to provide an abnormal condition estimation device that estimates an abnormal condition of an occupant based on information about the occupant of a vehicle and a machine learning model, and that is capable of preventing erroneous estimation of the abnormal condition of an occupant when estimating whether the occupant is in an abnormal state. [Means for solving the problem]

[0006] The abnormal condition estimation device according to the present invention estimates an abnormal condition of an occupant based on information about an occupant of a vehicle and a machine learning model, and includes an information acquisition unit that acquires information about the occupant, and a threshold setting unit that inputs the information about the occupant acquired by the information acquisition unit within a first threshold setting time, which is a time during which the occupant is assumed to be in a normal state, into a machine learning model that receives the information about the occupant as input and outputs an abnormal condition score used to estimate the occupant's abnormal condition, to acquire an abnormal condition score, and sets an abnormal condition determination threshold used for comparison with the abnormal condition score in estimating the abnormal condition of the occupant, based on the acquired abnormal condition score. the threshold setting unit sets, as the abnormal state determination threshold, a maximum value among the abnormal state scores obtained by inputting the information about the occupant acquired by the information acquisition unit within the first threshold setting time period into the machine learning model and adding a buffer thereto. It is something. Effect of the Invention

[0007] According to this invention, in an abnormal condition estimation device that estimates an abnormal condition of an occupant based on information about the occupant of a vehicle and a machine learning model, it is possible to prevent erroneous estimation of the abnormal condition of the occupant when estimating whether the occupant is in an abnormal condition. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of an abnormal state estimating device according to a first embodiment. [Diagram 2]FIG. 2 is a diagram for explaining an image of a process in which a learning device performs a learning process to generate a machine learning model in the first embodiment. [Diagram 3] 4 is a flowchart for explaining the operation of the abnormal state estimating device according to the first embodiment. [Figure 4] 4 is a flowchart for explaining a specific operation of step ST301 in FIG. 3. [Diagram 5] 4 is a flowchart illustrating a specific operation of step ST303 in FIG. 3. [Figure 6] 6 is a flowchart for explaining a specific operation of step ST305 in FIG. 5. [Figure 7] 4 is a flowchart for explaining a specific operation of step ST306 in FIG. 3. [Figure 8] FIG. 11 is a diagram illustrating an example of the configuration of an abnormal state estimating device according to a second embodiment. [Figure 9] FIG. 11 is a diagram for explaining an image of a process in which a learning device performs a learning process to generate a machine learning model in the second embodiment. [Figure 10] 10 is a flowchart for explaining the operation of the abnormal state estimating device according to the second embodiment. [Figure 11] 11 is a flowchart for explaining a specific operation of step ST1005 in FIG. [Figure 12] 11 is a flowchart for explaining a specific operation of step ST1007 in FIG. [Figure 13] 11 is a flowchart for explaining a specific operation of step ST1008 in FIG. [Figure 14] FIG. 11 is a diagram illustrating an example of the configuration of an abnormal state estimating device according to a third embodiment. [Figure 15] 11 is a flowchart for explaining the operation of the abnormal state estimating device according to the third embodiment. [Figure 16] 16 is a flowchart for explaining a specific operation of step ST1506 in FIG. [Figure 17]16 is a flowchart for explaining a specific operation of step ST1508 in FIG. [Figure 18] 16 is a flowchart for explaining a specific operation of step ST1509 in FIG. [Figure 19] 19A and 19B are diagrams showing an example of the hardware configuration of the abnormal state estimating devices 1 to 1b according to the first to third embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Embodiment 1 FIG. 1 is a diagram showing an example of the configuration of an abnormal state estimating device 1 according to the first embodiment. The abnormal state estimating device 1 according to the first embodiment is mounted on a vehicle. The abnormal condition estimation device 1 estimates an abnormal condition of a vehicle occupant based on information about the vehicle occupant and a machine learning model (hereinafter referred to as the "machine learning model") 17. Specifically, the abnormal condition estimation device 1 estimates whether or not the vehicle occupant is in an abnormal condition based on an abnormal condition score obtained by inputting information about the vehicle occupant into the machine learning model 17. The abnormal condition estimation device 1 estimates that the vehicle occupant is in an abnormal condition when the abnormal condition score is greater than a threshold value (hereinafter referred to as the "threshold value for determining an abnormal condition"). When estimating an abnormal state of a vehicle occupant, the abnormal state estimation device 1 sets the abnormal state determination threshold based on an abnormal state score obtained by inputting information about the occupant acquired within a predetermined time (hereinafter, "first threshold setting time") into the machine learning model 17 prior to the estimation. After setting the abnormal state determination threshold, the abnormal state estimation device 1 performs the above-mentioned estimation process using the set abnormal state determination threshold. The setting of the abnormal state determination threshold value or the abnormal state estimation process in the abnormal state estimating device 1 will be described in detail later.

[0010] In the first embodiment, the vehicle occupant is assumed to be, for example, a driver. Hereinafter, the vehicle occupant will be simply referred to as an "occupant". In the first embodiment, the abnormal state of the occupant estimated by the abnormal state estimation device 1 refers to a state in which the occupant cannot drive appropriately. Specifically, the abnormal state of the occupant refers to, for example, a state in which the occupant is sleepy, a state in which the occupant has drunk alcohol, a state in which the occupant is fatigued, or a state in which the occupant is absent-minded. In the first embodiment, the information about the occupant is, for example, a captured image of the occupant's face, biometric information about the occupant, or vehicle information about the vehicle operation performed by the occupant. The abnormal state estimation device 1 estimates the abnormal state of the occupant based on at least one of the captured image, the biometric information, and the vehicle information, and a machine learning model. The captured image, the biometric information, and the vehicle information will be described in detail later.

[0011] The abnormal state estimating device 1 acquires information about an occupant from an imaging device 2, a biosensor 3, and a vehicle information acquiring device (not shown). The abnormal state estimating device 1 is connected to the imaging device 2, the biosensor 3, and the vehicle information acquiring device via a network. The imaging device 2 is installed in a vehicle and captures an image of the face of an occupant. The imaging device 2 may be, for example, one visible light camera or multiple visible light cameras. The imaging device 2 may be, for example, one infrared camera or multiple infrared cameras. The imaging device 2 may be shared with an imaging device included in a so-called "driver monitoring system" that is mounted in a vehicle to monitor the state of the driver in the vehicle. The imaging device 2 may be installed so as to be able to capture an image of at least the range inside the vehicle including the range where the face of the occupant should be present. When the imaging device 2 is an infrared camera, a light source (not shown) for irradiating an area including the face of the occupant with infrared light for imaging is provided in the imaging device 2. The light source is, for example, an LED (Light Emitting Diode).

[0012] The biosensor 3 is installed in a vehicle, for example. Specifically, the biosensor 3 is, for example, a microwave Doppler sensor, an electrocardiograph embedded in a seat or a steering wheel, or a temperature sensor. The biosensor 3 may also be, for example, a passenger-worn biosensor. An example of a passenger-worn biosensor is a wristwatch-type pulse wave meter. The biosensor 3 acquires bioinformation of the occupant in real time. Specifically, the bioinformation acquired by the biosensor 3 is information about the pulse, heart rate, breathing, iris, or body surface temperature, for example.

[0013] The vehicle information acquisition device is, for example, various sensors that acquire information about the vehicle. Specifically, the vehicle information acquisition device is, for example, a vehicle speed sensor, a steering angle sensor, an accelerator opening sensor, or a brake opening sensor that are mounted on the vehicle. The vehicle information acquisition device acquires, as vehicle information, information relating to, for example, vehicle speed, steering angle, accelerator opening, or brake opening.

[0014] As shown in FIG. 1, the abnormal state estimation device 1 includes an information acquisition unit 11, an authentication unit 12, a feature calculation unit 13, a threshold setting unit 14, an estimation unit 15, a personal authentication database (hereinafter referred to as the “personal authentication DB”) 16, and a machine learning model 17. The information acquisition unit 11 includes a face image acquisition unit 111, a biometric information acquisition unit 112, and a vehicle information acquisition unit 113. The face image acquisition unit 111 includes a face detection unit 1111. The feature amount calculation unit 13 includes a face feature amount calculation unit 131, a biometric feature amount calculation unit 132, and a vehicle operation feature amount calculation unit 133.

[0015] The information acquisition unit 11 acquires information about the occupants of the vehicle. That is, the information acquisition unit 11 acquires information about the driver. The face image acquisition unit 111 of the information acquisition unit 11 acquires a captured image (hereinafter referred to as a "face image") of the face of the occupant from the imaging device 2. At this time, the face detection unit 1111 of the face image acquisition unit 111 detects the area of ​​the occupant's face, the parts of the occupant's face, the direction of the occupant's face, the degree of opening of the occupant's eyes, the line of sight, the degree of opening of the occupant's mouth, and the like, based on the face image. The face detection unit 1111 may detect the area of ​​the occupant's face, the parts of the occupant's face, the direction of the occupant's face, the degree of opening of the occupant's eyes, the line of sight, the degree of opening of the occupant's mouth, and the like, for example, using a well-known image recognition technique. The parts of the face refer to, for example, the corners of the eyes, the inner corners of the eyes, the corners of the mouth, or the nose.

[0016] The biometric information acquiring unit 112 of the information acquiring unit 11 acquires biometric information of the occupant from the biometric sensor 3 . In this embodiment, the biometric information acquiring unit 112 acquires the biometric information of the occupant from the biometric sensor 3, but this is merely an example. For example, the biometric information acquiring unit 112 may acquire a face image from the imaging device 2, and acquire biometric information from the acquired face image using an existing biometric authentication technology.

[0017] The vehicle information acquisition unit 113 of the information acquisition unit 11 acquires vehicle information from various sensors. The vehicle information acquisition unit 113 acquires vehicle information from various sensors via, for example, a CAN (Controller Area Network).

[0018] The information acquisition unit 11 outputs the acquired information about the occupant to the authentication unit 12 and the feature amount calculation unit 13. Specifically, the information acquisition unit 11 outputs the face image acquired by the face image acquisition unit 111, the biometric information acquired by the biometric information acquisition unit 112, or the vehicle information acquired by the vehicle information acquisition unit 113 to the authentication unit 12 and the feature amount calculation unit 13. The information acquisition unit 11 outputs information about the occupant's face area, information about the occupant's face parts, information about the occupant's face direction, information about the occupant's eye opening degree, information about the occupant's line of sight, information about the occupant's mouth opening degree, or the like, detected by the face detection unit 1111, in association with the face image. The information about the occupant's face area is, for example, coordinates on the face image indicating the occupant's face area. The information about the occupant's face parts is, for example, coordinates on the face image indicating the positions of the occupant's face parts.

[0019] The authentication unit 12 authenticates each individual occupant and acquires information for identifying the occupant (hereinafter referred to as "personal identification information"). In the first embodiment, the personal identification information is, for example, an ID. Specifically, the authentication unit 12 acquires authentication information and authenticates the individual occupant by referring to the personal authentication DB 16. The authentication information is, for example, information about the occupant output from the information acquisition unit 11. In this case, when the information about the occupant is output from the information acquisition unit 11, the authentication unit 12 authenticates the individual occupant by referring to the personal authentication DB 16. Examples of the personal authentication method include image authentication and biometric authentication. The personal authentication DB 16 stores pre-registered information capable of identifying an individual (hereinafter referred to as "personal identification information"). Specifically, the personal identification information is, for example, information for image authentication such as a face image, or information for biometric authentication such as a fingerprint, palm shape, vein pattern, or iris. In addition, the personal identification information may be information capable of identifying an occupant by matching it with information related to the occupant. The personal identification information is stored in association with information identifying the occupant. Here, the information identifying the occupant is an ID.

[0020] For example, the authentication unit 12 authenticates an individual occupant by image authentication. In this case, for example, the authentication unit 12 performs face authentication to determine whether the face image and information on the facial features of the occupant output from the information acquisition unit 11 are registered in the personal identification information. If the face image and information on the facial features of the occupant output from the information acquisition unit 11 match the face image and information on the facial features registered in the personal identification information, the authentication unit 12 determines that the occupant has been identified, and acquires the corresponding personal identification information, in other words, the ID. On the other hand, if the face image and information on the facial features of the occupant output from the information acquisition unit 11 do not match the face image and information on the facial features registered in the personal identification information, the authentication unit 12 determines that the individual occupant cannot be identified.

[0021] In the above example, the authentication unit 12 authenticates individual occupants through image authentication, but the authentication unit 12 may also perform biometric authentication to check whether the biometric information output from the information acquisition unit 11 matches the biometric information registered in the personal identification information. Furthermore, for example, the authentication unit 12 may authenticate an individual occupant by ID card authentication, fingerprint authentication, or the like. When the authentication unit 12 performs ID card authentication, the authentication unit 12 acquires, for example, information on a read ID card from a card reader (not shown) as authentication information and compares the information on the ID card with information for identifying an individual. When the authentication unit 12 performs fingerprint authentication, the authentication unit 12 acquires, for example, fingerprint information from a fingerprint sensor (not shown) as authentication information and compares the fingerprint information with information for identifying an individual. When the authentication unit 12 is able to identify the individual occupant, it outputs the acquired individual identification information to the threshold setting unit 14 and the estimation unit 15.

[0022] The feature amount calculation unit 13 calculates a feature amount related to the occupant based on the information related to the occupant acquired by the information acquisition unit 11. In the first embodiment, the feature amount related to the occupant calculated by the feature amount calculation unit 13 is an effective index when estimating an abnormal state of the occupant. The facial feature amount calculation unit 131 of the feature amount calculation unit 13 calculates feature amounts related to the occupant based on the facial image of the occupant and information on the facial features of the occupant acquired by the facial image acquisition unit 111 of the information acquisition unit 11. Hereinafter, the feature amount calculated by the facial feature amount calculation unit 131 will be referred to as the "facial feature amount."

[0023] For example, the facial feature amount calculation unit 131 calculates, as a facial feature amount, a ratio of information on the degree of eye opening that indicates that the occupant's eyes are open, in other words, an eye opening rate, among information on the degree of eye opening within a predetermined time (hereinafter referred to as a "facial feature amount determination time"), based on time-series information on the degree of eye opening acquired by the facial image acquisition unit 111. The facial feature amount calculation unit 131 may calculate, as a facial feature amount, an eye opening / closing speed or a blinking interval within the facial feature amount determination time, based on the time-series data on the degree of eye opening. Also, for example, the facial feature amount calculation section 131 may calculate the degree of gaze wandering as a facial feature amount based on the time-series gaze information acquired by the facial image acquisition section 111.

[0024] Each time the information acquisition unit 11 acquires information about the occupant, it links the information with the acquisition date and time and stores the information in chronological order in a storage unit (not shown). The storage unit may be provided in the abnormal condition estimation device 1, or may be provided in a location outside the abnormal condition estimation device 1 that can be referenced by the abnormal condition estimation device 1. The facial feature amount calculation unit 131 may acquire time-series information on the degree of eye opening or time-series gaze information from the storage unit. The facial feature amount calculation unit 131 may calculate a plurality of types of facial feature amounts, such as an eye opening rate and a degree of gaze wandering. The facial feature amount calculation unit 131 may calculate a facial feature amount that is effective for estimating an abnormal state of an occupant.

[0025] The biometric feature amount calculation unit 132 of the feature amount calculation unit 13 calculates feature amounts related to an occupant based on the biometric information acquired by the biometric information acquisition unit 112 of the information acquisition unit 11. Hereinafter, the feature amount calculated by the biometric feature amount calculation unit 132 will be referred to as a "biometric feature amount." For example, the biometric feature calculation unit 132 calculates the average breathing interval for a predetermined period of time (hereinafter referred to as the “biometric feature determination period”) based on the occupant's breathing rate acquired by the biometric information acquisition unit 112 as a biometric feature.

[0026] Also, for example, the biometric characteristic calculation section 132 may calculate, based on the heart rate, an average value or a standard deviation of the heart rate during the biometric characteristic determination time as the biometric characteristic. The biometric feature amount calculation unit 132 may obtain the past respiratory rate of the occupant or the past heart rate of the occupant from the storage unit. The biometric feature amount calculation unit 132 may calculate a plurality of types of biometric feature amounts, such as an average breathing interval and a heart rate during the biometric feature amount determination time. The biometric feature amount calculation unit 132 may calculate a biometric feature amount that is effective for estimating an abnormal state of an occupant.

[0027] The vehicle operation feature amount calculation unit 133 of the feature amount calculation unit 13 calculates a feature amount related to the occupant based on the vehicle information acquired by the vehicle information acquisition unit 113. Hereinafter, the feature amount calculated by the vehicle operation feature amount calculation unit 133 will be referred to as a "vehicle operation feature amount." For example, based on information regarding the brake opening acquired by the vehicle information acquisition unit 113, the vehicle operation feature calculation unit 133 calculates the change in the brake opening over a predetermined time (hereinafter referred to as the ``vehicle operation feature determination time'') as the vehicle operation feature. Also, for example, the vehicle operation feature amount calculation unit 133 may calculate the change in the steering wheel angle during the vehicle operation feature amount determination time as the vehicle operation feature amount based on the information regarding the steering wheel angle acquired by the vehicle information acquisition unit 113. The vehicle operation feature amount calculation unit 133 may obtain information on the past brake opening or the past steering wheel angle from the storage unit. The vehicle operation feature amount calculation unit 133 may calculate a plurality of types of vehicle operation feature amounts, such as a change amount of a brake opening amount and a change amount of a steering wheel angle, etc. It is sufficient that the vehicle operation feature amount calculation unit 133 calculates a vehicle operation feature amount that is effective for estimating an abnormal state of an occupant. The feature amount calculation unit 13 outputs the calculated feature amount related to the occupant to the threshold setting unit 14 and the estimation unit 15.

[0028] The threshold setting unit 14 sets an abnormal condition determination threshold for estimating an abnormal condition of an occupant within a predetermined time (hereinafter referred to as the "first threshold setting time") based on an abnormal condition score obtained by inputting the features calculated by the feature calculation unit 13 into the machine learning model 17. The first threshold setting time is a time during which the occupant is assumed to be in a normal state, which is set in order to perform a process of setting a threshold for determining an abnormal state in the abnormal state estimating device 1. The process of setting a threshold for determining an abnormal state is a process performed in the abnormal state estimating device 1 before estimating an abnormal state of the occupant. The first threshold setting time is set in advance. In the first embodiment, the first threshold setting time is set to a time starting from the start of driving the vehicle and ending at a point when a predetermined certain time has passed. As an example, the first threshold setting time is 10 minutes from the start of driving the vehicle. In the first embodiment, the start of driving the vehicle refers to the time when a certain occupant who is the subject of estimating an abnormal state starts driving the vehicle, and does not simply refer to the time when the engine of the vehicle is started. For example, after a certain time has passed since the certain occupant drove the vehicle, the engine is still running, and the certain occupant switches driving to another occupant, so that the occupant who is the subject of estimating an abnormal state has changed. In this case, the time when the other occupant who has switched driving starts driving the vehicle after the change of driving is considered to be the time when the vehicle started driving. The first threshold setting time is set starting from the start of driving the vehicle when the other occupant starts driving the vehicle. The threshold setting unit 14 may detect that the vehicle has started to be driven by using various well-known methods. For example, the threshold setting unit 14 may detect that the vehicle has started to be driven by starting the engine of the vehicle, or may detect that the vehicle has started to be driven based on a face image acquired from the imaging device 2.

[0029] Here, the machine learning model 17 will be described. The machine learning model 17 is a machine learning model that, when information about an occupant is input, outputs an abnormal condition score used to estimate an abnormal condition of the occupant. The machine learning model 17 is generated in advance by, for example, a learning device (not shown) that executes learning in machine learning. The learning device performs a learning process to execute learning using teacher data and a correct label of an abnormal state. Note that the correct label of an abnormal state is a label indicating an abnormal state or a label indicating a normal state.

[0030] The learning device acquires information about multiple occupants that has been collected in advance. Specifically, for example, an administrator or the like collects information about multiple test subjects when the test subjects get into a vehicle and perform a test drive. The learning device acquires the information about the multiple test subjects collected by the administrator or the like as information about multiple occupants. The information about occupants acquired by the learning device is the same type of information as the information about occupants acquired by the information acquisition unit 11 described above. In other words, the information about occupants acquired by the learning device is one or more of the occupant's facial image, biometric information, or vehicle information. The learning device acquires information about a plurality of occupants, and calculates feature amounts for each of the plurality of occupants based on the acquired information about the plurality of occupants. The feature amounts for the occupants calculated by the learning device are the same type of feature amounts as the feature amounts for the occupants calculated by the feature amount calculation unit 13 described above. That is, the feature amounts calculated by the learning device are one or more of face feature amounts, biometric feature amounts, or vehicle operation feature amounts. The learning device uses the calculated feature amounts as teacher data.

[0031] FIG. 2 is a diagram for explaining an image of a process in which the learning device performs a learning process to generate a machine learning model 17 in the first embodiment. As shown in Fig. 2, the learning device performs a learning process on information about a plurality of occupants collected in advance, together with a correct label of an abnormal state, to generate a machine learning model 17. In Fig. 2, the learning device generates the machine learning model 17 based on information about N occupants. The N occupants include occupants in a normal state and occupants in an abnormal state. The number of occupants "N" is preferably as large as possible. The more occupants the machine learning model 17 is generated based on information about, the higher the accuracy of the estimation when the abnormal state estimation device 1 estimates an abnormal state based on the machine learning model 17. In addition, it is preferable that the attributes of the N occupants are various. In the first embodiment, the attributes of the occupants are, for example, gender or age.

[0032] There is no restriction on the technique of the learning method in the learning process performed by the learning device. For example, the learning device may perform the learning process by deep learning or decision tree learning. In addition, although the learning device generates the machine learning model 17 by supervised learning in which learning is performed together with the correct answer label, this is merely an example. The learning device may generate the machine learning model 17 by unsupervised learning.

[0033] In addition, although the machine learning model 17 is described here as being provided in the abnormal condition estimation device 1, this is merely an example. The machine learning model 17 may be provided in a location outside the abnormal condition estimation device 1 that can be referenced by the abnormal condition estimation device 1.

[0034] Returning to the explanation of Figure 1. As described above, the threshold setting unit 14 sets an abnormal condition determination threshold for estimating an abnormal condition of an occupant based on an abnormal condition score obtained by inputting the features calculated by the feature calculation unit 13 into the machine learning model 17 within the first threshold setting time. In the first embodiment, the threshold setting unit 14 sets the maximum value of one or more abnormal state scores obtained within the first threshold setting time as the abnormal state determination threshold. At this time, the threshold setting unit 14 may set the abnormal state determination threshold by providing a buffer for the maximum value of one or more abnormal state scores obtained within the first threshold setting time. Specifically, the threshold setting unit 14 may set the abnormal state determination threshold to the maximum abnormal state score + α, where α is a value greater than 0. Also, for example, the threshold setting unit 14 may set the abnormal state determination threshold to a value obtained by multiplying the maximum value of one or more abnormal state scores obtained within the first threshold setting time by a coefficient. Since the first threshold setting time is a time during which the occupant is assumed to be in a normal state, the abnormal state score obtained within the first threshold setting time is regarded as an abnormal state score obtained when the occupant is in a normal state. In other words, the occupant can be regarded as being in a normal state if the abnormal state score is within the maximum value of the abnormal state scores obtained within the first threshold setting time. The threshold setting unit 14 sets the maximum value of the abnormal state scores obtained when the occupant is in a normal state as the abnormal state determination threshold, thereby making it possible to set an abnormal state determination threshold that can appropriately estimate the abnormal state of the occupant. However, the maximum abnormal state score obtained within the first threshold setting time is not necessarily the maximum abnormal state score that can be obtained when the occupant is in a normal state. For example, if the abnormal state score obtained within the first threshold setting time is slightly exceeded, the occupant may be in a normal state. The threshold setting unit 14 can reduce the possibility of determining that the occupant is in an abnormal state even though the occupant is in a normal state by setting the abnormal state determination threshold with a margin for the abnormal state score, for example, by setting the abnormal state determination threshold to the maximum abnormal state score +α. The threshold setting unit 14 associates the set abnormal state determination threshold with the personal identification information output from the authentication unit 12 and stores it in the storage unit.

[0035] Thus, in the first embodiment, the threshold setting unit 14 assumes that the occupant is not in an abnormal state during the first threshold setting time, in other words, that the occupant is in a normal state. The threshold setting unit 14 inputs features related to the occupant during that time into the machine learning model 17 to obtain one or more abnormal state scores. Then, the threshold setting unit 14 sets a threshold for determining an abnormal state for the occupant based on the maximum value of the one or more obtained abnormal state scores. As described above, during the first threshold setting time when the threshold setting unit 14 is performing the process of setting the abnormal state determination threshold, the abnormal state score is output from the machine learning model 17. However, during the first threshold setting time when the threshold setting unit 14 is performing the process of setting the abnormal state determination threshold, the abnormal state estimation device 1 does not estimate the abnormal state of the occupant based on the abnormal state score. For example, the abnormal state estimation device 1 determines whether to estimate the abnormal state of the occupant based on whether the estimable flag is "1" or "0". When the estimable flag is "1", the abnormal state estimation device 1 determines to estimate the abnormal state of the occupant. Conversely, when the estimable flag is "0", the abnormal state estimation device 1 determines not to estimate the abnormal state of the occupant. This determination is made, for example, by a control unit of the abnormal state estimation device 1. In the abnormal state estimation device 1, the estimation of the abnormal state of the occupant is made by an estimation unit 15 described later.

[0036] Upon completing the setting of the abnormal state determination threshold, the threshold setting unit 14 sets the estimable flag to "1". Specifically, when the control unit determines that the estimable flag is "0", the threshold setting unit 14 executes a process of setting the abnormal state determination threshold. As described above, the threshold setting unit 14 executes a process of setting the abnormal state determination threshold to set the abnormal state determination threshold, stores the abnormal state determination threshold in the personal authentication DB 16 in association with the ID, and sets the estimable flag to "1". When the threshold setting unit 14 completes setting the threshold for determining an abnormal state, it sets the estimable flag to "1". Therefore, in the abnormal state estimation device 1, while the threshold setting unit 14 is unable to set the threshold for determining an abnormal state, the estimable flag is "0". When the estimable flag is "0", the control unit of the abnormal condition estimation device 1 prevents the estimation unit 15 from estimating the abnormal condition of the occupant. This allows the abnormal condition estimation device 1 to not estimate the abnormal condition of the occupant even if the abnormal condition score is output from the machine learning model 17 for the first threshold setting time. After the threshold setting unit 14 sets the estimable flag to "1", when the control unit determines that the estimable flag is "1", the control unit causes the estimation unit 15 to estimate an abnormal state of the occupant. That is, in the abnormal state estimation device 1, after the threshold setting unit 14 sets a threshold for determining an abnormal state, an estimation of an abnormal state of the occupant is performed.

[0037] Furthermore, as described above, the threshold setting unit 14 performs the process of setting the abnormal state determination threshold value based on the control of the control unit when the estimable flag is "0." In this manner, the threshold setting unit 14 only needs to perform the process of setting the abnormal state determination threshold value once when the vehicle starts to operate. The estimable flag is initialized to “0” when, for example, the vehicle stops driving or when the occupant who is the target of estimating an abnormal state is changed. For example, the control unit initializes the estimable flag.

[0038] The estimation unit 15 estimates that the occupant is in an abnormal state if the abnormal state score obtained by inputting the features related to the occupant calculated by the feature calculation unit 13 into the machine learning model 17 is greater than the threshold for determining an abnormal state set by the threshold setting unit 14. Specifically, the estimation unit 15 inputs the feature amounts related to the occupant calculated by the feature amount calculation unit 13 to the machine learning model 17 and acquires the output abnormal state score. The estimation unit 15 compares the acquired abnormal state score with the abnormal state determination threshold stored in the storage unit by the threshold setting unit 14. At that time, the estimation unit 15 compares the personal identification information output from the authentication unit 12 with the personal identification information linked to the abnormal state determination threshold in the storage unit, and acquires the abnormal state determination threshold linked to the matching personal identification information from the storage unit as the abnormal state determination threshold for estimating the abnormal state of the occupant. When the abnormal state score is greater than the abnormal state determination threshold, the estimation unit 15 estimates that the occupant is in an abnormal state. On the other hand, when the abnormal state score is equal to or less than the abnormal state determination threshold, the estimation unit 15 estimates that the occupant is in a normal state.

[0039] The personal authentication DB 16 stores the above-mentioned personal identification information. The personal authentication DB 16 is provided in, for example, a storage unit.

[0040] The operation of the abnormal state estimating device 1 according to the first embodiment will be described. FIG. 3 is a flowchart for explaining the operation of the abnormal state estimating device 1 according to the first embodiment. The information acquisition unit 11 acquires information about the occupant (step ST301).

[0041] FIG. 4 is a flow chart for explaining a specific operation of step ST301 in FIG. The face image acquisition section 111 of the information acquisition section 11 acquires a face image from the imaging device 2 (step ST401). The biological information acquiring unit 112 of the information acquiring unit 11 acquires biological information of the occupant from the biological sensor 3 (step ST402). The vehicle information acquisition unit 113 of the information acquisition unit 11 acquires vehicle information from various sensors (step ST403). The information acquisition unit 11 outputs the acquired information about the occupant to the authentication unit 12 and the feature amount calculation unit 13.

[0042] Returning to the explanation of the flowchart in FIG. The authentication unit 12 authenticates the individual occupant and acquires personal identification information (step ST302). If the authentication unit 12 can identify the individual occupant, the authentication unit 12 outputs the acquired personal identification information to the threshold setting unit 14 and the estimation unit 15.

[0043] The feature amount calculation unit 13 calculates a feature amount related to the occupant based on the information related to the occupant acquired by the information acquisition unit 11 (step ST303).

[0044] FIG. 5 is a flow chart for explaining a specific operation of step ST303 in FIG. The facial feature calculation unit 131 of the feature calculation unit 13 calculates the facial feature of the occupant based on the facial image of the occupant and information regarding the facial features of the occupant acquired by the facial image acquisition unit 111 of the information acquisition unit 11 in step ST401 of Figure 4 (step ST501). The biometric feature calculation unit 132 of the feature calculation unit 13 calculates a biometric feature based on the biometric information acquired by the biometric information acquisition unit 112 of the information acquisition unit 11 in step ST402 of FIG. 4 (step ST502). The vehicle operation feature amount calculation unit 133 of the feature amount calculation unit 13 calculates a vehicle operation feature amount based on the vehicle information acquired by the vehicle information acquisition unit 113 in step ST403 of FIG. 4 (step ST503). The feature amount calculation unit 13 outputs the calculated feature amount related to the occupant to the threshold setting unit 14 and the estimation unit 15.

[0045] Returning to the explanation of the flowchart in FIG. The control unit determines whether or not the abnormal state determination threshold has been set (step ST304). Specifically, the control unit determines whether or not the estimable flag is "1". Note that, before the operation described in FIG. 6 is performed, the abnormal state determination threshold has not been set, and therefore the control unit determines that the abnormal state determination threshold has not been set. After the operation described in FIG. 6 is performed, the abnormal state determination threshold has been set, and therefore the control unit determines that the abnormal state determination threshold has been set. In step ST304, if the control unit determines that the threshold for determining an abnormal state has not been set, in other words, if it determines that the estimation possible flag is “0” (if “NO” in step ST304), the operation of the flowchart in FIG. 3 proceeds to step ST305. Then, within the first threshold setting time, the threshold setting unit 14 sets an abnormal state determination threshold for estimating the abnormal state of the occupant based on the abnormal state score obtained by inputting the features related to the occupant calculated by the feature calculation unit 13 in step ST303 into the machine learning model 17 (step ST305).

[0046] FIG. 6 is a flow chart for explaining a specific operation of step ST305 in FIG. First, the threshold setting unit 14 judges whether or not the time is within the first threshold setting time (step ST601). In step ST601, when the threshold setting unit 14 determines that the first threshold setting time is elapsed (in the case of "YES" in step ST601), the threshold setting unit 14 inputs the feature amount related to the occupant calculated by the feature amount calculation unit 13 in the most recent step ST303 in FIG. 3 to the machine learning model 17, and acquires the output abnormal state score. Then, the threshold setting unit 14 stores the acquired abnormal state score in the storage unit (step ST602). After that, the operation of the flowchart in FIG. 6 returns to step ST301 in FIG. 3, and after performing the operation of step ST301, the operation proceeds to the operation after step ST301 again. After that, the threshold setting unit 14 performs the operation of step ST602 until the first threshold setting time has elapsed (in the case of "NO" in step ST601). As a result, one or more abnormal state scores obtained by the threshold setting unit 14 within the first threshold setting time are stored in the storage unit.

[0047] When the first threshold setting time has elapsed (if "NO" in step ST601), the threshold setting unit 14 sets the abnormal state determination threshold based on the abnormal state score stored in the storage unit. Specifically, the threshold setting unit 14 sets the abnormal state determination threshold to the maximum abnormal state score stored in the storage unit, in other words, the maximum value of one or more abnormal state scores obtained within the first threshold setting time (step ST603). The threshold setting unit 14 associates the set abnormal state determination threshold with the personal identification information output from the authentication unit 12 and stores it in the storage unit.

[0048] Returning to the explanation of the flowchart in FIG. In step ST304, if the control unit determines that the threshold for determining an abnormal state has been set, in other words, if it determines that the estimation possible flag is "1" ("YES" in step ST304), the operation of the flowchart in FIG. 3 proceeds to step ST306. Then, the estimation unit 15 estimates the abnormal state of the occupant based on the abnormal state score obtained by inputting the feature amounts related to the occupant calculated by the feature amount calculation unit 13 in step ST303 into the machine learning model 17 (step ST306).

[0049] FIG. 7 is a flow chart for explaining a specific operation of step ST306 in FIG. The estimation unit 15 inputs the feature amount related to the occupant calculated by the feature amount calculation unit 13 in the most recent step ST303 in Fig. 3 to the machine learning model 17, and acquires the output abnormal state score. Then, the estimation unit 15 determines whether or not the acquired abnormal state score is greater than the abnormal state determination threshold value set by the threshold setting unit 14 in step ST305 in Fig. 3 (step ST701).

[0050] In step ST701, when it is determined that the abnormal state score is greater than the abnormal state determination threshold value ("YES" in step ST701), the estimation unit 15 determines that the occupant is in an abnormal state (step ST702). On the other hand, if it is determined in step ST701 that the abnormal state score is equal to or lower than the threshold for determining an abnormal state ("NO" in step ST701), the estimation unit 15 estimates that the occupant is not in an abnormal state, in other words, that the occupant is in a normal state (step ST703).

[0051] In this way, the abnormal state estimation device 1 of embodiment 1 assumes that the occupant is in a normal state during the first threshold setting time, and sets the first threshold setting time as the threshold for determining an abnormal state based on the maximum abnormal state score obtained by inputting features related to the occupant into the machine learning model 17. Then, after setting the abnormal state determination threshold, the abnormal state estimation device 1 inputs the feature amount related to the occupant into the machine learning model 17 to acquire an abnormal state score. If the acquired abnormal state score is greater than the abnormal state determination threshold, the abnormal state estimation device 1 estimates that the occupant is in an abnormal state. The abnormal state estimation device 1 does not estimate whether or not the occupant is in an abnormal state using a fixed threshold, but sets an abnormal state determination threshold for each occupant and estimates whether or not the occupant is in an abnormal state using the abnormal state determination threshold. Therefore, when estimating whether or not the occupant is in an abnormal state, the abnormal state estimation device 1 can prevent erroneous estimation of the abnormal state of the occupant.

[0052] Regarding the operation described in the flowchart of Fig. 3, even if the authentication unit 12 is unable to identify the individual occupant in step ST302 of Fig. 3, the operation of the abnormal state estimation device 1 proceeds to the operation from step ST304 onwards in Fig. 3, similarly to the case where the authentication unit 12 is able to identify the individual occupant. However, if the authentication unit 12 is unable to identify the individual occupant, the threshold setting unit 14 is unable to link the set abnormal state determination threshold to personal identification information when storing it in the storage unit. When retrieving the abnormal state determination threshold from the storage unit, the estimation unit 15 is configured to retrieve, for example, the latest abnormal state determination threshold stored in the storage unit. 3 from the start of driving the vehicle until the end of driving the vehicle, for example, at the timing when the facial image acquisition unit 111 acquires one frame of a facial image from the imaging device 2. End of driving the vehicle includes, for example, a stop of driving the vehicle as well as a change in the occupant who is the target of estimating an abnormal state. In this case, the estimation unit 15 frequently estimates the abnormal state of the occupant. However, as described above, the abnormal state of the occupant is, for example, a state in which the occupant is sleepy, has drunk alcohol, is fatigued, or is absent-minded, and the abnormal state does not change frequently. Therefore, for example, if the percentage of occupants estimated to be in an abnormal state during a predetermined time (hereinafter referred to as the "abnormal state estimation time") is equal to or greater than a threshold, the estimation unit 15 may estimate that the occupant is in an abnormal state, and thereafter not estimate the abnormal state of the occupant until driving of the vehicle is terminated. The abnormal state estimation time may be 5 minutes, 10 minutes, or the like.

[0053] In the above-described first embodiment, the information input to the machine learning model 17 in the abnormal condition estimation device 1 is the feature amount related to the occupant calculated by the feature amount calculation unit 13, but this is merely an example. For example, in the abnormal condition estimation device 1, the information input to the machine learning model 17 may be information related to the occupant acquired by the information acquisition unit 11. In this case, the abnormal condition estimation device 1 may be configured not to include the feature amount calculation unit 13. However, the abnormal condition estimation device 1 is equipped with a feature calculation unit 13, and by inputting features related to the occupant, which have a high correlation with the estimation of the abnormal condition, into the machine learning model 17, the accuracy of estimation when estimating the abnormal condition of the occupant can be improved compared to the case where information about the occupant is directly input into the machine learning model 17. In the abnormal state estimation device 1, when the information to be input to the machine learning model 17 is information about the occupant, the learning device uses the information about the occupant as teacher data.

[0054] As described above, according to embodiment 1, the abnormal state estimation device 1 that estimates the abnormal state of an occupant based on information about the vehicle occupant and the machine learning model 17 is configured to include an information acquisition unit 11 that acquires information about the occupant, and a threshold setting unit 14 that sets an abnormal state determination threshold for estimating the abnormal state of the occupant based on the abnormal state score obtained by inputting the information about the occupant acquired by the information acquisition unit 11 within the first threshold setting time into the machine learning model 17. Therefore, when estimating whether or not an occupant is in an abnormal state, the abnormal state estimating device 1 can prevent erroneous estimation of the abnormal state of the occupant.

[0055] Embodiment 2 In the first embodiment, it is assumed that one machine learning model 17 is generated in advance based on information on a plurality of occupants. However, the facial feature amount or biometric feature amount of an occupant when the occupant is in a normal state or an abnormal state differs for each occupant. For example, some occupants cannot be said to be in an abnormal state even if they are drowsy, and some occupants cannot be said to be in an abnormal state even if their pulse rate has decreased significantly. In addition, some occupants cannot be said to be in an abnormal state even if the change in the steering angle of the steering wheel is large. In this way, for example, even if the occupants are in the same abnormal state, the facial feature amount or biometric feature amount of multiple occupants have various variations. In this case, when the learning device inputs the feature amounts related to multiple occupants based on information about the multiple occupants into the machine learning model 17 to perform learning, there is a possibility that the input feature amount is rounded off. The machine learning model 17 generated by performing the rounding of the feature amount may be a machine learning model that cannot properly estimate the abnormal state when estimating the abnormal state of an occupant based on the machine learning model 17. There is a possibility that the abnormal state estimation device 1 cannot properly set the abnormal state determination threshold for properly estimating the abnormal state from a machine learning model that cannot properly estimate the abnormal state. Therefore, in the second embodiment, it is assumed that a plurality of machine learning models 17 are generated in advance based on information on a plurality of occupants. Specifically, it is assumed that N machine learning models (1 to N) 17 are generated in advance based on information on N occupants. The abnormal condition estimation device 1a sets an abnormal condition determination threshold based on the N machine learning models (1 to N) 17, and estimates an abnormal condition of an occupant using the set abnormal condition determination threshold.

[0056] Like the abnormal condition estimating device 1 according to the first embodiment, the abnormal condition estimating device 1a according to the second embodiment is assumed to be mounted on a vehicle. FIG. 8 is a diagram showing an example of the configuration of an abnormal condition estimating device 1a according to the second embodiment. In a configuration example of an abnormal state estimating device 1a according to embodiment 2, the same components as those of the abnormal state estimating device 1 described in embodiment 1 using FIG. 1 are given the same reference numerals and redundant description will be omitted. An abnormal state estimating device 1a according to the second embodiment differs from the abnormal state estimating device 1 according to the first embodiment in that a threshold setting unit 14a includes a model-specific threshold setting unit 141 and a model-specific determining unit 142. Also, the abnormal state estimating device 1a according to the second embodiment differs from the abnormal state estimating device 1 according to the first embodiment in that an estimating unit 15a includes a model number calculating unit 151. Furthermore, the abnormal condition estimating device 1a according to the second embodiment differs from the abnormal condition estimating device 1 according to the first embodiment in that the abnormal condition estimating device 1a includes a plurality of machine learning models (1 to N) 17. Note that, although the plurality of machine learning models (1 to N) 17 are assumed to be included in the abnormal condition estimating device 1a here, this is merely an example. The plurality of machine learning models (1 to N) 17 may be included in a location outside the abnormal condition estimating device 1a that can be referenced by the abnormal condition estimating device 1a.

[0057] Here, the machine learning model (1 to N) 17 will be described. The machine learning models (1 to N) 17 are generated in advance by a learning device, similarly to the first embodiment. FIG. 9 is a diagram for explaining an image of a process in which the learning device performs a learning process to generate machine learning models (1 to N) 17 in the second embodiment. In the first embodiment, the learning device performs a learning process on information about a plurality of occupants collected in advance together with the correct labels of abnormal states, and generates one machine learning model 17 (see FIG. 2). In contrast, in the second embodiment, the learning device performs a learning process on information about a plurality of occupants collected in advance, together with a correct label of an abnormal state for each piece of information about the plurality of occupants, to generate a machine learning model (1 to N) 17 corresponding to each occupant. The specific contents of the learning process performed by the learning device are similar to the specific contents of the learning process performed by the learning device in the first embodiment, and therefore a duplicated description will be omitted.

[0058] The threshold setting unit 14a sets an abnormal state determination threshold for estimating an abnormal state of an occupant, similar to the threshold setting unit 14 of the abnormal state estimating device 1 according to embodiment 1. However, the contents of the abnormal state determination threshold set by the threshold setting unit 14a and a specific setting method of the abnormal state determination threshold by the threshold setting unit 14a differ from the contents of the abnormal state determination threshold set by the threshold setting unit 14 of the abnormal state estimating device 1 according to embodiment 1 and a specific setting method of the abnormal state determination threshold by the threshold setting unit 14, which will be described below.

[0059] In the second embodiment, the threshold setting unit 14a performs a "model-specific threshold setting process" and then performs an "abnormal state determination threshold setting process" to set the abnormal state determination threshold. The threshold setting unit 14a performs the "model-specific threshold setting process" and the "abnormal state determination threshold setting process" under the control of the control unit. Specifically, the control unit first causes the threshold setting unit 14a to perform a "model-specific threshold setting process." The threshold setting unit 14a performs the "model-specific threshold setting process," and sets a threshold (hereinafter referred to as a "model-specific threshold") used when setting an abnormal state determination threshold for each machine learning model (1 to N) 17. The "model-specific threshold setting process" will be described in detail later. When the control unit determines that the "model-specific threshold setting process" has been performed, it causes the threshold setting unit 14a to perform an "abnormal state determination threshold setting process." The threshold setting unit 14a performs the "abnormal state determination threshold setting process" and sets the abnormal state determination threshold. The "abnormal state determination threshold setting process" will be described in detail later.

[0060] The "model-specific threshold setting process" and the "abnormal state determination threshold setting process" performed by the threshold setting unit 14a will be described in detail below. First, the "model-specific threshold setting process" performed by the threshold setting unit 14a will be described. In the threshold setting unit 14a, the model-specific threshold setting unit 141 of the threshold setting unit 14a sets a model-specific threshold for each of the multiple machine learning models (1 to N) 17, based on the abnormal state score obtained by inputting the feature calculated by the feature calculation unit 13 within the first threshold setting time. The first threshold setting time has been explained in the first embodiment, so a duplicate explanation will be omitted. However, the first threshold setting time in the second embodiment may be the same length of time as the first threshold setting time in the first embodiment, or may be a different length of time.

[0061] Specifically, the model-specific threshold setting unit 141 first inputs the feature amount calculated by the feature amount calculation unit 13 to a certain machine learning model (1 to N) 17 among the multiple machine learning models (1 to N) 17 to obtain an abnormal state score. The model-specific threshold setting unit 141 performs the above-mentioned acquisition of the abnormal state score for all machine learning models (1 to N) 17, and obtains an abnormal state score corresponding to each machine learning model (1 to N) 17. The model-specific threshold setting unit 141 generates information (hereinafter referred to as "first score information") linking the acquired abnormal state score corresponding to each machine learning model (1 to N) 17, information capable of identifying the machine learning model (1 to N) 17, and the acquisition date and time of the abnormal state score, and stores the information in the storage unit. The model-specific threshold setting unit 141 performs the acquisition and storage of the abnormal state score for a first threshold setting time.

[0062] Then, when the first threshold setting time has elapsed, the model-specific threshold setting unit 141 sets the maximum value of one or more abnormal state scores stored in the storage unit during the first threshold setting time for a certain machine learning model (1 to N) 17 among the multiple machine learning models (1 to N) 17 as the model-specific threshold. The model-specific threshold setting unit 141 may identify one or more abnormal state scores of the certain machine learning model (1 to N) 17 stored in the storage unit during the first threshold setting time from the first score information. At this time, the model-specific threshold setting unit 141 may set the model-specific threshold by providing a buffer for the maximum value of one or more abnormal state scores obtained within the first threshold setting time. In addition, the model-specific threshold setting unit 141 may set the model-specific threshold to a value obtained by multiplying the maximum value of one or more abnormal state scores obtained within the first threshold setting time by a coefficient. The model-specific threshold setting unit 141 sets the model-specific threshold for all machine learning models (1 to N) 17 as described above, and sets a model-specific threshold corresponding to each machine learning model (1 to N) 17. The model-specific threshold setting unit 141 generates information (hereinafter referred to as "model-specific threshold information") that links the set model-specific thresholds for each machine learning model (1 to N) 17 to information that can identify the machine learning model (1 to N) 17, and stores the model-specific threshold information in the memory unit. When the model-specific threshold setting unit 141 generates model-specific threshold information corresponding to all machine learning models (1 to N) 17 and stores it in the storage unit, the control unit determines that the "model-specific threshold setting process" is completed.

[0063] Next, the "abnormal state determination threshold setting process" performed by the threshold setting unit 14a will be described. In the "threshold setting process for determining an abnormal state", the threshold setting unit 14a sets a threshold for determining an abnormal state to estimate an abnormal state of an occupant for each of the multiple machine learning models (1 to N) 17, based on an abnormal state score obtained by inputting features related to the occupant calculated by the feature calculation unit 13 within a predetermined time (hereinafter referred to as the "second threshold setting time"). In the second embodiment, the second threshold setting time, like the first threshold setting time, is a time during which the occupant is assumed to be in a normal state, which is set in order to perform a process for setting an abnormal state determination threshold as a process prior to estimating an abnormal state of the occupant. The second threshold setting time is the time after the first threshold setting time has elapsed. The length of the second threshold setting time is appropriately set, such as 5 minutes or 10 minutes. The length of the second threshold setting time may be the same as the length of the first threshold setting time.

[0064] First, in the threshold setting unit 14a, the model-specific determination unit 142 of the threshold setting unit 14a determines, for each of the multiple machine learning models (1 to N) 17, whether or not the abnormal state score obtained by inputting the feature amount calculated by the feature amount calculation unit 13 within the second threshold setting time is greater than the model-specific threshold amount set by the model-specific threshold setting unit 141. Note that the model-specific determination unit 142 may specify the model-specific threshold amount corresponding to each machine learning model (1 to N) 17 from the model-specific threshold amount information stored in the storage unit by the model-specific threshold setting unit 141.

[0065] Specifically, the model-specific determination unit 142 first inputs the feature amount calculated by the feature amount calculation unit 13 to a certain machine learning model (1 to N) 17 among the multiple machine learning models (1 to N) 17 to obtain an abnormal state score. The model-specific determination unit 142 performs the above-mentioned acquisition of the abnormal state score for all machine learning models (1 to N) 17, and obtains an abnormal state score corresponding to each machine learning model (1 to N) 17. The model-specific determination unit 142 generates information (hereinafter referred to as "second score information") linking the acquired abnormal state score corresponding to each machine learning model (1 to N) 17, information capable of identifying the machine learning model (1 to N) 17, and the acquisition date and time of the abnormal state score, and stores the score information for setting the determination threshold in the storage unit. The model-specific determination unit 142 performs the acquisition and storage of the abnormal state score for the second threshold setting time.

[0066] Then, when the second threshold setting time has elapsed, the model-specific determination unit 142 determines, for a certain machine learning model (1 to N) 17 among the multiple machine learning models (1 to N) 17, whether or not the certain machine learning model (1 to N) 17 stored in the storage unit during the second threshold setting time corresponds to a machine learning model (1 to N) 17 for which any of one or more abnormal state scores obtained from the certain machine learning model (1 to N) 17 is greater than the model-specific threshold set by the model-specific threshold setting unit 141. Hereinafter, a machine learning model (1 to N) 17 for which any of one or more abnormal state scores obtained from the machine learning model (1 to N) 17 is greater than the model-specific threshold set by the model-specific threshold setting unit 141 is referred to as a "model to be counted." The model-specific determination unit 142 may specify, from the second score information, one or more abnormal state scores obtained from the certain machine learning model (1 to N) 17 stored in the storage unit during the second threshold setting time.

[0067] The model-specific determination unit 142 determines whether a certain machine learning model (1 to N) 17 corresponds to a "model to be counted". Specifically, when at least one of the one or more abnormal state scores obtained from the certain machine learning model (1 to N) 17 is greater than the model-specific threshold, the model-specific determination unit 142 determines that the certain machine learning model (1 to N) 17 corresponds to a "model to be counted". Note that, here, the model-specific determination unit 142 determines that the machine learning model (1 to N) 17 corresponds to a model to be counted when at least one of the one or more abnormal state scores for the machine learning model (1 to N) 17 is greater than the model-specific threshold, but this is merely an example. For example, when the abnormal state score that is greater than the model-specific threshold among the one or more abnormal state scores for the machine learning model (1 to N) 17 is equal to or greater than a preset threshold, the model-specific determination unit 142 may determine that the certain machine learning model (1 to N) 17 corresponds to a model to be counted. The model-specific determination unit 142 performs the above-mentioned determination on all machine learning models (1 to N) 17 as to whether or not they correspond to the count target models. The model-specific determination unit 142 generates information (hereinafter referred to as "model-specific counting necessity determination result information") that links, for each machine learning model (1 to N) 17, the determination result of whether or not the model corresponds to a counting target model with information that can identify the machine learning model (1 to N) 17, and outputs the model-specific counting necessity determination result information to the threshold setting unit 14a.

[0068] The threshold setting unit 14a sets the number of models to be counted as the abnormal state determination threshold based on the model-by-model count necessity determination result information output from the model-by-model determination unit 142. At this time, the threshold setting unit 14a may set the abnormal state determination threshold by providing a buffer for the number of models to be counted. Specifically, the threshold setting unit 14a may set the abnormal state determination threshold to the number of models to be counted + α, where α is a value greater than 0. The threshold setting unit 14a associates the set abnormal state determination threshold with the personal identification information output from the authentication unit 12 and stores it in the storage unit, and sets the estimation possible flag to "1."

[0069] In the second embodiment, during the first threshold setting time and the second threshold setting time when the threshold setting unit 14a is performing the process of setting the abnormal state determination threshold, the abnormal state score is output from the machine learning model (1 to N) 17. However, during this time, the abnormal state estimation device 1a does not estimate the abnormal state of the occupant based on the abnormal state score. For example, the abnormal state estimation device 1a determines whether to estimate the abnormal state of the occupant based on whether the estimable flag is "1" or "0". When the estimable flag is "1", the abnormal state estimation device 1a determines to estimate the abnormal state of the occupant. Conversely, when the estimable flag is "0", the abnormal state estimation device 1a determines not to estimate the abnormal state of the occupant. The control based on the estimable flag is performed by the control unit. The control based on the estimable flag by the control unit has been explained in the first embodiment, so a duplicate explanation will be omitted. In the abnormal state estimation device 1a, the threshold setting unit 14a performs a "model-specific threshold setting process" and an "abnormal state determination threshold setting process" in accordance with control by the control unit based on the estimability flag. If a model-specific threshold has not been set, the control unit causes the threshold setting unit 14a to perform a "model-specific threshold setting process". If a model-specific threshold has been set and the estimability flag is "0", the control unit causes the threshold setting unit 14a to perform a "abnormal state determination threshold setting process". Note that the "model-specific threshold setting process" is a process that is performed before the "abnormal state determination threshold setting process", and therefore the estimability flag is "0" at the time of execution of the "model-specific threshold setting process". In the abnormal state estimation device 1a, an estimation unit 15a described below performs a process of estimating an abnormal state of an occupant according to control by the control unit based on an estimable flag. When the estimable flag is "1", the control unit causes the estimation unit 15a to perform a process of estimating an abnormal state of an occupant.

[0070] The estimation unit 15a estimates the abnormal state of the occupant, similarly to the estimation unit 15 of the abnormal state estimating device 1 according to embodiment 1. However, the specific method of estimating the abnormal state of the occupant in the estimation unit 15a differs from that of the estimation unit 15 of the abnormal state estimating device 1 according to embodiment 1, and will be described below. In embodiment 2, the estimation unit 15a estimates the abnormal state of the occupant based on the abnormal state score obtained by inputting the features related to the occupant calculated by the feature calculation unit 13 into each machine learning model (1 to N) 17. The estimation unit 15a will be described in detail below.

[0071] In the estimation unit 15a, the model number calculation unit 151 of the estimation unit 15a calculates the number of machine learning models (1 to N) 17 for which the abnormal state score obtained by inputting the features calculated by the feature calculation unit 13 is determined to be greater than the model-specific threshold (hereinafter referred to as the "number of models for abnormality determination"). Specifically, the model number calculation unit 151 first inputs the feature amount calculated by the feature amount calculation unit 13 to a certain machine learning model (1 to N) 17 among the multiple machine learning models (1 to N) 17 to obtain an abnormal state score. Then, the model number calculation unit 151 determines whether or not the obtained abnormal state score is greater than the model-specific threshold set by the model-specific threshold setting unit 141 of the threshold setting unit 14a. Note that the model number calculation unit 151 may specify a model-specific threshold corresponding to a certain machine learning model (1 to N) 17 to be compared with the abnormal state score, from the model-specific threshold information stored in the storage unit. The model number calculation unit 151 performs the above-mentioned determination of whether or not the abnormal state score is greater than the model-specific threshold for all machine learning models (1 to N) 17. Then, the model number calculation unit 151 calculates the number of models for abnormality determination. The model number calculation unit 151 outputs the calculated number of models for abnormality determination to the estimation unit 15a.

[0072] When the number of models for abnormality determination calculated by the model number calculation unit 151 is greater than the abnormal state determination threshold value set by the threshold setting unit 14a, the estimation unit 15a estimates that the occupant is in an abnormal state. Specifically, the estimation unit 15a compares the number of models for abnormality determination calculated by the model number calculation unit 151 with the abnormal state determination threshold stored in the storage unit by the threshold setting unit 14a. At that time, the estimation unit 15a compares the personal identification information output from the authentication unit 12 with the personal identification information linked to the abnormal state determination threshold in the storage unit, and acquires the abnormal state determination threshold linked to the matching personal identification information from the storage unit as the abnormal state determination threshold to be compared with the number of models for abnormality determination. When the number of models for abnormality determination is greater than the abnormal state determination threshold, the estimation unit 15a estimates that the occupant is in an abnormal state. On the other hand, when the number of models for abnormality determination is equal to or less than the abnormal state determination threshold, the estimation unit 15a estimates that the occupant is in a normal state.

[0073] The operation of the abnormal condition estimating device 1a according to the second embodiment will be described. FIG. 10 is a flowchart for explaining the operation of the abnormal condition estimating device 1a according to the second embodiment. Specific operations of steps ST1001 to ST1003 in FIG. 10 are similar to those of steps ST301 to ST303 in FIG. 3 explained in the first embodiment, and therefore overlapping explanations will be omitted.

[0074] In step ST1004, the control unit judges whether or not the model-specific threshold has been set (step ST1004). Specifically, the control unit judges whether or not the model-specific threshold information corresponding to all the machine learning models (1 to N) 17 is stored in the storage unit. In step ST1004, when it is determined that the model-specific threshold has not been set ("NO" in step ST1004), the operation of the flowchart in Fig. 10 proceeds to step ST1005. Note that, before the operation described in Fig. 11 described later is performed, the model-specific threshold has not been set, so the control unit determines that the model-specific threshold has not been set. After the operation described in Fig. 11 described later is performed, the model-specific threshold has been set, so the control unit determines that the model-specific threshold has been set. Then, the threshold setting unit 14a performs a "model-specific threshold setting process" (step ST1005).

[0075] FIG. 11 is a flow chart for explaining a specific operation of step ST1005 in FIG. The model-specific threshold setting unit 141 judges whether or not the time is within the first threshold setting time (step ST1101). In step ST1101, when the model-specific threshold setting unit 141 determines that the first threshold setting time is within the first threshold setting time (in the case of "YES" in step ST1101), the model-specific threshold setting unit 141 inputs the feature amount calculated by the feature amount calculation unit 13 in the most recent step ST1003 of FIG. 10 to a certain machine learning model (1 to N) 17 among the multiple machine learning models (1 to N) 17 to obtain an abnormal state score. The model-specific threshold setting unit 141 performs the above-mentioned acquisition of the abnormal state score for all the machine learning models (1 to N) 17, and obtains an abnormal state score corresponding to each machine learning model (1 to N) 17. The model-specific threshold setting unit 141 generates first score information and stores the first score information in the storage unit (step ST1102). Thereafter, the operation of the flowchart in FIG. 11 returns to step ST1001 in FIG. 10, and after performing the operation of the step T1001, the operation proceeds to the operation of the step ST1001 and thereafter again. After that, until the first threshold setting time has elapsed (in the case of "NO" in step ST1101), the model-specific threshold setting unit 141 performs the operation of step ST1102. As a result, the first score information generated by the model-specific threshold setting unit 141 within the first threshold setting time is stored in the storage unit.

[0076] In step ST1101, when the model-specific threshold setting unit 141 determines that the first threshold setting time has not elapsed, that is, when the first threshold setting time has elapsed (in the case of "NO" in step ST1101), the model-specific threshold setting unit 141 sets the maximum value of one or more abnormal state scores stored in the storage unit during the first threshold setting time for a certain machine learning model (1 to N) 17 among the multiple machine learning models (1 to N) 17 as the model-specific threshold (step ST1103). The model-specific threshold setting unit 141 sets the above-mentioned model-specific threshold for all machine learning models (1 to N) 17, and sets the model-specific threshold corresponding to each machine learning model (1 to N) 17. The model-specific threshold setting unit 141 generates model-specific threshold information, and stores the model-specific threshold information in the storage unit. Thereafter, the operation of the flowchart in FIG. 11 returns to step ST1001 in FIG. 10, and after performing the operation of step T1001, the operation proceeds to the operation of step ST1001 and thereafter again.

[0077] Returning to the explanation of the flowchart in FIG. In step ST1004, if the control section determines that the model-specific threshold has been set ("YES" in step ST1004), the operation of the flowchart in FIG. 10 proceeds to step ST1006. In step ST1006, the control unit determines whether or not the abnormal state determination threshold has been set. Specifically, the control unit determines whether or not the estimation possible flag is "1" (step ST1006). In step ST1006, if the control unit determines that the abnormal state determination threshold has not been set, in other words, if the control unit determines that the estimable flag is "0" ("NO" in step ST1006), the operation of the flowchart in Fig. 10 proceeds to step ST1007. Note that before the operation described in Fig. 12 below is performed, the abnormal state determination threshold has not been set, so the control unit determines that the abnormal state determination threshold has not been set. After the operation described in Fig. 12 below is performed, the abnormal state determination threshold has been set, so the control unit determines that the abnormal state determination threshold has been set. Then, the threshold setting unit 14a performs "an abnormal state determination threshold setting process" (step ST1007).

[0078] FIG. 12 is a flow chart for explaining a specific operation of step ST1007 in FIG. The model-specific determination unit 142 of the threshold setting unit 14a determines whether or not the time is within the second threshold setting time (step ST1201). In step ST1201, when it is determined that the time is within the second threshold setting time (in the case of "YES" in step ST1201), the model-specific determination unit 142 first inputs the feature amount calculated by the feature amount calculation unit 13 in the most recent step ST1003 of FIG. 10 to a certain machine learning model (1 to N) 17 among the multiple machine learning models (1 to N) 17 to obtain an abnormal state score. The model-specific determination unit 142 performs the above-mentioned acquisition of the abnormal state score for all the machine learning models (1 to N) 17, and obtains an abnormal state score corresponding to each machine learning model (1 to N) 17. The model-specific determination unit 142 generates second score information and stores the second score information in the storage unit (step ST1202). Thereafter, the operation of the flowchart in FIG. 12 returns to step ST1001 in FIG. 10, and after performing the operation of the step T1001, it proceeds again to the operation of the step ST1001 and subsequent steps. After that, until the second threshold setting time has elapsed (if "NO" in step ST1201), the model-specific determination unit 142 performs the operation of step ST1202. As a result, the second score information generated by the model-specific determination unit 142 within the second threshold setting time is stored in the storage unit.

[0079] In step ST1201, if the model-specific judgment unit 142 judges that the time is not within the second threshold setting time, i.e., when the second threshold setting time has elapsed (in the case of ``NO'' in step ST1201), the model-specific judgment unit 142 judges, for a certain machine learning model (1 to N) 17 among the multiple machine learning models (1 to N) 17, whether or not any of the one or more abnormal state scores of the certain machine learning model (1 to N) 17 stored in the memory unit during the second threshold setting time is greater than the model-specific threshold set by the model-specific threshold setting unit 141 in step ST1103 of Figure 11. When a certain machine learning model (1 to N) 17 is a machine learning model (1 to N) 17 in which at least one of the abnormal state scores obtained from the certain machine learning model (1 to N) 17 is greater than the model-specific threshold, the model-specific determination unit 142 determines that the certain machine learning model (1 to N) 17 is a machine learning model (1 to N) 17 that corresponds to a model to be counted (step ST1203). The model-specific determination unit 142 performs the above-mentioned determination for all machine learning models (1 to N) 17 as to whether or not they correspond to a model to be counted. The model-specific determination unit 142 generates model-specific count necessity determination result information, and outputs the model-specific count necessity determination result information to the threshold setting unit 14a.

[0080] The threshold setting unit 14a sets the number of models to be counted as the abnormal state determination threshold based on the model-by-model count necessity determination result information output from the model-by-model determination unit 142 in step ST1203 (step ST1204). The threshold setting unit 14a associates the set abnormal state determination threshold with the personal identification information output from the authentication unit 12 and stores it in the storage unit, and sets the estimation possible flag to "1." After that, the operation of the flowchart in Fig. 12 returns to step ST1001 in Fig. 10, and after performing the operation of step T1001, it again proceeds to the operation of step ST1001 and subsequent steps.

[0081] Returning to the explanation of the flowchart in FIG. In step ST1006, if the control unit determines that the threshold for determining an abnormal state has been set, in other words, if it determines that the estimation possible flag is "1" ("YES" in step ST1006), the operation of the flowchart in FIG. 10 proceeds to step ST1008. Then, the estimation unit 15a estimates the abnormal state of the occupant based on the abnormal state score obtained by inputting the features related to the occupant calculated by the feature calculation unit 13 in step ST1003 into each machine learning model (1 to N) 17 (step ST1008).

[0082] FIG. 13 is a flow chart for explaining a specific operation of step ST1008 in FIG. The model number calculation unit 151 of the estimation unit 15a first inputs the feature calculated by the feature calculation unit 13 in the most recent step ST1003 in Fig. 10 to a certain machine learning model (1 to N) 17 among the multiple machine learning models (1 to N) 17 to acquire an abnormal state score. Then, the model number calculation unit 151 determines whether the acquired abnormal state score is greater than the model-specific threshold set by the model-specific threshold setting unit 141 in step ST1103 in Fig. 11. The model number calculation unit 151 performs the above-mentioned determination as to whether or not the abnormal state score is greater than the model-specific threshold for all machine learning models (1 to N) 17. Then, the model number calculation unit 151 calculates the number of models for abnormality determination (step ST1301). The model number calculation unit 151 outputs the calculated number of models for abnormality determination to the estimation unit 15a.

[0083] The estimation unit 15a compares the number of models for abnormality determination calculated by the model number calculation unit 151 in step ST1301 with the abnormal state determination threshold value set by the threshold value setting unit 14a in step ST1204 in FIG. 12 (step ST1302). In step ST1302, if the number of models for abnormality determination is greater than the threshold for abnormal state determination ("YES" in step ST1302), the estimation unit 15a estimates that the occupant is in an abnormal state (step ST1303). On the other hand, in step ST1302, if the number of abnormality determination models is equal to or less than the abnormal state determination threshold value ("NO" in step ST1302), the estimation unit 15a estimates that the occupant is in a normal state (step ST1304).

[0084] In the above description of the operation, in the "abnormal state determination threshold setting process", the threshold setting unit 14a sets the abnormal state determination threshold by performing the operation described in the flowchart of Fig. 12, but this is merely an example. The threshold setting unit 14a may set the abnormal state determination threshold by performing, for example, the operation described below. If it is within the second threshold setting time, the model-specific determination unit 142 of the threshold setting unit 14a inputs the feature calculated by the feature calculation unit 13 in the most recent step ST1003 of FIG. 10 to a certain machine learning model (1 to N) 17 among the multiple machine learning models (1 to N) 17 to acquire an abnormal state score. Then, the model-specific determination unit 142 determines whether the certain machine learning model (1 to N) 17 corresponds to a count target model depending on whether the acquired abnormal state score is greater than the model-specific threshold set by the model-specific threshold setting unit 141 in step ST1103 of FIG. If the abnormal state score obtained from the certain machine learning model (1 to N) 17 is greater than the model-specific threshold set by the model-specific threshold setting unit 141, the model-specific determination unit 142 determines that the certain machine learning model (1 to N) 17 corresponds to a count target model. The model-specific determination unit 142 obtains the abnormal state score and determines whether or not the model corresponds to the count target model for all machine learning models (1 to N) 17. After obtaining the abnormal state score for all machine learning models (1 to N) 17 and determining whether or not the model corresponds to the count target model, the model-specific determination unit 142 calculates the number of machine learning models (1 to N) 17 that correspond to the count target models among all machine learning models (1 to N) 17. The model-specific determination unit 142 stores the calculated number of models to be counted in the storage unit, for example, in association with the time when the number was calculated. The model-specific determination unit 142 performs the above operation during the second threshold setting time and each time the feature amount calculation unit 13 calculates a feature amount. When the second threshold setting time has elapsed, the threshold setting unit 14a sets the maximum value of the number of count target models calculated by the model-specific determination unit 142 within the second threshold setting time as the abnormal state determination threshold. At this time, the threshold setting unit 14a may set the abnormal state determination threshold by providing a buffer for the maximum value of the number of count target models calculated within the second threshold setting time. Specifically, the threshold setting unit 14a may set the abnormal state determination threshold to the maximum value of the number of count target models calculated within the second threshold setting time + α. α is a value greater than 0.

[0085] In this way, the abnormal state estimation device 1a of embodiment 2 assumes that the occupant is in a normal state during the first threshold setting time, and sets a model-specific threshold for each machine learning model (1 to N) 17 based on the maximum abnormal state score obtained by inputting features related to the occupant during the first threshold setting time. Furthermore, after setting the model-specific threshold, the abnormal condition estimation device 1a assumes that the occupant is in a normal state during the second threshold setting time, and compares the abnormal condition score obtained by inputting the feature amount related to the occupant during the second threshold setting time with the model-specific threshold for each machine learning model (1 to N) 17, and calculates the number of machine learning models (1 to N) 17 that correspond to models to be counted among the machine learning models (1 to N) 17. Then, the abnormal condition estimation device 1a sets the calculated number of models to be counted as the abnormal condition determination threshold.

[0086] After setting the abnormal state determination threshold, the abnormal state estimation device 1a inputs the feature amount related to the occupant for each machine learning model (1 to N) 17 and obtains the abnormal state score for each machine learning model (1 to N) 17. The abnormal state estimation device 1a determines whether or not the abnormal state score obtained for each machine learning model (1 to N) 17 is a model for abnormal state determination that is greater than the model-specific threshold. Then, the abnormal state estimation device 1a calculates the number of models for abnormal state determination. When the abnormality determination device 1a calculates the number of abnormality determination models, it compares the number of abnormality determination models with the abnormality determination threshold value, and if the number of abnormality determination models is greater than the abnormality determination threshold value, it estimates that the occupant is in an abnormal state.

[0087] The abnormal condition estimation device 1a does not use a fixed threshold to estimate whether an occupant is in an abnormal state, but sets an abnormal condition determination threshold for each occupant and estimates whether the occupant is in an abnormal state using the abnormal condition determination threshold. Therefore, when estimating whether an occupant is in an abnormal state, the abnormal condition estimation device 1a can prevent erroneous estimation of the abnormal state of the occupant. Furthermore, the abnormal condition estimation device 1a sets the abnormal condition determination threshold and estimates the abnormal condition of the occupant based on a plurality of machine learning models (1 to N) 17 that are individually generated in advance according to the attributes of a plurality of occupants having various attributes. Therefore, the abnormal condition estimation device 1a can improve the estimation accuracy of the abnormal condition of the occupant compared to the case where the abnormal condition determination threshold is set and the abnormal condition of the occupant is estimated using one machine learning model generated from information on a plurality of occupants having various attributes.

[0088] In the abnormal state estimation device 1a of embodiment 2, like the abnormal state estimation device 1 of embodiment 1, the estimation unit 15a may, for example, estimate that the occupant is in an abnormal state if the percentage of the occupant estimated to be in an abnormal state during the abnormal state estimation time is equal to or greater than a threshold value, and thereafter not estimate the occupant's abnormal state until driving of the vehicle is terminated.

[0089] In addition, in the above-described second embodiment, the information input to the machine learning model (1 to N) 17 in the abnormal condition estimation device 1a is the feature amount related to the occupant calculated by the feature amount calculation unit 13, but this is merely an example. For example, in the abnormal condition estimation device 1a, the information input to the machine learning model (1 to N) 17 may be information related to the occupant acquired by the information acquisition unit 11. In this case, the abnormal condition estimation device 1a may be configured not to include the feature amount calculation unit 13. However, the abnormal condition estimation device 1a is equipped with a feature calculation unit 13, and by inputting features related to the occupant, which have a high correlation with the estimation of the abnormal condition, into the machine learning model (1 to N) 17, the estimation accuracy when estimating the abnormal condition of the occupant can be improved compared to the case where information about the occupant is directly input into the machine learning model (1 to N) 17. In the abnormal state estimating device 1a, when the information to be input to the machine learning model (1 to N) 17 is information about the occupant, the learning device uses the information about the occupant as teacher data.

[0090] As described above, according to the second embodiment, the abnormal state estimation device 1a that estimates an abnormal state of an occupant based on information about the occupant of a vehicle and the machine learning model (1 to N) 17 includes an information acquisition unit 11 that acquires information about the occupant, and a first threshold setting time setting unit 12 that sets the abnormal state of the occupant based on the information about the occupant of the vehicle. 11 The system is configured to include a threshold setting unit 14a that sets an abnormal state determination threshold for estimating the abnormal state of the occupant based on the abnormal state score obtained by inputting the information about the occupant acquired by the machine learning model (1 to N) 17. Therefore, when estimating whether or not an occupant is in an abnormal state, the abnormal state estimating device 1a can prevent erroneous estimation of the abnormal state of the occupant.

[0091] In particular, in embodiment 2, there are multiple machine learning models (1 to N) 17, and the abnormal state estimation device 1a is equipped with a model-specific threshold setting unit 141 that sets a model-specific threshold for each of the multiple machine learning models (1 to N) based on the abnormal state score obtained by inputting information about the occupant acquired by the information acquisition unit 11 within a first threshold setting time, and a model-specific judgment unit 142 that judges whether or not the abnormal state score obtained by inputting information about the occupant acquired by the information acquisition unit within a second threshold setting time for each of the multiple machine learning models (1 to N) 17 is greater than the model-specific threshold set by the model-specific threshold setting unit 141 after the model-specific threshold setting unit 141 sets the model-specific threshold, and the threshold setting unit 14a sets the number of machine learning models (1 to N) 17 for which the model-specific judgment unit 142 has judged that the abnormal state score is greater than the model-specific threshold as the abnormal state judgment threshold. As a result, the abnormal condition estimation device 1a can improve the accuracy of estimating the abnormal condition of occupants compared to when setting a threshold for determining an abnormal condition and estimating the abnormal condition of occupants using a single machine learning model generated from information about multiple occupants with various attributes.

[0092] Embodiment 3 In the second embodiment, a plurality of machine learning models (1 to N) 17 are generated in advance based on information on a plurality of occupants. The machine learning models (1 to N) 17 are generated based on information on a plurality of occupants having various attributes. The abnormal state estimation device 1a then uses all of the multiple machine learning models (1 to N) 17 that have been generated to set an abnormal state determination threshold value and estimate an abnormal state of an occupant. Here, the tendency of an abnormal state of an occupant may differ depending on the attributes of the occupant. For example, an occupant belonging to an older age group may be more likely to have differences in biometric features when in an abnormal state, whereas an occupant belonging to a younger age group may be more likely to have differences in facial features when in an abnormal state. As a result, for example, the tendency of an abnormal state may differ between an occupant belonging to an older age group and an occupant belonging to a younger age group. Also, for example, the tendency of an abnormal state may differ between males and females. Therefore, in embodiment 3, an embodiment is described in which the abnormal state estimation device 1b selects one or more machine learning models (hereinafter referred to as "associated machine learning models") from the multiple machine learning models (1 to N) 17 that are related to the attributes of the occupant for whom an abnormal state is to be estimated, and uses the selected one or more associated machine learning models to set a threshold for determining an abnormal state and estimate the abnormal state of the occupant.

[0093] Like the abnormal condition estimating device 1a according to the second embodiment, the abnormal condition estimating device 1b according to the third embodiment is assumed to be mounted on a vehicle. FIG. 14 is a diagram showing an example of the configuration of an abnormal state estimating device 1b according to the third embodiment. In a configuration example of an abnormal state estimating device 1b according to embodiment 3, the same components as those of the abnormal state estimating device 1a described in embodiment 2 using FIG. 8 are given the same reference numerals and redundant description will be omitted. The abnormal condition estimating device 1b according to the third embodiment differs from the abnormal condition estimating device 1a according to the second embodiment in that an authentication unit 12b includes an attribute information acquisition unit 121. Also, the abnormal state estimating device 1b according to the third embodiment differs from the abnormal state estimating device 1a according to the second embodiment in that a model selecting unit 18 is provided. Furthermore, the abnormal state estimation device 1b of embodiment 3 differs from the abnormal state estimation device 1a of embodiment 2 in that the threshold setting unit 14b includes an associated model-specific threshold setting unit 143 and an associated model-specific judgment unit 144 instead of the model-specific threshold setting unit 141 and the model-specific judgment unit 142. In addition, the abnormal state estimating device 1b according to embodiment 3 differs from the abnormal state estimating device 1a according to embodiment 2 in that the estimation unit 15b includes an associated model number calculation unit 152 instead of the model number calculation unit 151.

[0094] In the third embodiment, similarly to the second embodiment, it is assumed that a plurality of machine learning models (1 to N) 17 are generated in advance. The multiple machine learning models (1 to N) 17 are generated in advance by a learning device (see FIG. 9), similarly to the second embodiment. However, in the third embodiment, the learning device acquires attribute information on the attributes of occupants, and assigns the attribute information to the generated machine learning models (1 to N) 17.

[0095] The attribute information acquisition unit 121 of the authentication unit 12b acquires attribute information related to the attributes of the occupant. The attributes of the occupant include, for example, gender and age. The authentication unit 12b authenticates the individual occupant and acquires personal identification information. A specific method for the authentication unit 12b to acquire personal identification information is similar to the specific method for the authentication unit 12 to acquire personal identification information described in the first embodiment, and therefore a duplicated description will be omitted. In the third embodiment, when the authentication unit 12b acquires personal identification information, the attribute information acquisition unit 121 in the authentication unit 12b acquires attribute information relating to the attributes of the occupant.

[0096] Specifically, the attribute information acquisition unit 121 acquires the attribute information of the occupant from the individual identification information stored in the individual authentication DB 16. In the third embodiment, for example, it is assumed that the attribute information is set in the individual identification information. This is merely an example, and for example, the attribute information acquisition unit 121 may be capable of accepting the attribute information of the occupant input from an input device (not shown). Specifically, for example, the occupant inputs his / her own attribute information using a navigation device (not shown) mounted on the vehicle or a mobile terminal carried by the occupant as an input device. The attribute information acquisition unit 121 may be configured to accept the attribute information input by the occupant. When the authentication unit 12b is able to identify the individual occupant, the authentication unit 12b outputs the acquired individual identification information and the occupant's attribute information to the threshold setting unit 14b, the estimation unit 15b, and the model selection unit .

[0097] The model selection unit 18 selects one or more associated machine learning models related to the attributes of the occupant from among the multiple machine learning models (1 to N) 17 based on the attribute information acquired by the attribute information acquisition unit 121 of the authentication unit 12b. Specifically, the model selection unit 18 compares the attribute information acquired by the attribute information acquisition unit 121 with the attribute information assigned to the multiple machine learning models (1 to N) 17, and selects one or more associated machine learning models. The model selection unit 18 selects one or more associated machine learning models according to predetermined model selection rules.

[0098] For example, suppose that the attribute information acquired from the attribute information acquisition unit 121 is information indicating the age and sex of the occupant, and that the occupant is a 60-year-old male. Meanwhile, the attribute information assigned to the machine learning models (1 to N) 17 is age and sex. Furthermore, suppose that the model selection rule is set to "be of the same age and sex as the occupant." In this case, the model selection unit 18 selects the machine learning model (1 to N) 17 to which attribute information indicating a man in his 60s is assigned. Also, for example, it is assumed that the attribute information acquired from the attribute information acquisition unit 121 is a date of birth. On the other hand, it is assumed that the attribute information assigned to the machine learning models (1 to N) 17 is an age group. It is also assumed that the model selection rule is set to "be of the same age and sex as the occupant." In this case, the model selection unit 18 can calculate the age of the occupant from the date of birth, for example, and select the machine learning model (1 to N) 17 related to the attributes of the occupant from the calculated age.

[0099] The model selection unit 18 outputs information on the selected associated machine learning model to the threshold setting unit 14b and the estimation unit 15b. The associated machine learning model selected by the model selection unit 18 is used in setting the abnormal state determination threshold in the threshold setting unit 14b and in estimating the abnormal state in the estimation unit 15b. As described above, the model selection unit 18 selects one or more associated machine learning models from the machine learning models (1 to N) 17. However, if the number of associated machine learning models selected by the model selection unit 18 is too small, the threshold setting unit 14b may not be able to appropriately set the abnormal state determination threshold. As a result, the estimation unit 15b may not be able to appropriately estimate the abnormal state. Therefore, it is preferable that the model selection unit 18 selects a certain number of associated machine learning models that are set in advance, such as 100 or more.

[0100] Therefore, for example, the model selection rule may be set to a plurality of rules with priorities assigned, and the model selection unit 18 selects related machine learning models in accordance with the rules in order of the highest priority, so that the number of related machine learning models is equal to or greater than a preset number. For example, the model selection rules may have the following set of rules with the highest priority: "must be the same age and sex as the occupant." The second set of rules with the highest priority: "must be within ±5 years of the same age as the occupant and have the same sex as the occupant" or "must be the same age as the occupant." It is assumed that the attribute information acquired from the attribute information acquisition unit 121 is information indicating the age and gender of the occupant, and that the occupant is a 60-year-old male. It is also assumed that the attribute information assigned to the machine learning models (1 to N) 17 is the age and gender. In this case, first, the model selection unit 18 selects, as the relevant machine learning model, the machine learning model (1 to N) 17 to which attribute information indicating a man in his 60s is assigned, in accordance with the rule of first priority.

[0101] Here, the model selection unit 18 determines whether the number of relevant machine learning models is equal to or greater than a preset number. If the number of associated machine learning models is less than a preset number, the model selection unit 18 then selects, according to the second priority rule, machine learning model (1 to N) 17 to which attribute information indicating a male aged 55 to 75 is assigned, and machine learning model (1 to N) 17 to which attribute information indicating a female in her 60s is assigned, as associated machine learning models. If the sum of the related machine learning models selected according to the rule with the first priority and the related machine learning models selected according to the rule with the second priority is equal to or greater than a predetermined number, the model selection unit 18 terminates the selection of related machine learning models. If the sum of the related machine learning models selected according to the rule with the first priority and the related machine learning models selected according to the rule with the second priority is less than a predetermined number, the model selection unit 18 continues to select related machine learning models according to the rule with the third priority.

[0102] Returning to the explanation of Figure 14. The threshold setting unit 14b, like the threshold setting unit 14a of the abnormal condition estimating device 1a according to the second embodiment, sets an abnormal condition determination threshold for estimating an abnormal condition of an occupant. The threshold setting unit 14b differs from the threshold setting unit 14a according to the second embodiment in that the machine learning model used when setting the threshold for determining an abnormal state is the associated machine learning model selected by the model selection unit 18. The threshold setting unit 14b performs "threshold setting process for related model" and then "threshold setting process for abnormal state determination" to set the threshold for abnormal state determination. The threshold setting unit 14b performs "threshold setting process for related model" and "threshold setting process for abnormal state determination" under the control of the control unit. Specifically, the control unit first causes the threshold setting unit 14b to perform a "threshold setting process for each associated model." The threshold setting unit 14b performs the "threshold setting process for each associated model," and sets a threshold (hereinafter referred to as a "threshold for each associated model") to be used when setting a threshold for determining an abnormal state for each associated machine learning model. The "threshold setting process for each associated model" will be described in detail later. When the control unit determines that the "related model-specific threshold setting process" has been performed, it causes the threshold setting unit 14b to perform the "abnormal state determination threshold setting process." The threshold setting unit 14b performs the "abnormal state determination threshold setting process" and sets the abnormal state determination threshold. The "abnormal state determination threshold setting process" will be described in detail later. The "related model-specific threshold setting process" and the "abnormal state determination threshold setting process" performed by the threshold setting unit 14b will be described in detail below.

[0103] First, the "related model-specific threshold setting process" performed by the threshold setting unit 14b will be described. In the threshold setting unit 14b, the associated model threshold setting unit 143 of the threshold setting unit 14b sets an associated model threshold for each of one or more associated machine learning models selected by the model selection unit 18, based on the abnormal state score obtained by inputting the feature calculated by the feature calculation unit 13 within the first threshold setting time. The first threshold setting time has been described in the first embodiment, so a duplicated description will be omitted. However, the first threshold setting time in the third embodiment may be the same length of time as the first threshold setting time in the first embodiment, or may be a different length of time.

[0104] Specifically, the associated model threshold setting unit 143 first inputs the feature calculated by the feature calculation unit 13 to a certain associated machine learning model among one or more associated machine learning models to obtain an abnormal state score. The associated model threshold setting unit 143 performs the above-mentioned acquisition of abnormal state scores for all associated machine learning models, and obtains an abnormal state score corresponding to each associated machine learning model. The associated model threshold setting unit 143 generates information (hereinafter referred to as "third score information") linking the acquired abnormal state score corresponding to each associated machine learning model, information capable of identifying the associated machine learning model, and the acquisition date and time of the abnormal state score, and stores the third score information in the storage unit. The associated model threshold setting unit 143 performs the acquisition and storage of the abnormal state score for a first threshold setting time.

[0105] Then, when the first threshold setting time has elapsed, the associated model threshold setting unit 143 sets the maximum value of one or more abnormal state scores stored in the storage unit during the first threshold setting time for a certain associated machine learning model among the one or more associated machine learning models as the associated model threshold. The associated model threshold setting unit 143 may identify one or more abnormal state scores of a certain associated machine learning model stored in the storage unit during the first threshold setting time from the third score information. At this time, the associated model threshold setting unit 143 may set the associated model threshold by providing a buffer for the maximum value of one or more abnormal state scores obtained within the first threshold setting time. Furthermore, the associated model threshold setting unit 143 may set the associated model threshold to a value obtained by multiplying the maximum value of one or more abnormal state scores obtained within the first threshold setting time by a coefficient. The associated model threshold setting unit 143 sets the associated model threshold to the above-mentioned associated model threshold for all associated machine learning models, and sets the associated model threshold corresponding to each associated machine learning model. The related model-specific threshold setting unit 143 generates information (hereinafter referred to as "related model-specific threshold information") that links the set related model-specific threshold for each related machine learning model to information that can identify the related machine learning model, and stores the related model-specific threshold information in a memory unit. When the associated model-specific threshold setting unit 143 generates the associated model-specific threshold information and stores it in the storage unit, the control unit determines that the associated model-specific threshold setting process is completed.

[0106] Next, the "abnormal state determination threshold setting process" performed by the threshold setting unit 14b will be described. In the "threshold setting process for determining an abnormal state", the threshold setting unit 14b sets a threshold for determining an abnormal state to estimate an abnormal state of an occupant, for each of one or more associated machine learning models, based on an abnormal state score obtained by inputting features related to the occupant calculated by the feature calculation unit 13 within a predetermined time (hereinafter referred to as the "third threshold setting time"). In the third embodiment, the third threshold setting time is a time during which the occupant is assumed to be in a normal state, which is set in order to perform a process for setting an abnormal state determination threshold as a process prior to estimating an abnormal state of the occupant, similar to the first threshold setting time. The third threshold setting time is the time after the first threshold setting time has elapsed. The length of the third threshold setting time is appropriately set, such as 5 minutes or 10 minutes. The length of the third threshold setting time may be the same as the length of the first threshold setting time. Also, the length of the third threshold setting time may be the same as the length of the second threshold setting time described in the second embodiment.

[0107] First, in the threshold setting unit 14b, the related model-specific determination unit 144 of the threshold setting unit 14b determines, for each of one or more related machine learning models, whether or not the abnormal state score obtained by inputting the feature amount calculated by the feature amount calculation unit 13 within the third threshold setting time is greater than the related model-specific threshold amount set by the related model-specific threshold setting unit 143. Note that the related model-specific determination unit 144 may specify the related model-specific threshold amount corresponding to each related machine learning model from the related model-specific threshold amount information stored in the storage unit by the related model-specific threshold setting unit 143. Specifically, the associated model-specific determination unit 144 first inputs the feature amount calculated by the feature amount calculation unit 13 to a certain associated machine learning model among one or more associated machine learning models to obtain an abnormal state score. The associated model-specific determination unit 144 performs the above-mentioned acquisition of the abnormal state score for all associated machine learning models, and obtains an associated abnormal state score corresponding to each associated machine learning model. The associated model-specific determination unit 144 generates information (hereinafter referred to as "fourth score information") linking the acquired abnormal state score corresponding to each associated machine learning model, information capable of identifying the associated machine learning model, and the acquisition date and time of the abnormal state score, and stores the fourth score information in the storage unit. The associated model-specific determination unit 144 performs the acquisition and storage of the abnormal state score for a third threshold setting time.

[0108] Then, when the third threshold setting time has elapsed, related model-specific determination unit 144 determines, for a certain related machine learning model among the one or more related machine learning models, whether or not the certain related machine learning model corresponds to a related machine learning model in which any of the one or more abnormal state scores obtained from the certain related machine learning model stored in the storage unit during the third threshold setting time is greater than the related model-specific threshold set by related model-specific threshold setting unit 143. Hereinafter, in the third embodiment, a related machine learning model in which any of the one or more abnormal state scores obtained from the certain related machine learning model is greater than the related model-specific threshold set by related model-specific threshold setting unit 143 is referred to as a "model to be counted". The associated model determination unit 144 may identify, from the fourth score information, one or more abnormal state scores obtained from the associated machine learning model stored in the storage unit during the third threshold setting time. The related model determination unit 144 determines whether a certain related machine learning model corresponds to a "model to be counted". Specifically, when at least one of one or more abnormal state scores obtained from the certain related machine learning model is greater than the related model threshold, the related model determination unit 144 determines that the certain related machine learning model corresponds to a "model to be counted". Note that, here, the related model determination unit 144 determines that the certain related machine learning model corresponds to a model to be counted when at least one of one or more abnormal state scores of the certain related machine learning model is greater than the related model threshold, but this is merely an example. For example, when the abnormal state score that is greater than the related model threshold among one or more related abnormal state scores of the certain related machine learning model is equal to or greater than a preset threshold, the related model determination unit 144 may determine that the certain related machine learning model corresponds to a model to be counted.

[0109] The associated model determination unit 144 performs the above-mentioned determination on all associated machine learning models as to whether or not they correspond to models to be counted. The related model determination unit 144 generates information (hereinafter referred to as "related model counting necessity determination result information") that links, for each related machine learning model, the determination result of whether or not the model corresponds to a model to be counted with information that can identify the related machine learning model, and outputs the related model counting necessity determination result information to the threshold setting unit 14b.

[0110] The threshold setting unit 14b sets the number of models to be counted as the abnormal state determination threshold based on the associated model count necessity determination result information output from the associated model determination unit 144. At this time, the threshold setting unit 14b may set the abnormal state determination threshold by providing a buffer for the number of models to be counted. Specifically, the threshold setting unit 14b may set the abnormal state determination threshold to the number of models to be counted + α, where α is a value greater than 0. The threshold setting unit 14b associates the set abnormal state determination threshold with the personal identification information output from the authentication unit 12b and stores the same in the storage unit, and sets the estimation possible flag to "1."

[0111] In the third embodiment, during the first threshold setting time and the third threshold setting time during which the threshold setting unit 14b is performing the process of setting the abnormal state determination threshold, the abnormal state score is output from the associated machine learning model. However, during this time, the abnormal state estimation device 1b is configured not to estimate the abnormal state of the occupant based on the abnormal state score. For example, the abnormal state estimation device 1b determines whether to estimate the abnormal state of the occupant based on whether the estimable flag is "1" or "0". When the estimable flag is "1", the abnormal state estimation device 1b determines to estimate the abnormal state of the occupant. Conversely, when the estimable flag is "0", the abnormal state estimation device 1b determines not to estimate the abnormal state of the occupant. The control based on the estimable flag is performed by the control unit. The control based on the estimable flag by the control unit has been described in the first embodiment, so a duplicated description will be omitted. In the abnormal state estimation device 1b, the threshold setting unit 14b performs a "threshold setting process for each associated model" and a "threshold setting process for abnormal state determination" in accordance with control by the control unit based on the estimability flag. If a threshold for each associated model has not been set, the control unit causes the threshold setting unit 14b to perform a "threshold setting process for each associated model". If a threshold for each associated model has been set and the estimability flag is "0", the control unit causes the threshold setting unit 14b to perform a "threshold setting process for abnormal state determination". Note that since the "threshold setting process for each associated model" is a process that is performed before the "threshold setting process for abnormal state determination", the estimability flag is "0" at the time of execution of the threshold setting process for each associated model. In the abnormal state estimation device 1b, an estimation unit 15b described later performs a process of estimating an abnormal state of an occupant according to control by the control unit based on an estimable flag. When the estimable flag is "1", the control unit causes the estimation unit 15b to perform a process of estimating an abnormal state of an occupant.

[0112] The estimation unit 15b estimates an abnormal state of an occupant, similarly to the estimation unit 15a of the abnormal state estimation device 1a according to the second embodiment. The estimation unit 15b differs from the estimation unit 15a according to the second embodiment in that the machine learning model used to estimate the abnormal state of the occupant is one or more associated machine learning models selected by the model selection unit 18. The estimation unit 15b estimates the abnormal state of the occupant based on the abnormal state score obtained by inputting the features related to the occupant calculated by the feature calculation unit 13 into one or more associated machine learning models selected by the model selection unit 18.

[0113] The estimation unit 15b will be described in detail below. In the estimation unit 15b, the related model number calculation unit 152 of the estimation unit 15b calculates the number of related machine learning models (hereinafter referred to as the "number of related models for abnormality determination") among one or more related machine learning models for which the abnormal state score obtained by inputting the features calculated by the feature calculation unit 13 is determined to be greater than the threshold value for each related model. Specifically, the related model number calculation unit 152 first inputs the feature calculated by the feature calculation unit 13 to a related machine learning model among the one or more related machine learning models to obtain an abnormal state score. Then, the related model number calculation unit 152 determines whether the obtained abnormal state score is greater than the related model-specific threshold set by the related model-specific threshold setting unit 143 of the threshold setting unit 14b. Note that the related model number calculation unit 152 may specify, from the related model-specific threshold information stored in the storage unit, a related model-specific threshold corresponding to a certain related machine learning model to be compared with the abnormal state score. The related model number calculation unit 152 performs the above-mentioned determination of whether the abnormal state score is greater than the related model-specific threshold for all related machine learning models. Then, the related model number calculation unit 152 calculates the number of related models for abnormality determination. The associated model number calculation unit 152 outputs the calculated number of associated models for abnormality determination to the estimation unit 15b.

[0114] The estimation unit 15b estimates that the occupant is in an abnormal state when the number of abnormality determining associated models calculated by the associated model number calculation unit 152 is greater than the abnormal state determining threshold set by the threshold setting unit 14b. Specifically, the estimation unit 15b compares the number of related models for abnormality determination calculated by the related model number calculation unit 152 with the abnormal state determination threshold stored in the storage unit by the threshold setting unit 14b. At that time, the estimation unit 15b compares the personal identification information output from the authentication unit 12b with the personal identification information linked to the abnormal state determination threshold in the storage unit, and obtains the abnormal state determination threshold linked to the matching personal identification information from the storage unit as the abnormal state determination threshold to be compared with the number of related models for abnormality determination. When the number of related models for abnormality determination is greater than the abnormal state determination threshold, the estimation unit 15b estimates that the occupant is in an abnormal state. On the other hand, when the number of related models for abnormality determination is equal to or less than the abnormal state determination threshold, the estimation unit 15b estimates that the occupant is in a normal state.

[0115] The operation of the abnormal state estimating device 1b according to the third embodiment will be described. FIG. 15 is a flowchart for explaining the operation of the abnormal state estimating device 1b according to the third embodiment. Specific operations of step ST1501 and step ST1503 in FIG. 15 are similar to those of step ST1001 and step ST1003 in FIG. 10 described in the second embodiment, and therefore repeated description will be omitted. In step ST1502, the authentication unit 12b performs authentication of the individual occupant (step ST1502). When the authentication unit 12b is able to authenticate an individual occupant, the authentication unit 12b acquires personal identification information. At that time, the attribute information acquisition unit 121 of the authentication unit 12b acquires attribute information of the occupant. When the authentication unit 12b is able to identify the individual occupant, the authentication unit 12b outputs the acquired individual identification information and the occupant's attribute information to the threshold setting unit 14b, the estimation unit 15b, and the model selection unit . If the authentication unit 12b fails to identify the individual occupant, the abnormal condition estimating device 1b ends the operation of the flowchart in FIG.

[0116] The model selection unit 18 selects one or more associated machine learning models related to the attributes of the occupant from among the multiple machine learning models (1 to N) 17 based on the attribute information acquired by the attribute information acquisition unit 121 of the authentication unit 12b in step ST1502 (step ST1504). The model selection unit 18 outputs information on the selected one or more relevant machine learning models to the threshold setting unit 14b and the estimation unit 15b.

[0117] The control unit determines whether or not the associated model-specific threshold has been set (step ST1505). Specifically, the control unit determines whether or not the associated model-specific threshold information corresponding to all associated machine learning models is stored in the storage unit. In step ST1505, when it is determined that the threshold for each related model has not been set ("NO" in step ST1505), the operation of the flowchart in Fig. 15 proceeds to step ST1506. Note that, before the operation described in Fig. 16 described later is performed, the threshold for each related model has not been set, so the control unit determines that the threshold for each related model has not been set. After the operation described in Fig. 16 described later is performed, the threshold for each related model has been set, so the control unit determines that the threshold for each related model has been set. Then, the threshold setting unit 14b performs "related model-specific threshold setting processing" (step ST1506).

[0118] FIG. 16 is a flow chart for explaining a specific operation of step ST1506 in FIG. The associated model-specific threshold setting unit 143 judges whether or not the time is within the first threshold setting time (step ST1601). In step ST1601, when the associated model threshold setting unit 143 determines that the first threshold setting time is within the first threshold setting time (in the case of "YES" in step ST1601), the associated model threshold setting unit 143 inputs the feature calculated by the feature calculation unit 13 in the most recent step ST1503 of FIG. 15 to a certain associated machine learning model among one or more associated machine learning models to obtain an abnormal state score. The associated model threshold setting unit 143 performs the above-mentioned acquisition of the abnormal state score for all associated machine learning models to obtain an abnormal state score corresponding to each associated machine learning model. The associated model threshold setting unit 143 generates third score information and stores the third score information in the storage unit (step ST1602). Thereafter, the operation of the flowchart in FIG. 16 returns to step ST1501 in FIG. 15, and after performing the operation of step ST1501, the operation proceeds to the operation of step ST1501 and thereafter again. Thereafter, until the first threshold setting time has elapsed (if "NO" in step ST1601), the associated model-specific threshold setting unit 143 performs the operation of step ST1602. As a result, the third score information generated by the associated model-specific threshold setting unit 143 within the first threshold setting time is stored in the storage unit.

[0119] In step ST1601, when the associated model threshold setting unit 143 determines that the first threshold setting time has not elapsed, that is, when the first threshold setting time has elapsed (in the case of "NO" in step ST1601), the associated model threshold setting unit 143 sets the maximum value of one or more abnormal state scores stored in the storage unit during the first threshold setting time for a certain associated machine learning model among one or more associated machine learning models as the associated model threshold (step ST1603). The associated model threshold setting unit 143 sets the associated model threshold for all associated machine learning models as described above, and sets the associated model threshold corresponding to each associated machine learning model. The associated model threshold setting unit 143 generates associated model threshold information and stores it in the storage unit. Thereafter, the operation of the flowchart in FIG. 16 returns to step ST1501 in FIG. 15, and after performing the operation of step ST1501, the operation proceeds to the operation of step ST1501 and thereafter again.

[0120] Returning to the explanation of the flowchart in FIG. In step ST1505, when the control section judges that the associated model-specific threshold has been set ("YES" in step ST1505), the operation of the flowchart in FIG. 15 proceeds to step ST1507. In step ST1507, the control unit judges whether or not the abnormal state judgment threshold has been set. Specifically, the control unit judges whether or not the estimation possible flag is "1" (step ST1507). In step ST1507, if the control unit determines that the abnormal state determination threshold has not been set, in other words, if the control unit determines that the estimable flag is "0" ("NO" in step ST1507), the operation of the flowchart in Fig. 15 proceeds to step ST1508. Note that, before the operation described in Fig. 17 described later is performed, the abnormal state determination threshold has not been set, so the control unit determines that the abnormal state determination threshold has not been set. After the operation described in Fig. 17 is performed, the abnormal state determination threshold has been set, so the control unit determines that the abnormal state determination threshold has been set. Then, the threshold setting unit 14b performs "an abnormal state determination threshold setting process" (step ST1508).

[0121] FIG. 17 is a flow chart for explaining a specific operation of step ST1508 in FIG. The associated model-specific determination unit 144 of the threshold setting unit 14b determines whether or not the time is within the third threshold setting time (step ST1701). In step ST1701, when it is determined that the time is within the third threshold setting time (in the case of "YES" in step ST1701), the related model determination unit 144 first inputs the feature amount calculated by the feature amount calculation unit 13 in the most recent step ST1503 of FIG. 15 to a related machine learning model among one or more related machine learning models to obtain an abnormal state score. The related model determination unit 144 performs the above-mentioned acquisition of the abnormal state score for all related machine learning models to obtain an abnormal state score corresponding to each related machine learning model. The related model determination unit 144 generates fourth score information and stores the fourth score information in the storage unit (step ST1702). Thereafter, the operation of the flowchart in FIG. 17 returns to step ST1501 in FIG. 15, and after performing the operation of the step ST1501, the operation proceeds to the operation after the step ST1501 again. After that, until the third threshold setting time has elapsed (if "NO" in step ST1701), the associated model-specific determination unit 144 performs the operation of step ST1702. As a result, the fourth score information generated by the associated model-specific determination unit 144 within the third threshold setting time is stored in the storage unit.

[0122] In step ST1701, if the related model-specific judgment unit 144 judges that the third threshold setting time is not reached, i.e., when the third threshold setting time has elapsed (if "NO" in step ST1701), the related model-specific judgment unit 144 judges, for a certain related machine learning model among the one or more related machine learning models, whether or not any of the one or more abnormal state scores of the certain related machine learning model stored in the memory unit at the third threshold setting time is greater than the related model-specific threshold set by the related model-specific threshold setting unit 143. For a certain related machine learning model, when at least one of one or more abnormal state scores obtained from the certain related machine learning model is greater than the related model-specific threshold, the related model determination unit 144 determines that the certain related machine learning model is a related machine learning model that corresponds to a model to be counted (step ST1703). The related model determination unit 144 performs the above-mentioned determination on all related machine learning models as to whether or not they correspond to models to be counted. The associated model-specific determination unit 144 generates associated model-specific count necessity determination result information, and outputs the associated model-specific count necessity determination result information to the threshold setting unit 14b.

[0123] The threshold setting unit 14b sets the number of models to be counted as the abnormal state determination threshold based on the associated model count necessity determination result information output from the associated model determination unit 144 in step ST1703 (step ST1704). The threshold setting unit 14b associates the set abnormal state determination threshold with the personal identification information output from the authentication unit 12b and stores it in the storage unit, and sets the estimation possible flag to "1." After that, the operation of the flowchart in Fig. 17 returns to step ST1501 in Fig. 15, and after performing the operation of step ST1501, the operation proceeds to the operation of step ST1501 and thereafter.

[0124] Returning to the explanation of the flowchart in FIG. In step ST1507, if the control unit determines that the threshold for determining an abnormal state has been set, in other words, if it determines that the estimation possible flag is "1" ("YES" in step ST1507), the operation of the flowchart in FIG. 15 proceeds to step ST1509. Then, the estimation unit 15b estimates the abnormal state of the occupant based on the abnormal state score obtained by inputting the features related to the occupant calculated by the feature calculation unit 13 in step ST1503 into each associated machine learning model (step ST1509).

[0125] FIG. 18 is a flow chart for explaining a specific operation of step ST1509 in FIG. The related model number calculation unit 152 of the estimation unit 15b first inputs the feature calculated by the feature calculation unit 13 in the most recent step ST1503 in Fig. 15 to a related machine learning model among the one or more related machine learning models to obtain an abnormal state score. Then, the related model number calculation unit 152 determines whether the obtained abnormal state score is greater than the related model-specific threshold set by the related model-specific threshold setting unit 143 in step ST1603 in Fig. 16. The related model number calculation unit 152 performs the above-mentioned determination as to whether or not the abnormal state score is greater than the associated model-specific threshold for all associated machine learning models. Then, the related model number calculation unit 152 calculates the number of associated models for abnormality determination (step ST1801). The associated model number calculation unit 152 outputs the calculated number of associated models for abnormality determination to the estimation unit 15b.

[0126] The estimation unit 15b compares the number of associated models for abnormality determination calculated by the associated model number calculation unit 152 in step ST1801 with the abnormal state determination threshold set by the threshold setting unit 14b in step ST1704 in FIG. 17 (step ST1802). In step ST1802, if the number of abnormality determination related models is greater than the abnormal state determination threshold value ("YES" in step ST1802), the estimation unit 15b estimates that the occupant is in an abnormal state (step ST1803). On the other hand, in step ST1802, if the number of abnormality determination related models is equal to or smaller than the abnormal state determination threshold value ("NO" in step ST1802), the estimation unit 15b estimates that the occupant is in a normal state (step ST1804).

[0127] In the above description of the operation, in the "abnormal state determination threshold setting process", the threshold setting unit 14b performs the operation described in the flowchart of Fig. 17 to set the abnormal state determination threshold, but this is merely an example. The threshold setting unit 14b may, for example, perform the operation described below to set the abnormal state determination threshold. If it is within the third threshold setting time, the related model determination unit 144 of the threshold setting unit 14b inputs the feature calculated by the feature calculation unit 13 in the most recent step ST1503 of FIG. 15 to a certain related machine learning model among the one or more related machine learning models to acquire an abnormal state score. Next, the related model determination unit 144 determines whether or not the certain related machine learning model corresponds to a "model to be counted" depending on whether or not the acquired abnormal state score is greater than the related model threshold set by the related model threshold setting unit 143 in step ST1603 of FIG. 16. If the abnormal state score obtained from the certain related machine learning model is greater than the related model threshold set by the related model threshold setting unit 143, the related model determination unit 144 determines that the certain related machine learning model corresponds to a "model to be counted". The associated model determination unit 144 performs the above-mentioned acquisition of the abnormal state score and the determination of whether or not each of the associated machine learning models corresponds to a count target model for all of the associated machine learning models. After acquiring the abnormal state score for each of the associated machine learning models and determining whether or not each of the associated machine learning models corresponds to a count target model, the associated model determination unit 144 calculates the number of associated machine learning models that correspond to the count target models among all of the associated machine learning models. The associated model determination unit 144 stores the calculated number of count target models in the storage unit, for example, in association with the time when the number was calculated. The related model-specific determination unit 144 performs the above operation during the third threshold setting time and each time the feature amount calculation unit 13 calculates a feature amount. When the third threshold setting time has elapsed, the threshold setting unit 14b sets the maximum value of the number of count target models calculated by the related model-specific determination unit 144 within the third threshold setting time as the abnormal state determination threshold. At this time, the threshold setting unit 14b may set the abnormal state determination threshold by providing a buffer for the maximum value of the number of count target models calculated within the third threshold setting time. Specifically, the threshold setting unit 14b may set the abnormal state determination threshold to the maximum value of the number of count target models calculated within the third threshold setting time +α. α is a value greater than 0.

[0128] In this way, the abnormal condition estimation device 1b according to the third embodiment selects one or more associated machine learning models related to the attributes of the occupant from the machine learning models (1 to N) 17. The abnormal condition estimation device 1b assumes that the occupant is in a normal state during the first threshold setting time, and sets a threshold for each associated machine learning model based on the maximum abnormal condition score obtained by inputting the feature amount related to the occupant during the first threshold setting time. Furthermore, after setting the associated model-specific threshold, the abnormal condition estimation device 1b assumes that the occupant is in a normal state for the third threshold setting time, and compares the abnormal condition score obtained by inputting the feature amount related to the occupant for the third threshold setting time with the associated model-specific threshold for each associated machine learning model to calculate the number of associated machine learning models that are to be counted among the associated machine learning models. Then, the abnormal condition estimation device 1b sets the calculated number of models to be counted as the abnormal condition determination threshold. After setting the abnormal state determination threshold, the abnormal state estimation device 1b inputs the feature amount related to the occupant for each associated machine learning model and obtains the abnormal state score for each associated machine learning model. The abnormal state estimation device 1b determines whether the obtained abnormal state score for each associated machine learning model is greater than the associated model-specific threshold. Then, the abnormal state estimation device 1b calculates the number of associated models for abnormality determination.

[0129] When the abnormal state estimation device 1b calculates the number of relevant models for abnormality determination, it compares the number of relevant models for abnormality determination with the threshold value for abnormality state determination, and if the number of relevant models for abnormality determination is greater than the threshold value for abnormality state determination, it estimates that the occupant is in an abnormal state. The abnormal condition estimation device 1b does not use a fixed threshold to estimate whether an occupant is in an abnormal state, but sets an abnormal condition determination threshold for each occupant and estimates whether the occupant is in an abnormal state using the abnormal condition determination threshold. Therefore, when estimating whether an occupant is in an abnormal state, the abnormal condition estimation device 1b can prevent erroneous estimation of the abnormal state of the occupant. Furthermore, the abnormal condition estimation device 1b sets the abnormal condition determination threshold and estimates the abnormal condition of the occupant based on an associated machine learning model among a plurality of machine learning models that have been individually generated in advance according to the attributes of a plurality of occupants having various attributes. Therefore, the abnormal condition estimation device 1b can further improve the estimation accuracy of the abnormal condition of the occupant compared to a case where the abnormal condition determination threshold is set and the abnormal condition of the occupant is estimated using all of a plurality of machine learning models generated from information on a plurality of occupants having various attributes.

[0130] In the abnormal state estimation device 1b of embodiment 3, like the abnormal state estimation device 1 of embodiment 1, the estimation unit 15b may, for example, estimate that the occupant is in an abnormal state if the percentage of the occupant estimated to be in an abnormal state during the abnormal state estimation time is equal to or greater than a threshold value, and thereafter not estimate the occupant's abnormal state until driving of the vehicle is terminated.

[0131] In addition, in the above-described third embodiment, the information input to the associated machine learning model in the abnormal condition estimation device 1b is the feature amount related to the occupant calculated by the feature amount calculation unit 13, but this is merely an example. For example, in the abnormal condition estimation device 1b, the information input to the associated machine learning model may be information related to the occupant acquired by the information acquisition unit 11. In this case, the abnormal condition estimation device 1b may be configured not to include the feature amount calculation unit 13. However, if the abnormal condition estimation device 1b is equipped with a feature calculation unit 13 and features related to the occupant that have a high correlation with the estimation of the abnormal condition are input to the associated machine learning model, it is expected that the accuracy of estimation when estimating the abnormal condition of the occupant will be improved compared to the case where information about the occupant is directly input to the associated machine learning model. In addition, in the abnormal state estimation device 1b, when the information to be input to the associated machine learning model is information about the occupant, the learning device uses the information about the occupant as teacher data.

[0132] As described above, according to embodiment 3, in the abnormal condition estimation device 1b that estimates the abnormal condition of an occupant based on information about the vehicle occupant and the machine learning model (1 to N) 17, the abnormal condition estimation device 1b is configured to include an information acquisition unit 11 that acquires information about the occupant, and a threshold setting unit 14b that sets an abnormal condition determination threshold for estimating the abnormal condition of the occupant based on the abnormal condition score obtained by inputting the information about the occupant acquired by the information acquisition unit within a first threshold setting time into the machine learning model (1 to N) 17. Therefore, when estimating whether or not an occupant is in an abnormal state, the abnormal state estimating device 1b can prevent erroneous estimation of the abnormal state of the occupant.

[0133] In particular, in the third embodiment, there are a plurality of machine learning models (1 to N) 17, and the abnormal state estimation device 1b includes an attribute information acquisition unit 121 that acquires attribute information related to the attributes of an occupant, a model selection unit 18 that selects one or more associated machine learning models related to the attributes of the occupant from the plurality of machine learning models (1 to N) 17 based on the attribute information acquired by the attribute information acquisition unit 121, and a related model-specific threshold value for each of the one or more associated machine learning models selected by the model selection unit 18 based on the abnormal state score obtained by inputting information about the occupant acquired by the information acquisition unit 11 within a first threshold setting time. The system is equipped with a related model-specific threshold setting unit 143 that sets the related model-specific threshold, and a related model-specific judgment unit 144 that, after the related model-specific threshold setting unit 143 sets the related model-specific threshold, judges, for each of one or more related machine learning models, whether or not the abnormal condition score obtained by inputting information about the occupant acquired by the information acquisition unit 11 within the third threshold setting time is greater than the related model-specific threshold set by the related model-specific threshold setting unit 143, and the threshold setting unit 14b sets the number of related machine learning models for which the related model-specific judgment unit 144 has judged that the abnormal condition score is greater than the related model-specific threshold as the abnormal condition judgment threshold. As a result, the abnormal condition estimation device 1b can further improve the accuracy of estimating the abnormal condition of an occupant compared to when setting a threshold for determining an abnormal condition and estimating the abnormal condition of an occupant using all of multiple machine learning models generated from information about multiple occupants with various attributes.

[0134] In the above first to third embodiments, it is assumed that the occupant of the vehicle is the driver, and the abnormal state estimating devices 1 to 1b estimate whether the driver is in an abnormal state, but this is merely one example. In the first to third embodiments, the occupant of the vehicle may be, for example, a passenger sitting in the front passenger seat or the back seat. In this case, the abnormal state of the occupant estimated by the abnormal state estimation device 1 to 1b refers to a state in which the occupant is in poor physical condition. Specifically, the abnormal state of the occupant refers to, for example, a state in which the occupant has drunk alcohol or a state in which the occupant is fatigued. In this case, the start of driving the vehicle refers to, for example, a time when the driver starts driving the vehicle.

[0135] In the above first to third embodiments, the information acquisition unit 11 acquires a face image, biometric information, and vehicle information in the abnormal state estimation devices 1 to 1b, but this is merely an example. The information acquisition unit 11 may acquire at least one of the face image, biometric information, and vehicle information as information related to the occupant. The feature amount calculation unit 13 calculates a feature amount related to the occupant according to the information related to the occupant acquired by the information acquisition unit 11.

[0136] In the above first to third embodiments, the estimation results of whether or not the occupant is in an abnormal state, which are estimated by the abnormal state estimating devices 1 to 1b, are used in various situations. For example, it is assumed that the vehicle on which the abnormal state estimation device 1-1b is mounted is an autonomous vehicle, and the abnormal state estimation device 1-1b estimates whether the driver is in an abnormal state. In this case, for example, when the abnormal state estimation device 1-1b estimates that the driver is in an abnormal state, it outputs the estimation result that the driver is in an abnormal state to an autonomous driving control unit (not shown) mounted on the vehicle. When the abnormal state estimation device 1-1b outputs the estimation result that the driver is in an abnormal state, the autonomous driving control unit performs control to advance the timing of applying the automatic brake. Also, for example, when the abnormal state estimation device 1-1b outputs the estimation result that the driver is in an abnormal state, the autonomous driving control unit prohibits switching to manual driving and performs control to pull the vehicle over to the shoulder of the road. Also, for example, the abnormal condition estimating devices 1-1b estimate the abnormal condition of a passenger. In this case, for example, when the abnormal condition estimating devices 1-1b estimate that the passenger is in an abnormal condition, they output the estimation result that the passenger is in an abnormal condition to a communication device (not shown) external to the abnormal condition estimating devices 1-1b. When the estimation result that the passenger is in an abnormal condition is output from the abnormal condition estimating devices 1-1b, the communication device reports it to an emergency center.

[0137] 19A and 19B are diagrams showing an example of the hardware configuration of the abnormal state estimating devices 1 to 1b according to the first to third embodiments. In the first to third embodiments, the functions of the information acquisition unit 11, the authentication units 12 and 12b, the feature calculation unit 13, the threshold setting units 14, 14a and 14b, the estimation units 15, 15a and 15b, and the model selection unit 18 are realized by a processing circuit 1901. That is, the abnormal state estimation devices 1 to 1b include the processing circuit 1901 for controlling the setting of an abnormal state determination threshold used when estimating an abnormal state of a vehicle occupant. The processing circuit 1901 may be dedicated hardware as shown in FIG. 19A, or may be a CPU (Central Processing Unit) 1905 that executes a program stored in a memory 1906 as shown in FIG. 19B.

[0138] When the processing circuit 1901 is dedicated hardware, the processing circuit 1901 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0139] When the processing circuit 1901 is a CPU 1905, the functions of the information acquisition unit 11, the authentication units 12 and 12b, the feature amount calculation unit 13, the threshold setting units 14, 14a and 14b, the estimation units 15, 15a and 15b, and the model selection unit 18 are realized by software, firmware, or a combination of software and firmware. That is, the information acquisition unit 11, the authentication units 12 and 12b, the feature amount calculation unit 13, the threshold setting units 14, 14a and 14b, the estimation units 15, 15a and 15b, and the model selection unit 18 are realized by a processing circuit such as the CPU 1905 that executes a program stored in a hard disk drive (HDD) 1902 or a memory 1906, a system LSI (Large-Scale Integration), etc. It can also be said that the programs stored in the HDD 1902, memory 1906, etc. cause the computer to execute the procedures or methods of the information acquisition unit 11, the authentication units 12 and 12b, the feature amount calculation unit 13, the threshold setting units 14, 14a and 14b, the estimation units 15, 15a and 15b, and the model selection unit 18. Here, the memory 1906 corresponds to, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), etc., or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, and a DVD (Digital Versatile Disc), etc.

[0140] The functions of the information acquisition unit 11, the authentication units 12 and 12b, the feature amount calculation unit 13, the threshold setting units 14, 14a and 14b, the estimation units 15, 15a and 15b, and the model selection unit 18 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the functions of the authentication units 12 and 12b may be realized by a processing circuit 1901 as dedicated hardware, and the functions of the information acquisition unit 11, the feature amount calculation unit 13, the threshold setting units 14, 14a and 14b, the estimation units 15, 15a and 15b, and the model selection unit 18 may be realized by the processing circuit 1901 reading and executing a program stored in a memory 1906. Moreover, the personal authentication DB 16 and a storage unit (not shown) use the memory 1906. Note that this is just an example, and the personal authentication DB 16 and a storage unit (not shown) may be configured with the HDD 1902, an SSD (Solid State Drive), a DVD, or the like. Furthermore, the abnormal state estimating devices 1 to 1b each include a device such as the imaging device 2 or the biosensor 3, an input interface device 1903 and an output interface device 1904 that perform wired or wireless communication with the device.

[0141] In the above embodiments 1 to 3, the abnormal state estimation devices 1 to 1b are in-vehicle devices mounted on a vehicle, and the information acquisition unit 11, the authentication units 12, 12b, the feature calculation unit 13, the threshold setting units 14, 14a, 14b, the estimation units 15, 15a, 15b, and the model selection unit 18 are provided in the abnormal state estimation devices 1 to 1b. Without being limited to this, some of the information acquisition unit 11, authentication units 12, 12b, feature calculation unit 13, threshold setting units 14, 14a, 14b, estimation units 15, 15a, 15b, and model selection unit 18 may be mounted on the vehicle's on-board device, and the rest may be provided in a server connected to the on-board device via a network, and the abnormal state estimation system may be configured by the on-board device and the server.

[0142] Furthermore, within the scope of the present invention, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted. [Industrial Applicability]

[0143] The abnormal condition estimation device according to the present invention can be applied to an abnormal condition estimation device that estimates whether an occupant of a vehicle is in an abnormal condition based on information about the occupant and a machine learning model. [Explanation of symbols]

[0144] 1, 1a, 1b Abnormal state estimation device, 2 Imaging device, 3 Biometric sensor, 11 Information acquisition unit, 111 Face image acquisition unit, 1111 Face detection unit, 112 Biometric information acquisition unit, 113 Vehicle information acquisition unit, 12, 12b Authentication unit, 13 Feature amount calculation unit, 131 Face feature amount calculation unit, 132 Biometric feature amount calculation unit, 133 Vehicle operation feature amount calculation unit, 14, 14a, 14b Threshold setting unit, 141 Model-specific threshold setting unit, 142 Model-specific judgment unit, 143 Related model-specific threshold setting unit, 144 Related model-specific judgment unit, 15, 15a, 15b Estimation unit, 151 Model number calculation unit, 152 Related model number calculation unit, 16 Personal authentication DB, 17 Machine learning model, 1901 Processing circuit, 1902 HDD, 1903 Input interface device, 1904 Output interface device, 1905 CPU, 1906 memory.

Claims

1. 1. An abnormal state estimation device that estimates an abnormal state of a vehicle occupant based on information about the occupant and a machine learning model, An information acquisition unit that acquires information about the occupant; a threshold setting unit that inputs information about the occupant acquired by the information acquisition unit within a first threshold setting time, which is a time during which the occupant is assumed to be in a normal state, into the machine learning model that receives information about the occupant as an input and outputs an abnormal state score used to estimate an abnormal state of the occupant, to acquire the abnormal state score, and sets an abnormal state determination threshold that is used to compare with the abnormal state score in estimating the abnormal state of the occupant, based on the acquired abnormal state score; The threshold setting unit sets, as the abnormal state determination threshold, a value obtained by inputting information about the occupant acquired by the information acquisition unit within the first threshold setting time into the machine learning model and adding a buffer to the maximum value of the abnormal state scores. An abnormal state estimation device comprising:

2. The threshold setting unit obtains one or more of the abnormal state scores, and sets a maximum value of the one or more abnormal state scores with a buffer as the abnormal state determination threshold.

2. The abnormal state estimating device according to claim 1 .

3. an estimation unit that estimates that the occupant is in an abnormal state when the abnormal state score obtained by inputting the information about the occupant acquired by the information acquisition unit into the machine learning model is greater than the abnormal state determination threshold set by the threshold setting unit; The abnormal state estimating device according to claim 1 , further comprising:

4. an authentication unit that performs authentication of the individual occupant and acquires personal identification information that identifies the occupant; the threshold setting unit stores the set abnormal state determination threshold in association with the personal identification information acquired by the authentication unit; The estimation unit estimates an abnormal state of the occupant when the authentication unit acquires the personal identification information and the threshold setting unit stores the abnormal state determination threshold in association with the personal identification information.

4. The abnormal state estimating device according to claim 3.

5. 1. An abnormal state estimation device that estimates an abnormal state of a vehicle occupant based on information about the occupant and a plurality of machine learning models, An information acquisition unit that acquires information about the occupant; a threshold setting unit that inputs information about the occupant acquired by the information acquisition unit within a first threshold setting time, which is a time during which the occupant is assumed to be in a normal state, into each of the plurality of machine learning models that input information about the occupant and output an abnormal state score used to estimate an abnormal state of the occupant, to acquire the abnormal state score, and sets an abnormal state determination threshold based on the acquired abnormal state score to be used for comparison with the number of machine learning models among the plurality of machine learning models in which the abnormal state score obtained by inputting information about the occupant is determined to be greater than a model-specific threshold in estimating the abnormal state of the occupant, The threshold setting unit is a model-specific threshold setting unit that sets the model-specific threshold for each of the plurality of machine learning models based on the abnormal state score obtained by inputting information about the occupant acquired by the information acquisition unit within the first threshold setting time; a model-specific determination unit that determines whether or not the abnormal state score obtained by inputting information about the occupant acquired by the information acquisition unit within a second threshold setting time is greater than the model-specific threshold set by the model-specific threshold setting unit after the model-specific threshold setting unit sets the model-specific threshold, for each of the plurality of machine learning models; The model-specific determination unit sets the number of the machine learning models for which the abnormal state score has been determined to be greater than the model-specific threshold as the abnormal state determination threshold. An abnormal state estimation device comprising:

6. a model number calculation unit that calculates the number of the machine learning models among the plurality of the machine learning models, the abnormal state score obtained by inputting the information about the occupant acquired by the information acquisition unit being greater than the model-specific threshold value set by the model-specific threshold value setting unit; an estimation unit that estimates that the occupant is in an abnormal state when the number of the machine learning models calculated by the model number calculation unit is greater than the abnormal state determination threshold set by the threshold setting unit; The abnormal state estimating device according to claim 5 , further comprising:

7. 1. An abnormal state estimation device that estimates an abnormal state of a vehicle occupant based on information about the occupant and a plurality of machine learning models, An information acquisition unit that acquires information about the occupant; a threshold setting unit that inputs information about the occupant acquired by the information acquisition unit within a first threshold setting time, which is a time during which the occupant is assumed to be in a normal state, into each of the multiple machine learning models that input information about the occupant and output an abnormal state score used to estimate an abnormal state of the occupant, to acquire the abnormal state score, and sets, based on the acquired abnormal state score, an abnormal state determination threshold used for comparing, in estimating the abnormal state of the occupant, with the number of machine learning models among the multiple machine learning models in which the abnormal state score obtained by inputting information about the occupant is determined to be greater than a related model-specific threshold; an attribute information acquisition unit that acquires attribute information relating to the attributes of the occupant; a model selection unit that selects one or more associated machine learning models related to the attributes of the occupant from among the plurality of machine learning models based on the attribute information acquired by the attribute information acquisition unit; The threshold setting unit is a related model threshold setting unit that sets the related model threshold for each of the one or more related machine learning models selected by the model selection unit based on the abnormal state score obtained by inputting information about the occupant acquired by the information acquisition unit within the first threshold setting time; a related model determination unit that determines, after the related model threshold setting unit sets the related model threshold, whether or not the abnormal state score obtained by inputting information about the occupant acquired by the information acquisition unit within a third threshold setting time, for one or more of the related machine learning models, is greater than the related model threshold set by the related model threshold setting unit; The number of the associated machine learning models for which the associated model-specific determination unit has determined that the abnormal state score is greater than the associated model-specific threshold is set as the abnormal state determination threshold. An abnormal state estimation device comprising:

8. The attribute of the occupant is the age of the occupant.

8. The abnormal state estimating device according to claim 7.

9. a related model number calculation unit that counts the number of the related machine learning models among the plurality of related machine learning models, the abnormal condition score obtained by inputting the information about the occupant acquired by the information acquisition unit into the related machine learning model is greater than the related model-specific threshold value set by the related model-specific threshold setting unit; an estimation unit that estimates that the occupant is in an abnormal state when the number of the relevant machine learning models calculated by the relevant model number calculation unit is greater than the abnormal state determination threshold set by the threshold setting unit; The abnormal state estimating device according to claim 7, further comprising:

10. a feature amount calculation unit that calculates a feature amount related to the occupant based on the information about the occupant acquired by the information acquisition unit, The threshold setting unit is The abnormal state determination threshold is set based on the abnormal state score obtained by inputting the feature amount calculated by the feature amount calculation unit into the machine learning model within the first threshold setting time.

2. The abnormal state estimating device according to claim 1 .

11. 1. An abnormal state estimation method for estimating an abnormal state of an occupant of a vehicle based on information about the occupant and a machine learning model, comprising: An information acquisition unit acquires information about the occupant; a threshold setting unit inputs information about the occupant acquired by the information acquisition unit within a first threshold setting time, which is a time during which the occupant is assumed to be in a normal state, into the machine learning model that receives information about the occupant as an input and outputs an abnormal state score used to estimate an abnormal state of the occupant, thereby acquiring the abnormal state score, and based on the acquired abnormal state score, sets an abnormal state determination threshold used for comparison with the abnormal state score in estimating the abnormal state of the occupant; The threshold setting unit sets, as the abnormal state determination threshold, a value obtained by inputting information about the occupant acquired by the information acquisition unit within the first threshold setting time into the machine learning model and adding a buffer to the maximum value of the abnormal state scores.

4. An abnormality state estimation method comprising:

12. 1. An abnormal state estimation method for estimating an abnormal state of a vehicle occupant based on information about the occupant and a plurality of machine learning models, comprising: An information acquisition unit acquires information about the occupant; a threshold setting unit inputs information about the occupant acquired by the information acquisition unit within a first threshold setting time, which is a time during which the occupant is assumed to be in a normal state, to each of the plurality of machine learning models that input information about the occupant and output an abnormal state score used to estimate an abnormal state of the occupant, to acquire the abnormal state score, and based on the acquired abnormal state score, sets an abnormal state determination threshold to be used for comparison with the number of the machine learning models among the plurality of machine learning models in which the abnormal state score obtained by inputting information about the occupant is determined to be greater than a model-specific threshold in estimating the abnormal state of the occupant; The threshold setting unit is a model-specific threshold setting unit setting the model-specific threshold for each of the plurality of machine learning models based on the abnormal state score obtained by inputting information about the occupant acquired by the information acquisition unit within the first threshold setting time; a step of a model-specific determination unit determining, after the model-specific threshold setting unit sets the model-specific threshold, whether or not the abnormal state score obtained by inputting information about the occupant acquired by the information acquisition unit within a second threshold setting time is greater than the model-specific threshold set by the model-specific threshold setting unit, for each of the plurality of machine learning models; The model-specific determination unit sets the number of the machine learning models for which the abnormal state score has been determined to be greater than the model-specific threshold as the abnormal state determination threshold.

4. An abnormality state estimation method comprising:

13. 1. An abnormal state estimation method for estimating an abnormal state of a vehicle occupant based on information about the occupant and a plurality of machine learning models, comprising: An information acquisition unit acquires information about the occupant; a threshold setting unit inputs information about the occupant acquired by the information acquisition unit within a first threshold setting time, which is a time during which the occupant is assumed to be in a normal state, into each of the multiple machine learning models that input information about the occupant and output an abnormal state score used to estimate an abnormal state of the occupant, to acquire the abnormal state score, and based on the acquired abnormal state score, sets an abnormal state determination threshold to be used for comparison with the number of machine learning models among the multiple machine learning models in which the abnormal state score obtained by inputting information about the occupant is determined to be greater than a related model-specific threshold in estimating the abnormal state of the occupant; An attribute information acquisition unit acquires attribute information related to attributes of the occupant; A model selection unit selects one or more associated machine learning models related to the attributes of the occupant from among the plurality of machine learning models based on the attribute information acquired by the attribute information acquisition unit, The threshold setting unit is a related model threshold setting unit setting the related model threshold for each of the one or more related machine learning models selected by the model selection unit based on the abnormal state score obtained by inputting information about the occupant acquired by the information acquisition unit within the first threshold setting time; a step of determining, for each of the one or more relevant machine learning models, whether or not the abnormal state score obtained by inputting information about the occupant acquired by the information acquisition unit within a third threshold setting time after the related model threshold setting unit has set the related model threshold, is greater than the related model threshold set by the related model threshold setting unit; The number of the associated machine learning models for which the associated model-specific determination unit has determined that the abnormal state score is greater than the associated model-specific threshold is set as the abnormal state determination threshold.

4. An abnormality state estimation method comprising:

Images