State estimation device and state estimation method

Abstract

The present invention comprises: a captured image acquisition unit (11) that acquires a captured image in which a subject is captured; a feature point detection unit (13) that detects a feature point of the subject on the basis of the captured image acquired by the captured image acquisition unit (11), wherein the feature point indicates a body part of the subject; a feature amount calculation unit (14) that calculates a plurality of types of feature amounts for state estimation of the subject on the basis of feature point information related to the feature point of the subject detected by the feature point detection unit (13); a plurality of state temporary estimation units (15) which each temporarily estimate the state of the subject on the basis of the feature amounts calculated by the feature amount calculation unit (14) and a model for estimating the state of a person using different types of feature amounts as inputs; and an integrated estimation unit (17) which, on the basis of the results of temporary estimations of the states of a plurality of subjects by the plurality of state temporary estimation units (15) and state determination conditions, determines an estimated state that is the state of the subject used as the estimation result.

Description

State estimation device and state estimation method

[0001] The present disclosure relates to a state estimation device and a state estimation method.

[0002] Conventionally, there has been known a technique for estimating the state of a person (hereinafter referred to as the "subject") whose state is to be estimated from feature quantities based on points (hereinafter referred to as "feature points") that indicate body parts of the subject detected in an image of the subject. For example, Patent Literature 1 discloses a technique for acquiring the positions of multiple skeleton points of a body image of an occupant included in an image captured by a camera capturing the interior of a vehicle as skeleton point coordinates in a two-dimensional coordinate system of the captured image, estimating front coordinates that represent the arrangement of the skeleton points when the occupant is viewed from the front based on the acquired skeleton point coordinates, and determining the physique of the occupant from distance information of the skeleton point coordinates from the estimated front coordinates. In the technique disclosed in Patent Literature 1, the skeleton points correspond to feature points, and the distance information of the skeleton point coordinates corresponds to feature quantities.

[0003] Japanese Patent Application Laid-Open No. 2021-81836

[0004] Conventional technologies, such as that disclosed in Patent Document 1, which estimate the state of a subject from feature quantities based on feature points detected in a captured image, have had the problem that if feature points of the subject are not detected in the captured image, the feature quantities cannot be calculated properly, making it impossible to estimate the state of the subject, or the estimation accuracy may be reduced.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a state estimation device that, compared to conventional technology, can accurately estimate the state of a subject even when feature points of the subject are not detected in a captured image.

[0006] The state estimation device according to the present disclosure includes an image acquisition unit that acquires an image of a subject, a feature point detection unit that detects feature points of the subject that indicate body parts of the subject based on the image acquired by the image acquisition unit, a feature amount calculation unit that calculates multiple types of feature amounts for estimating the subject's state based on feature point information related to the feature points of the subject detected by the feature point detection unit, a plurality of state provisional estimation units that each provisionally estimate the subject's state based on the feature amount calculated by the feature amount calculation unit and a model that estimates a human state using different types of feature amounts as input, and an integrated estimation unit that determines an estimated state, which is the state of the subject to be used as an estimation result, based on the provisional estimation results of the subject's states by the multiple state provisional estimation units and a state determination condition.

[0007] According to the present disclosure, a state estimation device can estimate the state of a subject with higher accuracy than conventional techniques, even when feature points of the subject are not detected in a captured image.

[0008] 4A and 4B are diagrams illustrating an example of the configuration of a state estimation device according to embodiment 1. FIG. 4B is a diagram illustrating an example of the contents of input feature quantity definition information stored in a storage unit in embodiment 1. FIG. 4C is a flowchart for explaining the operation of the state estimation device according to embodiment 1. FIG. 4A and 4B are diagrams illustrating an example of the hardware configuration of the state estimation device according to embodiment 1.

[0009] Embodiments of the present disclosure will be described in detail below with reference to the drawings. Embodiment 1. A state estimation device according to embodiment 1 is connected to an imaging device and estimates the state of a person (hereinafter referred to as the "subject") whose state is to be estimated based on a captured image of the subject captured by the imaging device. More specifically, the state estimation device detects points (hereinafter referred to as "feature points") indicating body parts of the subject based on the captured image, and estimates the state of the subject using feature amounts calculated based on the detected feature points and multiple models. Here, feature points may not be detected in the captured image depending on the positional relationship between the imaging device and the subject, the angle of view of the imaging device, etc. If feature points are not calculated in the captured image, attempting to estimate the state of the subject using feature amounts calculated based on feature points detected from the captured image may result in poorly calculated feature amounts, which may result in a problem in which the state of the subject cannot be estimated or the estimation accuracy may be reduced. Taking into consideration the occurrence of such problems, the state estimation device according to embodiment 1 prepares multiple models, each of which estimates a person's state using different types of feature quantities as input, and provisionally estimates the state of the subject using the multiple models. The state estimation device then determines the state of the subject to be used as an estimation result (hereinafter referred to as the "estimated state") from the result of provisionally estimating the state of the subject using the multiple models (hereinafter referred to as the "provisional estimation result"). In other words, the estimated state is the state of the subject that the state estimation device ultimately estimates. In this way, the state estimation device accurately estimates the state of the subject. The estimated state of the subject estimated by the state estimation device is used for various controls, etc.

[0010] In the following first embodiment, as an example, the subject is a vehicle occupant, and the state estimated by the state estimation device is a physique. That is, the state estimation device estimates the physique of the vehicle occupant. In the following first embodiment, the vehicle occupant is also simply referred to as an "occupant." The physique estimated by the state estimation device includes physical characteristics indicated by continuous values, such as height or weight, or a physique classification indicated by a discrete classification, such as small, medium, or large. Note that the physique classification estimated by the state estimation device is not limited to small, medium, or large, and is set appropriately by an administrator or the like. The physique classification may be divided into two categories, or may be divided into four or more categories. For example, the physique classification may be a classification such as adult male, adult female, child, or infant.

[0011] 1 is a diagram illustrating an example of the configuration of a state estimation device 1 according to embodiment 1. The state estimation device 1 is mounted on, for example, a vehicle. The state estimation device 1 is connected to an imaging device 2 and a sensor 3.

[0012] The imaging device 2 is, for example, a near-infrared camera or a visible light camera, and captures images of vehicle occupants. The imaging device 2 may be, for example, a shared imaging device included in a so-called "driver monitoring system (DMS)" that is installed in a vehicle to monitor the state of the driver inside the vehicle. The imaging device 2 is installed so as to capture an image of at least an area inside the vehicle that includes an area where the upper half of the vehicle occupant's body should be present. The area where the upper half of the vehicle occupant's body should be present is, for example, an area corresponding to the seat back and the space near the front of the headrest. For example, the imaging device 2 is installed in the center of an overhead console or dashboard inside the vehicle cabin so as to capture an image of the vehicle occupant from the center of the overhead console or dashboard. The imaging device 2 may be installed so as to capture an image of only the driver, or may be installed so as to capture an image of the occupant in the rear seat. The imaging device 2 outputs the captured image to the state estimation device 1.

[0013] The sensor 3 is a so-called interior sensor mounted on the vehicle so as to be able to detect objects, including people, in the vehicle cabin. The sensor 3 includes a radio wave sensor, etc. The sensor 3 outputs information about the detected objects in the vehicle cabin (hereinafter referred to as "sensor information") to the state estimation device 1. The sensor information includes, for example, information indicating the position of an object present in the vehicle cabin, the movement of an object in the vehicle cabin, or a load at a certain position in the vehicle cabin (e.g., a seat). The sensor information output from the sensor 3 is used by the state estimation device 1 to acquire information (hereinafter referred to as "subject-related information") that is the basis for calculating feature amounts used to estimate the physique of an occupant.

[0014] 1, only one imaging device 2 and one sensor 3 are connected to the state estimation device 1, but this is merely an example. A plurality of imaging devices 2 may be connected to the state estimation device 1, and a plurality of sensors 3 may be connected to the state estimation device 1.

[0015] The state estimation device 1 estimates the physique of an occupant based on feature amounts based on the captured image output from the imaging device 2 and feature amounts based on sensor information output from the sensor 3. Note that, here, the state estimation device 1 is connected to the sensor 3 and acquires sensor information from the sensor 3, but this is merely an example. The state estimation device 1 does not necessarily use feature amounts based on the subject-related information to estimate the physique of an occupant. If the state estimation device 1 does not use feature amounts based on the subject-related information to estimate the physique of an occupant, it does not need to acquire the subject-related information. In other words, the state estimation device 1 does not necessarily acquire sensor information from the sensor 3, and it is not essential that the state estimation device 1 be connected to the sensor 3. It is sufficient that the state estimation device 1 is at least connected to the imaging device 2, detects feature points of the occupant based on the captured image acquired from the imaging device 2, and estimates the physique of the occupant using feature amounts calculated based on the detected feature points. However, the state estimation device 1 estimates the occupant's physique based on features calculated from feature points detected based on the captured image, as well as features based on subject-related information obtained from sensor information, thereby increasing the number of features used to estimate the occupant's physique and further improving the accuracy of estimating the occupant's physique.

[0016] An example configuration of the state estimation device 1 will be described. As shown in FIG. 1 , the state estimation device 1 includes an image acquisition unit 11, a subject-related information acquisition unit 12, a feature point detection unit 13, a feature amount calculation unit 14, n (n is an integer equal to or greater than 2) state estimation units 151, 152, ..., 15n, a success / failure determination unit 16, an integrated estimation unit 17, and an estimation result output unit 18. In the first embodiment, the first state estimation unit 151, the second state estimation unit 152, ..., 15n are collectively referred to simply as the state estimation unit 15. In the following first embodiment, for simplicity of explanation, the state estimation device 1 will be described as including two state estimation units 15, the first state estimation unit 151 and the second state estimation unit 152.

[0017] The captured image acquisition unit 11 acquires a captured image from the imaging device 2. The captured image acquisition unit 11 outputs the acquired captured image to the feature point detection unit 13.

[0018] The subject-related information acquisition unit 12 acquires sensor information from the sensor 3 and acquires subject-related information about the occupant based on the sensor information. The subject-related information includes, for example, information indicating the position of the occupant in the vehicle cabin or the movement of the occupant. In the first embodiment, the position of the occupant is assumed to be, for example, the seating position of the occupant, and the position of the occupant is represented by the seat or the position of the seat. As described above, the sensor information includes information indicating the position of an object present in the vehicle cabin, the movement of the object in the vehicle cabin, or the load at a certain position in the vehicle cabin (e.g., a seat). For example, the subject-related information acquisition unit 12 may use a known object recognition technology or the like to detect a moving object, in other words, the position or movement of a living body (occupant), from the sensor information, generate information indicating the position or movement of the occupant, and acquire this information as the subject-related information. Note that this is merely an example; for example, the sensor 3 may detect a moving object, in other words, a living body (occupant), present in the vehicle cabin, generate subject-related information, and output it to the state estimation device 1. In this case, in the state estimation device 1 , the subject-related information acquisition unit 12 may acquire the subject-related information from the sensor 3 .

[0019] Furthermore, for example, the subject-related information acquisition unit 12 may acquire the subject-related information based on the captured image output from the imaging device 2. For example, the subject-related information acquisition unit 12 may perform a known edge detection process on the captured image to detect the contour of the occupant's physique in the captured image, generate information indicating the contour of the occupant's physique in the captured image, and acquire this as the subject-related information. In this case, the subject-related information acquisition unit 12 may acquire the captured image from the imaging device 2 via the captured image acquisition unit 11, for example. In FIG. 1 , an arrow is illustrated from the captured image acquisition unit 11 to the subject-related information acquisition unit 12.

[0020] The subject-related information acquisition unit 12 outputs the acquired subject-related information to the feature calculation unit 14. As described above, it is not essential for the state estimation device 1 to acquire the subject-related information. In other words, it is not essential for the state estimation device 1 to be equipped with the subject-related information acquisition unit 12. The state estimation device 1 may be configured not to include the subject-related information acquisition unit 12.

[0021] The feature point detection unit 13 detects feature points of the occupant that indicate body parts of the occupant based on the captured image acquired by the captured image acquisition unit 11. In the first embodiment, the feature points are assumed to be, for example, skeletal points of the occupant that indicate joint points determined for each body part of the occupant, or points corresponding to body parts of the occupant other than joint points, such as the eyes, nose, nasal passage, or collarbone center point. The feature point detection unit 13 detecting the feature points of the occupant means, in more detail, for example, that the feature point detection unit 13 extracts skeletal points of the occupant that indicate joint points determined for each body part of the occupant, or points corresponding to body parts of the occupant other than joint points, such as the eyes, nose, nasal passage, or collarbone center point, based on the captured image acquired by the captured image acquisition unit 11. The joint points determined for each body part of the occupant are, for example, shoulder joint points, elbow joint points, waist joint points, wrist joint points, knee joint points, and ankle joint points. These joint points are defined in advance.

[0022] The feature point detection unit 13 may detect the feature points of the occupant by a known method using known technology, such as image recognition technology or technology using a trained model in machine learning (hereinafter referred to as a "machine learning model"). In the first embodiment, the machine learning model for detecting the feature points of the occupant is also referred to as a "feature point detection model." The feature point detection model is a machine learning model that receives a captured image as input and outputs information about the feature points. The model is generated in advance and stored in a storage unit (not shown). The storage unit may be provided in the state estimation device 1 or may be provided in a location outside the state estimation device 1 that can be referenced by the state estimation device 1. The feature points are points in the captured image and are represented, for example, by coordinates in the captured image. The feature point detection unit 13 detects the coordinates of the feature points of the occupant and which parts of the occupant's body the feature points indicate.

[0023] The feature point detection unit 13 detects feature points for each occupant. For example, in the captured image, an area corresponding to each seat (hereinafter referred to as a "seat-corresponding area") is set in advance. The seat-corresponding area is set in advance depending on the installation position and angle of view of the image capture device 2. For example, the feature point detection unit 13 detects feature points for each seat-corresponding area. For example, if a certain seat-corresponding area corresponds to the driver's seat, the feature point detection unit 13 determines that the feature points detected in the seat-corresponding area are the feature points of the driver. Also, for example, if a certain seat-corresponding area corresponds to the passenger seat, the feature point detection unit 13 determines that the feature points detected in the seat-corresponding area are the feature points of the passenger in the passenger seat. In this way, the feature point detection unit 13 detects feature points for each occupant by detecting feature points from the seat-corresponding area for each seat-corresponding area. For example, feature points for multiple human bodies may be extracted from the entire captured image and associated with the seat-corresponding area, thereby detecting feature points for each occupant.

[0024] The feature point detection unit 13 outputs information about the detected feature points (hereinafter referred to as "feature point information") to the feature amount calculation unit 14. The feature point information includes information in which information indicating the occupant, information indicating the coordinates of the feature points of the occupant, and information indicating which part of the occupant's body the feature points represent, are associated with each other, and the captured image acquired by the captured image acquisition unit 11. In the first embodiment, the information indicating the occupant is, for example, information indicating the seat in which the occupant is sitting.

[0025] The feature amount calculation unit 14 calculates feature amounts for estimating the occupant's physique based on feature point information related to the occupant's feature points detected by the feature point detection unit 13. The feature amount calculation unit 14 also calculates feature amounts for estimating the occupant's physique based on the subject-related information acquired by the subject-related information acquisition unit 12. The feature amount calculation unit 14 calculates multiple types of feature amounts.

[0026] The feature amount may be, for example, a feature amount indicating eye height, a feature amount indicating shoulder width, a feature amount indicating upper body length, a feature amount indicating upper arm (right upper arm or left upper arm), or a feature amount indicating body area. For example, the feature amount calculation unit 14 identifies the feature amount to be calculated based on the input feature amount definition information, and calculates the identified feature amount in accordance with the feature amount calculation conditions.

[0027] The input feature definition information is information in which, for each of a plurality of models used by a plurality of state provisional estimation units 15 in the state estimation device 1 when provisionally estimating the physique of an occupant, feature values ​​to be input to the model are set (see FIG. 2 described later). The input feature definition information is generated in advance by an administrator or the like, for example, when a model is generated, and stored in a storage unit. The input feature definition information may be implemented, for example, as source code. Based on the input feature definition information, the feature calculation unit 14 can identify feature values ​​to be calculated and necessary for estimating the physique of an occupant in the state estimation device 1, in other words, feature values ​​to be input to each model used by the plurality of state provisional estimation units 15 when provisionally estimating the physique of an occupant. Details of the state provisional estimation unit 15 will be described later.

[0028] The feature calculation conditions are conditions that define how and based on which information (feature point information or subject-related information) the feature amounts set in the input feature amount definition information are calculated, more specifically, the feature amounts input by the multiple models used by the multiple state provisional estimation units 15 to provisionally estimate the occupant's physique. The feature calculation conditions are set for each feature amount. The feature calculation conditions are determined in advance by an administrator or the like, for example, when a model is generated. For example, after determining the feature calculation conditions, the administrator or the like stores information indicating the feature calculation conditions (hereinafter referred to as "feature calculation condition information") in a storage unit. The feature calculation unit 14 calculates the feature amounts based on the feature point information or the subject-related information by referring to the feature calculation condition information stored in the storage unit. For example, the feature calculation conditions may be changed as appropriate by an administrator or the like.

[0029] For example, the feature amount calculation conditions define the following condition: "For the feature amount indicating eye height, based on the feature point information, the distance from the midpoint between the feature points corresponding to both eyes (hereinafter referred to as the "eye center point") to the midpoint between the feature points corresponding to both waists (hereinafter referred to as the "waist center point") on the captured image is calculated as the feature amount indicating eye height." In this case, the feature amount calculation unit 14 calculates the coordinates of the eye center point and the waist center point based on the feature point information, and calculates the distance from the eye center point to the waist center point on the captured image as the feature amount indicating eye height.

[0030] Furthermore, for example, the feature amount calculation conditions define a condition that "for the feature amount indicating shoulder width, the distance between the feature points corresponding to both shoulders on the captured image is calculated as the feature amount indicating shoulder width based on the feature point information." In this case, the feature amount calculation unit 14 calculates the distance between both shoulders on the captured image as the feature amount indicating shoulder width from the coordinates of the feature points corresponding to both shoulders based on the feature point information.

[0031] Furthermore, for example, the feature amount calculation conditions define the following condition: "For the feature amount indicating the length of the upper body, the distance from the center point of both eyes to the feature point corresponding to the navel on the captured image is calculated as the feature amount indicating the length of the upper body based on the feature point information." In this case, the feature amount calculation unit 14 calculates the coordinates of the center point of both eyes based on the feature point information, and calculates the distance from the center point of both eyes to the feature point corresponding to the navel on the captured image as the feature amount indicating the length of the upper body.

[0032] Furthermore, for example, the feature amount calculation conditions define the following condition: "For a feature amount indicating the length of the right upper arm (left upper arm), calculate, based on the feature point information, the distance from the feature point corresponding to the right shoulder (left shoulder) to the right elbow (left elbow) on the captured image as the feature amount indicating the length of the right upper arm (left upper arm)." In this case, based on the feature point information, the feature amount calculation unit 14 calculates the distance from the feature point corresponding to the right shoulder (left shoulder) on the captured image to the right elbow (left elbow) as the feature amount indicating the length of the right upper arm (left upper arm).

[0033] Further, for example, the feature amount calculation conditions may define a condition that "with respect to the feature amount indicating body area, the area of ​​a region surrounded by the outline of the occupant in the captured image is calculated as the feature amount indicating body area based on the subject-related information." In this case, the feature amount calculation unit 14 calculates the area of ​​the region surrounded by the outline of the occupant in the captured image based on the subject-related information, and sets the calculated area as the feature amount indicating body area. Note that the feature amount calculation unit 14 may calculate the area of ​​the region surrounded by the outline of the occupant in the captured image using, for example, a known technique for calculating the area of ​​a polygon. Further, for example, the feature amount calculation conditions may define a condition that "with respect to the feature amount indicating body area, the area of ​​a region surrounded by feature points corresponding to the neck, both shoulders, and both waists in the captured image is calculated as the feature amount indicating body area based on the feature point information." In this case, the feature amount calculation unit 14 calculates the area of ​​the region surrounded by feature points corresponding to the neck, both shoulders, and both waists in the captured image based on the feature point information, and sets the area as the feature amount indicating body area.

[0034] The feature amount calculation unit 14 calculates the feature amount for each occupant based on feature point information indicating the feature points of the occupant or on subject-related information. After calculating the feature amount, the feature amount calculation unit 14 outputs information related to the calculated feature amount (hereinafter referred to as "feature amount information") to the state tentative estimation unit 15. The feature amount information includes the feature amount, information that can identify what the feature amount indicates, and information indicating the occupant. The feature amount information may further include the feature point information, the subject-related information, or the acquired captured image.

[0035] Each state tentative estimation unit 15 tentatively estimates the physique of the occupant based on the feature amounts calculated by the feature amount calculation unit 14 and the model. Specifically, the first state tentative estimation unit 151 and the second state tentative estimation unit 152 each tentatively estimate the physique of the occupant based on the feature amounts calculated by the feature amount calculation unit 14 and a model that estimates a human state using different types of feature amounts as input.

[0036] More specifically, the first state tentative estimation unit 151 uses a certain model (hereinafter referred to as the "first model") to input, to the first model, feature quantities (hereinafter referred to as the "first feature quantities") that are to be input to the first model, among the feature quantities calculated by the feature quantity calculation unit 14, to tentatively estimate the occupant's physique. The second state tentative estimation unit 152 uses a certain model (hereinafter referred to as the "second model") to input, to the second model, feature quantities (hereinafter referred to as the "second feature quantities") that are to be input to the second model, among the feature quantities calculated by the feature quantity calculation unit 14, to tentatively estimate the occupant's physique. The first feature quantities and the second feature quantities are different types of feature quantities. In the first embodiment, the feature quantities input to the model are one or more types of feature quantities.

[0037] The first model and the second model are trained in advance. For example, an administrator or the like generates the first model and the second model and stores the generated first model and the second model in a storage unit. In the first embodiment, the first model and the second model may be rule-based models (hereinafter referred to as "rule-based models") that estimate a person's state from feature quantities based on set conditions for provisional state estimation, or may be machine learning models that estimate a person's state using feature quantities as input. The conditions for provisional state estimation are conditions that define the occupant's physical size when the input feature quantities have certain values, and are set in advance by an administrator or the like. The machine learning model is trained to input feature quantities and output information indicating a person's physical size. The models used in the state estimation device 1 may include both rule-based models and machine learning models. For example, the first model may be a rule-based model, and the second model may be a machine learning model.

[0038] Furthermore, the administrator or the like generates the above-mentioned input feature definition information in advance, such as when generating the first model and the second model, and stores it in the storage unit. Here, FIG. 2 is a diagram illustrating an example of the contents of the input feature definition information stored in the storage unit in the first embodiment. The input feature definition information associates a model with a feature to be input to the model. The input feature definition information illustrated in FIG. 2 defines that the first model receives as input a feature indicating eye height, a feature indicating shoulder width, and a feature indicating the length of the left upper arm. That is, the feature indicating eye height, the feature indicating shoulder width, and the feature indicating the length of the left upper arm are the first feature values. Furthermore, the input feature definition information illustrated in FIG. 2 defines that the second model receives as input a feature indicating eye height, a feature indicating shoulder width, a feature indicating the length of the upper body, and a feature indicating the body area. That is, the feature indicating eye height, the feature indicating shoulder width, the feature indicating the length of the upper body, and the feature indicating the body area are the second feature values. In the following, as an example, the input feature definition information will be described as information having the contents shown in FIG.

[0039] By referring to the input feature definition information as shown in FIG. 2 , the feature calculation unit 14 can identify that the features to be calculated are a feature indicating eye height, a feature indicating shoulder width, a feature indicating the length of the left upper arm, a feature indicating the length of the upper body, and a feature indicating body area.

[0040] Each state temporary estimation unit 15 outputs a provisional estimation result of the occupant's physique to the success / failure determination unit 16. More specifically, here, the first state temporary estimation unit 151 outputs a result of the provisional estimation of the occupant's physique (hereinafter referred to as the "first provisional estimation result") to the success / failure determination unit 16. Furthermore, the second state temporary estimation unit 152 outputs a result of the provisional estimation of the occupant's physique (hereinafter referred to as the "second provisional estimation result") to the success / failure determination unit 16. The provisional estimation result includes the result of the provisional estimation of the occupant's physique by the state temporary estimation unit 15, information indicating the occupant, information (e.g., ID) that can identify the model used to provisionally estimate the occupant's physique, and feature amounts input to the model at the time of the provisional estimation. The provisional estimation result may further include feature amounts input to the model. Note that the features input to the model, in other words, the features defined in the input feature definition information, are not necessarily the same as the features input to the model during provisional estimation. For example, there are models, such as decision trees, that can perform provisional estimation of the occupant's physique even if one of the features input to the model is missing. If such a model has a missing feature, the feature input to the model may differ from the feature actually input to the model when provisional estimation is performed. A possible cause of a missing feature is that a feature point on the captured image that is the basis for calculating the feature was not detected.

[0041] The success / failure determination unit 16 determines the success / failure of the provisional estimation of the occupant's physique by each state provisional estimation unit 15. More specifically, here, the success / failure determination unit 16 determines the success / failure of the provisional estimation of the occupant's physique by the first state provisional estimation unit 151 and the success / failure of the provisional estimation of the occupant's physique by the second state provisional estimation unit 152.

[0042] In the first embodiment, the success / failure determination performed by the success / failure determination unit 16 includes a "first success / failure determination" in which each state provisional estimation unit 15 determines whether the provisional estimation of the occupant's physique has been successful or unsuccessful based on whether the provisional estimation of the occupant's physique was able to be performed without a model execution error, and a "second success / failure determination" in which each state provisional estimation unit 15 determines whether the provisional estimation of the occupant's physique has been successful or unsuccessful based on whether the reliability of the provisionally estimated occupant's physique is sufficient to allow it to be used to determine the estimated state of the occupant, as a result of performing a provisional estimation of the occupant's physique using a model.

[0043] In the first success / failure determination process, the success / failure determination unit 16 determines whether the provisional estimation of the occupant's physique by each state provisional estimation unit 15 was successful or failed, depending on whether error information was output from each state provisional estimation unit 15. A model (here, the first model or the second model) typically cannot perform estimation (here, provisional estimation of the occupant's physique) unless all input features are available, except for models that can perform provisional estimation of the occupant's physique even if any of the input features is missing, such as the decision tree described above. That is, a model typically requires all input features to be available for estimation. Such models are designed or trained to output information indicating an execution error (hereinafter referred to as "error information") when an input feature is missing, i.e., when all features are not available. Note that, for example, in a model that can perform provisional estimation of the occupant's physique even if any of one or more input features is missing, such as a decision tree, error information is not output.

[0044] When error information is output from a model, each state temporary estimation unit 15 outputs the error information to the success / failure determination unit 16. For example, assume that error information is output from the first model. In this case, the first state temporary estimation unit 151 outputs the error information to the success / failure determination unit 16. Note that the first state temporary estimation unit 151 is unable to perform a temporary estimation of the occupant's physique, and therefore does not output a first temporary estimation result. On the other hand, assume that no error information is output from the second model. In this case, the second state temporary estimation unit 152 does not output error information to the success / failure determination unit 16. The second state temporary estimation unit 152 outputs a second temporary estimation result to the success / failure determination unit 16. The success / failure determination unit 16 determines that the temporary estimation of the occupant's physique by the first state temporary estimation unit 151 has failed because error information was output from the first state temporary estimation unit 151. On the other hand, in the first success / failure determination, the success / failure determination unit 16 determines that the second state tentative estimation unit 152 has successfully tentatively estimated the occupant's physique.

[0045] In the second success / failure determination process, the success / failure determination unit 16 determines whether the provisional estimation of the occupant's physique by each state provisional estimation unit 15 was successful or unsuccessful by comparing the reliability of the occupant's physique provisionally estimated by the state provisional estimation unit 15 with a preset threshold (hereinafter referred to as the "provisional estimation result reliability determination threshold") based on the provisional estimation results output from each state provisional estimation unit 15. The provisional estimation result reliability determination threshold is set in advance by an administrator or the like and stored in the storage unit. Note that the provisional estimation result reliability determination threshold may be a different value for each state provisional estimation unit 15, in other words, for each model.

[0046] In the first embodiment, the model is assumed to have been trained to output, for example, information indicating the estimated occupant's physique, along with a reliability indicating how reliable the estimated occupant's physique is. Each state provisional estimation unit 15 assigns the reliability output by the model to a provisional estimation result and outputs the result to the success / failure determination unit 16. For example, if the reliability is equal to or greater than a threshold for determining the reliability of the provisional estimation result, the success / failure determination unit 16 determines that the occupant's physique indicated by the provisional estimation result to which the reliability is assigned is reliable, in other words, that it is safe to use the provisional estimation result to determine the estimated state of the occupant. In other words, if the reliability is equal to or greater than the threshold for determining the reliability of the provisional estimation result, the success / failure determination unit 16 determines that the state provisional estimation unit 15 that output the provisional estimation result to which the reliability is assigned has successfully provisionally estimated the occupant's physique. On the other hand, if the reliability is less than the threshold for determining the reliability of the provisional estimation result, the success / failure determination unit 16 determines that the state provisional estimation unit 15, which output the provisional estimation result to which the reliability is assigned, failed to provisionally estimate the occupant's physique.

[0047] For example, suppose the second state provisional estimation unit 152 outputs a second provisional estimation result, and the reliability assigned to the output second provisional estimation result is equal to or greater than the provisional estimation result reliability determination threshold. In this case, the success / failure determination unit 16 determines that the occupant's physique indicated by the second provisional estimation result is sufficiently reliable, and the second state provisional estimation unit 152 determines that the provisional estimation of the occupant's physique has been successful. For example, a model that can provisionally estimate the occupant's physique even if any of the input feature values ​​is missing, such as the decision tree described above, may be able to provisionally estimate the occupant's physique, but the missing feature value may reduce the accuracy of the provisional estimation. In other words, the reliability of the provisionally estimated occupant's physique may be low. By performing the second success / failure determination process, the success / failure determination unit 16 prevents the integrated estimation unit 17 from determining the estimated state using a provisional estimation result with low accuracy, or in other words, low reliability, when determining the estimated state. This prevents a decrease in the accuracy of the occupant's physique as an estimation result. The integrated estimation unit 17 will be described in detail later.

[0048] The success / failure determination unit 16 outputs information indicating whether each state provisional estimation unit 15 has determined that the provisional estimation of the occupant's physique was successful or unsuccessful as a result of the success / failure determination (hereinafter referred to as the "success / failure determination result") to the integrated estimation unit 17, together with the provisional estimation results output from each state provisional estimation unit 15. The success / failure determination result includes the first success / failure determination process result or the second success / failure determination process result. Note that if error information is output from a state provisional estimation unit 15, the provisional estimation result is not output from that state provisional estimation unit 15. In this case, the success / failure determination unit 16 may output the success / failure determination result to the integrated estimation unit 17 together with the error information.

[0049] The integrated estimation unit 17 determines the estimated state of the occupant based on the provisional estimation results of the occupant's physique by each state provisional estimation unit 15, which are determined by the success / failure determination unit 16 as having been successful in provisionally estimating the occupant's physique, and on the state determination conditions. Here, the estimated state refers to the occupant's physique that the state estimation device 1 ultimately estimates. Note that the determination by the success / failure determination unit 16 that the state provisional estimation unit 15 has been successful in provisionally estimating the occupant's physique means that both the first success / failure determination and the second success / failure determination have been successful. The integrated estimation unit 17 rejects the provisional estimation results for which the success / failure determination unit 16 has determined that the state provisional estimation unit 15 has failed in provisionally estimating the occupant's physique. In the following description, as an example, the details of the integrated estimation unit 17 will be described assuming that the success / failure determination unit 16 has determined that both the first state temporary estimation unit 151 and the second state temporary estimation unit 152 have successfully made temporary estimations of the occupant's physique. That is, the details of the integrated estimation unit 17 will be described assuming that the integrated estimation unit 17 determines an estimated state of the occupant based on the first and second temporary estimation results and the state determination conditions. Note that the integrated estimation unit 17 determines an estimated state for each occupant.

[0050] The state determination conditions are set conditions for determining how to ultimately determine an estimated state from multiple provisional estimation results by each state provisional estimation unit 15, in this case, how to determine the physique of the occupant ultimately estimated by the state estimation device 1. The state determination conditions are set in advance by an administrator or the like, and information indicating the state determination conditions (hereinafter referred to as "state determination condition information") is stored in the storage unit. For example, the state determination conditions may be changed as appropriate by the administrator or the like. An example of the state determination conditions and an example of a method for determining an estimated state of an occupant based on the provisional estimation results by the integrated estimation unit 17 and the state determination conditions will be described below using specific examples.

[0051] <Example of State Determination Condition (1): Majority Decision> For example, the state determination condition may include a condition that "the estimated state of the occupant is determined by majority decision using the most common occupant physique among the occupant physiques indicated by the multiple provisional estimation results." This condition is also referred to as a "first determination condition." In this case, the integrated estimation unit 17 determines the estimated state of the occupant based on the first provisional estimation result by the first state provisional estimation unit 151, the second provisional estimation result by the second state provisional estimation unit 152, and the first determination condition. In addition to the condition that "the estimated state of the occupant is determined by majority decision using the most common occupant physique among the occupant physiques indicated by the multiple provisional estimation results," a tie result selection condition may also be set in the first determination condition. The tie result selection condition defines how to determine the estimated state when there are multiple occupant physiques that are the most common among the occupant physiques indicated by the multiple provisional estimation results. The tie result selection condition may be, for example, "determine the occupant's physique indicated by any provisional estimation result as the occupant's estimated state." In this case, the integrated estimation unit 17 determines the occupant's physique from the most common occupant physique among the multiple provisional estimation results as the occupant's estimated state. The tie result selection condition may also be, for example, "if the occupant's physique estimated by the state estimation device 1 is expressed as a continuous value, determine the average value of the most common occupant physique as the estimated state." In this case, the integrated estimation unit 17 determines the occupant's estimated state as the average value of the most common occupant physique among the multiple provisional estimation results. Note that the tie result selection condition described above is merely an example, and the tie result selection condition may be any appropriate condition. The tie result selection condition may also be changed as appropriate by an administrator or the like.

[0052] The state determination conditions include a first determination condition that the estimated state of the occupant is determined by majority vote to adopt the most common occupant physique among the occupant physiques indicated by the multiple provisional estimation results, and the integrated estimation unit 17 determines the most common occupant physique among the occupant physiques indicated by the multiple provisional estimation results as the estimated state of the occupant based on the multiple provisional estimation results of the occupant physique by the multiple state provisional estimation units 15 and the first determination condition, thereby allowing the state estimation device 1 to perform highly accurate estimation of the occupant physique taking into account the provisional estimation results of the occupant physique based on the multiple models that can currently be used to estimate the occupant physique.

[0053] <Example of State Determination Condition (2): Average> For example, the state determination condition may include a condition that states, "When the occupant's physique estimated by the state estimation device 1 is expressed as a continuous value, the average value of the occupant's physique indicated by the multiple provisional estimation results is determined as the estimated state of the occupant." This condition is also referred to as a "second determination condition." In this case, the integrated estimation unit 17 determines, as the estimated state of the occupant, the occupant's physique indicated by the first provisional estimation result by the first state provisional estimation unit 151, the second provisional estimation result by the second state provisional estimation unit 152, and the second determination condition. The average value may be a median. Even if the values ​​are discrete values, values ​​that have an ordered relationship may be treated as continuous values. For example, by assigning numerical values ​​to physique classifications in ascending order, such as 1 for child, 2 for small, 3 for medium, and 4 for large, and calculating the median instead of the average value, an average physique can be calculated equivalent to a continuous value.

[0054] The occupant's physique is represented by a continuous value, and the state determination conditions include a second determination condition that the average value of the occupant's physique indicated by the multiple provisional estimation results is taken as the estimated state of the occupant.The integrated estimation unit 17 determines the average value of the occupant's physique indicated by the multiple provisional estimation results as the estimated state of the occupant based on the multiple provisional estimation results of the occupant's physique by the multiple state provisional estimation units 15 and the second determination condition.This allows the state estimation device 1 to perform highly accurate estimation of the occupant's physique, taking into account the provisional estimation results of the occupant's physique based on the multiple models that are currently available for estimating the occupant's physique.

[0055] <Example of State Determination Condition (3): Determined Ranking> For example, the state determination condition may include a condition that "the occupant's physique indicated by the provisional estimation result provisionally estimated based on the model with the highest assigned rank (hereinafter referred to as the "model rank") is determined as the estimated state of the occupant." This condition is also referred to as a "third determination condition." The integrated estimation unit 17 determines the estimated state based on the first provisional estimation result by the first state provisional estimation unit 151, the second provisional estimation result by the second state provisional estimation unit 152, and the third determination condition. For example, it is assumed that each model (here, the first model and the second model) has been ranked in advance by an administrator or the like. For example, the administrator or the like sets the model ranking so that a model estimated to be able to perform a more reliable estimation has a higher model ranking. For example, the administrator or the like associates the model ranking of each model in the input feature definition information. Note that information indicating the model ranking of each model is omitted from the input feature definition information shown in FIG. 2. The integrated estimation unit 17 can determine the model ranking of each model by, for example, referring to the input feature definition information. Furthermore, the integrated estimation unit 17 can determine, from the provisional estimation result output via the success / failure determination unit 16, which model the provisional estimation result was based on. As described above, the provisional estimation result includes information that can identify the model used to provisionally estimate the occupant's physique. For example, assume that the model ranking of the first model is "first place" and the model ranking of the second model is "second place." In this case, the integrated estimation unit 17 determines the occupant's physique indicated in the first provisional estimation result by the first state provisional estimation unit 151, which is provisionally estimated based on the first model, as the estimated state of the occupant.

[0056] The state determination conditions include a third determination condition that determines the occupant's estimated state to be the occupant's physique indicated by the provisional estimation result provisionally estimated based on the model with the highest assigned model ranking, and the integrated estimation unit 17 determines the occupant's estimated state to be the occupant's physique indicated by the provisional estimation result provisionally estimated based on the model with the highest assigned model ranking, based on the multiple provisional estimation results of the occupant's physique by the multiple state provisional estimation units 15 and the third determination condition.This allows the state estimation device 1 to determine the occupant's physique that is estimated to be most likely from the occupant's physiques estimated based on the multiple models that are currently available for estimating the occupant's physique as the final occupant's physique, and to perform highly accurate occupant physique estimation.

[0057] <Example of State Determination Condition (4): Priority> For example, the state determination conditions may include a condition that "the estimated state of the occupant is determined based on the priorities of the multiple provisional estimation results determined according to the priority determination conditions" and a priority-state association condition that defines how the estimated state of the occupant is determined based on the determined priorities. This state determination condition may also be referred to as a "fourth determination condition." Note that the priority-state association condition may be uniquely determined in advance and stored, for example, in an internal buffer of the integrated estimation unit 17. In this case, the priority-state association condition may not be included in the fourth determination condition. The priority-state association condition will be described later with a specific example. The integrated estimation unit 17 determines the priorities of the multiple provisional estimation results (the first and second provisional estimation results) based on the first provisional estimation result by the first state provisional estimation unit 151, the second provisional estimation result by the second state provisional estimation unit 152, and the fourth determination condition, and determines the estimated state of the occupant based on the determined priorities. For example, the priority determination conditions are set in advance by an administrator or the like, and information indicating the priority determination conditions (hereinafter referred to as "priority determination condition information") is stored in a storage unit. For example, the administrator or the like may include the priority determination conditions in the fourth determination conditions.

[0058] The state determination conditions include a fourth determination condition that determines the estimated state of the occupant based on the priority of the multiple provisional estimation results determined in accordance with the priority determination conditions, and the integrated estimation unit 17 determines the priority of the multiple provisional estimation results of the occupant's physique based on the multiple provisional estimation results of the occupant's physique by the multiple state provisional estimation units 15 and the fourth determination condition, and determines the estimated state of the occupant based on the determined priority.By this, the state estimation device 1 can determine the final occupant's physique by taking into account the occupant's physique that should be prioritized from among the occupant's physiques estimated based on the multiple models that are currently available for estimating the occupant's physique, and can perform highly accurate occupant physique estimation.

[0059] Here, the priority determination conditions will be explained using some specific examples.

[0060] <Example of Priority Determination Condition (1)> For example, the priority determination condition may include a condition that "the more features a model has as input, the higher the priority is assigned to the provisional estimation result." In the first embodiment, a large number of input features refers to, for example, a large number of types of features. Here, the features input by the first model, in other words, the first features, are three features indicating eye height, shoulder width, and left upper arm length. The features input by the second model, in other words, the second features, are four features indicating eye height, shoulder width, upper body length, and body area (see FIG. 2 ). The number of second features is greater than the number of first features. Therefore, the integrated estimation unit 17 prioritizes the second provisional estimation result estimated by the second state provisional estimation unit 152 based on the second model that uses the larger number of second feature values ​​as input, over the first provisional estimation result estimated by the first state provisional estimation unit 151 based on the first model that uses the first feature values ​​as input. Note that the provisional estimation result includes information that can identify the model used to provisionally estimate the occupant's physique. The integrated estimation unit 17 can identify the feature values ​​input to the first model and the second model based on, for example, the provisional estimation result and the input feature value definition information stored in the storage unit.

[0061] Generally, when performing estimation based on a model, the use of a model with a larger number of input feature quantities results in higher estimation accuracy. The priority determination conditions include a condition that the higher the priority of a provisional estimation result obtained based on a model with a larger number of input feature quantities, and the estimated state of the occupant is determined based on this priority. This allows the state estimation device 1 to estimate the occupant's physique using the model with the highest accuracy among multiple models currently available for estimating the occupant's physique.

[0062] <Example of Priority Determination Condition (2)> Furthermore, for example, the priority determination condition may include a condition that "the more feature quantities used in provisionally estimating the occupant's physique among the input feature quantities, the higher the priority of the provisional estimation result provisionally estimated based on the model." For example, it is assumed that the first state provisional estimation unit 151 inputs all of the feature quantities input to the first model (feature quantities indicating eye height, feature quantities indicating shoulder width, and feature quantities indicating the length of the left upper arm) to the first model and provisionally estimates the occupant's physique. On the other hand, it is assumed that the second state provisional estimation unit 152 inputs only the feature quantities indicating eye height and feature quantities indicating shoulder width to the second model because the feature quantities indicating the upper body and the feature quantities indicating the body area are missing from the feature quantities input to the second model (feature quantities indicating eye height, feature quantities indicating shoulder width, feature quantities indicating the length of the upper body, and feature quantities indicating the body area). As described above, there are models that can input missing values, such as decision trees, in other words, models that can perform estimation even when input features are missing. In this case, the first model uses a larger number of features to provisionally estimate the occupant's physique than the second model. The integrated estimation unit 17 prioritizes the first provisional estimation result, which is provisionally estimated by the first state provisional estimation unit 151 based on the first model that used a larger number of features to provisionally estimate the occupant's physique, over the second provisional estimation result, which is estimated by the second state provisional estimation unit 152 based on the second model.

[0063] As described above, some models can perform estimation even when the input feature values ​​are missing values ​​(invalid feature values). However, if there are many missing values, the estimation accuracy may decrease. The priority determination conditions include a condition that the provisional estimation result obtained based on a model having a larger number of input feature values ​​used to provisionally estimate the occupant's physique is given a higher priority. By determining the estimated state of the occupant based on this priority, the state estimation device 1 can perform highly accurate estimation of the occupant's physique, taking into account the fact that the model used to estimate the occupant's physique is missing input feature values.

[0064] For example, in addition to the condition that "the provisional estimation result obtained based on a model having a larger number of input feature quantities used to provisionally estimate the occupant's physique is given a higher priority" in the above-described priority determination conditions, a tie model selection condition may be set. The tie model selection condition defines which model should have a higher priority when multiple models have the same number of input feature quantities used to provisionally estimate the occupant's physique. For example, the tie model selection condition may include the condition that "when multiple models have the same number of input feature quantities used to provisionally estimate the occupant's physique, the provisional estimation result obtained based on the model having a larger ratio of the number of input feature quantities to the number of input feature quantities is given a higher priority." Note that the above-described tie model selection condition is merely an example, and the tie model selection condition can be any appropriate condition. The tie model selection condition may also be changed as appropriate by an administrator or the like.

[0065] <Example of Priority Determination Condition (3)> Furthermore, for example, the priority determination condition may include a condition that "a provisional estimation result provisionally estimated based on a model whose input feature quantities include many feature quantities that have a high degree of contribution to the provisional estimation result is given a higher priority" and a contribution determination condition. The contribution determination condition is a condition for determining whether the model whose input feature quantities include many feature quantities that have a high degree of contribution to the provisional estimation result is a model whose input feature quantities include many feature quantities that have a high degree of contribution to the provisional estimation result.

[0066] For example, the contribution of the feature values ​​input to each model to the provisional estimation result is set in advance. The "contribution" is an index indicating how much the feature value affects the objective variable, in this case, the physique of the provisionally estimated occupant. In the first embodiment, the contribution is expressed, for example, by a numerical value ranging from "0" to "1." For example, when the objective variable is a continuous value, the contribution may be a correlation coefficient indicating the degree of similarity between the explanatory variable and the objective variable.

[0067] For example, an administrator or the like may prepare in advance a machine learning model capable of outputting contributions (hereinafter referred to as a "contribution setting model"). The contribution setting model is a machine learning model for estimating a person's physique. The contribution setting model receives multiple feature values ​​as input and is trained to output information indicating the person's physique and the contribution of each of the input feature values ​​to the estimation result. For example, before generating the first model and the second model, the administrator or the like inputs all feature values ​​into the contribution setting model and obtains the contributions of all the feature values. The administrator or the like then generates information associating the feature values ​​with the contributions (hereinafter referred to as "contribution information") and stores the information in a storage unit. For example, assume that the contribution setting model is used to obtain the following contributions: a feature value indicating eye height of "0.9," a feature value indicating shoulder width of "0.8," a feature value indicating left upper arm length of "0.3," a feature value indicating upper body length of "0.6," and a feature value indicating body area of ​​"0.7." In this case, contribution information is generated and stored in which the feature indicating eye height and contribution rate of "0.9", the feature indicating shoulder width and contribution rate of "0.8", the feature indicating the length of the left upper arm and contribution rate of "0.3", the feature indicating the length of the upper body and contribution rate of "0.6", and the feature indicating the body area and contribution rate of "0.7" are associated with each other.

[0068] Furthermore, for example, when generating the first model and the second model, the administrator or the like may acquire the contribution of each feature obtained by the first model and the second model through learning, and generate contribution information. For example, if the first model or the second model is a model that learns to output the contribution of each feature input during learning, the administrator or the like can generate contribution information from the contribution output by the model. Alternatively, the administrator or the like can acquire the contribution of each feature input to the first model and the second model using a known method for estimating the contribution of each feature input from the model's estimation results, and generate contribution information from the acquired contribution. For example, when generating the first model, assume that the contribution of the feature indicating eye height, the contribution of the feature indicating shoulder width, and the contribution of the feature indicating the length of the left upper arm, input by the first model, are obtained as "0.9," "1," and "0.3." On the other hand, when generating the second model, the contribution rate of the feature indicating eye height, the contribution rate of the feature indicating shoulder width, the contribution rate of the feature indicating upper body length, and the contribution rate of the feature indicating body area, which are input to the second model, are assumed to be "0.9," "0.6," "0.6," and "0.7." For overlapping feature values, the administrator or the like may, for example, determine the average of the corresponding contribution rates as the contribution rate of that feature value. In the above example, the administrator or the like may determine the contribution rate of the feature indicating shoulder width to be "0.8." In the above example, contribution rate information is generated and stored, in which the feature indicating eye height and the contribution rate of "0.9," the feature indicating shoulder width and the contribution rate of "0.8," the feature indicating left upper arm length and the contribution rate of "0.3," the feature indicating upper body length and the contribution rate of "0.6," and the feature indicating body area and the contribution rate of "0.7" are associated with each other.

[0069] The integrated estimation unit 17 determines the features input to the first model and the second model and the contributions corresponding to the features, for example, from the input feature definition information and the contribution information. Then, the integrated estimation unit 17 determines, in accordance with the contribution determination conditions, whether the input features of the first model and the second model include many features that have a high contribution to the provisional estimation result.

[0070] As a specific example, the contribution determination condition may be set to the following condition: "A contribution equal to or greater than a threshold (hereinafter referred to as the "contribution determination threshold") is considered to be a 'high' contribution, a contribution below the contribution determination threshold is considered to be a 'low' contribution, and the more feature quantities associated with a 'high' contribution, the more feature quantities that contribute highly to the provisional estimation result are included in the input feature quantities." The contribution determination threshold is set to 0.7. The contribution information may also include a feature quantity indicating eye height with a contribution of 0.9, a feature quantity indicating shoulder width with a contribution of 0.8, a feature quantity indicating left upper arm length with a contribution of 0.3, a feature quantity indicating upper body length with a contribution of 0.6, and a feature quantity indicating body area with a contribution of 0.7. In this case, the integrated estimation unit 17 determines, in accordance with the contribution determination condition, that the feature quantity indicating eye height, the feature quantity indicating shoulder width, and the feature quantity indicating body area have a "high" contribution. Then, the integrated estimation unit 17 assigns a higher priority to the second provisional estimation result provisionally estimated by the second state provisional estimation unit 152 based on the second model, which includes three features with a "high" contribution level among the input features, than to the first provisional estimation result provisionally estimated by the first state provisional estimation unit 151 based on the first model, which includes two features with a "high" contribution level.

[0071] The above-described contribution determination conditions are merely examples. The contribution determination conditions may be appropriately set by an administrator or the like. For example, the contribution determination conditions may include a condition such that “feature values ​​calculated from feature points that can be detected more stably are considered to have a high contribution.” More stably detectable feature points are selected in advance by an administrator or the like from the perspective of feature point detection accuracy or robustness, and information indicating the selected more stably detectable feature points (hereinafter referred to as “stable feature point information”) is stored in a storage unit. For example, when calculating feature values, the feature calculation unit 14 refers to the stable feature point information and includes information indicating whether the feature value is calculated based on more stably detectable feature points in the feature value information. Based on the feature value information, the integrated estimation unit 17 can determine whether the feature value is calculated from more stably detectable feature points. The contribution determination conditions may be appropriately changed by an administrator or the like. Also, here, the contribution degree is a numerical value between "0" and "1", but if the contribution degree is expressed as a binary value of, for example, "high" or "low", there is no need to set the above-mentioned conditions for determining the contribution degree.

[0072] The priority determination conditions include a condition that a provisional estimation result tentatively estimated based on a model that includes more input feature quantities that have a higher contribution to the provisional estimation result is given a higher priority, and the estimated state of the occupant is determined based on this priority.By doing so, the state estimation device 1 can perform a highly accurate estimation of the occupant's physique, taking into account the provisional estimation result of the occupant's physique based on a model that uses more important feature quantities as input.

[0073] <Example of Priority Determination Condition (4)> Furthermore, for example, the priority determination condition may be set as follows: "The wider the range of positions on the reference body that can be associated with body parts indicated by feature points (also referred to as "calculation source feature points") that were used to calculate feature amounts input to the model, the higher the priority of a provisional estimation result provisionally estimated based on a model that uses feature amounts based on the calculation source feature points as input." Note that in the first embodiment, the "reference body" is assumed to be the body of a standard adult. For example, information indicating the positional relationships between body parts indicated by each feature point on the reference body (hereinafter referred to as "part positional relationship information") is generated in advance by an administrator or the like and stored in the storage unit. The integrated estimation unit 17 refers to the part positional relationship information to determine the range of positions on the reference body that can be associated with body parts indicated by the calculation source feature points.

[0074] For example, the first model receives inputs of a feature indicating eye height, a feature indicating shoulder width, and a feature indicating waist width, and provisionally estimates the occupant's physique. The second model receives inputs of a feature indicating shoulder width, a feature indicating neck length, and a feature indicating upper body length, and provisionally estimates the occupant's physique. For convenience, the feature quantities input by the first model and the second model are assumed to be different from those defined in the input feature quantity definition information shown in FIG. 2 . The feature quantity calculation conditions are assumed to be the same as those in the above example. The feature quantity calculation conditions also define that the feature quantity indicating waist width is calculated from the distance between feature points corresponding to both waists, and that the feature quantity indicating neck length is calculated from the distance from the center of both eyes to the feature point corresponding to the neck. In this case, the first state provisional estimation unit 151 inputs all of the feature quantities input by the first model (the feature quantity indicating eye height, the feature quantity indicating shoulder width, and the feature quantity indicating waist width) into the first model, and provisionally estimates the occupant's physique. The feature points from which the feature amount indicating eye height is calculated are the feature points corresponding to both eyes and both waists, the feature points from which the feature amount indicating shoulder width is calculated are the feature points corresponding to both shoulders, and the feature amount indicating waist width are the feature points corresponding to both waists. Furthermore, the second state provisional estimation unit 152 inputs all of the feature amounts input to the second model (the feature amount indicating shoulder width, the feature amount indicating neck length, and the feature amount indicating upper body length) to the second model, and provisionally estimates the physique of the occupant. The feature points from which the feature amount indicating shoulder width is calculated are the feature points corresponding to both shoulders, the feature points from which the feature amount indicating neck length is calculated are the feature points corresponding to both eyes and the neck, and the feature points from which the feature amount indicating upper body length is calculated are the feature points corresponding to both eyes and the navel.

[0075] In this case, the body parts indicated by the feature points used to calculate the feature amounts input to the first model are the eyes, shoulders, and hips, and the range of positions associated with these points on the reference body (hereinafter referred to as the "first model position range") is the range surrounded by the eyes, shoulders, and hips. On the other hand, the body parts indicated by the feature points used to calculate the feature amounts input to the second model are the eyes, shoulders, neck, and navel, and the range of positions associated with these points on the reference body (hereinafter referred to as the "second model position range") is the range surrounded by the eyes, shoulders, and navel. The second model position range is wider than the first model position range. Therefore, the integrated estimation unit 17 prioritizes the second provisional estimation result provisionally estimated by the second state provisional estimation unit 152 based on the second model higher than the first provisional estimation result provisionally estimated by the first state provisional estimation unit 151 based on the first model.

[0076] For example, the above-described priority determination conditions may include a condition that, in addition to "the wider the range of positions on the reference body to which a body part indicated by a calculation source feature point can be associated, the higher the priority of a provisional estimation result provisionally estimated based on a model that uses the calculation source feature point as input," "the wider the range of positions on the reference body to which a body part indicated by a calculation source feature point can be associated means that the range of positions on the reference body to which a body part indicated by a calculation source feature point can be associated is wide in the vertical direction." For example, if the body parts indicated by the calculation source feature points of feature amounts input to a first model are both eyes, both shoulders, and the left elbow, and the body parts indicated by the calculation source feature points of feature amounts input to a second model are both eyes, both shoulders, the neck, and the navel, the vertical range of the first model position range is the range from the eyes to the elbows, and the vertical range of the second model position range is the range from the eyes to the navel. In terms of vertical width, the first model position range is wider. Therefore, in this case, the integrated estimation unit 17 assigns a higher priority to the first provisional estimation result provisionally estimated by the first state provisional estimation unit 151 based on the first model than to the second provisional estimation result provisionally estimated by the second state provisional estimation unit 152 based on the second model.

[0077] Furthermore, for example, the above-described priority determination conditions may include a condition that "the wider the range of positions on the reference body to which a body part indicated by a calculation source feature point can be associated, the higher the priority of a provisional estimation result provisionally estimated based on a model that uses the calculation source feature point as an input," in addition to "the wider the range of positions on the reference body to which a body part indicated by a calculation source feature point can be associated, the wider the range of positions on the reference body to which the body part indicated by the calculation source feature point can be associated in the horizontal direction." Furthermore, for example, the above-described priority determination conditions may include a condition that "the wider the range of positions on the reference body to which a body part indicated by a calculation source feature point can be associated, the higher the priority of a provisional estimation result provisionally estimated based on a model that uses the calculation source feature point as an input, the wider the range of positions on the reference body to which the body part indicated by the calculation source feature point can be associated, the larger the area of ​​the range of positions on the reference body to which the body part indicated by the calculation source feature point can be associated."

[0078] Because physique estimation estimates the characteristics (physique) of the entire body from feature amounts corresponding to parts of the body, the closer the feature amounts used for estimation are to the entire body, the higher the estimation accuracy. The priority determination conditions include a condition that the wider the range of positions on the reference body to which body parts indicated by calculation source feature points, which are feature points from which feature amounts input to the model are calculated, the higher the priority of the provisional estimation result provisionally estimated based on a model that inputs feature amounts based on the calculation source feature points. By determining the estimated state of the occupant based on this priority, the state estimation device 1 can perform highly accurate occupant physique estimation, giving priority to the provisional estimation result of the occupant physique based on a model that inputs feature amounts calculated from feature points in a wider range.

[0079] Furthermore, in physique estimation, feature quantities indicating vertical length are particularly important, because differences in physique are likely to appear in vertical length. In the priority determination conditions, "the range of positions on the reference body to which the body parts indicated by the calculation source feature points correspond is wide" means that the range of positions on the reference body to which the body parts indicated by the calculation source feature points correspond is wide in the vertical direction. This enables the state estimation device 1 to perform more accurate occupant physique estimation by prioritizing provisional estimation results of the occupant physique based on a model that uses feature quantities calculated from feature points in a wider range as input.

[0080] <Example of Priority Determination Condition (5)> Furthermore, for example, the priority determination condition may include a condition that states, "The more feature quantities with high corresponding reliability (hereinafter referred to as "feature quantity reliability") are included in one or more feature quantities used as input to a model, the higher the priority of a provisional estimation result provisionally estimated based on the model," and a feature quantity reliability determination condition. The feature quantity reliability determination condition is a condition for determining whether the feature quantity reliability is high or low. For example, the feature quantity reliability determination condition may include a condition that states, "If the feature quantity reliability is equal to or greater than a threshold value (hereinafter referred to as "feature quantity reliability determination threshold value"), the feature quantity reliability is determined to be high."

[0081] For example, an administrator or the like may set a corresponding feature reliability for each feature input to each model (here, the first model and the second model). The administrator or the like generates information (hereinafter referred to as "feature reliability information") associating features with feature reliability and stores it in a storage unit. The administrator or the like may set the feature reliability, for example, as a continuous value from "0" to "1" or as a binary value such as "high" or "low." Note that if the feature reliability is set as a binary value such as "high" or "low," the above-described feature reliability determination conditions do not need to be set. For example, the administrator or the like may set a low feature reliability for feature values ​​that are expected to be poorly calculated and a high feature reliability for feature values ​​that are expected to be well calculated. For example, if one shoulder (e.g., the right shoulder) is expected to be far from the image capture device 2, that shoulder may appear dark in the captured image, and the feature point corresponding to that shoulder may not be detected. In this case, it is assumed that the feature amount based on the feature point corresponding to one shoulder will not be calculated properly. In this case, the administrator or the like sets a low value or "low" for the feature amount reliability of the feature amount calculated based on the feature point corresponding to one shoulder (for example, the feature amount indicating the right upper arm).

[0082] The integrated estimation unit 17 determines the number of feature quantities with "high" feature reliability among the feature quantities input to the first model and the second model, and then assigns a higher priority to a provisional estimation result tentatively estimated based on a model that inputs a larger number of feature quantities with "high" feature reliability.

[0083] In the above example, the feature reliability is set in advance by an administrator or the like, but this is merely an example. For example, conditions for calculating the feature reliability may be set as appropriate, and the integrated estimation unit 17 may calculate the feature reliability according to the conditions. Specifically, the above-mentioned priority determination conditions may include the condition that "the more feature quantities with high corresponding feature reliability are included in one or more feature quantities used as input to the model, the higher the priority of the provisional estimation result provisionally estimated based on the model" and the feature reliability determination conditions, as well as the condition that "the feature reliability is calculated based on the feature reliability calculation conditions."

[0084] The feature reliability calculation condition defines a calculation method for the feature reliability. For example, the feature reliability calculation condition may include a condition stating that "the feature reliability is calculated based on the reliability of the feature points (hereinafter referred to as "feature point reliability") in accordance with a feature reliability calculation rule." In this case, the feature point detection model is a machine learning model that receives a captured image as input and outputs information about the feature points and feature point reliability. The feature point reliability of each feature point is stored in the storage unit by the feature point detection unit 13. The integrated estimation unit 17 calculates the feature reliability from the feature point reliability of each feature point stored in the storage unit in accordance with the feature reliability calculation rule. The feature reliability calculation rule is generated in advance by an administrator or the like and stored in the storage unit. For example, the feature reliability calculation rule may include a rule stating that "the feature reliability of a feature indicating shoulder width is calculated as the average value of the feature point reliability of the feature points corresponding to both shoulders."

[0085] The feature reliability calculation conditions may include, for example, the following condition: "The feature reliability is calculated in accordance with a feature reliability calculation rule based on the detection status of the feature point, which is the calculation source of the feature input to the model, during a predetermined period in the past (hereinafter referred to as the "detection status determination period")." In this case, the feature point detection unit 13 assigns the output date and time of the feature point information to the feature point information and stores the information in the storage unit. In this case, the feature reliability calculation rule includes, for example, a rule that determines the level of feature reliability of a feature based on a feature point based on the detection status of the feature point, such as, "If the detection position of a feature point varies or is lost, the feature reliability of the feature calculated based on the feature point is determined to be 'low'."

[0086] The priority determination conditions include a condition that the more features that are input to the model and that have a high corresponding feature reliability, the higher the priority of the provisional estimation result provisionally estimated based on that model.By determining the estimated state of the occupant based on that priority, the state estimation device 1 does not prioritize the provisional estimation result of the occupant's physique that is based on a model that inputs features that are estimated to be poorly calculated, and can reduce the probability of making an incorrect physique estimation.

[0087] <Example of Priority Determination Condition (6)> Furthermore, for example, the priority determination condition may be set to a condition that "when the model includes a rule-based model and a machine learning model, the priority of a provisional estimation result of the occupant's physique based on the rule-based model is set higher than the priority of a provisional estimation result of the occupant's physique based on the machine learning model." In this case, the integrated estimation unit 17 sets a higher priority of the provisional estimation result of the occupant's physique based on the rule-based model by the state temporary estimation unit 15 than the priority of the provisional estimation result of the occupant's physique based on the machine learning model by the state temporary estimation unit 15. For example, if the first model is a rule-based model and the second model is a machine learning model, the integrated estimation unit 17 sets a higher priority of the first provisional estimation result tentatively estimated by the first state temporary estimation unit 151 based on the first model than the priority of the second provisional estimation result estimated by the second state temporary estimation unit 152 based on the second model.

[0088] The rule-based model performs estimation based on whether specific conditions (here, the conditions for provisional state estimation) are met, and therefore, estimation results that are determined to meet the conditions can be said to be highly reliable. In contrast, the machine learning model is highly dependent on the quantity or quality of input data (here, feature quantities) and requires appropriate training and testing, and therefore, the reliability of the estimation results may be lower than that of estimation results obtained from the rule-based model. The priority determination conditions may be set to a condition that prioritizes the provisional estimation results of the occupant's physique based on the rule-based model higher than the provisional estimation results of the occupant's physique based on the machine learning model. By determining the estimated state of the occupant based on this priority, the state estimation device 1 can perform more accurate estimation of the occupant's physique.

[0089] Although machine learning models tend to be less reliable when the amount of input data is uneven or when proper training has not been performed, they generally have the ability to capture complex relationships because they learn patterns from large data sets. On the other hand, rule-based models can make accurate estimates under specific conditions, but may have difficulty dealing with complex problems.

[0090] <Example of Priority Determination Condition (7)> ​​Furthermore, for example, the priority determination conditions may be set as a combination of two or more of the above-mentioned <Example of Priority Determination Condition (1)> to <Example of Priority Determination Condition (6)>. For example, the priority determination conditions may be set as a combination of the above-mentioned <Example of Priority Determination Condition (1)> and <Example of Priority Determination Condition (3)>, such that "the provisional estimation result provisionally estimated based on a model with a larger number of input feature quantities is assigned a higher priority. When determining the number of input feature quantities, weighting is performed based on the contribution of the feature quantities to the provisional estimation result."

[0091] By combining multiple priority determination conditions, determining the priority of the provisional estimation result tentatively estimated based on a model, and determining the estimated state of the occupant based on that priority, the state estimation device 1 can perform more accurate estimation of the occupant's physique.

[0092] As described above, the priority determination conditions are listed as examples of the priority determination conditions (Example 1 of Priority Determination Condition (1)) to (Example 7 of Priority Determination Condition (7)). However, these are merely examples. The priority determination conditions may be set with conditions that allow prioritization of multiple provisional estimation results.

[0093] After determining the priorities of the multiple provisional estimation results according to the priority determination conditions such as the above-described "Example Priority Determination Condition (1)" to "Example Priority Determination Condition (7)," the integrated estimation unit 17 determines the estimated state based on the determined priorities. The integrated estimation unit 17 determines the estimated state according to the priority-state association condition. For example, the priority-state association condition may include a condition that "the estimation result with the highest priority is set as the estimated state." In this case, the integrated estimation unit 17 determines the occupant's physique indicated by the provisional estimation result with the highest priority among the multiple provisional estimation results of the occupant's physique as the estimated state. Here, the integrated estimation unit 17 determines the occupant's physique indicated by the provisional estimation result with the higher determined priority between the first and second provisional estimation results as the estimated state.

[0094] The integrated estimation unit 17 determines the occupant's physique indicated by the provisional estimation result with the highest priority among multiple provisional estimation results of the occupant's physique as the estimated state of the occupant, and the state estimation device 1 can estimate the occupant's physique based on the most accurate model among multiple models that can currently be used to estimate the occupant's physique.

[0095] Furthermore, for example, the priority-state association condition may include the following condition: "When the occupant's physique is expressed as a continuous value, the occupant's physique indicated by the multiple provisional estimation results is weighted and averaged using the priority to determine the estimated state of the occupant." In this case, the integrated estimation unit 17 determines the estimated state of the occupant by weighting and averaging the occupant's physique indicated by the multiple provisional estimation results using the priority of the provisional estimation results.

[0096] By having the integrated estimation unit 17 determine the estimated state of the occupant by taking a weighted average using priority of the occupant's physique indicated by multiple provisional estimation results, the state estimation device 1 can perform highly accurate estimation of the occupant's physique, taking into consideration whether the provisional estimation result of the occupant's physique based on multiple models that can currently be used to estimate the occupant's physique is a provisional estimation result of the occupant's physique based on a highly accurate model.

[0097] The priority-state association condition described above is merely an example. The priority-state association condition may define a condition for how to ultimately determine the estimated state of the occupant based on the determined priority. The priority-state association condition may be changed as appropriate by an administrator or the like. Furthermore, as described above, if the priority-state association condition is uniquely determined in advance, the priority-state association condition does not need to be set in the fourth determination condition.

[0098] <Example of State Determining Condition (5): Selection or Combination> For example, the state determining conditions may include the first determining condition, the second determining condition, the third determining condition, and the fourth determining condition, as described above, and a use condition (also referred to as a fifth determining condition). The use condition is a condition that specifies how to use the first determining condition, the second determining condition, the third determining condition, or the fourth determining condition.

[0099] The use conditions may specify, for example, a combination of at least two of the first, second, third, or fourth determination conditions. In this case, the integrated estimation unit 17 determines the estimated state of the occupant by combining the first, second, third, or fourth determination conditions in the combination specified by the use conditions. For example, suppose the use conditions specify "combining the first and fourth determination conditions." In this case, the integrated estimation unit 17 determines the priorities of multiple provisional estimation results, weights the votes according to the determined priorities, and then determines the occupant's physique by majority vote. For example, suppose the occupant's physique is estimated based on a physique classification, and the first provisional estimation result indicates "large build" and the second provisional estimation result indicates "medium build." Furthermore, suppose that the integrated estimation unit 17 determines the priorities, and the first provisional estimation result is higher in priority than the second provisional estimation result. In this case, the integrated estimation unit 17 determines that the "large build" indicated by the first provisional estimation result has received two votes, and determines that the occupant's physique is "large build."

[0100] By setting the state determination conditions as a combination of multiple determination conditions (e.g., the first determination condition, the second determination condition, the third determination condition, or the fourth determination condition), the state estimation device 1 can perform more accurate estimation of the occupant's physique.

[0101] The above-described use conditions are merely examples, and other use conditions may be set. For example, the use conditions may specify "use of the first determination condition." The use conditions may specify the use of multiple conditions (here, for example, the first determination condition, the second determination condition, the third determination condition, and the fourth determination condition) for determining the estimated state, such as which of the multiple conditions to select or how to combine them.

[0102] After determining the estimated state, the integrated estimation unit 17 outputs information indicating the determined estimated state (hereinafter referred to as “estimated state information”) to the estimation result output unit 18 .

[0103] The estimation result output unit 18 outputs the estimated state information output from the integrated estimation unit 17 to an external device. Note that the integrated estimation unit 17 may have the function of the estimation result output unit 18. In this case, the state estimation device 1 does not necessarily have to include the estimation result output unit 18.

[0104] The external device may be, for example, a seat belt control device (not shown), an airbag control device (not shown), an abandoned vehicle detection device (not shown), or a storage device (not shown). The seat belt control device controls the seat belt. For example, the seat belt control device has a seat belt reminder function that outputs an alarm in consideration of the physique of the occupant based on the estimated state information output from the state estimation device 1.

[0105] The airbag control device is, for example, an ECU (Engine Control Unit) for an airbag system, and controls the airbag based on the estimated state information output from the state estimating device 1.

[0106] The abandonment detection device detects whether a child or the like has been left behind in the vehicle cabin. For example, the abandonment detection device determines whether a child or the like has been left behind in the vehicle cabin based on the estimated state information output from the state estimation device 1.

[0107] The storage device is owned by, for example, a marketing company. The marketing company stores the estimated state information output from the state estimation device 1 in the storage device and performs analysis, etc.

[0108] The operation of the state estimation device 1 according to the first embodiment will now be described. Fig. 3 is a flowchart for explaining the operation of the state estimation device 1 according to the first embodiment. The state estimation device 1 starts the operation shown in the flowchart of Fig. 3 when, for example, the power of the vehicle is turned on and the imaging device 2 starts capturing images. The state estimation device 1 repeats the operation shown in the flowchart of Fig. 3 until, for example, the power of the vehicle is turned off.

[0109] The captured image acquisition unit 11 acquires a captured image from the imaging device 2 (step ST1). The captured image acquisition unit 11 outputs the acquired captured image to the feature point detection unit 13.

[0110] The subject-related information acquisition unit 12 acquires sensor information from the sensor 3 and acquires subject-related information about the occupant based on the sensor information (step ST2). The subject-related information acquisition unit 12 outputs the acquired subject-related information to the feature calculation unit 14.

[0111] The feature point detection unit 13 detects feature points of the occupant, which indicate body parts of the occupant, based on the captured image acquired by the captured image acquisition unit 11 in step ST1 (step ST3). The feature point detection unit 13 outputs feature point information to the feature amount calculation unit 14.

[0112] The feature amount calculation unit 14 calculates feature amounts for estimating the occupant's physique based on feature point information related to the occupant's feature points detected by the feature point detection unit 13 in step ST3. The feature amount calculation unit 14 also calculates feature amounts for estimating the occupant's physique based on the subject-related information output from the subject-related information acquisition unit 12 in step ST2 (step ST4). After calculating the feature amounts, the feature amount calculation unit 14 outputs the feature amount information to the state tentative estimation unit 15.

[0113] Each state temporary estimation unit 15 performs a temporary estimation process to temporarily estimate the occupant's physique based on the feature amount calculated by the feature amount calculation unit 14 in step ST4 and the model (step ST5). In the first embodiment, the temporary estimation process performed in step ST6 includes a process by the state temporary estimation unit 15 to output error information due to a missing feature amount that the model inputs. If error information is output from the model, each state temporary estimation unit 15 outputs the error information to the success / failure determination unit 16. If the occupant's physique can be temporarily estimated without error information being output from the model, each state temporary estimation unit 15 outputs the temporary estimation result of the occupant's physique to the success / failure determination unit 16.

[0114] The success / failure determination unit 16 performs a success / failure determination process (step ST6) to determine whether the provisional estimation of the occupant's physique by each state provisional estimation unit 15 in step ST5 has been successful. The success / failure determination unit 16 outputs the success / failure determination result to the integrated estimation unit 17 together with the provisional estimation results output from each state provisional estimation unit 15.

[0115] The integrated estimation unit 17 performs an estimated state determination process to determine the estimated state of the occupant, i.e., the occupant's physique to be used as the estimation result, based on the provisional estimation results of the occupant's physique by each state provisional estimation unit 15 that have been determined to have successfully estimated the occupant's physique by the success / failure determination unit 16, and on the state determination conditions (step ST7). After determining the occupant's physique to be used as the estimation result, the integrated estimation unit 17 outputs estimated state information to the estimation result output unit 18.

[0116] The estimation result output unit 18 outputs the estimated state information output from the integrated estimation unit 17 in step ST7 to an external device (step ST8).

[0117] Regarding the operation of the state estimation device 1 described using the flowchart of FIG. 3, the processing is performed in the order of step ST1, step ST2, and step ST3, but the processing order is not limited to this. For example, the processing of step ST1 and the processing of step ST2 may be performed in parallel, or the processing of step ST1 and step ST3 and the processing of step ST2 may be performed in parallel. It is sufficient that the processing of steps ST1 to ST3 is performed before the processing of step ST4 is performed. Furthermore, if the state estimation device 1 is configured not to include the subject-related information acquisition unit 12, the processing of step ST2 can be omitted.

[0118] In this way, the state estimation device 1 detects occupant feature points indicating body parts of the occupant based on captured images of the occupant, and calculates feature amounts for estimating the occupant's state based on feature point information related to the detected occupant feature points. The state estimation device 1 provisionally estimates the occupant's physique using multiple state provisional estimation units 15, each of which provisionally estimates the occupant's physique based on the calculated feature amounts and a model that estimates a person's physique using different types of feature amounts as input. The state estimation device 1 then determines an estimated state, which is the occupant's physique as an estimation result, based on the multiple provisional estimation results of the occupant's physique and the state determination conditions. Therefore, the state estimation device 1 can accurately estimate the occupant's physique even if feature points of the occupant are not detected in the captured images.

[0119] For example, depending on the installation position of the imaging device 2, the imaging device 2 may have difficulty capturing certain body parts of the occupant. For example, an imaging device 2 installed to capture images of an occupant in a vehicle cabin is generally set to be able to capture images of the occupant's face to the center of their body, making it difficult to capture body parts of the occupant that are far from the imaging device 2, such as the occupant's waist. Furthermore, even if a body part of the occupant is located between the occupant's face and the center of their body, the imaging device 2 may have difficulty capturing the body part if there is an obstruction between the body part and the imaging device 2, such as the occupant's hand or luggage. In this way, it is possible that the imaging device 2 may not be able to capture an image of a body part of the occupant. If the imaging device 2 cannot capture an image of a body part of the occupant, the occupant's feature points will not be detected from the captured image. If the occupant's feature points are not detected, feature amounts based on the feature points will not be calculated. If only one pattern of feature amounts calculated based on feature points detected from a captured image is prepared for use in estimating the occupant's physique, it is possible that the feature amounts for estimating the occupant's physique will not be calculated because feature points are not detected from the captured image, and the occupant's physique will not be able to be estimated. Alternatively, there is a possibility that the accuracy of estimating the occupant's physique will be reduced because there are not enough feature amounts for estimating the occupant's physique.

[0120] In contrast, as described above, the state estimation device 1 according to the first embodiment provisionally estimates the occupant's physique using a plurality of state provisional estimation units 15, each of which provisionally estimates the occupant's physique based on calculated feature quantities and models that estimate a person's physique using different types of feature quantities as input. The state estimation device 1 determines an estimated state, which is the occupant's physique as the estimation result, based on the provisional estimation results of the occupant's physique and the state determination conditions. The state estimation device 1 comprehensively determines the occupant's physique as the estimation result from the provisional estimation results of the occupant's physique based on models that use different types of feature quantities as input. Furthermore, for example, when a feature quantity input by a certain model cannot be calculated or the feature quantities are not all available, the state estimation device 1 determines the occupant's physique as the estimation result from the provisional estimation result of the occupant's physique based on another model that has all the feature quantities available. Therefore, the state estimation device 1 can estimate the occupant's physique using as many feature quantities as possible that can be used to estimate the occupant's physique. The state estimation device 1 can accurately estimate the physique of the occupant even when no characteristic points of the occupant are detected in the captured image.

[0121] For example, it is also possible to estimate the occupant's physique based on feature quantities calculated from only feature points detected from the captured image. However, in this case, the number of feature quantities required to estimate the occupant's physique or the range of positions in the reference body to which the feature quantities are calculated may be narrowed. Generally, from the viewpoint of improving estimation accuracy, it is preferable to use more feature quantities or feature quantities calculated from a wider range of calculation source feature points when estimating the occupant's physique. The state estimation device 1 according to the first embodiment provisionally estimates the occupant's physique using multiple state provisional estimation units 15 and determines an estimated state, which is the occupant's physique to be estimated, from the multiple provisional state estimation results. The state estimation device 1 uses different types of feature quantities as input to models used by the multiple state provisional estimation units 15 when provisionally estimating the occupant's physique, and ultimately determines an estimated state, which is the occupant's physique to be estimated, from the provisional estimation results of the occupant's physique that are provisionally estimated based on various patterns of feature quantities. Therefore, the state estimation device 1 can estimate the occupant's physique based on a sufficient number of feature quantities, and can estimate the occupant's physique with high accuracy.

[0122] In the first embodiment described above, the state estimation device 1 is configured to include the success / failure determination unit 16, but this is merely an example. The state estimation device 1 does not require the success / failure determination unit 16. If the state estimation device 1 is configured not to include the success / failure determination unit 16, the processing of step ST6 can be omitted from the operation of the state estimation device 1 as described using the flowchart in FIG. 3. In this case, in the state estimation device 1, the integrated estimation unit 17 determines an estimated state based on the provisional estimation result of the occupant's physique by the state provisional estimation unit 15 and the state determination conditions.

[0123] Furthermore, in the above-described first embodiment, the state estimation device 1 is an in-vehicle device, and the captured image acquisition unit 11, the subject-related information acquisition unit 12, the feature point detection unit 13, the state tentative estimation unit 15, the success / failure determination unit 16, the integrated estimation unit 17, and the estimation result output unit 18 are provided in the in-vehicle device. However, without being limited to this, some of the captured image acquisition unit 11, the subject-related information acquisition unit 12, the feature point detection unit 13, the state tentative estimation unit 15, the success / failure determination unit 16, the integrated estimation unit 17, and the estimation result output unit 18 may be mounted in the in-vehicle device of the vehicle, and the others may be provided in a server connected to the in-vehicle device via a network, and a state estimation system may be configured by the in-vehicle device and the server. In addition, the captured image acquisition unit 11, the subject-related information acquisition unit 12, the feature point detection unit 13, the state provisional estimation unit 15, the success / failure determination unit 16, the integrated estimation unit 17, and the estimation result output unit 18 may all be provided on the server.

[0124] In the first embodiment described above, the state of the subject estimated by the state estimation device 1 is the subject's physique, but this is merely an example. The state of the subject estimated by the state estimation device 1 may be the subject's behavior (such as raising a hand) or posture (such as sitting or standing).

[0125] Furthermore, in the above-described first embodiment, the subject is described as a vehicle occupant as an example, but this is merely an example. The subject may be, for example, a occupant of a moving body other than a vehicle, such as a commercial vehicle such as a bus, a train, or an airplane, or may be a person in a factory, a room, or a living space, or may be a person outdoors. The state estimation device 1 according to the first embodiment can be applied to a device that estimates the state of a subject who is located in a position where the imaging device 2 can capture an image.

[0126] 4A and 4B are diagrams illustrating an example of a hardware configuration of the state estimation device 1 according to the first embodiment. In the first embodiment, the functions of the captured image acquisition unit 11, the subject-related information acquisition unit 12, the feature point detection unit 13, the state tentative estimation unit 15, the success / failure determination unit 16, the integrated estimation unit 17, and the estimation result output unit 18 are realized by a processing circuit 101. That is, the state estimation device 1 includes the processing circuit 101 for performing control to tentatively estimate the state of the subject using a plurality of models for estimating a human state using, as input, feature quantities based on feature points of the subject detected from a captured image of the subject, and each of which has a different type of feature quantity, and to determine an estimated state from a plurality of tentative estimation results of the subject's state. The processing circuit 101 may be dedicated hardware as shown in FIG. 4A or a processor 104 that executes a program stored in memory as shown in FIG. 4B.

[0127] When processing circuitry 101 is dedicated hardware, processing circuitry 101 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0128] When the processing circuit is the processor 104, the functions of the captured image acquisition unit 11, the subject-related information acquisition unit 12, the feature point detection unit 13, the state tentative estimation unit 15, the success / failure determination unit 16, the integrated estimation unit 17, and the estimation result output unit 18 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 105. The processor 104 reads and executes the program stored in the memory 105 to execute the functions of the captured image acquisition unit 11, the subject-related information acquisition unit 12, the feature point detection unit 13, the state tentative estimation unit 15, the success / failure determination unit 16, the integrated estimation unit 17, and the estimation result output unit 18. That is, the state estimation device 1 includes the memory 105 for storing a program that, when executed by the processor 104, results in the execution of steps ST1 to ST8 of FIG. 4 described above. In addition, the program stored in memory 105 can also be said to cause a computer to execute the processing procedures or methods of the captured image acquisition unit 11, the subject-related information acquisition unit 12, the feature point detection unit 13, the state provisional estimation unit 15, the success / failure determination unit 16, the integrated estimation unit 17, and the estimation result output unit 18. Here, the memory 105 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

[0129] Note that the functions of the captured image acquisition unit 11, the subject-related information acquisition unit 12, the feature point detection unit 13, the state tentative estimation unit 15, the success / failure determination unit 16, the integrated estimation unit 17, and the estimation result output unit 18 may be partially implemented by dedicated hardware and partially implemented by software or firmware. For example, the functions of the captured image acquisition unit 11 and the subject-related information acquisition unit 12 may be implemented by a processing circuit 101 as dedicated hardware, while the functions of the feature point detection unit 13, the state tentative estimation unit 15, the success / failure determination unit 16, the integrated estimation unit 17, and the estimation result output unit 18 may be implemented by the processor 104 reading and executing a program stored in the memory 105. The storage unit (not shown) may be, for example, the memory 105 or an HDD. The state estimation device 1 also includes an input interface device 102 and an output interface device 103 that communicate with devices such as the imaging device 2 or the sensor 3 via wired or wireless communication.

[0130] As described above, according to the first embodiment, the state estimation device 1 is configured to include: an image acquisition unit 11 that acquires an image of a subject; a feature point detection unit 13 that detects feature points of the subject that indicate body parts of the subject based on the image acquired by the image acquisition unit 11; a feature amount calculation unit 14 that calculates multiple types of feature amounts for estimating the state of the subject based on feature point information regarding the feature points of the subject detected by the feature point detection unit 13; a plurality of state provisional estimation units 15 that each provisionally estimate the state of the subject based on the feature amounts calculated by the feature amount calculation unit 14 and a model that estimates a human state using different types of feature amounts as input; and an integrated estimation unit 17 that determines an estimated state of the subject, which is the state of the subject that is to be an estimation result, based on a plurality of provisional estimation results of the state of the subject by the plurality of state provisional estimation units 15 and a state determination condition. This allows the state estimation device 1 to estimate the physique of the occupant using as many feature amounts as possible that can be used to estimate the physique of the occupant. The state estimation device 1 can accurately estimate the physique of the occupant even when feature points of the target person are not detected in the captured image.

[0131] Furthermore, in the first embodiment, for example, the state determination conditions include a first determination condition that determines the estimated state of the subject by majority vote to adopt the most common state of the subject among the states of the subject indicated by the multiple provisional estimation results, and the integrated estimation unit 17 determines the most common state of the subject among the states of the subject indicated by the multiple provisional estimation results as the estimated state of the subject based on the multiple provisional estimation results of the subject's state by the multiple state provisional estimation units 15 and the first determination condition. This allows the state estimation device 1 to perform a highly accurate state estimation of the subject's state that takes into account the provisional estimation results of the physique of the subject's state based on multiple models that can currently be used to estimate the subject's state.

[0132] Furthermore, in the first embodiment, for example, the state of the subject is represented by a continuous value, the state determination condition includes a second determination condition that determines the estimated state of the subject to be an average value of the state of the subject indicated by the plurality of provisional estimation results, and the integrated estimation unit 17 determines the estimated state of the subject to be the average value of the state of the subject indicated by the plurality of provisional estimation results, based on the plurality of provisional estimation results of the state of the subject by the plurality of state provisional estimation units 15 and the second determination condition. This allows the state estimation device 1 to perform a highly accurate state estimation of the subject, taking into account the provisional estimation results of the state of the subject based on the plurality of models that can currently be used to estimate the state of the subject.

[0133] Furthermore, in the first embodiment, for example, the state determination conditions include a third determination condition that determines the state of the subject indicated by the provisional estimation result provisionally estimated based on the model with the highest assigned rank as the estimated state of the subject, and the integrated estimation unit 17 determines the state of the subject indicated by the provisional estimation result provisionally estimated based on the model with the highest assigned rank as the estimated state of the subject, based on the multiple provisional estimation results of the subject's state by the multiple state provisional estimation units 15 and the third determination condition. As a result, the state estimation device 1 determines the state of the subject that is estimated most likely from the states of the subject estimated based on the multiple models that can currently be used to estimate the state of the subject as the final state of the subject, and can perform highly accurate state estimation of the subject.

[0134] Furthermore, in the first embodiment, for example, the state determination conditions include a fourth determination condition that determines the estimated state of the subject based on the priorities of the multiple provisional estimation results determined in accordance with the priority determination conditions, and the integrated estimation unit 17 determines the priorities of the multiple provisional estimation results of the subject's state based on the multiple provisional estimation results of the subject's state obtained by the multiple state provisional estimation units 15 and the fourth determination condition, and determines the estimated state of the subject based on the determined priorities. As a result, the state estimation device 1 determines the final state of the subject in consideration of the state of the subject that should be prioritized from among the states of the subject estimated based on the multiple models that can currently be used to estimate the state of the subject, and can perform highly accurate state estimation of the passenger's state.

[0135] In the first embodiment, the priority determination conditions include, for example, a condition that a provisional estimation result obtained based on a model having a larger number of input feature quantities is given a higher priority. This allows the state estimation device 1 to estimate the subject's state using the most accurate model among multiple models that are currently available for estimating the subject's state.

[0136] Furthermore, in the first embodiment, the priority determination conditions include, for example, a condition that a provisional estimation result provisionally estimated based on a model having a larger number of input feature quantities used in provisionally estimating the subject's state is given a higher priority. This enables the state estimation device 1 to perform highly accurate estimation of the subject's state, taking into consideration that the model used in estimating the subject's state may be missing input feature quantities.

[0137] Furthermore, in the first embodiment, the priority determination conditions include, for example, a condition that a provisional estimation result provisionally estimated based on a model in which input feature quantities include more feature quantities that have a high degree of contribution to the provisional estimation result is given a higher priority. This enables the state estimation device 1 to perform highly accurate estimation of the subject's state, taking into account the provisional estimation result of the subject's state based on a model in which more important feature quantities are input.

[0138] Furthermore, in the first embodiment, the priority determination conditions include, for example, a condition that the wider the range of positions on the reference body that correspond to body parts indicated by calculation source feature points, which are feature points that are used as inputs to the model to calculate feature amounts, the higher the priority of the provisional estimation result provisionally estimated based on a model that inputs feature amounts based on the calculation source feature points. This enables the state estimation device 1 to perform highly accurate state estimation of a subject, giving priority to the provisional estimation result of the subject's state that is based on a model that inputs feature amounts calculated from a wider range of feature points.

[0139] Furthermore, in the priority determination conditions, "the range of positions on the reference body to which the body parts indicated by the calculation source feature points correspond is wide" means that the range of positions on the reference body to which the body parts indicated by the calculation source feature points correspond is wide in the vertical direction. This enables the state estimation device 1 to perform more accurate state estimation of the subject by prioritizing the provisional estimation result of the subject's state based on a model that uses as input feature amounts calculated from feature points in a wider range.

[0140] Furthermore, in the first embodiment, the priority determination conditions include, for example, a condition that the more feature quantities that are input to a model include feature quantities with high corresponding feature quantity reliability, the higher the priority of the provisional estimation result provisionally estimated based on the model. This allows the state estimation device 1 to not prioritize a provisional estimation result of the subject's state that is based on a model that inputs feature quantities that are estimated to be poorly calculated, thereby reducing the probability of making an erroneous state estimation.

[0141] In the first embodiment, the priority determination condition includes, for example, a condition that the priority of a provisional estimation result of the subject's state based on a rule-based model is to be higher than the priority of a provisional estimation result of the subject's state based on a machine learning model, thereby enabling the state estimation device 1 to estimate the subject's state with higher accuracy.

[0142] Furthermore, in the first embodiment, the priority determination conditions include, for example, a condition that the provisional estimation result provisionally estimated based on a model having a larger number of input feature quantities is given a higher priority; a condition that the provisional estimation result provisionally estimated based on a model having a larger number of input feature quantities used in provisionally estimating the state of the subject is given a higher priority; a condition that the provisional estimation result provisionally estimated based on a model having a larger number of input feature quantities that have a high degree of contribution to the provisional estimation result ... is given a higher priority; a condition that the wider the range of positions in the reference body that can be associated with a body part indicated by a calculation source feature point, which is a feature point from which a feature quantity input to a model is calculated, the higher the priority is given to the input of a feature quantity based on a calculation source feature point; a condition that the priority of a provisional estimation result provisionally estimated based on a model that uses the feature quantities as input is increased, a condition that the priority of a provisional estimation result provisionally estimated based on the model is increased the more feature quantities that have high corresponding feature quantity reliability are included in the feature quantities that are input to the model, or, if the multiple models include a rule-based model that estimates a person's state based on conditions for provisional state estimation set from the feature quantities and a machine learning model that estimates a person's state using the feature quantities as input, the priority of a provisional estimation result of the subject's state based on the rule-based model is increased over the priority of a provisional estimation result of the subject's state based on the machine learning model. This enables the state estimation device 1 to estimate the subject's state with higher accuracy.

[0143] Furthermore, in the first embodiment, the integrated estimation unit 17 determines the state of the subject indicated by the provisional estimation result with the highest priority among the multiple provisional estimation results of the state of the subject as the estimated state of the subject. This allows the state estimation device 1 to currently estimate the state of the subject based on the most accurate model among the multiple models that can be used to estimate the state of the subject.

[0144] Furthermore, in the first embodiment, the subject's state is represented by a continuous value, and the integrated estimation unit 17 determines the estimated state of the subject by taking a weighted average, using priorities, of the subject's states represented by the multiple provisional estimation results. This allows the state estimation device 1 to perform highly accurate estimation of the subject's state, taking into consideration whether the provisional estimation result of the subject's state based on multiple models that can currently be used to estimate the subject's state is a provisional estimation result of the subject's state based on a highly accurate model.

[0145] Furthermore, in the first embodiment, the state determination conditions include, for example, a first determination condition that determines the estimated state of the subject by majority vote, adopting the most common state of the subject among the states of the subject indicated by the multiple provisional estimation results; a second determination condition that, if the state of the subject is indicated by a continuous value, determines the average value of the states of the subject indicated by the multiple provisional estimation results as the estimated state of the subject; a third determination condition that determines the estimated state of the subject as the state indicated by the provisional estimation result tentatively estimated based on the model with the highest assigned ranking; or a fourth determination condition that determines the estimated state of the subject based on the priority of the multiple provisional estimation results determined in accordance with the priority determination conditions. The integrated estimation unit 17 determines the estimated state of the subject based on the multiple provisional estimation results of the subject's state obtained by the multiple state provisional estimation units 15 and the fifth determination condition. This enables the state estimation device 1 to perform more accurate state estimation of the subject.

[0146] Furthermore, in the first embodiment, the state estimation device 1 includes a success / failure determination unit 16 that determines whether the provisional estimation of the subject's state by the multiple state provisional estimation units 15 is successful, and the integrated estimation unit 17 determines the estimated state of the subject based on the provisional estimation result of the subject's state by the state provisional estimation unit 15 that the success / failure determination unit 16 determines to be successful among the multiple provisional estimation results of the subject's state and on the state determination conditions. This allows the state estimation device 1 to prevent a decrease in the estimation accuracy of the subject's state as the estimation result.

[0147] Furthermore, in the first embodiment, the state estimation device 1 includes a subject-related information acquisition unit 12 that acquires subject-related information about the subject that serves as a basis for calculating feature amounts, based on the captured image acquired by the captured image acquisition unit 11 or sensor information indicating an object detected by the sensor 3, and the feature amount calculation unit 14 may calculate feature amounts based on the feature point information and the subject-related information. This enables the state estimation device 1 to increase the number of feature amounts used in estimating the state of the subject, and further improve the accuracy of estimating the state of the subject.

[0148] In addition, in the present disclosure, any of the components of the embodiments may be modified or omitted.

[0149] A state estimation device according to the present disclosure can accurately estimate the state of a subject even when feature points of the subject are not detected in a captured image.

[0150] 1 State estimation device, 11 Captured image acquisition unit, 12 Subject-related information acquisition unit, 13 Feature point detection unit, 14 Feature amount calculation unit, 15 State provisional estimation unit, 151 First state provisional estimation unit, 152 Second state provisional estimation unit, 15n nth state provisional estimation unit, 16 Success / failure determination unit, 17 Integrated estimation unit, 18 Estimation result output unit, 2 Imaging device, 3 Sensor, 101 Processing circuit, 102 Input interface device, 103 Output interface device, 104 Processor, 105 Memory.

Claims

1. A state estimation device comprising: an image acquisition unit that acquires an image of a subject; a feature point detection unit that detects feature points of the subject that indicate body parts of the subject based on the image acquired by the image acquisition unit; a feature amount calculation unit that calculates multiple types of feature amounts for estimating the state of the subject based on feature point information related to the feature points of the subject detected by the feature point detection unit; a plurality of state provisional estimation units that each provisionally estimate the state of the subject based on the feature amount calculated by the feature amount calculation unit and a model that estimates a human state using different types of feature amounts as input; and an integrated estimation unit that determines an estimated state of the subject, which is the state of the subject to be estimated, based on a plurality of provisional estimation results of the state of the subject by the plurality of state provisional estimation units and a state determination condition.

2. The state estimation device described in claim 1, characterized in that the state determination conditions include a first determination condition that the estimated state of the subject is determined by a majority vote to adopt the state of the subject that is most common among the states of the subject indicated by the multiple provisional estimation results, and the integrated estimation unit determines the state of the subject that is most common among the states of the subject indicated by the multiple provisional estimation results based on the multiple provisional estimation results of the state of the subject by the multiple state provisional estimation units and the first determination condition.

3. The state estimation device described in claim 1, characterized in that the state of the subject is represented by a continuous value, the state determination conditions include a second determination condition that determines the estimated state of the subject to be an average value of the state of the subject indicated by the plurality of provisional estimation results, based on the plurality of provisional estimation results of the state of the subject by the plurality of state provisional estimation units and the second determination condition.

4. The state estimation device described in claim 1, characterized in that the state determination conditions include a third determination condition that determines the estimated state of the subject to be the state of the subject indicated by the provisional estimation result provisionally estimated based on the model with the highest assigned ranking, and the integrated estimation unit determines the estimated state of the subject to be the state of the subject indicated by the provisional estimation result provisionally estimated based on the model with the highest assigned ranking, based on the multiple provisional estimation results of the subject's state by the multiple state provisional estimation units and the third determination condition.

5. The state estimation device described in claim 1, characterized in that the state determination conditions include a fourth determination condition that determines the estimated state of the subject based on the priorities of the multiple provisional estimation results determined in accordance with the priority determination conditions, and the integrated estimation unit determines the priorities of the multiple provisional estimation results of the subject's state based on the multiple provisional estimation results of the subject's state obtained by the multiple state provisional estimation units and the fourth determination condition, and determines the estimated state of the subject based on the determined priorities.

6. The state estimation device according to claim 5, characterized in that the priority determination conditions include a condition that the higher the number of input feature quantities of the provisional estimation result provisionally estimated based on the model, the higher the priority of the provisional estimation result.

7. The state estimation device according to claim 5, characterized in that the condition for determining priority is set so that the priority of the provisional estimation result provisionally estimated based on the model having a larger number of feature quantities used in provisionally estimating the state of the subject among the feature quantities that are input is higher.

8. The state estimation device according to claim 5, characterized in that the priority determination conditions include a condition that the priority of the provisional estimation result provisionally estimated based on the model includes a large number of feature quantities that have a high degree of contribution to the provisional estimation result among the input feature quantities.

9. The state estimation device according to claim 8, characterized in that the contribution degree is the contribution degree obtained by inputting multiple feature amounts into a trained contribution degree setting model, which inputs multiple feature amounts and outputs information indicating the person's state and the contribution degrees of the multiple feature amounts before the multiple state estimation units provisionally estimate the state of the subject.

10. A state estimation device as described in claim 8, characterized in that the contribution degree is set based on the contribution degree obtained by learning the model when the model was generated, before the multiple state estimation units provisionally estimated the state of the subject.

11. The state estimation device according to claim 5, wherein the priority determination conditions include a condition that the wider the range of positions in a reference body to which the body part indicated by the calculation source feature point, which is the feature point from which the feature amount used as input to the model is calculated, the higher the priority of the provisional estimation result provisionally estimated based on the model that uses as input the feature amount based on the calculation source feature point.

12. A state estimation device as described in claim 11, characterized in that, in the conditions for determining priority, a wide range of positions on the reference body to which the body part indicated by the calculation source feature point can be associated means that a wide range of positions on the reference body to which the body part indicated by the calculation source feature point can be associated is vertically wide.

13. A state estimation device according to claim 5, characterized in that the condition for determining priority is set so that the more features that are input to the model contain features with high corresponding feature reliability, the higher the priority of the provisional estimation result provisionally estimated based on the model.

14. The state estimation device described in claim 5, characterized in that the model includes a rule-based model that estimates the state of the person based on conditions for provisional state estimation set from the feature amounts, and a machine learning model that estimates the state of the person using the feature amounts as input, and the conditions for priority determination include a condition that sets the priority of the provisional estimation result of the subject's state based on the rule-based model higher than the priority of the provisional estimation result of the subject's state based on the machine learning model.

15. The priority determination conditions include: a condition that the provisional estimation result tentatively estimated based on the model having a larger number of input feature quantities is given a higher priority; a condition that the provisional estimation result tentatively estimated based on the model having a larger number of input feature quantities that were used to tentatively estimate the state of the subject is given a higher priority; a condition that the provisional estimation result tentatively estimated based on the model having a larger number of input feature quantities that were used to tentatively estimate the state of the subject is given a higher priority; a condition that the provisional estimation result tentatively estimated based on the model having a larger number of input feature quantities that have a higher contribution to the provisional estimation result is given a higher priority; a condition that the provisional estimation result tentatively estimated based on the model having as input feature quantities based on the calculation source feature points, which are the feature points from which the feature quantities input to the model are calculated, is given a higher priority the wider the range of positions on a reference body that can be associated with the body parts indicated by the calculation source feature points; a condition that the provisional estimation result tentatively estimated based on the model having as input feature quantities based on the calculation source feature points is given a higher priority the more the feature quantities input to the model include feature quantities with higher corresponding feature reliability; or a condition that, when the plurality of models include a rule-based model that estimates the state of the person based on conditions for provisional state estimation set from the feature amounts, and a machine learning model that estimates the state of the person using the feature amounts as input, the priority of the provisional estimation result of the subject's state based on the rule-based model is made higher than the priority of the provisional estimation result of the subject's state based on the machine learning model.

16. The state estimation device according to claim 5, characterized in that the integrated estimation unit determines the state of the subject indicated by the provisional estimation result with the highest priority among the multiple provisional estimation results of the state of the subject as the estimated state of the subject.

17. A state estimation device as described in claim 5, characterized in that the state of the subject is represented by a continuous value, and the integrated estimation unit determines the estimated state of the subject by taking a weighted average of the states of the subject represented by the multiple provisional estimation results using the priority.

18. The state estimation device according to claim 1, characterized in that the state determination conditions include a fifth determination condition specifying a combination of at least two of: a first determination condition stipulating that the estimated state of the subject is determined by majority vote to adopt the state of the subject that is most common among the states of the subject indicated by the multiple provisional estimation results; a second determination condition stipulating that, if the state of the subject is indicated by a continuous value, the average value of the states of the subject indicated by the multiple provisional estimation results is determined as the estimated state of the subject; a third determination condition stipulating that the state of the subject indicated by the provisional estimation result tentatively estimated based on the model with the highest assigned ranking is determined as the estimated state of the subject; and a fourth determination condition stipulating that the estimated state of the subject is determined based on the priority of the multiple provisional estimation results determined in accordance with a priority determination condition.

19. A state estimation device as described in claim 1, characterized in that it comprises a success / failure determination unit that determines the success or failure of the provisional estimation of the subject's state by the multiple state provisional estimation units, and the integrated estimation unit determines the estimated state of the subject based on the provisional estimation result of the subject's state by the state provisional estimation unit that the success / failure determination unit determines to be successful among the multiple provisional estimation results of the subject's state and the state determination conditions.

20. A state estimation device as described in claim 1, characterized in that it is provided with a subject-related information acquisition unit that acquires subject-related information about the subject that serves as the basis for calculating the feature amount based on the captured image acquired by the captured image acquisition unit or sensor information indicating an object detected by a sensor, and the feature amount calculation unit calculates the feature amount based on the feature point information and the subject-related information.

21. The state estimation device according to claim 1, wherein the state of the subject is represented by a discrete classification or a continuous value.

22. The state estimation device according to claim 1, wherein the state of the subject is the physique of the subject.

23. A state estimation device according to any one of claims 1 to 22, characterized in that the subject is a vehicle occupant.

24. A state estimation method comprising the steps of: an image acquisition unit acquiring an image of a subject; a feature point detection unit detecting feature points of the subject indicating body parts of the subject based on the image acquired by the image acquisition unit; a feature amount calculation unit calculating multiple types of feature amounts for estimating the state of the subject based on feature point information regarding the feature points of the subject detected by the feature point detection unit; a plurality of state provisional estimation units each provisionally estimating the state of the subject based on the feature amount calculated by the feature amount calculation unit and a model that estimates a human state using different types of feature amounts as input; and an integrated estimation unit determining an estimated state of the subject, which is the state of the subject to be estimated, based on multiple provisional estimation results of the state of the subject by the multiple state provisional estimation units and state determination conditions.

Images