Physique determination device and physique determination method

Abstract

The present invention comprises: a skeletal point extraction unit (102) that extracts skeletal points of a vehicle occupant on the basis of a captured image of the vehicle occupant; a normal sitting determination unit (104) that determines, on the basis of situation information relating to the situation of the vehicle or the vehicle occupant, whether the posture of the vehicle occupant is in a normal sitting state; a feature quantity calculation unit (105) that determines whether or not the skeletal points of the vehicle occupant that were extracted by the skeletal point extraction unit (102) are normal sitting skeletal points extracted on the basis of a captured image of the vehicle occupant in a normal sitting state, and, if the skeletal points of the vehicle occupant that were extracted by the skeletal point extraction unit (102) are determined to be the normal sitting skeletal points, calculates a feature quantity for physique determination on the basis of information relating to the normal sitting skeletal points; and a physique determination unit (107) that determines the physique of the vehicle occupant on the basis of the feature quantity for physique determination.

Description

[Technical Field]

[0001] The present disclosure relates to an occupant physique determination device and physique determination method. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there is known a technique for determining the posture of a vehicle occupant based on a captured image of the occupant, and for determining the physique of the occupant from the determined posture. For example, Patent Document 1 discloses an interior monitoring device that estimates the posture of an occupant based on an image obtained by capturing an image of the interior of a vehicle, using a physique estimation index including the coordinates of multiple feature points of the occupant in the captured image extracted by deep learning or the like, and then estimates the physique of the occupant based on the estimated posture. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-104680 Summary of the Invention [Problem to be solved by the invention]

[0004] When an occupant is sitting in a seat with poor posture, there are countless possible patterns of poor posture. Therefore, when an occupant is sitting in a seat with poor posture, it is difficult to cover all the possible patterns of poor posture, and the occupant's posture determined based on the captured image is more likely to contain errors, or the errors will be larger, compared to when the occupant is sitting in a good posture. As a result, the accuracy of the physique determination decreases. Conventional technologies such as that disclosed in Patent Document 1 do not take this into consideration, and therefore there was a problem that if the occupant's posture becomes unstable, the accuracy of determining the occupant's physique may decrease.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a physical size determination device that prevents a decrease in the accuracy of determining the physical size of an occupant even if the occupant's posture becomes unstable. [Means for solving the problem]

[0006] The physique determination device according to the present disclosure includes an image acquisition unit that acquires an image of a vehicle occupant, a skeleton point extraction unit that extracts skeleton points of the occupant that indicate body parts of the occupant based on the image acquired by the image acquisition unit, a proper seating determination unit that determines whether the posture of the occupant is in a proper seating state based on situation information related to the situation of the vehicle or the occupant, and a proper seating determination unit that determines whether the posture of the occupant is in a proper seating state based on information about the skeleton points of the occupant extracted by the skeleton point extraction unit and the posture of the occupant based on information about the skeleton points of the occupant extracted by the proper seating determination unit. Positive In a normal seated position with Based on the seating state determination result, the skeleton points of the occupant extracted by the skeleton point extraction unit are normal seating skeleton points extracted based on the captured image in which the occupant is normally seated. At points When it is determined that there is a normal seating skeleton point, a feature calculation unit calculates a feature for determining physique based on information about the normal seating skeleton point, and a physique determination unit determines the physique of the occupant based on the feature for determining physique calculated by the feature calculation unit. [Effects of the Invention]

[0007] According to the present disclosure, the physique determination device can prevent a decrease in accuracy in determining the physique of an occupant even if the occupant's posture becomes unstable. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the configuration of a system in which a physique determination device according to a first embodiment is used. [Figure 2] 1 is a diagram illustrating an example of the configuration of a physique determination device according to a first embodiment. [Figure 3] 4 is a flowchart illustrating an example of the operation of the physique determination device according to the first embodiment. [Figure 4]FIG. 2 is a diagram showing an example of the configuration of a physique determination device in the first embodiment when tilt information is used as situation information. [Figure 5] FIG. 2 is a diagram showing an example of the configuration of a physique determination device in the first embodiment when vehicle information is used as situation information. [Figure 6] 6A and 6B are diagrams illustrating an example of a hardware configuration of the physique determination device according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1 FIG. 1 is a diagram showing an example of the configuration of a system in which a physique determination device 100 according to the first embodiment is used. In the system shown in FIG. 1, a physique determination device 100 according to the first embodiment is connected to an imaging device 200, a seatbelt control device 300, an airbag control device 400, an abandonment detection device 500, and an output device 600. In the first embodiment, it is assumed that the physique determination device 100 is mounted on a vehicle. The seat belt control device 300, the airbag control device 400, the abandonment detection device 500, and the output device 600 are mounted on a vehicle.

[0010] The image capturing device 200 is, for example, a near-infrared camera or a visible light camera, and captures images of vehicle occupants. The image capturing device 200 may be, for example, a shared image capturing device included in a so-called "Driver Monitoring System (DMS)" that is mounted on a vehicle to monitor the state of the driver inside the vehicle. The image capturing device 200 is installed so as to be able to capture an image of at least the area inside the vehicle that includes the area where the upper half of the vehicle occupant's body should be located, which is, for example, the area corresponding to the seat back and the space in front of the headrest. For example, the imaging device 200 is installed in the center of the vehicle's instrument panel (hereinafter referred to as "instrument panel") so as to be able to image the front seats including the driver's seat and passenger seat from the center of the instrument panel, in other words, so as to be able to image the driver and passenger in the passenger seat (hereinafter referred to as "passenger in the passenger seat"). Note that this is just one example, and the imaging device 200 may be installed so as to be able to image only the driver, or so as to be able to image passengers in the rear seats.

[0011] The physique determination device 100 according to the first embodiment determines the physique of a vehicle occupant based on a captured image of the vehicle occupant captured by the imaging device 200. At this time, the physique determination device 100 determines whether the posture of the vehicle occupant is in a properly seated state, and determines the physique of the occupant based on an image captured when the occupant is in a properly seated state. In the first embodiment, "normal seating" means that the occupant is seated in the seat in a normal state without any poor posture. In the first embodiment, a state in which the occupant's face is not facing forward is also considered to be poor posture. When the occupant's face is not facing forward, it is estimated that there is a high possibility that the occupant's entire body is not facing forward. After determining the physique of the vehicle occupant, the physique determination device 100 outputs the determination result of the physique of the vehicle occupant (hereinafter referred to as the "physique determination result") to the seat belt control device 300, the airbag control device 400, and the abandonment detection device 500. In addition, the physique determination device 100 outputs information to the output device 600 for outputting a message or the like to urge the occupant to sit properly. An example of the configuration of the physique determination device 100 will be described in detail later with reference to FIG. In the following first embodiment, a vehicle occupant will also be simply referred to as an "occupant."

[0012] The seat belt control device 300 controls the seat belt. For example, the seat belt control device 300 has a seat belt reminder function that outputs an alarm in consideration of the physique of the occupant based on the physique determination result output from the physique determination device 100.

[0013] The airbag control device 400 is, for example, an ECU (Engine Control Unit) for an airbag system, and controls the airbag based on the physique determination result output from the physique determination device 100.

[0014] The abandonment detection device 500 detects whether a child or the like has been left behind in a vehicle. For example, the abandonment detection device 500 determines whether a child or the like has been left behind in a vehicle based on the physique determination result output from the physique determination device 100.

[0015] The output device 600 is, for example, a display device or an audio output device provided in a center cluster, an instrument panel, etc. For example, the output device 600 displays a message urging the occupant to sit properly, outputs the message as audio, or outputs a warning sound urging the occupant to sit properly.

[0016] An example of the configuration of the physique determination device 100 according to the first embodiment will be described. FIG. 2 is a diagram showing an example of the configuration of the physique determination device 100 according to the first embodiment. The physique determination device 100 includes an image acquisition unit 101, a skeleton point extraction unit 102, a situation information acquisition unit 103, a normal seating determination unit 104, a feature calculation unit 105, a feature leveling unit 106, a physique determination unit 107, and a posture warning unit 108. The situation information acquisition unit 103 includes a face direction detection unit 1031.

[0017] The image acquisition unit 101 acquires an image of an occupant captured by the imaging device 200. The image acquisition unit 101 outputs the acquired captured image to the skeleton point extraction unit 102.

[0018] The skeleton point extraction unit 102 extracts skeleton points of the occupant, which indicate body parts of the occupant, based on the captured images acquired by the image acquisition unit 101. In detail, the skeleton point extraction unit 102 extracts skeleton points of the occupant, which indicate joint points determined for each body part of the occupant, based on the captured images acquired by the image acquisition unit 101. The joint points determined for each body part of the occupant are, for example, nose joint points, neck joint points, shoulder joint points, elbow joint points, waist joint points, wrist joint points, knee joint points, and ankle joint points. The joint points are defined in advance. The skeleton point extraction unit 102 may extract the skeleton points of the occupant by a known method using a known technology such as an image recognition technology or a technology using a trained model (hereinafter referred to as a "machine learning model"). The skeleton points are, for example, points in a captured image and are represented by coordinates in the captured image. The skeleton point extraction unit 102 detects the coordinates of the skeleton points of the occupant and which parts of the occupant's body the skeleton points represent.

[0019] The skeleton point extraction unit 102 extracts skeleton points for each occupant. For example, in the captured image, an area corresponding to each seat (hereinafter referred to as a "seat-corresponding area") is set in advance. The seat-corresponding area is set in advance according to the installation position and angle of view of the imaging device 200. For example, the skeleton point extraction unit 102 extracts skeleton points for each seat corresponding area. For example, if a certain seat corresponding area is a seat corresponding area corresponding to the driver's seat, the skeleton point extraction unit 102 determines that the skeleton points extracted in that seat corresponding area are the skeleton points of the driver. Also, for example, if a certain seat corresponding area is a seat corresponding area corresponding to the passenger seat, the skeleton point extraction unit 102 determines that the skeleton points extracted in that seat corresponding area are the skeleton points of the passenger in the passenger seat. In this way, the skeleton point extraction unit 102 extracts skeleton points from each seat corresponding area, thereby extracting skeleton points for each occupant.

[0020] The skeleton point extraction unit 102 outputs information relating to the extracted skeleton points (hereinafter referred to as “skeleton point information”) to the situation information acquisition unit 103. The skeleton point information includes information that corresponds information indicating the occupant, information indicating the coordinates of the occupant's skeleton points, and information indicating which part of the occupant's body the skeleton points represent, as well as the captured image acquired by the image acquisition unit 101. In the first embodiment, the information indicating the occupant is, for example, information indicating the seat in which the occupant is seated.

[0021] When the skeleton point extraction unit 102 extracts the skeleton points of the occupant, in other words, when skeleton point information is output from the skeleton point extraction unit 102, the situation information acquisition unit 103 acquires information related to the situation of the occupant (hereinafter referred to as "situation information"). The situation information is, for example, information related to the facial direction of the occupant.

[0022] The face direction detection unit 1031 detects the face direction of the occupant based on the captured image acquired by the image acquisition unit 101. The captured image acquired by the image acquisition unit 101 is included in the skeleton point information. The face direction detection unit 1031 may detect the face direction of the occupant by a known method using known technology such as image recognition technology or technology using a machine learning model. The face direction of the occupant is expressed, for example, by an angle. The face direction detection unit 1031 detects the face direction for each occupant.

[0023] The situation information acquisition unit 103 acquires, as situation information, information relating to the facial direction of the occupant detected by the facial direction detection unit 1031 (hereinafter referred to as "facial direction information"). The facial direction information includes information in which information indicating the occupant is associated with information indicating the facial direction. The information indicating the occupant is, for example, information indicating the seat in which the occupant is sitting. The facial direction detection unit 1031 may determine, for example, from the seat corresponding area in which the occupant is sitting. In addition to the above, the facial direction information may also include the captured image acquired by the image acquisition unit 101. The situation information acquisition unit 103 outputs the acquired situation information, in this case, face direction information, to the normal seating determination unit 104.

[0024] The normal seating determination unit 104 determines whether the posture of the occupant is a normal seating state based on the situation information acquired by the situation information acquisition unit 103. More specifically, here, the proper seating determination unit 104 determines whether the posture of the occupant is a proper seating state or not based on the face direction information. The normal seating determination unit 104 determines whether the posture of each occupant is a normal seating state or not. The normal seating determination unit 104 can identify the facial direction of each occupant from the facial direction information.

[0025] An example of a method for determining whether the posture of an occupant is in a normal seating state by the normal seating determination unit 104 will be described. For example, the normal seating determination unit 104 determines whether the occupant's posture is in a normal seating state based on situation information, in this case, facial direction information, and in accordance with pre-set conditions for determining whether the occupant's posture is in a normal seating state (hereinafter referred to as "conditions for determining normal seating"). The normal seating determination conditions define, for example, conditions under which the state of the occupant's posture is considered to be a normal seating state. The normal seating determination conditions are generated in advance by an administrator or the like and stored in a location that can be referenced by the physique determination device 100.

[0026] The normal seating determination conditions include, for example, the following <Condition 1>. <Condition 1> The passengers' faces must be facing forward. When the occupant's face direction is within a predetermined range (referred to as a "face direction determination range") based on the front, the occupant's face direction is determined to be facing forward. In addition, for the face direction determination range in <Condition 1>, an appropriate range in which the face is considered to be facing forward is set in advance.

[0027] If the facial direction of the occupant satisfies the conditions for determining whether the occupant is properly seated, the normal seating determination unit 104 determines that the posture of the occupant is a normal seating state based on the facial direction information. On the other hand, if the facial direction of the occupant does not satisfy the conditions for determining normal seating, the normal seating determination unit 104 determines that the posture of the occupant is not a normal seating state based on the facial direction information.

[0028] The normal seating determination unit 104 outputs a determination result as to whether or not the posture of the occupant has been determined to be a normal seating state (hereinafter referred to as a "seating state determination result") to the feature calculation unit 105. At this time, the normal seating determination unit 104 outputs the skeleton point information output from the skeleton point extraction unit 102 to the feature calculation unit 105 together with the seating state determination result. Furthermore, the normal seating determination unit 104 outputs the seating state determination result to the posture warning unit 108.

[0029] Based on the skeleton point information output from the skeleton point extraction unit 102 and the seating state determination result by the normal seating determination unit 104 as to whether the occupant's posture is a normal seating state, the feature calculation unit 105 determines whether the skeleton points of the occupant extracted by the skeleton point extraction unit 102 are skeleton points extracted based on an image captured when the occupant is in a normal seating state (hereinafter referred to as "normal seating skeleton points"). For example, when the seating state determination result output from the normal seating determination unit 104 indicates that the occupant's posture is normal seating, the feature amount calculation unit 105 determines that the skeleton points extracted by the skeleton point extraction unit 102 are normal seating skeleton points. In other words, the feature amount calculation unit 105 determines that the skeleton point information output from the skeleton point extraction unit 102 is normal seating skeleton point information. In the first embodiment, the feature amount calculation unit 105 acquires the skeleton point information output from the skeleton point extraction unit 102 via the normal seating determination unit 104, but this is merely an example. For example, the feature amount calculation unit 105 may acquire the skeleton point information directly from the skeleton point extraction unit 102. In this case, in FIG. 1 , an arrow is drawn from the skeleton point extraction unit 102 to the feature amount calculation unit 105.

[0030] When the feature calculation unit 105 determines that the skeleton points of the occupant extracted by the skeleton point extraction unit 102 are regular seating skeleton points, it calculates feature amounts (hereinafter referred to as "features for physique determination") to be used for determining the occupant's physique based on the skeleton point information of the regular seating skeleton points (hereinafter referred to as "regular seating skeleton point information"). The feature amount calculation unit 105 calculates the feature amount for determining the physique for each occupant. The feature amount calculation unit 105 can identify the skeleton points of each occupant from the normal seating skeleton point information. It should be noted that the feature amount calculation unit 105 does not calculate the physique determination feature amount when it is determined that the skeleton points of the occupant extracted by the skeleton point extraction unit 102 are not regular seating skeleton points.

[0031] For example, the feature amount calculation unit 105 calculates a feature amount for physique determination from the length in the captured image of a line segment connecting two of the skeleton points in the captured image based on the normal sitting skeleton point information. In detail, for example, the feature amount calculation unit 105 calculates the arm length, shoulder width, and neck length of the occupant in the captured image as the feature amounts for determining the physique, based on the normal seating skeleton point information. The feature amount calculation unit 105 may, for example, determine the length of a line segment connecting a skeleton point representing a shoulder and a skeleton point representing an elbow as the occupant's arm length, which is used as a feature amount for determining physique. The feature amount calculation unit 105 may, for example, determine the length of a line segment connecting the skeleton points representing both shoulders, the length of a line segment connecting the skeleton point representing the neck and a skeleton point representing the right shoulder, or the length of a line segment connecting the skeleton point representing the neck and a skeleton point representing the left shoulder as the shoulder width, which is used as a feature amount for determining physique. The feature amount calculation unit 105 may, for example, determine the length of a line segment connecting the skeleton point representing the nose and a skeleton point representing the neck as the neck length, which is used as a feature amount for determining physique. Note that this is merely an example, and the feature amount calculation unit 105 does not need to calculate all of the occupant's arm length, shoulder width, and neck length as the feature amount for determining physique, but may calculate at least one of the occupant's arm length, shoulder width, and neck length as the feature amount for determining physique. Furthermore, the feature amount calculation unit 105 may use the length of a line segment connecting two skeleton points, other than the occupant's arm length, shoulder width, and neck length, as the feature amount for determining physique.

[0032] The feature amount calculation unit 105 outputs information relating to the calculated physique determination feature amounts (hereinafter referred to as “feature amount information”) to the feature amount leveling unit 106. In the feature information, for each occupant, information indicating the occupant is associated with a physical build determining feature. The information indicating the occupant is, for example, information indicating the seat in which the occupant is seated.

[0033] The feature leveling unit 106 levels a plurality of physique determination features for a predetermined past period (hereinafter referred to as the "leveling period"), in other words, a plurality of physique determination features calculated by the feature calculation unit 105 going back through the leveling period. The smoothing period can be set to an appropriate length and is set by an administrator or the like and stored in a location that can be referenced by the feature amount smoothing unit 106.

[0034] For example, the feature amount leveling unit 106 associates the feature amount information output from the feature amount calculation unit 105 with the acquisition date and time of the feature amount information and stores the information in chronological order in a location that can be referenced by the feature amount leveling unit 106. The feature amount leveling unit 106 may acquire a plurality of physique determination features for the leveling period from the stored feature amount information. Note that if physique determination features for the leveling period are not stored, such as immediately after the vehicle starts traveling, the feature amount leveling unit 106 may use, for example, only the stored physique determination features as the physique determination features for the leveling period. The feature amount information stored by the feature amount calculation unit 105 is deleted, for example, when the power supply of the vehicle is turned on.

[0035] The feature amount leveling unit 106 levels the plurality of physique determination feature amounts for the leveling period by, for example, averaging, taking the median, or taking a value corresponding to a set percentile. In other words, the feature leveling unit 106 calculates, for example, the average value of multiple physique determination features for the leveling period, the median value of multiple physique determination features for the leveling period, or a value corresponding to a set percentile of multiple physique determination features for the leveling period as the physique determination feature after leveling (hereinafter referred to as the "post-leveling physique determination feature").

[0036] The feature amount leveling unit 106 calculates the leveled physique determination feature amount for each occupant. The feature amount leveling unit 106 can identify the physique determination feature amount of each occupant from the feature amount information. Furthermore, when the feature calculation unit 105 calculates multiple types of features for determining physique, such as the occupant's arm length, shoulder width, and neck length, the feature leveling unit 106 levels each of the features for determining physique.

[0037] By leveling the physique determination features by the feature leveling unit 106, even if a physique determination feature that temporarily deviates from the feature corresponding to the occupant's original physique is calculated, the physique determination device 100 can reduce the influence on the occupant's physique determination using the physique determination feature (more specifically, the leveled physique determination feature). The physique determination of the occupant is performed by the physique determination unit 107. Details of the physique determination unit 107 will be described later.

[0038] For example, when the feature amount leveling unit 106 levels a plurality of physique determination features by averaging the plurality of physique determination features for a leveling period, the feature amount leveling unit 106 can set accurate values, in other words, values ​​closer to the features corresponding to the occupant's original physique, as the post-leveling physique determination features even when the number of the plurality of physique determination features is small. Therefore, in this case, for example, the administrator or the like can narrow the face direction determination range defined in the normal seating determination conditions used by the normal seating determination unit 104 when determining whether the occupant's posture is normal seating. Although it is expected that narrowing the range for determining face direction will reduce the frequency with which the proper seating determination unit 104 determines that the occupant's posture is properly seated, the feature amount leveling unit 106 can then calculate accurate normalized physique determination features from the few physique determination features, even if the number of physique determination features is small when the occupant's posture is determined to be properly seated. As a result, the physique determination device 100 can prevent a decrease in the accuracy of occupant physique determination using the normalized physique determination features.

[0039] Furthermore, for example, when the feature amount leveling unit 106 levels the plurality of physique determination features by taking the median value of the plurality of physique determination features for the leveling period or by taking a value corresponding to a set percentile, the feature amount leveling unit 106 can select an accurate value from the many physique determination features, in other words, a value that is closer to the feature amount corresponding to the occupant's main body physique, as the physique determination feature after leveling, even if the number of physique determination features is large. Therefore, in this case, for example, an administrator or the like can widen the face direction determination range defined in the normal seating determination conditions used by the normal seating determination unit 104 when determining whether the occupant's posture is normal, thereby making it easier for the normal seating determination unit 104 to determine that the occupant's posture is normal. Although it is expected that the accuracy of determining whether an occupant's posture is properly seated will decrease if the proper seating determination unit 104 makes it easier to determine that the occupant's posture is properly seated, the feature amount leveling unit 106 can then calculate accurate post-leveling physique determination features from many of the physique determination features obtained when the occupant's posture is determined to be properly seated. As a result, the physique determination device 100 can prevent a decrease in the accuracy of determining the occupant's physique using the post-leveling physique determination features.

[0040] The feature amount leveling unit 106 outputs information on the leveled physique determination feature amounts (hereinafter referred to as “leveled feature amount information”) to the physique determination unit 107. In the leveled feature information, information indicating each occupant is associated with the leveled feature information. The information indicating the occupant is, for example, information indicating the seat in which the occupant is seated.

[0041] The physique determination unit 107 determines the physique of the occupant based on the post-leveling physique determination feature amounts after the feature amount leveling unit 106 has leveled them. The physique determination unit 107 determines the physique of each occupant. The physique determination unit 107 can identify the normalized physique determination feature amount of each occupant from the normalized feature amount information.

[0042] The physique determination unit 107 may determine the physique of the occupant by a known method using known technology such as image recognition technology or technology using a trained model (hereinafter referred to as a "machine learning model"). The physique determination unit 107 determines the physique of the occupant using a known method, for example, either "child," "small," "standard," or "large." Note that this is just one example, and the definition of the physique determined by the physique determination unit 107 can be set as appropriate.

[0043] The physique determination unit 107 outputs the physique determination result to the seat belt control device 300, the airbag control device 400, and the abandoned occupant detection device 500. In the physical build determination result, for each occupant, information indicating the occupant is associated with information indicating the occupant's physical build. The information indicating the occupant is, for example, information indicating the seat in which the occupant is sitting.

[0044] If the occupant's posture, as determined by the normal seating determination unit 104, is not a normal seating state and continues for a predetermined time (hereinafter referred to as the "abnormal posture determination time"), the posture warning unit 108 outputs information (hereinafter referred to as "posture warning information") to the output device 600 to urge the occupant to adopt a normal seating posture. Based on the seating state determination result output from the normal seating determination unit 104, the posture warning unit 108 can determine that the normal seating determination unit 104 has determined that the occupant's posture is not a normal seating state.

[0045] For example, the posture warning unit 108 associates the seating state determination results output from the proper seating determination unit 104 with the acquisition date and time of the seating state determination results and stores them in chronological order in a location that can be referenced by the posture warning unit 108. Based on the stored seating state determination results, the posture warning unit 108 can determine whether the occupant's posture is not a proper seating state for the abnormal posture determination time period. Note that the seating state determination results stored by the posture warning unit 108 are deleted, for example, when the vehicle power is turned on. Then, when the posture warning unit 108 determines that the state in which the posture of the occupant is not in a normal seated state continues for the abnormal posture determination time, it outputs posture warning information to the output device 600.

[0046] By outputting the posture warning information, the posture warning unit 108 causes the output device 600 to, for example, display a message urging the occupant to sit properly, output the message as audio, or output a warning sound urging the occupant to sit properly.

[0047] The posture warning unit 108 outputs posture warning information to prompt the occupant to sit properly, so that the physique determination device 100 can make the occupant take a correct posture. As a result, the physique determination device 100 can more easily determine that the posture of the occupant is a properly seated state.

[0048] The operation of the physique determination device 100 according to the first embodiment will be described. FIG. 3 is a flowchart illustrating an example of the operation of the physique determination device 100 according to the first embodiment. For example, the physique determination device 100 periodically repeats the operation shown in the flowchart of FIG. 3 while the vehicle is running. Furthermore, for example, when the power of the vehicle is turned on, the physique determination device 100 may repeat the operation shown in the flowchart of FIG. 3 until the power of the vehicle is turned off.

[0049] The image acquisition unit 101 acquires an image of an occupant captured by the imaging device 200 (step ST1). The image acquisition unit 101 outputs the acquired captured image to the skeleton point extraction unit 102.

[0050] The skeleton point extraction unit 102 extracts skeleton points of the occupant, which indicate body parts of the occupant, based on the captured image acquired by the image acquisition unit 101 in step ST1 (step ST2). The skeleton point extraction unit 102 outputs the skeleton point information to the situation information acquisition unit 103 .

[0051] When the skeleton point extraction unit 102 extracts the skeleton points of the occupant in step ST2, the situation information acquisition unit 103 acquires situation information (step ST3). In step ST3, the face direction detection unit 1031 detects the face direction of the occupant based on the captured image acquired in step ST1 by the image acquisition unit 101. The situation information acquisition unit 103 acquires, as situation information, face direction information regarding the face direction of the occupant detected by the face direction detection unit 1031. The situation information acquisition unit 103 outputs the acquired situation information, in this case, face direction information, to the normal seating determination unit 104.

[0052] The normal seating determination unit 104 determines whether the posture of the occupant is a normal seating state or not (step ST4) based on the situation information acquired by the situation information acquisition unit 103 in step ST3. More specifically, here, the normal seating determination unit 104 determines whether the posture of the occupant is a normal seating state or not based on the face direction information. The regular seating determination unit 104 outputs the seating state determination result to the feature amount calculation unit 105. At this time, the regular seating determination unit 104 outputs the skeleton point information output from the skeleton point extraction unit 102 to the feature amount calculation unit 105 together with the seating state determination result. Furthermore, the normal seating determination unit 104 outputs the seating state determination result to the posture warning unit 108.

[0053] The feature calculation unit 105 determines whether the skeleton points of the occupant extracted by the skeleton point extraction unit 102 in step ST2 are normal seating skeleton points or not, based on the skeleton point information output from the skeleton point extraction unit 102 in step ST2 and the seating state determination result by the normal seating determination unit 104 in step ST4, which determines whether the state of the occupant's posture is a normal seating state or not. Then, when the feature calculation unit 105 determines in step ST2 that the skeleton point of the occupant extracted by the skeleton point extraction unit 102 is a normal seating skeleton point, it calculates a feature for physique determination based on the normal seating skeleton point information of the normal seating skeleton point (step ST5). The feature amount calculation unit 105 outputs the feature amount information to the feature amount leveling unit 106 .

[0054] If the feature calculation unit 105 determines in step ST2 that the skeleton points of the occupant extracted by the skeleton point extraction unit 102 are not regular seating skeleton points, the calculation of the feature values ​​for physique determination is not performed, and the operation of the physique determination device 100 ends the processing shown in the flowchart of Figure 3.

[0055] The feature leveling unit 106 levels the multiple physique determination feature values ​​for the past leveling period, in other words, the multiple physique determination feature values ​​calculated by the feature calculation unit 105 going back through the leveling period, based on the normal sitting skeleton point information output from the normal sitting determination unit 104 (step ST6). The feature amount leveling unit 106 outputs the leveled feature amount information to the physique determination unit 107.

[0056] The physique determination unit 107 determines the physique of the occupant based on the normalized physique determination features normalized by the feature normalization unit 106 in step ST6 (step ST7). The physique determination unit 107 outputs the physique determination result to the seat belt control device 300, the airbag control device 400, and the abandoned occupant detection device 500.

[0057] If the occupant's posture, determined by the normal seating determination unit 104 in step ST4, is not a normal seating state and continues for the abnormal posture determination time, the posture warning unit 108 outputs posture warning information to the output device 600 (step ST8).

[0058] In this way, the physique determination device 100 extracts the skeleton points of the vehicle occupant based on a captured image of the vehicle occupant. The physique determination device 100 also determines whether the occupant's posture is in a properly seated state based on situation information, in this case, facial orientation information. The physique determination device 100 determines whether the extracted skeleton points of the occupant are proper seating skeleton points based on the skeleton point information and the determination result of whether the occupant's posture is in a properly seated state, and if it determines that the extracted skeleton points are proper seating skeleton points, it calculates physique determination feature quantities based on the proper seating skeleton point information. The physique determination device 100 then determines the occupant's physique based on the calculated physique determination feature quantities.

[0059] As described above, when an occupant is sitting in a seat with poor posture, there are countless possible patterns of poor posture. Therefore, when an occupant is sitting in a seat with poor posture, it is difficult to cover all the countless possible patterns of poor posture, and the occupant's posture determined based on the captured image is more likely to contain errors, or the errors will be larger, compared to when the occupant is sitting properly. For example, the occupant's skeleton points extracted from the captured image are more likely to contain errors, or the errors will be larger. When the occupant's skeleton points extracted from the captured image are more likely to contain errors, or the errors will be larger, the feature amounts calculated based on the occupant's skeleton points are also more likely to contain errors, or the errors will be larger. As a result, the occupant's posture determined based on the feature amounts is also more likely to contain errors, or the errors will be larger. As a result, the accuracy of the physique determination is reduced. It should be noted that when an occupant is seated in a seat without losing their posture, i.e., when they are seated properly, the positions of the skeleton points, which are characteristic points of the occupant, in the captured image are expected to fall within a certain range. Therefore, when an occupant is seated properly, errors are unlikely to occur in the skeleton points of the occupant extracted from the captured image and in the feature amounts calculated from the skeleton points. As a result, errors are unlikely to occur in the posture of the occupant determined based on the feature amounts. Even if errors do occur, the impact of the errors on the physique determination is not significant.

[0060] The occupant's physical size determination result based on the captured image is used in functions such as seat belt control, airbag control, and abandoned vehicle detection. Because functions such as seat belt control, airbag control, and abandoned vehicle detection are safety-related functions, the physical size determination result used in these functions requires high accuracy. Therefore, an occupant's physical size determination result with low accuracy determined when the occupant is sitting in a slumped position in the seat cannot be used in functions such as seat belt control, airbag control, and abandoned vehicle detection. If the physical size determination device provided a physical size determination result determined when the occupant is sitting in a slumped position in the seat as the physical size determination result used in functions such as seat belt control, airbag control, and abandoned vehicle detection, erroneous control or erroneous detection may occur in the above functions.

[0061] In contrast, as described above, the physique determination device 100 according to the first embodiment determines whether the posture of an occupant is in a properly seated state, and determines the occupant's physique based on the feature amounts based on skeleton points extracted from the captured image when the posture of the occupant is determined to be in a properly seated state, and determines the occupant's physique based on the feature amounts. The physique determination device 100 calculates the feature amounts for determining physique based on the captured image when the posture of the occupant is in a properly seated state, and determines the occupant's physique based on the feature amounts for determining physique, thereby preventing a decrease in the accuracy of the occupant's physique determination. In other words, the physique determination device 100 can prevent a decrease in the accuracy of the occupant's physique determination, even if, for example, the occupant's posture becomes unstable while the vehicle is traveling. Furthermore, the physical build determination device 100 provides occupant physical build determination results that are designed to prevent a decrease in accuracy, and therefore can provide physical build determination results that can be used in functions such as seat belt control, airbag control, and abandoned vehicle detection, which require high accuracy in the physical build determination results.

[0062] In the above-described first embodiment, in the physique determination device 100, the situation information acquisition unit 103 includes a face direction detection unit 1031, which detects the face direction of the occupant based on the captured image acquired by the image acquisition unit 101. However, this is merely an example. For example, the situation information acquisition unit 103 may acquire the face direction information of the occupant from a device provided outside the physique determination device 100 in a location accessible to the physique determination device 100, such as an occupant state detection device provided in the DMS. In this case, it is assumed that the image capture device 200 is shared with the DMS. In this case, in the physique determination device 100, the situation information acquisition unit 103 may be configured without including the face direction detection unit 1031.

[0063] In the first embodiment, the situation information acquired by the situation information acquisition unit 103 in the physique determination device 100 is information relating to the facial orientation of the occupant, but this is merely an example. In the first embodiment, the situation information acquired by the situation information acquisition unit 103 in the physique determination device 100 may be, for example, skeleton point information. In this case, the normal seating determination unit 104 may have the function of the situation information acquisition unit 103. In other words, it is not essential that the physique determination device 100 shown in FIG. In this case, the skeleton point extraction unit 102 outputs the skeleton point information to the normal seating determination unit 104 . The normal seating determination unit 104 determines whether the posture of the occupant is a normal seating state based on the skeleton point information output from the skeleton point extraction unit 102.

[0064] For example, in this case, the normal seating determination conditions are defined as follows: <Condition 2>. <Condition 2> The position of the skeleton point indicating the elbow of the occupant on the captured image is not located within a preset range (hereinafter referred to as the "elbow position determination range"). The elbow position determination range in <Condition 2> is set in advance to, for example, a predetermined range including the armrest or a predetermined range near the door of the vehicle.

[0065] If the position of the skeleton point indicating the elbow of the occupant satisfies the normal seating determination condition based on the skeleton point information, the normal seating determination unit 104 determines that the posture of the occupant is a normal seating state. On the other hand, if the inclination of the skeletal line of the occupant does not satisfy the conditions for determining normal seating based on the inclination information, the normal seating determination unit 104 determines that the posture of the occupant is not a normal seating state. For example, if the position of the skeleton point indicating the elbow of the occupant is located within the elbow position determination range, the normal seating determination unit 104 determines that the posture of the occupant is not a normal seating state.

[0066] The flow chart showing the operation of the physique determination device 100 when situation information is used as skeleton point information is as shown in FIG. However, in step ST3, the normal seating determination unit 104 acquires, as situation information, the skeleton point information output by the skeleton point extraction unit 102 in step ST2. Then, in step ST4, the normal seating determination unit 104 determines whether the posture of the occupant is a normal seating state or not, based on the skeleton point information acquired in step ST3.

[0067] Other specific operations of the physique determination device 100 in steps ST1 to ST2 and steps ST5 to ST8 are the same as those already explained, and therefore will not be explained again.

[0068] In this way, the situation information may be the skeleton point information of the occupant, and in the physique determination device 100, the proper seating determination unit 104 may determine whether the occupant's posture is in a properly seated state based on the skeleton point information of the occupant extracted by the skeleton point extraction unit 102. In this case, too, the physique determination device 100 calculates physique determination features based on a captured image when the occupant's posture is in a properly seated state, and determines the occupant's physique based on the physique determination features, thereby preventing a decrease in the accuracy of the occupant's physique determination. In other words, even if the occupant's posture becomes unstable while the vehicle is traveling, the physique determination device 100 can prevent a decrease in the accuracy of the occupant's physique determination.

[0069] Furthermore, in the above-described first embodiment, the physique determination device 100 may acquire, as the situation information, information about the inclination of the skeleton lines connecting the skeleton points of the occupant (hereinafter referred to as "inclination information"). In this case, even if a state occurs in which only the occupant's face is not facing forward, it is not regarded as a loss of posture. FIG. 4 is a diagram showing an example of the configuration of the physique determination device 100 in the first embodiment when tilt information is used as situation information. The configuration example of the physique determination device 100 shown in Figure 4 differs from the configuration example of the physique determination device 100 shown in Figure 1 in that the situation information acquisition unit 103 includes an inclination detection unit 1032 instead of the face direction detection unit 1031. Furthermore, in the physique determination device 100 shown in Fig. 4, the specific operation of the normal sitting determination unit 104 is different from the specific operation of the normal sitting determination unit 104 in the physique determination device 100 shown in Fig. 1. The specific operations of the image acquisition unit 101, skeleton point extraction unit 102, feature calculation unit 105, feature leveling unit 106, physique determination unit 107, and posture attention unit 108 are the same as the specific operations already described, and therefore redundant description will be omitted.

[0070] The inclination detection unit 1032 detects the inclination of a skeleton line, which is a line segment connecting a plurality of skeleton points of the occupant, based on the skeleton point information output from the skeleton point extraction unit 102 . For example, the inclination detection unit 1032 detects the inclination of the skeletal line of the occupant's shoulders, which connects the skeletal points that indicate both shoulders of the occupant. Note that this is merely an example, and the inclination detection unit 1032 can detect the inclination of any appropriate skeletal line, such as the inclination of the skeletal line that indicates the torso (for example, the skeletal line that indicates the sitting height). However, an imbalance in the occupant's posture is likely to be reflected in the inclination of the skeletal line of the occupant's shoulders. Therefore, by detecting the inclination of the skeletal line of the occupant's shoulders, the inclination detection unit 1032 can reduce erroneous determination by the proper seating determination unit 104 as to whether the occupant's posture is in a proper seating state.

[0071] The situation information acquisition unit 103 acquires tilt information as situation information. The tilt information includes information indicating the occupant and information indicating the tilt of the skeleton line, which are associated with each other. In addition to these, the tilt information may also include the captured image acquired by the image acquisition unit 101. The situation information acquisition unit 103 outputs the acquired situation information, in this case, tilt information, to the normal seating determination unit 104.

[0072] When the configuration example of the physique determination device 100 is the configuration example shown in Figure 4, the normal seating determination unit 104 determines whether the occupant's posture is normal seating or not based on the inclination information regarding the inclination of the occupant's skeletal line detected by the inclination detection unit 1032. For example, in this case, the normal seating determination conditions are defined as the following <Condition 3>. <Condition 3> The inclination of the occupant's skeleton line is within a preset range (hereinafter referred to as the "inclination determination range"). For the range for determining the inclination in <Condition 3>, a predetermined range is set in advance, for example, based on the inclination of the skeleton line of an occupant who is considered to be facing forward.

[0073] If the inclination of the skeletal line of the occupant satisfies the conditions for determining normal seating based on the inclination information, the normal seating determination unit 104 determines that the posture of the occupant is a normal seating state. On the other hand, if the inclination of the skeletal line of the occupant does not satisfy the conditions for determining normal seating based on the inclination information, the normal seating determination unit 104 determines that the posture of the occupant is not a normal seating state.

[0074] The flowchart showing the operation of the physique determination device 100 when the configuration example of the physique determination device 100 is as shown in FIG. 4 is the same as the flowchart shown in FIG. However, when the configuration example of the physique determination device 100 is the configuration example shown in FIG. 4, in step ST3, the situation information acquisition unit 103 acquires tilt information as situation information. Then, the situation information acquisition unit 103 outputs the acquired situation information, in this case, the tilt information, to the normal seating determination unit 104. In step ST4, the normal seating determination unit 104 determines whether the posture of the occupant is a normal seating state based on the tilt information.

[0075] Other specific operations of the physique determination device 100 in steps ST1 to ST2 and steps ST5 to ST8 are the same as those already explained, and therefore will not be explained again.

[0076] In this way, the situation information may be inclination information, the physique determination device 100 may include an inclination detection unit 1032 that detects the inclination of the occupant's skeleton line based on information about the occupant's skeleton points extracted by the skeleton point extraction unit 102, and the proper seating determination unit 104 may determine whether the occupant's posture is in a properly seated state based on the inclination of the occupant's skeleton line detected by the inclination detection unit 1032. In this case, too, the physique determination device 100 calculates physique determination features based on a captured image when the occupant's posture is in a properly seated state and determines the occupant's physique based on the physique determination features, thereby preventing a decrease in the accuracy of the occupant's physique determination. In other words, the physique determination device 100 can prevent a decrease in the accuracy of the occupant's physique determination even if, for example, the occupant's posture becomes unstable while the vehicle is traveling.

[0077] In the first embodiment, the situation information may be not only information about the situation of the occupant but also information about the situation of the vehicle, for example. In this case, even if a situation occurs in which only the occupant's face is not facing forward, this is not considered to be a loss of posture. For example, in the first embodiment, the physique determination device 100 may acquire information relating to the state of the vehicle (hereinafter referred to as "vehicle information") as the state information. Vehicle information includes, for example, information regarding the vehicle's steering angle, vehicle speed, vehicle door opening / closing status, vehicle ignition status, or operating status of in-vehicle devices (such as a car navigation system or room lamps).

[0078] FIG. 5 is a diagram showing an example of the configuration of the physique determination device 100 in the first embodiment when vehicle information is used as situation information. The configuration example of the physique determination device 100 shown in Figure 5 differs from the configuration example of the physique determination device 100 shown in Figure 1 in that the situation information acquisition unit 103 includes a vehicle information acquisition unit 1033 instead of the face direction detection unit 1031. 1. In addition, in the physique determination device 100 shown in FIG. 5, the specific operation of the regular seating determination unit 104 is different from the specific operation of the regular seating determination unit 104 in the physique determination device 100 shown in FIG. The specific operations of the image acquisition unit 101, the skeleton point extraction unit 102, the feature calculation unit 105, the feature leveling unit 106, the physique determination unit 107, and the posture attention unit 108 are the same as those already explained, and therefore will not be explained again.

[0079] The vehicle information acquisition unit 1033 acquires vehicle information from various devices mounted on the vehicle. For example, the vehicle information acquisition unit 1033 may acquire information relating to the state of the steering wheel steering angle from a steering angle sensor. Also, for example, the vehicle information acquisition unit 1033 may acquire vehicle speed from a vehicle speed sensor. Also, for example, the vehicle information acquisition unit 1033 may acquire information relating to the opening and closing state of a door from a sensor provided in the door. Also, for example, the vehicle information acquisition unit 1033 may acquire information relating to the ignition state from an ignition sensor. Also, for example, the vehicle information acquisition unit 1033 may acquire information relating to the operation state of an in-vehicle device from an in-vehicle device such as a car navigation device.

[0080] The situation information acquisition unit 103 acquires the vehicle information acquired by the vehicle information acquisition unit 1033 as situation information. The situation information acquisition unit 103 outputs the acquired situation information, in this case, vehicle information, to the normal seating determination unit 104.

[0081] When the configuration example of the physique determination device 100 is the configuration example shown in Figure 5, the normal seating determination unit 104 determines whether the occupant's posture is normal seating or not based on the vehicle information acquired by the vehicle information acquisition unit 1033. For example, in this case, the normal seating determination conditions are defined as the following <Condition 4> to <Condition 8>. <Condition 4> The vehicle's steering angle is within a preset range (hereinafter referred to as the "steering angle determination range"). <Condition 5> The vehicle speed exceeds a preset range (hereinafter referred to as the "vehicle speed determination range"). <Condition 6> The vehicle doors are not open <Condition 7> The time that has elapsed since the ignition was turned on must be within a preset time (hereinafter referred to as the "start determination time"). <Condition 8> The in-vehicle device is not being operated. The steering angle determination range in <Condition 4> is set to a range of steering angles that cannot be achieved without bending the arms, for example. Furthermore, the vehicle speed determination range in <Condition 5> is set to a vehicle speed that assumes driving on an expressway, for example. Regarding <Condition 6>, for example, if the vehicle door is open, it is assumed that the occupant is getting on or off the vehicle and is not properly seated.

[0082] If the vehicle information satisfies the conditions for determining whether the occupant is properly seated, the normal seating determination unit 104 determines that the posture of the occupant is in a normal seating state. On the other hand, if the vehicle information does not satisfy the conditions for determining normal seating, the normal seating determination unit 104 determines that the posture of the occupant is not a normal seating state.

[0083] The flowchart showing the operation of the physique determination device 100 when the configuration example of the physique determination device 100 is as shown in FIG. 5 is the same as the flowchart shown in FIG. However, when the configuration example of the physique determination device 100 is the configuration example shown in FIG. 5, in step ST3, the situation information acquisition unit 103 acquires the vehicle information acquired by the vehicle information acquisition unit 1033 as the situation information. Then, the situation information acquisition unit 103 outputs the acquired situation information, here the vehicle information, to the normal seating determination unit 104. In step ST4, the normal seating determination unit 104 determines whether the posture of the occupant is a normal seating state based on the vehicle information acquired by the vehicle information acquisition unit 1033 in step ST3.

[0084] Other specific operations of the physique determination device 100 in steps ST1 to ST2 and steps ST5 to ST8 are the same as those already explained, and therefore will not be explained again.

[0085] In this way, the situation information may be vehicle information, the physique determination device 100 may include a vehicle information acquisition unit 1033 that acquires the vehicle information, and the proper seating determination unit 104 may determine whether the occupant's posture is in a properly seated state based on the vehicle information acquired by the vehicle information acquisition unit 1033. In this case, the physique determination device 100 calculates physique determination features based on a captured image when the occupant's posture is in a properly seated state and determines the occupant's physique based on the physique determination features, thereby preventing a decrease in the accuracy of the occupant's physique determination. In other words, even if the occupant's posture becomes unstable while the vehicle is traveling, the physique determination device 100 can prevent a decrease in the accuracy of the occupant's physique determination.

[0086] In this way, the physique determining device 100 may acquire not only face direction information but also skeleton point information, tilt information, or vehicle information as situation information. However, devices other than the physique determination device 100 often have the function of detecting the facial direction of an occupant based on an image captured in a vehicle. As described above, for example, an occupant state detection device provided in a DMS may have the function of detecting the facial direction of an occupant. For example, by using facial direction information as situation information, the physique determination device 100 can acquire the facial direction information from a device external to the physique determination device 100, and does not need to be provided with the function of detecting the facial direction. Furthermore, because the physique determination device 100 does not need to perform processing for detecting the facial direction of an occupant, the processing load for determining the physique of an occupant can be reduced.

[0087] Note that even if skeleton point information is used as situation information, the physique determination device 100 does not need to add a function for acquiring situation information. However, generally, detecting the position of the eyes or mouth based on a captured image provides higher detection accuracy than detecting the position of skeleton points based on a captured image. For example, determining whether an occupant is facing forward based on the facial direction of the occupant provides higher determination accuracy than determining whether the occupant is facing forward based on the occupant's skeleton points. Note that a person's facial direction is generally detected using the position of the eyes or mouth. Therefore, the physique determining device 100 can determine whether the posture of the occupant is in a normal seated state more accurately when facial direction information is used as situation information than when skeleton point information is used as situation information.

[0088] In the first embodiment, the physique determining device 100 may acquire, as situation information, two or more of face direction information, skeleton point information, tilt information, and vehicle information. That is, in the physique determination device 100, the situation information acquisition unit 103 may be configured to include two or more of a face direction detection unit 1031, a tilt detection unit 1032, and a vehicle information acquisition unit 1033, for example.

[0089] Furthermore, in the above-described embodiment 1, in the physical type determination device 100, the normal seating determination unit 104 determines whether the occupant's posture is normal seating or not based on the situation information and in accordance with the normal seating determination conditions, but this is merely one example. For example, the proper seating determination unit 104 may determine whether the posture of the occupant is a proper seating state based on the situation information and a machine learning model (hereinafter referred to as a "proper seating determination model"). For example, the model for determining proper seating is a machine learning model that receives situation information as input and outputs information indicating whether the occupant's posture is a proper seating state. The normal seating determination model is generated in advance by an administrator or the like and stored in a location that can be referenced by the physique determination device 100.

[0090] In addition, in the above-described embodiment 1, the normal seating determination unit 104 may determine whether the occupant is in a normal seating state or not based on the situation information, using both the normal seating determination conditions and the normal seating determination model. For example, the regular seating determination unit 104 determines whether the occupant's posture is in a regular seating state (first determination) based on the situation information and in accordance with the regular seating determination conditions. Also, the regular seating determination unit 104 determines whether the occupant's posture is in a regular seating state (second determination) based on the situation information and the regular seating determination model. Then, if the regular seating determination unit 104 determines that the occupant's posture is in a regular seating state in both the first determination and the second determination, it determines that the occupant's posture is in a regular seating state, and if it determines that the occupant's posture is not in a regular seating state in either the first determination or the second determination, it determines that the occupant's posture is not in a regular seating state. Furthermore, for example, the seating state determination result of the first determination and the seating state determination result of the second determination may be weighted to determine whether the posture of the occupant is in a normal seating state.

[0091] In the first embodiment, the physique determination device 100 includes the feature amount leveling unit 106. However, this is merely an example. The physique determination device 100 may be configured without the feature amount leveling unit 106. In this case, the physique determination unit 107 may determine the physique of the occupant based on the physique determination feature amount calculated by the feature amount calculation unit 105. 3, the physique determination device 100 can omit the processing of step ST6. In step ST7, the physique determination unit 107 determines the physique of the occupant based on the physique determination features calculated by the feature calculation unit 105 in step ST5.

[0092] Furthermore, in the above-described first embodiment, the physique determination device 100 includes the posture warning unit 108, but this is merely an example. The physique determination device 100 may be configured without the posture warning unit 108. Furthermore, the physique determination device 100 does not have to be connected to the output device 600. In this case, in the operation of the physique determination device 100 described using the flowchart of FIG. 3, the physique determination device 100 can omit the processing of step ST8.

[0093] In the first embodiment, the physique determining device 100 is connected to the seatbelt control device 300, the airbag control device 400, and the abandonment detection device 500, but this is merely an example. The physical size determination device 100 may be connected only to either the seat belt control device 300, the airbag control device 400, or the abandonment detection device 500, or may be connected to a device other than the seat belt control device 300, the airbag control device 400, or the abandonment detection device 500 that performs various processes using the physical size determination results. For example, the physique determination device 100 may be connected to a display device mounted on a vehicle. The display device, for example, displays information according to the physique determination result output from the physique determination device 100. For example, if there is a small child among the occupants, the display device displays an icon indicating that a child is riding in the vehicle.

[0094] Furthermore, in the above-described first embodiment, it is assumed that there is one imaging device 200, but this is merely an example. In the first embodiment, a plurality of imaging devices 200 may be installed in the vehicle. For example, a plurality of imaging devices 200 may be installed so as to be able to capture images of passengers seated in each seat in the vehicle. In this case, the physique determination device 100 acquires captured images from the multiple image capture devices 200 and determines the physique of the occupant.

[0095] 6A and 6B are diagrams illustrating an example of the hardware configuration of the physique determination device 100 according to the first embodiment. In the first embodiment, the functions of the image acquisition unit 101, skeleton point extraction unit 102, situation information acquisition unit 103, face direction detection unit 1031, tilt detection unit 1032, vehicle information acquisition unit 1033, normal seating determination unit 104, feature amount calculation unit 105, feature amount leveling unit 106, physique determination unit 107, and posture warning unit 108 are realized by a processing circuit 1001. That is, the physique determination device 100 includes the processing circuit 1001 for performing control to determine the physique of a vehicle occupant based on a captured image of the occupant. The processing circuit 1001 may be dedicated hardware as shown in FIG. 6A, or may be a processor 1004 that executes a program stored in memory as shown in FIG. 6B.

[0096] When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0097] When the processing circuit is the processor 1004, the functions of the image acquisition unit 101, skeleton point extraction unit 102, situation information acquisition unit 103, face direction detection unit 1031, tilt detection unit 1032, vehicle information acquisition unit 1033, normal seating determination unit 104, feature amount calculation unit 105, feature amount leveling unit 106, physique determination unit 107, and posture attention unit 108 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 1005. The processor 1004 reads and executes the programs stored in the memory 1005, thereby executing the functions of the image acquisition unit 101, the skeleton point extraction unit 102, the situation information acquisition unit 103, the face direction detection unit 1031, the tilt detection unit 1032, the vehicle information acquisition unit 1033, the normal seating determination unit 104, the feature calculation unit 105, the feature leveling unit 106, the physique determination unit 107, and the posture warning unit 108. That is, the physique determination device 100 includes the memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST8 in FIG. 3 described above. Furthermore, it can be said that the programs stored in the memory 1005 cause the computer to execute the processing procedures or methods of the image acquisition unit 101, the skeleton point extraction unit 102, the situation information acquisition unit 103, the face direction detection unit 1031, the tilt detection unit 1032, the vehicle information acquisition unit 1033, the normal seating determination unit 104, the feature amount calculation unit 105, the feature amount leveling unit 106, the physique determination unit 107, and the posture warning unit 108. Here, the memory 1005 corresponds to, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), a magnetic disk, a flexible disk, an optical disk, a compact disk, a minidisk, a DVD (Digital Versatile Disc), etc.

[0098] It is also possible to realize some of the functions of the image acquisition unit 101, skeleton point extraction unit 102, situation information acquisition unit 103, face direction detection unit 1031, tilt detection unit 1032, vehicle information acquisition unit 1033, normal seating determination unit 104, feature calculation unit 105, feature leveling unit 106, physique determination unit 107, and posture attention unit 108 by using dedicated hardware and some by using software or firmware. For example, the image acquisition unit 101 can be realized by a processing circuit 1001 as dedicated hardware, and the functions of the skeleton point extraction unit 102, situation information acquisition unit 103, face direction detection unit 1031, tilt detection unit 1032, vehicle information acquisition unit 1033, normal seating determination unit 104, feature calculation unit 105, feature leveling unit 106, physique determination unit 107, and posture attention unit 108 can be realized by the processor 1004 reading and executing programs stored in the memory 1005. The physical size determination device 100 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as an imaging device 200, a seat belt control device 300, an airbag control device 400, an abandoned vehicle detection device 500, or an output device 600.

[0099] In the above-described first embodiment, the physique determination device 100 is an in-vehicle device mounted on a vehicle, and the skeleton point extraction unit 102, the situation information acquisition unit 103, the normal seating determination unit 104, the feature calculation unit 105, the feature leveling unit 106, the physique determination unit 107, and the posture warning unit 108 are provided in the in-vehicle device. Without being limited to this, some of the skeleton point extraction unit 102, situation information acquisition unit 103, normal seating determination unit 104, feature calculation unit 105, feature leveling unit 106, physique determination unit 107, and posture attention unit 108 may be mounted on the in-vehicle device of the vehicle, and the rest may be provided on a server connected to the in-vehicle device via a network, so that the physique determination system is configured by the in-vehicle device and the server. In addition, the skeleton point extraction unit 102, situation information acquisition unit 103, normal seating determination unit 104, feature amount calculation unit 105, feature amount leveling unit 106, physique determination unit 107, and posture warning unit 108 may all be provided in the server.

[0100] As described above, according to the first embodiment, the physique determination device 100 includes an image acquisition unit 101 that acquires an image of a vehicle occupant, a skeleton point extraction unit 102 that extracts skeleton points of the occupant that indicate body parts of the occupant based on the image acquired by the image acquisition unit 101, a normal seating determination unit 104 that determines whether the posture of the occupant is in a normal seating state based on situation information about the situation of the vehicle or the occupant, and a normal seating determination unit 104 that determines whether the posture of the occupant is in a normal seating state based on the information about the skeleton points of the occupant extracted by the skeleton point extraction unit 102 and the information about the posture of the occupant extracted by the normal seating determination unit 104. The physique determination device 100 is configured to include a feature amount calculation unit 105 that calculates physique determination features based on information about the normal seating skeleton points when the occupant's posture is in a normal seating state and a physique determination unit 107 that determines the physique of the occupant based on the physique determination features calculated by the feature amount calculation unit 105. The physique determination device 100 calculates physique determination features based on the image captured when the occupant's posture is in a normal seating state and determines the occupant's physique based on the physique determination features. In other words, the physique determination device 100 can prevent a decrease in the accuracy of the physique determination of the occupant even if the occupant's posture becomes unstable.

[0101] In addition, in the present disclosure, any of the components of the embodiments may be modified or omitted. [Industrial Applicability]

[0102] The physical size determination device of the present disclosure calculates physical size determination features based on an image captured when the occupant's posture is in a normal seated state, and determines the occupant's physical size based on the physical size determination features, thereby preventing a decrease in the accuracy of the occupant's physical size determination. [Explanation of symbols]

[0103] 100 physique determination device, 101 image acquisition unit, 102 skeletal point extraction unit, 103 situation information acquisition unit, 1031 face direction detection unit, 1032 tilt detection unit, 1033 vehicle information acquisition unit, 104 normal seating determination unit, 105 feature calculation unit, 106 feature leveling unit, 107 physique determination unit, 108 posture warning unit, 200 imaging device, 300 seat belt control device, 400 airbag control device, 500 abandoned object detection device, 600 output device, 1001 processing circuit, 1002 input interface device, 1003 output interface device, 1004 processor, 1005 memory.

Claims

1. an image acquisition unit that acquires an image of a vehicle occupant; a skeleton point extraction unit that extracts skeleton points of the occupant that indicate body parts of the occupant based on the captured image acquired by the image acquisition unit; a normal seating determination unit that determines whether the occupant is properly seated based on situation information relating to the vehicle or the occupant; a feature calculation unit that calculates a physique determination feature based on information about the skeleton points of the occupant extracted by the skeleton point extraction unit and the seating state determination result by the proper seating determination unit, when it is determined that the skeleton points of the occupant extracted by the skeleton point extraction unit are proper seating skeleton points extracted based on the captured image in which the occupant is in the proper seating state; and a physique determination unit that determines the physique of the occupant based on the physique determination feature calculated by the feature calculation unit; A physique determination device comprising:

2. the situation information is information about a facial orientation of the occupant detected based on the captured image, The normal seating determination unit determines that the posture of the occupant is the normal seating state when the face of the occupant is facing forward.

2. The physique determination device according to claim 1.

3. a face direction detection unit that detects the face direction of the occupant based on the captured image acquired by the image acquisition unit, The normal seating determination unit determines that the posture of the occupant is the normal seating state when the face direction of the occupant detected by the face direction detection unit is facing forward.

3. The physique determination device according to claim 2.

4. the situation information is information about an inclination of a skeleton line connecting the skeleton points of the occupant, a tilt detection unit that detects a tilt of the skeleton line of the occupant based on information about the skeleton points of the occupant extracted by the skeleton point extraction unit, The normal seating determination unit determines that the posture of the occupant is the normal seating state when the inclination of the skeleton line of the occupant detected by the inclination detection unit is within an inclination determination range.

4. The physique determination device according to claim 1, wherein the body type is determined by the body weight.

5. The skeleton line of the occupant is a line connecting the skeleton points that indicate both shoulders of the occupant.

5. The physique determination device according to claim 4.

6. the situation information is vehicle information relating to the situation of the vehicle, including a steering angle of the vehicle, a vehicle speed, a door opening / closing state of the vehicle, an ignition state of the vehicle, or an operation state of an in-vehicle device; a vehicle information acquisition unit that acquires the vehicle information, The normal seating determination unit determines, based on the vehicle information acquired by the vehicle information acquisition unit, that the posture of the occupant is the normal seating state if the steering angle of the vehicle is within a steering angle determination range, if the vehicle speed exceeds a vehicle speed determination range, if the door of the vehicle is not open, if the elapsed time after the ignition is turned on is within a start determination time, or if the in-vehicle device is not operated.

2. The physique determination device according to claim 1.

7. the situation information is information about the skeleton points of the occupant extracted by the skeleton point extraction unit, The normal seating determination unit determines that the occupant is in the normal seating state when the position of the skeleton point indicating the elbow of the occupant extracted by the skeleton point extraction unit is not located within an elbow position determination range.

2. The physique determination device according to claim 1.

8. The proper seating determination unit determines the proper seating state of the occupant based on the situation information and a machine learning model that receives the situation information as an input and outputs information indicating whether the posture of the occupant is the proper seating state.

2. The physique determination device according to claim 1.

9. a feature amount leveling unit that levels the plurality of physique determination feature amounts calculated by the feature amount calculation unit going back through a leveling period; The physique determination unit determines the physique of the occupant based on the normalized physique determination feature amount after the feature amount normalization unit has normalized the physique determination feature amount.

2. The physique determination device according to claim 1.

10. The feature amount leveling unit sets an average value, a median value, or a value corresponding to a set percentile of the plurality of physique determination feature amounts as the leveled physique determination feature amount.

10. The physique determination device according to claim 9.

11. and a posture warning unit that outputs posture warning information to prompt the occupant to take the normal seating posture when the posture state of the occupant determined by the normal seating determination unit is not the normal seating state and continues for a posture abnormality determination time.

2. The physique determination device according to claim 1.

12. An image acquisition unit acquires an image of an occupant of the vehicle; a skeleton point extraction unit extracting skeleton points of the occupant that indicate body parts of the occupant based on the captured image acquired by the image acquisition unit; a step in which a normal seating determination unit determines whether the occupant is normally seated based on situation information relating to the situation of the vehicle or the occupant; a feature calculation unit, when determining, based on information about the skeleton points of the occupant extracted by the skeleton point extraction unit and the seating state determination result by the proper seating determination unit that the occupant is in the proper seating state, that the skeleton points of the occupant extracted by the skeleton point extraction unit are proper seating skeleton points extracted based on the captured image taken with the occupant in the proper seating state, calculating a feature for physique determination based on information about the proper seating skeleton points; a step in which a physique determination unit determines the physique of the occupant based on the physique determination feature amount calculated by the feature amount calculation unit; A physique determination method comprising the steps of:

Images