Body type determination device and body type determination method

Abstract

A skeleton point extraction unit (102) that extracts skeleton points of an occupant based on an image capture, a correct sitting determination unit (104) that determines whether a state of a posture of the occupant is a correct sitting state based on status information about a status of the vehicle or the occupant, a feature calculation unit (105) that determines whether the skeleton points of the occupant extracted by the skeleton point extraction unit (102) are correct sitting skeleton points extracted based on the image capture in which the occupant was recorded in the correct sitting state, and in a case that the skeleton points of the occupant extracted by the skeleton point extraction unit (102) are correct sitting skeleton points, calculates a body structure determination feature based on the information about the correct sitting skeleton points,and a body type determination unit (107) is provided which judges the body type of the occupant based on the body type determination feature.,

Description

Technical field

[0001] The present disclosure relates to a body type determination device and a body type determination method for an occupant. General state of the art

[0002] A technique is known from the prior art with which the posture of the occupant is determined on the basis of images in which an occupant of a vehicle is recorded and the body structure of the occupant is determined on the basis of the determined posture.

[0003] For example, Patent Document 1 discloses a passenger compartment monitoring device that estimates the posture of the occupant based on image captures obtained by capturing a passenger compartment using body composition estimation indexes including coordinates of a plurality of features of the occupant in the image captures extracted by deep learning or the like, and estimates the body composition of the occupant based on the estimated posture. Prior art documentsPatent documents

[0004] Patent document 1 JP 2020-104680 A Summary of the inventionObject of the present invention

[0005] In the case of an occupant sitting in a seat with poor posture, countless patterns of poor posture are conceivable. Therefore, it is difficult to fully capture the countless conceivable patterns of poor posture in which an occupant can sit in a seat. Random errors are likely to creep into the occupant's posture determined based on image recordings, or these random errors may increase compared to when the occupant is not sitting in poor posture. As a result, the accuracy of body composition determination decreases.

[0006] Since this is not considered in the prior art disclosed in Patent Document 1, there is a problem that when a state of poor posture of the occupant occurs, there is a risk that the determination accuracy of the occupant's body structure will decrease.

[0007] The invention has been made to solve this problem, and has as its object to provide a body composition determination device which can prevent the decrease in the accuracy of the body composition determination of the occupant even if a poor posture condition of the occupant occurs. Means of solving the task

[0008] A body composition determination device according to the present disclosure includes an image acquisition unit for acquiring an image in which an occupant of a vehicle is captured, a skeleton point extraction unit for extracting skeleton points of the occupant indicating parts of the occupant's body based on the image acquisition unit acquired by the image acquisition unit, a correct sitting determination unit for determining whether a state of a posture of the occupant is a correct sitting state based on status information about a status of the vehicle or the occupant, a feature calculation unit for determining whether the state of the posture of the occupant is a correct sitting state based on the information about the skeleton points of the occupant extracted by the skeleton point extraction unit and a sitting state determination result of the correct sitting determination unit,whether the occupant's skeleton points extracted by the skeleton point extraction unit are correctly seated skeleton points extracted based on the image capture in which the occupant was captured in the correctly seated state, and in a case where it is determined that the occupant's skeleton points extracted by the skeleton point extraction unit are correctly seated skeleton points, to calculate a physique determination feature based on the information on the correctly seated skeleton points, and a physique determination unit to determine a physique of the occupant based on the physique determination feature calculated by the feature calculation unit. Effect of the invention

[0009] According to the present disclosure, the body type determination device can avoid a decrease in the accuracy of the body type determination of the occupant even if a poor posture condition of the occupant occurs. Short description of the characters

[0010] They show: Fig. 1 shows a configuration example of a system using a body type determining device according to a first embodiment; Fig. 2 shows a configuration example of the body type determining device of the first embodiment; Fig. 3 is a flowchart for explaining an example of the operation of the body type determining device of the first embodiment; Fig. 4 shows a configuration example of the body type determining device of the first embodiment in the case where inclination information is used as status information; Fig. 5 shows a configuration example of the body type determination device of the first embodiment in the case where vehicle information is used as status information; and Fig. 6A and Fig. 6B Examples of a hardware configuration of the body type determination device of the first embodiment. Embodiment of the invention

[0011] With reference to the figures, a detailed description of embodiments of the present disclosure now follows. First embodiment

[0012] Fig. 1 is a configuration example of a system using a body type determination device 100 according to a first embodiment.

[0013] In the Fig. 1, the body type determination device 100 of the first embodiment is connected to an image pickup device 200, a belt control device 300, an airbag control device 400, a left-behind person detection device 500, and an output device 600.

[0014] In the first embodiment, it is assumed that the body type determination device 100 is installed in a vehicle.

[0015] The seat belt control device 300, the airbag control device 400, the left-behind detection device 500 and the output device 600 are also installed in the vehicle.

[0016] The image pickup device 200 is, for example, a near-infrared camera or a visible light camera and captures images of vehicle occupants. The image pickup device 200 can also be used as an image pickup device of a so-called driver monitoring system (DMS) installed in the vehicle to monitor the driver's condition in the vehicle.

[0017] The image pickup device 200 is installed so that it can capture at least one area in the vehicle that includes an area where the upper half of a vehicle occupant's body should be located. The area where the upper half of a vehicle occupant's body should be located is, for example, an area corresponding to a space close to a seat back or headrest of a seat.

[0018] For example, the image pickup device 200 is installed in the center of a vehicle dashboard (hereinafter referred to as "dashboard") so that it can capture front seats including the driver's seat and the front passenger seat from the center of the dashboard, or in other words, the driver and the passenger in the front passenger seat (hereinafter referred to as "front passenger"). This is merely an example, and the image pickup device 200 can also be installed, for example, so that it can capture only the driver or so that it can capture passengers in the rear seats.

[0019] The body type determining device 100 of the first embodiment determines the body type of the occupant based on the images of an occupant taken by the image pickup device 200.

[0020] At this time, the body type determining device 100 determines whether the posture state of the occupant of the vehicle is a correct sitting state, and determines the body type of this occupant based on images taken in a correct sitting state of the occupant.

[0021] In the first embodiment, "correct sitting" refers to sitting in a normal state without poor posture. In the first embodiment, a state in which the occupant's face is not facing forward is also considered poor posture. In a state in which the occupant's face is not facing forward, the probability that the occupant's entire body is not facing forward is estimated to be high.

[0022] When the body type determination device 100 determines the body type of a vehicle occupant, it outputs the result of the body type determination of the vehicle occupant (hereinafter referred to as the "body type determination result") to the seatbelt control device 300, the airbag control device 400, and the left-behind detection device 500. Furthermore, the body type determination device 100 outputs information to the output device 600 to cause a message or the like to be output to the occupant prompting the occupant to sit correctly.

[0023] Details of the configuration example of the body type determination device 100 will be explained below with reference to Fig. 2 described.

[0024] In the following first embodiment, the occupant of the vehicle is also referred to as “occupant” for short.

[0025] The belt control device 300 controls the seat belt. The belt control device 300 has, for example, a seat belt reminder function that issues a warning based on the body type determination result output by the body type determination device 100, taking the body type of the occupant into account.

[0026] The airbag control device 400 is, for example, an ECU (engine control unit) for the airbag system, which controls the airbags based on the body type determination result outputted by the body type determination device 100.

[0027] The left-behind detection device 500 detects persons left behind in the vehicle, such as infants or the like. For example, the left-behind detection device 500 determines whether an infant or the like is left behind in the vehicle based on the body type determination result output by the body type determination device 100.

[0028] The output device 600 is a display device or voice output device provided, for example, in the center console, the instrument panel, or the like. The output device 600 performs, for example, a display of a message prompting the occupant to sit correctly, a voice output of this message, or a warning sound prompting the occupant to sit correctly.

[0029] The following is a description of a configuration example of the body type determining device 100 of the first embodiment.

[0030] Fig. 2 shows a configuration example of the body type determining device 100 of the first embodiment.

[0031] The physique determination device 100 includes an image acquisition unit 101, a skeleton point extraction unit 102, a status information acquisition unit 103, a correct sitting determination unit 104, a feature calculation unit 105, a feature standardization unit 106, a physique determination unit 107, and a posture warning unit 108. The status information acquisition unit 103 includes a face orientation detection unit 1031.

[0032] The image acquisition unit 101 acquires images in which the occupant was captured by the image pickup device 200.

[0033] The image acquisition unit 101 outputs the acquired image recordings to the skeleton point extraction unit 102.

[0034] The skeleton point extraction unit 102 extracts, based on the images acquired by the image acquisition unit 101, skeleton points of the occupant indicating parts of the occupant's body. More specifically, based on the images acquired by the image acquisition unit 101, the skeleton point extraction unit 102 extracts, based on the images acquired by the image acquisition unit 101, skeleton points of the occupant indicating joint points specified for each body part of the occupant. The joint points specified for each body part of the occupant include, for example, joint points of the nose, joint points of the shoulders, joint points of the elbows, joint points of the hips, joint points of the wrists, joint points of the knees, and joint points of the ankles. These joint points are defined in advance.

[0035] The skeleton point extraction unit 102 may extract the occupant's skeleton points in a known method using a known technique such as, among others, an image recognition technique or a technique using a trained model (hereinafter referred to as a "machine learning model").

[0036] The skeleton points are, for example, points in the image recordings and are expressed by coordinates in the image recordings. The skeleton point extraction unit 102 acquires the coordinates of the occupant's skeleton points and which body parts of the occupant indicate the respective skeleton points.

[0037] The skeleton point extraction unit 102 extracts the skeleton points separately for each occupant.

[0038] For example, in the image capture, a seat-corresponding area (hereinafter referred to as "seat-corresponding area") is set in advance for each seat. The seat-corresponding area is set in advance based on the installation position of the image capture device 200 and the angle of view.

[0039] The skeleton point extraction unit 102 extracts the skeleton points for each seat corresponding area, for example. For example, if a seat corresponding area is a seat corresponding area corresponding to the driver's seat, the skeleton point extraction unit 102 determines the skeleton points extracted in this seat corresponding area as the driver's skeleton points. For example, if a seat corresponding area is a seat corresponding area corresponding to the front passenger seat, the skeleton point extraction unit 102 determines the skeleton points extracted in this seat corresponding area as the front passenger's skeleton points. For example, by extracting the skeleton points of the relevant seat corresponding area for each seat corresponding area in this way, the skeleton point extraction unit 102 extracts the skeleton points for each occupant.

[0040] The skeleton point extraction unit 102 outputs the information about the skeleton points (hereinafter referred to as “skeleton point information”) to the status information acquisition unit 103.

[0041] The skeleton point information includes information in which the information indicating the occupant, the information on the coordinates of the skeleton points of that occupant, and the information indicating which body part of the occupant the skeleton points indicate are linked together, and the images acquired by the image acquisition unit 101 are included.

[0042] In the first embodiment, the information indicating the occupant is, for example, information indicating the seat on which the occupant is sitting.

[0043] When the skeleton point extraction unit 102 extracts the skeleton points of the occupant, that is, when the skeleton point information is output from the skeleton point extraction unit 102, the status information acquisition unit 103 acquires information about the status of the occupant (hereinafter referred to as "status information"). The status information is, for example, information about the facial orientation of the occupant.

[0044] The face orientation detection unit 1031 detects the face orientation of the occupant based on the images acquired by the image acquisition unit 101.

[0045] The image recordings acquired by the image acquisition unit 101 are included in the skeleton point information.

[0046] The facial orientation detection unit 1031 can detect the facial orientation of the occupant using a known technique such as, among others, an image recognition technique or a technique using a machine learning model. The facial orientation of the occupant is expressed by an angle, for example.

[0047] The face orientation detection unit 1031 detects the face orientation for each occupant.

[0048] The status information acquisition unit 103 acquires, as status information, information about the facial orientation of the occupant acquired by the facial orientation acquisition unit 1031 (hereinafter referred to as "facial orientation information"). The facial orientation information includes information in which the information indicating the occupant and the information indicating the facial orientation are linked. The information indicating the occupant is, for example, information indicating the seat on which the occupant is seated. The facial orientation acquisition unit 1031 determines which seat the occupant is seated on based on, for example, the seat corresponding area. The facial orientation information may also include the images acquired by the image acquisition unit 101.

[0049] The status information acquisition unit 103 outputs the acquired status information, here the face orientation information, to the correct sitting determination unit 104.

[0050] The correct sitting determining unit 104 determines whether the state of the posture of the occupant is the correct sitting state based on the status information acquired by the status information obtaining unit 103.

[0051] More specifically, here, the correct sitting determining unit 104 determines whether the state of the occupant's posture is the state of correct sitting based on the face orientation information.

[0052] The correct seating determination unit 104 determines for each occupant whether the occupant's posture state is a correct seating state. The correct seating determination unit 104 determines the facial orientation of each occupant from the facial orientation information.

[0053] The following is an example of how the correct sitting determining unit 104 determines whether the state of the occupant's posture is the correct sitting state.

[0054] For example, the correct sitting determination unit 104 determines whether the occupant's posture state is the correct sitting state based on the status information, here the face orientation information, based on a condition for determining whether the occupant's posture state is the correct sitting state (hereinafter referred to as "correct sitting state determination condition") set in advance.

[0055] For example, the determination condition for the correct sitting state is defined as a condition when the occupant's posture is considered to be a correct sitting state. The determination condition for the correct sitting state is generated by an administrator or the like and stored in a location accessible by the physique determination device 100.

[0056] For example, a condition such as the following condition 1 is defined as the determining condition for the state of correct sitting. Condition 1

[0057] The occupant's face is facing forward.

[0058] If the occupant's face orientation is within a specified range using the front side as a scale (referred to as the "face orientation determination range"), the occupant's face is considered to be facing forward.

[0059] As the face orientation determination range of Condition 1, an appropriate range within which the orientation is regarded as a front orientation is set.

[0060] The correct sitting determining unit 104 determines that the posture state of the occupant is the correct sitting state when the facial orientation of the occupant satisfies the correct sitting state determination condition based on the facial orientation information.

[0061] On the other hand, if the facial orientation of the occupant does not satisfy the determination condition for the correct sitting state based on the facial orientation information, the correct sitting determining unit 104 determines that the posture state of the occupant is not the correct sitting state.

[0062] The determination result of the correct seating determination unit 104 as to whether the occupant's posture is the correct seating state (hereinafter referred to as the "seating state determination result") is output to the feature calculation unit 105. At this time, the correct seating determination unit 104 outputs the skeleton point information output from the skeleton point extraction unit 102 along with the seating state determination result to the feature calculation unit 105. Furthermore, the correct seating determination unit 104 outputs the seating state determination result to the posture warning unit 108.

[0063] The feature calculation unit 105 determines whether the occupant's posture state is the correct sitting state based on the occupant's skeleton point information output from the skeleton point extraction unit 102 and the sitting state determination result of the correct sitting determination unit 104, and whether the occupant's skeleton points extracted by the skeleton point extraction unit 102 are correct sitting skeleton points extracted based on images in which the occupant was captured in the correct sitting state (hereinafter referred to as "correct sitting skeleton points"). For example, in the case where the result output from the correct sitting determination unit 104 The sitting state determination result is a result that the state of the occupant's posture is the correct sitting state, that is, the skeleton points extracted by the skeleton point extraction unit 102 are at correct sitting. Thus, the feature calculation unit 105 determines that the skeleton point information output by the skeleton point extraction unit 102 is at correct sitting.

[0064] In the first embodiment, the feature calculation unit 105 obtains the skeleton point information output from the skeleton point extraction unit 102 via the correct sitting determination unit 104, but this is merely an example. The feature calculation unit 105 may also obtain the skeleton point information directly from the skeleton point extraction unit 102, for example. In this case, the arrow extends in Fig. 1 from the skeleton point extraction unit 102 to the feature calculation unit 105.

[0065] When it is determined that the occupant's skeleton points extracted by the skeleton point extraction unit 102 are skeleton points when sitting correctly, the feature calculation unit 105 calculates a feature used in determining the physique (hereinafter referred to as a "physique determination feature") based on the skeleton point information of the skeleton points when sitting correctly (hereinafter referred to as "skeleton point information when sitting correctly").

[0066] The feature calculation unit 105 calculates the body composition determination feature for each occupant. The feature calculation unit 105 can determine the skeletal points of each occupant based on the skeletal point information when correctly seated.

[0067] When the feature calculation unit 105 determines that the occupant's skeleton points extracted by the skeleton point extraction unit 102 are not skeleton points when sitting correctly, it does not perform calculation of the physique determination feature.

[0068] For example, the feature calculation unit 105 calculates the physique determination feature based on the skeleton point information when sitting correctly based on the length of a line present in the image recordings that connects two of the skeleton points in the image recordings.

[0069] More specifically, the feature calculation unit 105 calculates an arm length, a shoulder width, and a neck length of the occupant in the image captures as body conformation determining features based on the skeleton point information when sitting correctly.

[0070] The feature calculation unit 105 may, for example, use the length of a line connecting a skeletal point indicating a shoulder and a skeletal point indicating an elbow as the occupant's arm length and use it as a physique determination feature. Also, the feature calculation unit 105 may, for example, use the length of a line connecting the skeletal points indicating the shoulders, the length of a line connecting a skeletal point indicating the neck and a skeletal point indicating the right shoulder, or the length of a line connecting a skeletal point indicating the neck and a skeletal point indicating the left shoulder as the shoulder width and use it as a physique determination feature.For example, the feature calculation unit 105 may also use the length of a line connecting a skeleton point indicating the nose and a skeleton point indicating the neck as the neck length of the occupant and use it as a body type determining feature.

[0071] This is merely an example, and it is not necessary for the feature calculation unit 105 to calculate all of the occupant's arm length, shoulder width, and neck length as physique determination features. It is sufficient to calculate at least one of the occupant's arm length, shoulder width, and neck length as a physique determination feature. The feature calculation unit 105 may also use a line connecting other skeletal points as a physique determination feature in addition to the arm length, shoulder width, and neck length.

[0072] The feature calculation unit 105 outputs information about the calculated body type determination features (hereinafter referred to as “feature information”) to the feature standardization unit 106.

[0073] In the feature information, the information identifying the occupant and the body-determining features for each occupant are linked. The information identifying the occupant includes, for example, information indicating the seat the occupant is sitting on.

[0074] The feature standardization unit 106 performs standardization on the multiple body type determination features calculated by the feature calculation unit 105 on a plurality of body type determination features from a predetermined past period (hereinafter referred to as “standardization period”), in other words, retroactively for the standardization period.

[0075] The standardization period can be set to an appropriate length. The standardization period is set by an administrator or the like and stored in a location accessible by the feature standardization unit 106.

[0076] For example, the feature standardization unit 106 stores the feature information output by the feature calculation unit 105, linked to a date and time at which the feature information was acquired, as a time series in a location accessible to the feature standardization unit 106. The feature standardization unit 106 can acquire the plurality of body type determination features of the standardization period from the feature information. For example, when no body type determination features corresponding to an entire standardization period are stored, such as immediately after the vehicle starts driving, the feature standardization unit 106 can use only the already stored body type determination features as the body type determination features for the standardization period.

[0077] The feature information stored by the feature calculation unit 105 may be deleted, for example, when the vehicle is switched on.

[0078] For example, the feature standardization unit 106 may standardize a plurality of body type determination features for a standardization period by averaging them, determining their median value, or using a value corresponding to a specified percentile value.

[0079] Thus, the feature standardization unit 106 calculates, for example, an average value of a plurality of body type determining features for a standardization period, a median value of a plurality of body type determining features for a standardization period, or a value corresponding to a predetermined percentile value for a plurality of body type determining features for a standardization period as the body type determining feature after standardization (hereinafter referred to as “standardized body type determining feature”).

[0080] The feature standardization unit 106 calculates the standardized body type determination feature for each occupant. The feature standardization unit 106 can determine the body type determination feature from the feature information.

[0081] When the feature calculation unit 105 calculates a plurality of body type determining features such as arm length, shoulder width, and neck length of the occupant and the like, the feature standardization unit 106 standardizes these respective body type determining features.

[0082] By standardizing the physique determination features by the feature standardization unit 106, the physique determination device 100 can minimize the impact on the physique determination using this physique determination feature (more specifically, the standardized physique determination feature) when, for example, a physique determination feature is temporarily calculated that deviates from the features actually corresponding to the occupant's physique. The occupant's physique determination is performed by the physique determination unit 107. Details of the physique determination unit 107 will be described later.

[0083] When the feature standardization unit 106 standardizes the multiple body type determination features by averaging multiple body type determination features for a standardization period, the feature standardization unit 106 can use an accurate value, or in other words, a value closer to the feature actually corresponding to the occupant's body type, as the standardized body type determination feature even when there are only a few body type determination features. To this end, in this case, the administrator or the like can narrow the facial orientation determination range defined by the correct sitting state determination condition, which the correct sitting determination unit 104 uses when determining whether the occupant's posture state is the correct sitting state.By narrowing the facial orientation determination range, the correct sitting determination unit 104 is expected to determine that the occupant's posture state is the correct sitting state less frequently. However, even if the number of physique determination features for determining that the occupant's posture state is the correct sitting state is small, the feature standardization unit 106 can subsequently calculate an accurately standardized physique determination feature from the small number of physique determination features. As a result, the physique determination device 100 can prevent the occupant's physique determination from decreasing when using such a standardized physique determination feature.

[0084] When the feature standardization unit 106 determines the median value or a value corresponding to a predetermined percentile value of a plurality of body type determining features for a standardization period to standardize the body type determining feature, the feature standardization unit 106 can use an accurate value, or in other words, a value closer to the feature actually corresponding to the occupant's body type, from the large number of body type determining features as the standardized body type determining feature, even when there are a large number of body type determining features.To this end, in this case, the administrator or the like may expand the face orientation determination range defined by the correct sitting state determination condition, which the correct sitting determination unit 104 uses when determining whether the occupant's posture state is the correct sitting state, so that the correct sitting determination unit 104 can more easily determine that the occupant's posture state is the correct sitting state.If the correct sitting determination unit 104 more easily determines that the occupant's posture state is the correct sitting state, the determination accuracy of the correct sitting state is expected to decrease. However, the feature standardization unit 106 can then calculate an accurate standardized physique determination feature from the large number of physique determination features for which the occupant's posture state is determined to be the correct sitting state. As a result, the physique determination device 100 can prevent the occupant's physique determination from decreasing when using such a standardized physique determination feature.

[0085] The feature standardization unit 106 outputs information on the standardized body type determination feature (hereinafter referred to as “feature standardization information”) to the body type determination unit 107.

[0086] In the feature standardization information, the information indicating the occupant and the feature standardization information for each occupant are linked. For example, the information indicating the occupant is information indicating the seat the occupant is sitting on.

[0087] The body type determination unit 107 determines the body type of the occupant based on the body type determination feature standardized by the feature standardization unit 106.

[0088] The body type determination unit 107 performs the body type determination for each occupant. The body type determination unit 107 can determine the standardized body type determination feature for each occupant from the feature standardization information.

[0089] The body type determination unit 107 may determine the body type of the occupant in a known method using a known technique such as, among others, an image recognition technique or a technique using a trained model (hereinafter referred to as a “machine learning model”).

[0090] The body type determination unit 107 determines, for example, "toddler," "small body type," "average body type," "large body type," and the like in a known method. This is just an example, and the definition of the body type determined by the body type determination unit 107 can be set appropriately.

[0091] The body type determination unit 107 outputs the body type determination result to the belt control device 300, the airbag control device 400, and the left-behind detecting device 500.

[0092] In the body type determination result, the information indicating the occupant and the information about the occupant's body type are linked for each occupant. The information indicating the occupant includes, for example, information indicating the seat the occupant is sitting on.

[0093] the posture warning unit 108 outputs, in the case where a state in which the state of the posture of the occupant determined by the correct sitting determining unit 104 is not the correct sitting state, for a predetermined time (hereinafter referred to as "posture deviation determination time"), information prompting the occupant to a correct sitting posture (hereinafter referred to as "posture warning information") to the output device 600.

[0094] When the posture warning unit 108 refers to the sitting state determination result outputted by the correct sitting determination unit 104, it may determine that the correct sitting determination unit 104 has determined that the state of the occupant's posture is not the correct sitting state.

[0095] For example, the posture warning unit 108 may store the sitting state determination result output by the correct sitting determination unit 104, along with the date and time the sitting state determination result was obtained, as a time series in a location accessible to the posture warning unit 108. Referring to the stored sitting state determination result, the posture warning unit 108 may determine whether the state in which the occupant's posture is not the correct sitting state has continued for the posture deviation determination time. The sitting state determination result stored by the posture warning unit 108 may be deleted, for example, when the vehicle is turned on.

[0096] When the posture warning unit 108 determines that the state of the occupant's posture is not the state of correct sitting for the posture deviation determination time, it outputs the posture warning information to the output device 600.

[0097] By outputting the posture warning information, the posture warning unit 108 causes the output device 600 to, for example, display a message urging the occupant to sit correctly, output a voice of that message, or output a warning sound urging the occupant to sit correctly.

[0098] By outputting posture warning information to the occupant to sit correctly, the posture determination device 100 can cause the occupant to improve their posture. As a result, the posture determination device 100 can more easily determine that the occupant's posture is a correct sitting state.

[0099] The following is a description of the operation of the body type determining apparatus 100 of the first embodiment.

[0100] Fig. 3 is a flowchart for explaining an example of the operation of the body type determining device 100 of the first embodiment.

[0101] For example, during the running of the vehicle, the body type determination device 100 performs the process shown in the flowchart of Fig. 3 operation shown regularly.

[0102] The body type determining device 100 can execute the flowchart of Fig. 3 For example, repeat from switching on to switching off the vehicle.

[0103] The image acquisition unit 101 acquires images in which the occupant was captured by the image pickup device 200 (step ST1).

[0104] The image acquisition unit 101 outputs the acquired image recordings to the skeleton point extraction unit 102.

[0105] The skeleton point extraction unit 102 extracts skeleton points of the occupant indicating parts of the occupant's body based on the images acquired by the image acquisition unit 101 in step ST1 (step ST2).

[0106] The skeleton point extraction unit 102 outputs the skeleton point information to the status information acquisition unit 103.

[0107] Once the skeleton point extraction unit 102 extracts the occupant's skeleton points in step ST2, the status information acquisition unit 103 acquires the status information (step ST3).

[0108] In step ST3, the facial orientation detection unit 1031 detects the facial orientation of the passenger based on the images acquired by the image acquisition unit 101 in step ST1. The status information acquisition unit 103 acquires the facial orientation information related to the facial orientation of the passenger detected by the facial orientation detection unit 1031 as status information.

[0109] The status information acquisition unit 103 outputs the acquired status information, here the face orientation information, to the correct sitting determination unit 104.

[0110] The correct seating determination unit 104 determines whether the occupant's posture state is the correct seating state based on the status information acquired by the status information acquisition unit 103 in step ST3 (step ST4). Specifically, here, the correct seating determination unit 104 determines whether the occupant's posture state is the correct seating state based on the facial orientation information.

[0111] The correct sitting determination unit 104 outputs the sitting state determination result to the feature calculation unit 105. The correct sitting determination unit 104 outputs the skeleton point information output from the skeleton point extraction unit 102 together with the sitting state determination result to the feature calculation unit 105.

[0112] In addition, the correct sitting determination unit 104 outputs the sitting state determination result to the posture warning unit 108.

[0113] The feature calculation unit 105 determines whether the occupant's posture state is the correct sitting state based on the occupant's skeleton point information output from the skeleton point extraction unit 102 in step ST2 and the sitting state determination result of the correct sitting determination unit 104 in step ST4, whether the occupant's skeleton points extracted by the skeleton point extraction unit 102 are correct sitting skeleton points.

[0114] When it is determined that the occupant's skeleton points extracted by the skeleton point extraction unit 102 in step ST2 are skeleton points when sitting correctly, the feature calculation unit 105 calculates the physique determination features based on the skeleton point information when sitting correctly (step ST5).

[0115] The feature calculation unit 105 outputs the feature information to the feature standardization unit 106.

[0116] If it is not determined that the occupant's skeleton points extracted by the skeleton point extraction unit 102 in step ST2 are skeleton points when correctly sitting, the feature calculation unit 105 does not perform calculation of the physique determination features, and the operation of the physique determination device 100 ends the process shown in the flowchart of Fig. Processing shown in 3.

[0117] The feature standardization unit 106 standardizes, based on the correct sitting skeleton point information output by the correct sitting determination unit 104, a plurality of physique determination features for a past standardization period, or in other words, a plurality of physique determination features calculated by the feature calculation unit 105 retroactively for the standardization period (step ST6).

[0118] The feature standardization unit 106 outputs the feature standardization information to the body type determination unit 107.

[0119] The body type determination unit 107 determines the body type of the occupant based on the body type determination feature standardized by the feature standardization unit 106 in step ST6 (step ST7).

[0120] The body type determination unit 107 outputs the body type determination result to the belt control device 300, the airbag control device 400, and the left-behind detecting device 500.

[0121] When the state that the posture of the occupant determined in step ST4 by the correct sitting determining unit 104 is not the correct sitting state has continued for the posture deviation determination time, the posture warning unit 108 outputs the posture warning information to the output device 600 (step ST8).

[0122] In this way, the physique determination device 100 extracts the occupant's skeletal points based on the image captures. The physique determination device 100 determines whether the occupant's posture state is the correct sitting state based on the status information, here, the facial orientation information. The physique determination device 100 determines whether the occupant's posture state is the correct sitting state based on the skeletal point information and the determination result regarding whether the occupant's posture state is the correct sitting state, whether the extracted occupant's skeletal points are correct sitting skeletal points, and if it determines that they are correct sitting skeletal points, it calculates the physique determination feature based on the correct sitting skeletal point information. The physique determination device 100 then determines the occupant's physique based on the calculated physique determination feature.

[0123] As mentioned above, when an occupant sits in a seat with a poor posture, there are countless conceivable patterns for the poor posture. Therefore, it is difficult to fully capture the countless conceivable patterns of poor postures in which an occupant can sit in a seat. As a result, random errors are likely to intrude or increase in the occupant's posture determined based on imagery compared to when the occupant is sitting correctly. For example, random errors are likely to intrude or increase in the occupant's skeleton points extracted from the imagery. When random errors are likely to intrude or increase in the occupant's skeleton points extracted from the imagery, random errors are likely to intrude or increase in the features calculated based on the skeleton points.As a result, random errors creep in or increase in the occupant's posture determined based on the characteristics. As a result, the accuracy of the body composition determination decreases.

[0124] If the occupant is sitting correctly, the position of the skeletal points representing the features in the imagery is assumed to be within a certain range. Therefore, if the occupant is sitting correctly, there will be little random error in the skeletal points extracted from the imagery and the features calculated from the skeletal points. Therefore, there will be little random error in the occupant's posture determined based on the features. Even if a random error occurs, the impact of such a random error on the body composition determination is small.

[0125] The occupant body composition determination result based on the image capture is used for functions such as seat belt control, airbag control, and left-behind detection, among others. Functions such as seat belt control, airbag control, and left-behind detection are safety-related functions, and therefore, high accuracy is required from the occupant body composition determination result used for these functions. A low-accuracy occupant body composition determination result obtained while the occupant is sitting in an incorrect posture on the seat cannot be used for functions such as seat belt control, airbag control, and left-behind detection, among others.If the body composition determination device were to provide, as the body composition determination result for functions such as, among others, seat belt control, airbag control, and left-behind occupant detection, a body composition determination result based on a determination in a state where the occupant is sitting on the seat in a poor posture, miscontrol or misdetection could occur in the above-mentioned functions.

[0126] In contrast, as discussed above, the physique determination device 100 determines whether the occupant's posture state is the correct sitting state, and uses features based on skeleton points extracted from images determined to be in the correct sitting state as physique determination features, and performs occupant physique determination based on these physique determination features. The physique determination device 100 calculates the physique determination feature based on images in which the occupant's posture state was the correct sitting state, and determines the occupant's physique based on these physique determination features, so that it can prevent a decrease in the accuracy of the occupant's physique determination.Therefore, the body composition determination device 100 can prevent a decrease in the accuracy of the body composition determination of the occupant in the case where, for example, a state of poor posture of the occupant occurs during the running of the vehicle.

[0127] Since the body type determination device 100 provides a body type determination result for the occupant for which a decrease in accuracy has been prevented, it can provide a body type determination result that can also be used for functions such as seat belt control, airbag control, and left-behind detection, among others, which require a high degree of accuracy from the body type determination result.

[0128] In the above first embodiment, in the body type determination device 100, the status information acquisition unit 103 includes the facial orientation detection unit 1031, and the facial orientation detection unit 1031 detects the facial orientation of the occupant based on the images acquired by the image acquisition unit 101. However, this is merely an example. For example, the status information acquisition unit 103 may acquire information on the facial orientation of the occupant from an occupant state detection device or the like that is part of the DMS, or a device provided outside the body type determination device 100 in a location accessible to the body type determination device 100. In this case, the image acquisition device 200 is configured to simultaneously serve the DMS.

[0129] In this case, a configuration is possible in which the status information acquisition unit 103 of the body type determination device 100 does not include a face orientation detection unit 1031.

[0130] In the first embodiment described so far, the status information acquired by the status information acquisition unit 103 of the physique determination device 100 is information about the facial orientation of the occupant, but this is merely an example. In the first embodiment described so far, the physique determination device 100 may use, for example, skeleton point information as the status information acquired by the status information acquisition unit 103.

[0131] In this case, the correct sitting determination unit 104 may have the function of the status information acquisition unit 103. That is, the body type determination device 100 of Fig. 1 does not necessarily have to include the status information acquisition unit 103.

[0132] In this case, the skeleton point extraction unit 102 outputs the skeleton point information to the correct sitting determination unit 104.

[0133] The correct sitting determining unit 104 determines whether the state of the posture of the occupant is the correct sitting state based on the skeleton point information output from the skeleton point extraction unit 102.

[0134] In this case, for example, the determining condition for the state of correct sitting is defined as a condition such as the following condition 2. Condition 2

[0135] In the image recordings, the position of a skeletal point indicating the occupant's elbow is not within a predetermined range (hereinafter referred to as the "elbow position determination range")

[0136] The elbow position determination area of ​​Condition 2 is set in advance, for example, to a specific area including the armrest or a specific area near the vehicle door.

[0137] The correct sitting determining unit 104 determines that the posture state of the occupant is the correct sitting state when the skeleton point indicating the elbow of the occupant satisfies the correct sitting state determination condition based on the skeleton point information.

[0138] On the other hand, if the skeletal point indicating the occupant's elbow does not satisfy the determination condition for the correct sitting state based on the inclination information, the correct sitting determining unit 104 determines that the state of the occupant's posture is not the correct sitting state.

[0139] For example, if the position of the skeleton point indicating the occupant's elbow is in the elbow position determination range, the correct sitting determination unit 104 determines that the state of the occupant's posture is not the correct sitting state.

[0140] A flowchart of the operation of the body composition determining device 100 in the case where the body composition determining device 100 uses the skeleton point information as status information corresponds to the flowchart of Fig. 3.

[0141] However, in step ST3, the correct sitting determining unit 104 acquires the skeleton point information outputted from the skeleton point extraction unit 102 in step ST2 as the status information.

[0142] The correct sitting determining unit 104 then determines in step ST4 whether the state of the occupant's posture is the correct sitting state based on the skeleton point information acquired in step ST3.

[0143] The concrete operation of the body composition determining device 100 in the remaining steps ST1, ST2 and ST5 to ST8 corresponds to the concrete operation already described, which is why a repeated description is omitted.

[0144] Thus, in the physique determination device 100 that uses the occupant's skeletal point information as status information, the correct sitting determination unit 104 can determine whether the occupant's posture state is the correct sitting state based on the occupant's skeletal point information extracted by the skeletal point extraction unit 102. In this case, too, the physique determination device 100 calculates the physique determination feature based on image captures in which the occupant's posture state was the correct sitting state and determines the occupant's physique based on these physique determination features, thus preventing a decrease in the accuracy of the occupant's physique determination.Therefore, the body composition determination device 100 can prevent a decrease in the accuracy of the body composition determination of the occupant in the case where, for example, a state of poor posture of the occupant occurs during the running of the vehicle.

[0145] In the first embodiment described so far, the body type determination device 100 may also use information about the inclination of a skeletal line connecting the occupant's skeletal points (hereinafter referred to as "inclination information") as status information. In this case, poor posture is not assumed to exist solely because the occupant's face is not facing forward.

[0146] Fig. 4 shows a configuration example of the body type determination device 100 of the first embodiment in the case where the inclination information is used as status information.

[0147] The Fig. The configuration example of the body type determination device 100 shown in Fig. 4 differs from that shown in Fig. 1, the configuration example of the body type determination device 100 is that the status information acquisition unit 103 includes an inclination detection unit 1032 instead of the face orientation detection unit 1031.

[0148] At the Fig. 4, the concrete operation of the unit 104 for determining correct sitting also differs from the concrete operation of the unit 104 for determining correct sitting of the device 100 shown in Fig. 1. The concrete operation of the image acquisition unit 101, the skeleton point extraction unit 102, the feature calculation unit 105, the feature standardization unit 106, the body structure determination unit 107, and the posture warning unit 108 corresponds to the concrete operation already described, so a repeated description is omitted.

[0149] The inclination detection unit 1032 detects the inclination of a skeleton line, which is a line connecting a plurality of skeleton points of the occupant, based on the skeleton point information output from the skeleton point extraction unit 102.

[0150] For example, the inclination detection unit 1032 detects the inclination of a skeleton line of the occupant's shoulders connecting the skeleton points indicating the occupant's shoulders. This is only one example, and the inclination detection unit 1032 may detect the inclination of any suitable skeleton line, such as a skeleton line indicating the torso (for example, a skeleton line indicating the seat height). However, a poor posture of the occupant can be easily determined from the inclination of the skeleton line of the occupant's shoulders. Consequently, if the inclination detection unit 1032 detects the inclination of the skeleton line of the occupant's shoulders, erroneous determinations by the correct seating determination unit 104 as to whether the occupant's posture state is the correct seating state can be reduced.

[0151] The status information acquisition unit 103 acquires the inclination information as the status information. The inclination information includes information linking the occupant-indicating information and the skeletal line inclination information. The inclination information may also include the images acquired by the image acquisition unit 101. The status information acquisition unit 103 outputs the acquired status information, here the inclination information, to the correct seating determination unit 104.

[0152] If the configuration example of the body type determination device 100 is the one shown in Fig. 4, the correct sitting determining unit 104 determines whether the state of the posture of the occupant is the correct sitting state based on the inclination information of the inclination of the skeletal line of the occupant detected by the inclination detecting unit 1032.

[0153] In this case, the determining condition for the state of correct sitting is defined as, for example, a condition such as the following condition 3. Condition 3

[0154] The inclination of the occupant's skeleton line is within a predetermined range (hereinafter referred to as the “inclination determination range”)

[0155] For example, as the inclination determination range, a certain range is set in advance based on the inclination of the skeletal line of an occupant who is considered to be facing forward.

[0156] When the skeletal point indicating the occupant's elbow satisfies the determination condition for the correct sitting state based on the inclination information, the correct sitting determining unit 104 determines that the state of the occupant's posture is the correct sitting state.

[0157] On the other hand, if the skeletal point indicating the occupant's elbow does not satisfy the determination condition for the correct sitting state based on the inclination information, the correct sitting determining unit 104 determines that the state of the occupant's posture is not the correct sitting state.

[0158] A flowchart of the operation of the body type determination device 100 in the case where the configuration example of the body type determination device 100 is the one shown in Fig. 4 shown configuration example corresponds to the flowchart of Fig. 3.

[0159] If the configuration example of the body type determination device 100 is that shown in Fig. 4, the status information acquiring unit 103 acquires the inclination information as the status information in step ST3.

[0160] The status information acquisition unit 103 outputs the acquired status information, here the inclination information, to the correct sitting determination unit 104.

[0161] In step ST4, the correct sitting determining unit 104 determines whether the state of the occupant's posture is the correct sitting state based on the inclination information.

[0162] The concrete operation of the body composition determining device 100 in the remaining steps ST1, ST2 and ST5 to ST8 corresponds to the concrete operation already described, which is why a repeated description is omitted.

[0163] Thus, the inclination information may serve as the status information, and the inclination detection unit 1032 detects the inclination of the occupant's skeletal line based on the information on the skeletal points extracted by the skeletal point extraction unit 102, while the correct sitting determination unit 104 determines whether the occupant's posture state is the correct sitting state based on the inclination of the occupant's skeletal line detected by the inclination detection unit 1032. In this case, too, the physique determination device 100 calculates the physique determination feature based on image captures in which the occupant's posture state was the correct sitting state, and determines the occupant's physique based on these physique determination features, so that it can prevent a decrease in the accuracy of the occupant's physique determination.Therefore, the body composition determination device 100 can prevent a decrease in the accuracy of the body composition determination of the occupant in the case where, for example, a state of poor posture of the occupant occurs during the running of the vehicle.

[0164] In the first embodiment described so far, the status information may include not only information about the occupant's status but also information about the vehicle's status. In this case, too, the presence of poor posture is not assumed solely because the occupant's face is not facing forward.

[0165] For example, in the first embodiment described so far, the body type determination device 100 may acquire information on the status of the vehicle (hereinafter referred to as “vehicle information”) as status information.

[0166] The vehicle information includes, for example, information relating to the status of a steering control angle of the vehicle, the driving speed, the opening or closing status of doors of the vehicle, the status of an ignition of the vehicle, or the operation status of an on-board device (for example, a navigation device, an interior light, or the like).

[0167] Fig. 5 shows a configuration example of the body type determination device 100 of the first embodiment in the case where the vehicle information is used as status information.

[0168] The Fig. The configuration example of the body type determination device 100 shown in Fig. 5 differs from that shown in Fig. 1, the status information acquisition unit 103 includes a vehicle information acquisition unit 1033 instead of the face orientation detection unit 1031.

[0169] At the Fig. 5, the concrete operation of the unit 104 for determining correct sitting also differs from the concrete operation of the unit 104 for determining correct sitting of the device 100 shown in Fig. 1 shown body type determination device 100.

[0170] The concrete operation of the image acquisition unit 101, the skeleton point extraction unit 102, the feature calculation unit 105, the feature standardization unit 106, the body structure determination unit 107, and the posture warning unit 108 corresponds to the concrete operation already described, so a repeated description is omitted.

[0171] The vehicle information acquisition unit 1033 acquires vehicle information from various devices installed in the vehicle. For example, the vehicle information acquisition unit 1033 may acquire information about the status of the steering control angle from a steering angle sensor. The vehicle information acquisition unit 1033 may acquire the vehicle speed from, for example, a vehicle speed sensor. The vehicle information acquisition unit 1033 may acquire information about the opening and closing status of the doors from sensors provided on the doors. The vehicle information acquisition unit 1033 may acquire information about the ignition status from an ignition sensor. From on-board devices such as the navigation device, the vehicle information acquisition unit 1033 may acquire information about the operation status of the on-board device, for example.

[0172] The status information acquiring unit 103 acquires, as the status information, the vehicle information acquired by the vehicle information acquiring unit 1033.

[0173] The status information acquisition unit 103 outputs the acquired status information, here the vehicle information, to the correct seating determination unit 104.

[0174] If the configuration example of the body type determination device 100 is the one shown in Fig. 5, the correct seating determining unit 104 determines whether the state of the posture of the occupant is the correct seating state based on the vehicle information acquired by the vehicle information acquiring unit 1033.

[0175] In this case, the determining conditions for the state of correct sitting are defined as conditions such as the following conditions 4 to 8. Condition 4

[0176] The steering control angle is within a predetermined range (hereinafter referred to as the “steering angle determination range”) Condition 5

[0177] The driving speed exceeds a pre-determined range (hereinafter referred to as the “driving speed determination range”) Condition 6

[0178] No door of the vehicle is open Condition 7

[0179] The time since the ignition was turned on is within a predetermined time (hereinafter referred to as the “start determination time”) Condition 8

[0180] No on-board device is operated

[0181] For example, the steering angle determination range of Condition 5 is defined as a steering angle range that cannot be reached without bending an arm. For example, the vehicle speed determination range of Condition 5 is defined as a vehicle speed at which it can be assumed that the vehicle is traveling on an expressway.

[0182] For example, for condition 6 it is assumed that the occupant gets in or out with the door open and the occupant is therefore not sitting correctly.

[0183] The correct sitting determining unit 104 determines that the occupant's posture state is the correct sitting state when the vehicle information satisfies the correct sitting state determination condition.

[0184] On the other hand, when the vehicle information does not satisfy the determination condition for the correct sitting state, the correct sitting determining unit 104 determines that the state of the posture of the occupant is not the correct sitting state.

[0185] A flowchart of the operation of the body type determination device 100 in the case where the configuration example of the body type determination device 100 is the one shown in Fig. The configuration example shown in Figure 5 corresponds to the flowchart of Fig. 3.

[0186] If the configuration example of the body type determination device 100 is that shown in Fig. 5, the status information acquiring unit 103 acquires, as the status information, the vehicle information acquired by the vehicle information acquiring unit 1033 in step ST3.

[0187] Then, the status information acquisition unit 103 outputs the acquired status information, here the vehicle information, to the correct seating determination unit 104.

[0188] The correct seating determining unit 104 determines in step ST4 whether the state of the posture of the occupant is the correct seating state based on the vehicle information acquired in step ST3 by the vehicle information acquiring unit 1033.

[0189] The concrete operation of the body composition determining device 100 in the remaining steps ST1, ST2 and ST5 to ST8 corresponds to the concrete operation already described, which is why a repeated description is omitted.

[0190] Thus, the vehicle information can be used as the status information. The body type determination device 100 includes the vehicle information acquisition unit 1033 that acquires the vehicle information, and the correct sitting determination unit 104 can determine whether the occupant's posture state is the correct sitting state based on the vehicle information acquired by the vehicle information acquisition unit 1033. In this case, too, the body type determination device 100 calculates the body type determination feature based on images in which the occupant's posture state was the correct sitting state, and determines the occupant's body type based on these body type determination features, so that it can prevent a decrease in the accuracy of the occupant's body type determination.Therefore, the body composition determination device 100 can prevent a decrease in the accuracy of the body composition determination of the occupant in the case where, for example, a state of poor posture of the occupant occurs during the running of the vehicle.

[0191] Thus, there is no limitation to the facial orientation information, and the physique determination device 100 may instead acquire the skeleton point information, the inclination information, or the vehicle information as status information.

[0192] However, devices other than the body type determination device 100 often have a function of detecting the facial orientation of the occupant based on image capture. As discussed above, for example, an occupant state detection device of the DMS may have a function of detecting the facial orientation of the occupant. For example, the body type determination device 100 may use the facial orientation information as status information and acquire the facial orientation information from a device external to the body type determination device 100, so it does not need to have a function of detecting the facial orientation. Since the body type determination device 100 does not need to perform processing for detecting the facial orientation of the occupant, the processing load in the occupant body type determination can be reduced.

[0193] Even if the body composition determination device 100 uses the skeletal point information as status information, it does not require the addition of a status information acquisition function and the like. However, in general, the detection accuracy is higher when detecting the position of the eyes or mouth based on image capture than when detecting the position of skeletal points based on image capture. For example, the determination accuracy is higher when determining whether the occupant is facing forward based on the facial orientation of the occupant than when determining based on skeletal points. Also, the facial orientation of a person is generally detected based on the position of the eyes or mouth.

[0194] Consequently, the physique determination device 100 can more accurately determine whether the state of the posture of the occupant is the state of correct sitting when using the facial orientation information as the status information than when using the skeleton point information as the status information.

[0195] In the first embodiment described so far, the body type determination device 100 may also acquire two or more of the face orientation information, the skeleton point information, the inclination information, and the vehicle information as the status information.

[0196] For example, the configuration of the body type determination device 100 may be such that the status information acquisition unit 103 includes two or more of the face orientation detection unit 1031, the inclination detection unit 1032, and the vehicle information acquisition unit 1033.

[0197] In the first embodiment described so far, the correct sitting determination unit 104 of the physique determination device 100 determines whether the state of the posture of the occupant is the correct sitting state based on the status information according to a determination condition for the correct sitting state, but this is only an example.

[0198] The correct sitting determination unit 104 may also determine whether the state of the occupant's posture is the correct sitting state based on the status information and a machine learning model (hereinafter referred to as a "correct sitting determination model").

[0199] For example, the model for determining correct sitting is a machine learning model that uses the status information as inputs and outputs information about whether the occupant's posture is the correct sitting state.

[0200] The model for determining correct sitting is created by an administrator or the like and stored in a location accessible to the body type determining device 100.

[0201] In the first embodiment described so far, the correct seating determining unit 104 determines whether the state of the occupant is the correct seating state based on the status information using the correct seating state determination condition and the correct seating determining model.

[0202] For example, the correct seating determination unit 104 determines whether the occupant's posture state is the correct seating state based on the status information according to the correct seating state determination condition (Determination 1). Also, the correct seating determination unit 104 determines whether the occupant's posture state is the correct seating state based on the status information and the correct seating determination model (Determination 2).If it is determined in both determination 1 and determination 2 that the state of the posture of the occupant is the state of correct sitting, the correct sitting determining unit 104 determines that the state of the posture of the occupant is the state of correct sitting, and if it is determined in either of determination 1 and determination 2 that the state of the posture of the occupant is not the state of correct sitting, the correct sitting determining unit 104 determines that the state of the posture of the occupant is not the state of correct sitting.

[0203] It may also be performed to weight the sitting state determination result of Determination 1 and the sitting state determination result of Determination 2 to determine whether the state of the occupant's posture is the state of correct sitting.

[0204] In the first embodiment described so far, the body type determination device 100 includes the feature standardization unit 106, but this is merely an example. The body type determination device 100 may also be configured without the feature standardization unit 106.

[0205] In this case, the body type determination unit 107 may determine the body type of the occupant based on the body type determination feature calculated by the feature calculation unit 105.

[0206] In the flowchart of Fig. According to the operation of the body type determination device 100 described in FIG. 3, the body type determination device 100 may omit the processing of step ST6. In step ST7, the body type determination unit 107 may determine the body type of the occupant based on the body type determination feature calculated by the feature calculation unit 105 in step ST5.

[0207] In the first embodiment described so far, the physique determination device 100 includes the posture warning unit 108, but this is merely an example. The physique determination device 100 may also be configured without the posture warning unit 108. Also, the physique determination device 100 does not need to be connected to the output device 600.

[0208] In this case, the body type determination device 100 can be configured as shown in the flowchart of Fig. 3, omit the processing of step ST8.

[0209] In the first embodiment described so far, the body type determining device 100 is connected to the belt control device 300, the airbag control device 400, and the left-behind detecting device 500, but this is only an example.

[0210] The body type determination device 100 may also be connected to only one of the belt control device 300, the airbag control device 400, and the left-behind person detection device 500, or may be connected to a device other than the belt control device 300, the airbag control device 400, and the left-behind person detection device 500, which performs various processing based on the body type determination result.

[0211] For example, the body type determination device 100 may be connected to a display device installed in the vehicle. The display on the display device is, for example, based on the body type determination result output by the body type determination device 100. For example, if there is a small child among the occupants, the display device displays an icon indicating that a small child is present in the vehicle.

[0212] In the first embodiment described so far, it is assumed that a single image pickup device 200 is provided, but this is merely an example. In the first embodiment, multiple image pickup devices 200 may be installed in the vehicle. For example, the multiple image pickup devices 200 may be installed to capture the respective occupants sitting in each seat of the vehicle.

[0213] In this case, the body type determining device 100 acquires images from the plurality of image pickup devices 200 and determines the body type of the occupants.

[0214] Fig. 6A and Fig. 6B show examples of a hardware configuration of the body type determination device 100 of the first embodiment.

[0215] In the first embodiment, the functions of the image acquisition unit 101, the skeleton point extraction unit 102, the status information acquisition unit 103, the face orientation detection unit 1031, the inclination detection unit 1032, the vehicle information acquisition unit 1033, the correct sitting determination unit 104, the feature calculation unit 105, the feature standardization unit 106, the physique determination unit 107, and the posture warning unit 108 are implemented by a processing circuit 1001. That is, the physique determination device 100 includes the processing circuit 1001 for performing control for determining the physique of occupants based on image captures of the occupants of the vehicle.

[0216] The processing circuitry 1001 may be as in Fig. 6A can be dedicated hardware, or as shown in Fig.6B shows a processor 1004 that executes programs stored in a memory.

[0217] When the processing circuitry 1001 is dedicated hardware, the processing circuitry 1001 is a single circuit, a plurality of circuits, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof.

[0218] When the processing circuit is a processor 1004, the functions of the image acquisition unit 101, the skeleton point extraction unit 102, the status information acquisition unit 103, the facial orientation detection unit 1031, the inclination detection unit 1032, the vehicle information acquisition unit 1033, the correct sitting determination unit 104, the feature calculation unit 105, the feature standardization unit 106, the physique determination unit 107, and the posture warning unit 108 are implemented by software, firmware, or a combination of software and firmware. The software or firmware is recorded as a program and stored in the memory 1005.By reading and executing a program stored in the memory 1005, the processor 1004 performs the functions of the image acquisition unit 101, the skeleton point extraction unit 102, the status information acquisition unit 103, the face orientation detection unit 1031, the inclination detection unit 1032, the vehicle information acquisition unit 1033, the correct sitting determination unit 104, the feature calculation unit 105, the feature standardization unit 106, the physique determination unit 107, and the posture warning unit 108. The physique determination device 100 thus includes the memory 1005 in which the program is stored, which, when executed by the processor 1004, ultimately executes the steps ST1 to ST8 discussed above.It can be said that the program stored in the memory 1005 causes the sequence or method of the processings of the image acquisition unit 101, the skeleton point extraction unit 102, the status information acquisition unit 103, the face orientation detection unit 1031, the inclination detection unit 1032, the vehicle information acquisition unit 1033, the correct sitting determination unit 104, the feature calculation unit 105, the feature standardization unit 106, the physique determination unit 107, and the posture warning unit 108 to be executed by a processing circuit 1001 by a computer.The memory 1005 is, for example, RAM, ROM (read only memory), flash memory, EPROM (erasable programmable read only memory), EEPPROM (electrically erasable programmable read only memory) or another non-volatile or volatile semiconductor memory or a magnetic disk, a floppy disk, an optical disk, a minidisk, a DVD (digital versatile disc) or the like.

[0219] The functions of the image acquisition unit 101, the skeleton point extraction unit 102, the status information acquisition unit 103, the face orientation detection unit 1031, the inclination detection unit 1032, the vehicle information acquisition unit 1033, the correct sitting determination unit 104, the feature calculation unit 105, the feature standardization unit 106, the physique determination unit 107, and the posture warning unit 108 can be implemented partly by dedicated hardware and partly by software or firmware.For example, it is possible that the functions of the image acquisition unit 101 are implemented with the processing circuit 1001 as dedicated hardware, while the functions of the skeleton point extraction unit 102, the status information acquisition unit 103, the face orientation detection unit 1031, the inclination detection unit 1032, the vehicle information acquisition unit 1033, the correct sitting determination unit 104, the feature calculation unit 105, the feature standardization unit 106, the physique determination unit 107, and the posture warning unit 108 are implemented by the processor 1004 reading and executing the program stored in the memory 1005.

[0220] In addition, the body type determination device 100 includes an input interface device 1002 and an output interface device 1003 that communicate wirelessly or wired with devices such as the image pickup device 200, the belt control device 300, the airbag control device 400, the left-behind person detection device 500, the output device 600, or the like.

[0221] In the first embodiment described so far, the physique determination device 100 is implemented as an on-board device, while the skeleton point extraction unit 102, the status information acquisition unit 103, the correct sitting determination unit 104, the feature calculation unit 105, the feature standardization unit 106, the physique determination unit 107, and the posture warning unit 108 are arranged in the on-board device.

[0222] However, it is not limited to this, and a configuration of a physique determination system including the on-board device and a server is also possible, in which a part of the skeleton point extraction unit 102, the status information acquisition unit 103, the correct sitting determination unit 104, the feature calculation unit 105, the feature standardization unit 106, the physique determination unit 107, and the posture warning unit 108 is installed in the on-board device of the vehicle, while the rest are installed on a server connected to the on-board device via a network.

[0223] The skeleton point extraction unit 102, the status information acquisition unit 103, the correct sitting determination unit 104, the feature calculation unit 105, the feature standardization unit 106, the physique determination unit 107, and the posture warning unit 108 may also all be set up in the server.

[0224] As described above, the body composition determination device 100 according to the first embodiment is configured to include the image acquisition unit 101 that acquires images in which an occupant of a vehicle is captured; the skeleton point extraction unit 102 that extracts skeleton points of the occupant indicating parts of the occupant's body based on the images acquired by the image acquisition unit 101; the correct sitting determination unit 104 that determines whether a state of a posture of the occupant is a correct sitting state based on situation information about the situation of the vehicle or the occupant; the feature calculation unit 105 that calculates a seating state based on the information about the occupant's skeleton points extracted by the skeleton point extraction unit 102 and a seating state determination result of the correct sitting determination unit 104;whether the occupant's posture state is a correct sitting state, determines whether the occupant's skeleton points extracted by the skeleton point extraction unit 102 are correct sitting skeleton points extracted based on image captures in which the occupant was captured in the correct sitting state, and if it is determined that the occupant's skeleton points extracted by the skeleton point extraction unit 102 are correct sitting skeleton points, calculates a physique determination feature based on the information on the correct sitting skeleton points, and comprises the physique determination unit 107, which determines the occupant's physique based on the physique determination feature calculated by the feature calculation unit 105. The physique determination device 100 calculates the physique determination feature based on image captures,in which the occupant's posture was a correct sitting state, and determines the occupant's physique based on these physique determination features, thus preventing a decrease in the accuracy of the occupant's physique determination. Thus, the physique determination device 100 can prevent a decrease in the accuracy of the occupant's physique determination in the event that a poor occupant posture condition occurs.

[0225] In the present disclosure, modifications of any configuration elements of the embodiments or omissions of any configuration elements of the embodiment are possible within the scope of the inventions. Commercial applicability

[0226] The body type determination device of the present disclosure calculates the body type determination feature based on image captures in which the posture state of the occupant was the correct sitting state, and determines the body type of the occupant based on these body type determination features, so that it can prevent a decrease in the accuracy of the body type determination of the occupant. Explanation of reference symbols 100 Body composition determination device 101 Image Acquisition Unit 102 Skeleton Point Extraction Unit 103 Status information acquisition unit 1031 Face orientation detection unit 1032 Inclination detection unit 1033 Vehicle Information Acquisition Unit 104 Unit for determining correct sitting 105 Feature calculation unit 106 Feature standardization unit 107 Body composition determination unit 108 Posture warning unit 200 image recording device 300 belt control device 400 Airbag control device 500 Device for detecting left-behind persons 600 dispensing device 1001 processing circuit 1002 Input interface device 1003 Output interface device 1004 processor 1005 memory QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2020-104680 A

[0004]

Claims

[1] Body composition determination device comprising: an image acquisition unit for acquiring an image recording in which an occupant of a vehicle is recorded, a skeleton point extraction unit for extracting skeleton points of the occupant indicating parts of the occupant's body based on the image acquired by the image acquisition unit, a correct sitting determination unit for determining, based on status information about a status of the vehicle or the occupant, whether a state of a posture of the occupant is a state of correct sitting, a feature calculation unit for determining, based on the information on the occupant's skeleton points extracted by the skeleton point extraction unit and a sitting state determination result of the correct sitting determination unit as to whether the occupant's posture state is a correct sitting state, whether the occupant's skeleton points extracted by the skeleton point extraction unit are correct sitting skeleton points extracted based on the image capture in which the occupant was captured in the correct sitting state, and in a case where it is determined that the occupant's skeleton points extracted by the skeleton point extraction unit are correct sitting skeleton points, calculating a physique determination feature based on the information on the correct sitting skeleton points, and a body type determination unit for determining a body type of the occupant based on the body type determination feature calculated by the feature calculation unit. [2] The body type determining device according to claim 1, wherein the status information is information on a facial orientation of the occupant detected based on the image pickup, and the correct sitting determining unit determines that the posture state of the occupant is the correct sitting state in a case that the facial orientation of the occupant is a forward orientation. [3] The body type determining device according to claim 2, further comprising a face orientation detecting unit that detects the face orientation of the occupant based on the image acquired by the image acquiring unit, wherein the correct sitting determining unit determines that the posture state of the occupant is the correct sitting state in a case that the face orientation of the occupant acquired by the face orientation detecting unit is the front orientation. [4] A physique determination device according to any one of claims 1 to 3, wherein the status information is information on an inclination of a skeleton line connecting the skeleton points of the occupant, wherein the physique determining device further comprises an inclination detecting unit for detecting an inclination of the occupant's skeleton line based on the information on the skeleton points extracted by the skeleton point extraction unit, and wherein the correct sitting determining unit determines that the posture state of the occupant is the correct sitting state in a case that the skeletal line of the occupant detected by the inclination detecting unit is within an inclination determining range. [5] The body type determining device according to claim 4, wherein the occupant's skeleton line is a line connecting skeleton points indicating both shoulders of the occupant. [6] The body type determining device according to any one of claims 1 to 5, wherein the status information is vehicle information on the status of the vehicle, including a status of a steering control angle of the vehicle, a traveling speed, an opening or closing status of a door of the vehicle, a status of an ignition of the vehicle, or an operation status of an on-board device, wherein the body type determining device further comprises a vehicle information acquiring unit for acquiring the vehicle information, and wherein the correct seating determination unit determines that the state of the posture of the occupant is the correct seating state based on the vehicle information acquired by the vehicle information acquisition unit in a case that the steering control angle of the vehicle is within a steering angle determination range, in a case that the vehicle speed exceeds a vehicle speed determination range, in a case that no door of the vehicle is opened, in a case that an elapsed time since the ignition of the vehicle is turned on is within a start determination time, or in a case that no operation of the on-board device is performed. [7] The physique determination device according to any one of claims 1 to 6, wherein the status information is the information on the skeletal points of the occupant extracted by the skeletal point extraction unit, and wherein the correct sitting determination unit determines that the occupant is not in the correct sitting state in a case that a position of a skeletal point indicative of an elbow of the occupant extracted by the skeletal point extraction unit is within an elbow position determination range. [8] The body type determining device according to claim 1, wherein the correct sitting determining unit determines whether the state of the posture of the occupant is the correct sitting state based on the status information and a machine learning model that uses the status information as input and outputs information on whether the state of the posture of the occupant is the correct sitting state. [9] The physique determination device according to any one of claims 1 to 8, further comprising a feature standardization unit for retroactively standardizing a plurality of physique determination features calculated by the feature calculation unit for a standardization period, wherein the physique determination unit determines the physique of the occupant based on the physique determination features standardized by the feature standardization unit. [10] The physique determination device according to claim 9, wherein the feature standardization unit uses, as the physique determination feature after standardization, a value corresponding to an average value, a median value, or a set percentile value of the plurality of physique determination features. [11] The physique determination device according to any one of claims 1 to 10, further comprising a posture warning unit for outputting, in a case where a state in which the posture of the occupant determined by the correct sitting determination unit is not the correct sitting state continues for a posture abnormality determination time, posture warning information requesting the occupant to assume the correct sitting posture. [12] Body composition determination method, comprising: Obtaining, by an image acquisition unit, an image recording in which an occupant of a vehicle is recorded, Extracting, by a skeleton point extraction unit, based on the image acquired by the image acquisition unit, skeleton points of the occupant indicating parts of the occupant's body, Determining, by a correct sitting determination unit, based on status information about a status of the vehicle or the occupant, whether a state of a posture of the occupant is a state of correct sitting, Determining, by a feature calculation unit, based on the information on the occupant's skeleton points extracted by the skeleton point extraction unit and a sitting state determination result of the correct sitting determination unit, whether the occupant's posture state is a correct sitting state, whether the occupant's skeleton points extracted by the skeleton point extraction unit are correct sitting skeleton points extracted based on the image capture in which the occupant was captured in the correct sitting state, and in a case that it is determined that the occupant's skeleton points extracted by the skeleton point extraction unit are correct sitting skeleton points, calculating a physique determination feature based on the information on the correct sitting skeleton points, and Determining, by a body type determining unit, based on the body type determining feature calculated by the feature calculating unit, a body type of the occupant.

Images