Attitude estimation device

Abstract

To provide an attitude estimation device capable of easily achieving attitude estimation by using a single imaging device.SOLUTION: An attitude estimation device includes: an image acquisition part for acquiring a two-dimensional image obtained by imaging an attitude estimation object person seating on a seat in a vehicle by a single imaging device; a three-dimensional coordinate acquisition part for acquiring a three-dimensional coordinate of a body center part in a waist, a belly, and a shoulder of the attitude estimation object person and a three-dimensional coordinate of a left shoulder and a right shoulder on the basis of a two-dimensional coordinate of the two-dimensional image; a three-dimensional length acquisition part for acquiring a first three-dimensional length determined by the three-dimensional coordinate of the waist and the left shoulder and a second three-dimensional length determined by the three-dimensional coordinate of the waist and the right shoulder; and an attitude determination part for determining whether or not the attitude estimation object person keeps a normal seating attitude in which it is regarded that a seat center line passing through a center position in a lateral direction of the seat and regulated in a vertical direction is coincident with a body center line passing through the center position in the lateral direction of the body of the attitude estimation object person and regulated in the vertical direction, on the basis of the first three-dimensional length and the second three-dimensional length.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a pose estimation device. [Background technology]

[0002] Conventionally, various techniques have been proposed for detecting the posture and parts of a human body using an image of the human body. For example, a technique has been proposed in which an image of a person (human body) seated in a vehicle seat is captured by an imaging device, feature points of the human body are extracted, and the distance to the feature points is acquired to identify the posture, and vehicle control is performed according to the posture of the occupant. In addition, a technique has been proposed in which the distance distribution to the human body (person) is measured by stereo image processing, and the head, shoulders, legs, hands, etc. of the human body are detected. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-148865 A [Patent Document 2] JP 2003-294855 A [Patent Document 3] JP 2002-352231 A Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional technologies, detection of posture, etc. has involved the use of optical time-of-flight ranging or stereo ranging technology using multiple imaging devices, or the need for an imaging device for each detection target, resulting in problems such as increased costs for posture recognition (estimation).

[0005] Therefore, one of the objectives of the present disclosure is to provide a posture estimation device that can easily realize posture estimation using a single imaging device. [Means for solving the problem]

[0006] A posture estimation device as an example of the present disclosure includes an image acquisition unit that acquires a two-dimensional image of at least one posture estimation subject seated in a seat in the vehicle using a single imaging device provided in a vehicle cabin; a three-dimensional coordinate acquisition unit that acquires three-dimensional coordinates of at least the waist, abdomen, and shoulders of the posture estimation subject's body center and the three-dimensional coordinates of the left shoulder and right shoulder based on the two-dimensional coordinates of the two-dimensional image; a three-dimensional length acquisition unit that acquires a first three-dimensional length determined by the three-dimensional coordinates of the waist and the left shoulder and a second three-dimensional length determined by the three-dimensional coordinates of the waist and the right shoulder; and a posture determination unit that determines whether the posture estimation subject is in a normal seated posture in which a seat center line defined in the vertical direction passing through a center position of the seat in the left-right direction and a body center line defined in the vertical direction passing through a center position of the posture estimation subject's body in the left-right direction are consistent based on the first three-dimensional length and the second three-dimensional length. According to this configuration, for example, it is possible to easily determine (estimate) whether a person whose posture is to be estimated is seated in a vehicle seat in a normal seated posture that can be regarded as sitting in the center of the width of the seat and facing forward, that is, whether the person is seated in a desirable posture facing straight ahead.

[0007] Furthermore, when the posture determination unit of the posture estimation device determines that the posture is the normal seated posture, the posture determination unit may estimate the current posture of the posture estimation subject seated in the seat based on the difference from the normal seated posture using a reference body center index corresponding to the three-dimensional length from the abdomen to the height of the shoulder in the body center and a reference shoulder width index corresponding to the shoulder width length between the left shoulder and the right shoulder. With this configuration, for example, various seated postures can be easily estimated even in a state where various seating postures are permitted at the time of the emergence of a vehicle due to the advancement of autonomous driving technology.

[0008] In addition, the posture determination unit of the posture estimation device described above may estimate the current posture by, for example, assuming that the height of the waist of the posture estimation subject from the seat surface is the same regardless of the seated posture, and assuming that the positions of the waist and the abdomen are approximately the same. With this configuration, various seated postures can be easily estimated even using two-dimensional images captured by an in-vehicle imaging device that is easily subject to restrictions on the installation position. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is an exemplary schematic diagram illustrating the configuration of a vehicle in which a posture estimation device according to an embodiment can be mounted and an imaging range in the vehicle. [Diagram 2] FIG. 2 is an exemplary schematic diagram illustrating an image capturing process of a passenger seated in a front row seat in a vehicle equipped with a posture estimation device according to an embodiment, and the arrangement of the image capturing device. [Diagram 3] FIG. 3 is an exemplary schematic block diagram illustrating a configuration of a posture estimation device according to an embodiment. [Figure 4] FIG. 4 is an exemplary schematic explanatory diagram showing positions of skeleton points acquired by the posture estimation device according to the present embodiment using a two-dimensional image captured by an imaging device. [Diagram 5] FIG. 5 is an exemplary schematic explanatory diagram showing a normal sitting posture estimated by the posture estimation device according to the present embodiment. [Figure 6] FIG. 6 is an exemplary schematic explanatory diagram showing three-dimensional lengths between skeleton points used for pose estimation in the pose estimation device according to this embodiment. [Figure 7] FIG. 7 is an exemplary schematic explanatory diagram showing a reference body center index and a reference shoulder width index used for estimating a current posture in the posture estimation device according to this embodiment. [Figure 8] FIG. 8 is an exemplary schematic explanatory diagram showing a state in which the reference indices are aligned and adjusted to match the posture to be estimated in the posture estimation device according to this embodiment. [Figure 9]FIG. 9 is an exemplary flowchart showing the flow of a first stage process (normal seating determination process) of posture estimation by the posture estimation device according to this embodiment. [Figure 10] FIG. 10 is an exemplary flowchart showing the flow of post-stage processing (current posture estimation processing) of posture estimation by the posture estimation device according to this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments and modifications of the present disclosure will be described with reference to the drawings. The configurations of the embodiments and modifications described below, as well as the actions and effects brought about by the configurations, are merely examples and are not limited to the contents described below.

[0011] FIG. 1 is an exemplary schematic diagram for explaining the configuration of a vehicle 10 on which a posture estimation device according to an embodiment can be mounted and an imaging range E for recognizing a posture estimation target person in the vehicle 10. In this embodiment, for convenience, an X-axis, a Y-axis (see FIG. 2), and a Z-axis are defined. The X-axis, the Y-axis, and the Z-axis are mutually orthogonal. The X-axis indicates a direction along the vehicle width of the vehicle 10, and includes a +X direction (a left direction when the vehicle 10 is seen from the rear) and a -X direction (a right direction when the vehicle 10 is seen from the rear). The Y-axis indicates a direction along the vehicle height of the vehicle 10, and includes a +Y direction (an upward direction of the vehicle 10) and a -Y direction (a downward direction of the vehicle 10). The Z-axis indicates a direction along the front and rear of the vehicle 10, and includes a +Z direction (a direction facing backward of the vehicle 10) and a -Z direction (a direction facing forward of the vehicle 10).

[0012] As illustrated in FIG. 1, the vehicle 10 has a cabin 10a in which passengers (driver, passenger, posture estimation target) (not shown) ride. In the cabin 10a, for example, a driver's seat 14a (driver's seat) is arranged facing the steering wheel 12, and a passenger seat 14b (passenger seat) is arranged next to the seat 14a (in the +X direction) as a front row seat 14 (front seat). In addition, for example, passenger seats 16a, 16b, and 16c are arranged as a rear row seat 16 (rear seat). Note that FIG. 1 shows an example of a five-seat (five-seater) vehicle with two front row seats 14 and three rear row seats 16, but the seat layout can be appropriately selected according to, for example, the vehicle model. For example, the rear row seats 16 may be configured with two seats (four-seater together with the front row seats 14). Also, a three-row seat configuration or a configuration with three or more rows may be used. Also, the vehicle may be configured with a single row of front seats 14 for passenger use, with the space behind the front seats 14 provided as luggage space.

[0013] FIG. 2 is an exemplary schematic diagram illustrating an image capture of a posture estimation subject 18 (e.g., a passenger fastening a seat belt 20) seated in a front row seat 14 (seat 14a) in a vehicle 10 equipped with a posture estimation device according to an embodiment, and the arrangement of an imaging device (e.g., an infrared imaging unit 22).

[0014] The front row seat 14 (seat 14a) is composed of a seat cushion 14c, a seat back 14d, a headrest 14e, etc., and a frame (not shown) supporting the seat cushion 14c is fixed to the floor surface of the vehicle interior 10a by rails (not shown) so that the position can be adjusted in the vehicle front-rear direction (+Z direction, -Z direction). The front row seat 14 is also provided with various adjustment mechanisms such as a reclining mechanism, and is configured to be able to adjust the posture of a posture estimation subject 18 (driver) who wears a seat belt 20 and grips the steering wheel 12 so as to easily maintain the posture suitable for driving.

[0015] Incidentally, with airbags now being standard equipment in the vehicle 10, there is a demand for controlling the amount of airbag deployment by estimating the physique of the occupant. In addition, with the recent spread of autonomous driving, it is considered that the posture of the occupant when seated in a seat (for example, the front row seat 14) will change in various ways (to a posture that is not necessarily suitable for driving). In this embodiment, the posture estimation device captures an image of the inside of the vehicle compartment 10a with a single imaging device (infrared imaging unit 22) to realize the seating posture of the occupant (including the driver and other occupants, if any) more accurately while suppressing the cost required for posture estimation.

[0016] In the posture estimation device of this embodiment, an infrared imaging unit 22 is disposed at a position overlooking the interior of the vehicle cabin 10a in order to estimate the posture of an occupant in the vehicle cabin 10a. The infrared imaging unit 22 is an example of an imaging device. An infrared irradiator 24 is disposed near the infrared imaging unit 22. In this embodiment, the infrared imaging unit 22 and the infrared irradiator 24 are disposed at approximately the center of the upper inner part of the front window of the vehicle 10 (for example, near the rearview mirror). The infrared imaging unit 22 may be disposed at another position as long as the interior of the vehicle cabin 10a can be overlooked. In addition, as long as infrared light can be irradiated to the entire imaging area of ​​the infrared imaging unit 22, the position of the infrared irradiator 24 can be appropriately changed, and the infrared imaging unit 22 and the infrared irradiator 24 may be disposed at different positions. In addition, the infrared imaging unit 22 and the infrared irradiator 24 may be configured as an integrated camera unit.

[0017] The infrared imaging unit 22 is, for example, a digital camera incorporating an imaging element such as a CCD (Charge Coupled Device) or a CIS (CMOS image sensor) that is compatible with infrared imaging. The infrared imaging unit 22 can sequentially output video data (captured image data) at a predetermined frame rate. The infrared irradiating unit 24 can be, for example, an LED (Light Emitting Diode) light that irradiates infrared light. As shown by hatching in FIG. 1, the infrared imaging unit 22 can image, for example, an imaging range E (an area with the front row seats 14 and the rear row seats 16) in which the posture estimation subject 18 may be seated in the vehicle interior 10a by itself. In addition, the irradiation angle and irradiation intensity of the infrared irradiating unit 24 are set so that infrared light can be irradiated to an area including at least the imaging range E of the infrared imaging unit 22.

[0018] The viewing angle and posture of the infrared imaging unit 22 are adjusted and fixed so that the imaging range E includes at least the posture estimation subjects 18 seated in the front row seat 14 and the rear row seat 16. For example, as shown in FIG. 2, the imaging area of ​​the infrared imaging unit 22 includes the face and at least the waist (including the abdomen) of the posture estimation subject 18 when the posture estimation subject 18 is seated in the seat 14a (driver's seat). Similarly, the face and at least the waist (including the abdomen) of the posture estimation subject 18 when the posture estimation subject 18 is seated in the seat 14b (passenger seat). The same applies when the posture estimation subject 18 is seated in the seats 16a to 16c of the rear row seat 16.

[0019] The infrared irradiating unit 24 irradiates infrared light to the posture estimation subject 18 seated in the seat 14a, the seat 16a, etc., but since the infrared light is not recognized as light by the human eye, the posture estimation subject 18 does not feel dazzled even if the infrared light is irradiated in the direction of the face of the posture estimation subject 18. Therefore, it is possible to easily image the posture estimation subject 18, etc. by the infrared imaging unit 22 while ensuring the comfort of the posture estimation subject 18 in the vehicle interior 10a.

[0020] 3 is an exemplary schematic block diagram showing a configuration of posture estimation device 26 according to an embodiment. Like a general computer, posture estimation device 26 includes, for example, a central processing unit (CPU) 28, a read only memory (ROM) (not shown), a random access memory (RAM), and a storage unit N such as a solid state drive (SSD) or a flash memory. Note that storage unit N may be configured as a hard disk drive (HDD) or an SSD independent of posture estimation device 26.

[0021] The CPU 28 reads out a posture estimation program installed and stored in a non-volatile storage unit such as a ROM or SSD, and executes various modules for executing posture estimation processing according to the posture estimation program. The CPU 28 realizes, as various modules, for example, an image acquisition unit 28a, a two-dimensional skeleton point detection unit 28b, a distortion removal unit 28c, a three-dimensional coordinate acquisition unit 28d, a three-dimensional length acquisition unit 26e, a posture determination unit 28f, and the like. The posture determination unit 28f includes detailed modules such as a posture difference determination unit 28f1, a normal posture determination unit 28f2, and a current posture determination unit 28f3. The CPU 28 estimates the seating posture of the posture estimation subject 18 who is seated in a seat (front seat 14, rear seat 16) in the vehicle interior 10a, using the two-dimensional image captured by the infrared imaging unit 22.

[0022] The image acquisition unit 28a continuously acquires two-dimensional images in order to estimate the current posture state of the posture estimation subject 18 present in the vehicle interior 10a. As described above, the two-dimensional image simultaneously includes a plurality of posture estimation subjects 18 who may be seated in each seat in the vehicle interior 10a. Note that the infrared imaging unit 22 may specify a specific posture estimation subject 18 (specific seat) and capture a two-dimensional image to estimate the posture. The image acquisition unit 28a may automatically acquire two-dimensional images at a predetermined interval, or may acquire two-dimensional images at a timing specified by the user (posture estimation subject 18).

[0023] When multiple posture estimation subjects 18 are present in the two-dimensional image acquired by the image acquisition unit 28a, the two-dimensional skeleton point detection unit 28b selects an area in which one posture estimation subject 18 may be included according to a predetermined rule (for example, a predetermined priority order) from among the two-dimensional image. The two-dimensional skeleton point detection unit 28b may select an area in which the posture estimation subject 18 may be included in the following order: the driver's seat, the passenger seat, the rear left side, the rear center, and the rear right side. The two-dimensional skeleton point detection unit 28b then detects (extracts) skeleton points (coordinates) of the posture estimation subject 18 on the two-dimensional image, for example, using a skeleton point detection model using a well-known artificial intelligence technology or the like. For example, FIG. 4 is an exemplary and schematic explanatory diagram showing a state in which an area F in which one posture estimation subject 18 may be included is selected from the two-dimensional image M captured by the infrared imaging unit 22 (imaging device) acquired by the posture estimation device 26, and showing the position (coordinates) of the skeleton point 30 acquired within the area F. In the example shown in FIG. 4, the two-dimensional skeleton point detection unit 28b detects the lumbar skeleton points 30a (left lumbar skeleton point 30a1, right lumbar skeleton point 30a2), abdominal skeleton points 30b (left abdominal skeleton point 30b1, right abdominal skeleton point 30b2), and shoulder skeleton points 30c (left shoulder skeleton point 30c1, right shoulder skeleton point 30c2) as skeleton points 30 of the posture estimation subject 18 wearing the seat belt 20. When a plurality of posture estimation subjects 18 are present in the two-dimensional image, the two-dimensional skeleton point detection unit 28b similarly sequentially detects (extracts) skeleton points (coordinates) for the other posture estimation subjects 18. Note that, in this embodiment, the skeleton points 30 used on the two-dimensional image for posture estimation are the above-mentioned three types, but other skeleton points may be additionally used. Furthermore, the two-dimensional skeleton point detection unit 28b only needs to be able to detect (extract) skeleton points on a two-dimensional image, and may perform detection using a method other than a skeleton point detection model using artificial intelligence technology or the like.

[0024] Next, the distortion removal unit 28c removes lens distortion of the two-dimensional skeleton point coordinates detected by the two-dimensional skeleton point detection unit 28b. As described above, the two-dimensional image used by the posture estimation device 26 uses the image of the interior of the vehicle cabin 10a captured by the single infrared imaging unit 22. Therefore, for example, a fisheye lens (wide-angle lens) is attached to the infrared imaging unit 22, and a wider area is captured in one frame. Therefore, the captured two-dimensional image becomes more distorted toward the periphery. Therefore, the distortion removal unit 28c converts the image into orthogonal coordinates using a well-known distortion correction technique (coordinate conversion technique, etc.), thereby reducing the load of subsequent processing.

[0025] The posture estimation process in the posture estimation device 26 of this embodiment is divided into, for example, a front-stage process of acquiring a normal seating posture, which is a reference posture, and a back-stage process of estimating the current posture by referring to this normal seating posture.

[0026] In the first stage of processing, the three-dimensional coordinate acquisition unit 28d proceeds with the processing on the assumption that the posture estimation subject 18 being processed (selected) is seated in the center of the seat (for example, the front row seat 14) in the seat width direction. FIG. 5 is an exemplary and schematic explanatory diagram showing a normal seating posture estimated by the posture estimation device 26. FIG. 5 shows a state in which it is assumed that the posture estimation subject 18 is seated in a state in which the seat center line SC (center line in the seat width direction) of the front row seat 14 and the body center line HC (for example, a direction along the spinal cord) of the posture estimation subject 18 are substantially aligned. The seat center line SC is a position determined based on the center line in the vehicle width direction of the vehicle 10 (vehicle center line CC), and it is possible to acquire three-dimensional coordinates. It is also possible to acquire coordinates for other seats based on the vehicle center line CC of the vehicle 10. When the posture estimation subject 18 is seated facing almost straight ahead on the seat center line SC of the front row seat 14, for example, the distances from any position on the body center line HC of the posture estimation subject 18 to the left and right skeletal positions at any other height position on the body center line HC are considered to be approximately the same. FIG. 6 is an exemplary and schematic explanatory diagram showing a three-dimensional length 34 between skeletal points used for posture estimation in the posture estimation device 26. As shown in FIG. 5 and FIG. 6, when the posture estimation subject 18 is seated facing almost straight ahead on the seat center line SC of the front row seat 14, for example, the three-dimensional length 34a (34) from the waist center 32, which is the center of the waist in the left-right direction, to the left shoulder skeletal point 30c1 and the three-dimensional length 34b (34) from the waist center 32 to the right shoulder skeletal point 30c2 are approximately the same.

[0027] Therefore, the three-dimensional coordinate acquisition unit 28d first calculates the two-dimensional coordinate of the center in the X direction for each part from the two-dimensional coordinates (distortion-removed) acquired by the two-dimensional skeleton point detection unit 28b, using the average value of the coordinates of the left and right skeleton points 30. For example, the two-dimensional coordinate of the center of the waist (waist center 32) is calculated from the central coordinate of the left waist skeleton point 30a1 and the right waist skeleton point 30a2. Similarly, the three-dimensional coordinate acquisition unit 28d calculates the two-dimensional coordinate of the center of the abdomen (the central coordinate of the left abdomen skeleton point 30b1 and the right abdomen skeleton point 30b2) and the two-dimensional coordinate of the center of the shoulder (the central coordinate of the left shoulder skeleton point 30c1 and the right shoulder skeleton point 30c2). As described above, when it is assumed that the posture estimation subject 18 is seated facing almost straight ahead on the seat center line SC of the front row seat 14, the three-dimensional coordinate acquisition unit 28d regards the three-dimensional X coordinate of the center of each part as the same as the X coordinate of the seat center line SC. Then, once the X coordinate is determined, it becomes possible to calculate the three-dimensional Y and Z coordinates of the center of each part of the posture estimation subject 18. For example, it is assumed that the depth (Z direction) of the three-dimensional coordinates of the shoulders is equal for the left and right shoulders, and the three-dimensional coordinates of the left and right shoulders are calculated. That is, the Z coordinate of the three-dimensional coordinates of the shoulder center can be used as the Z coordinate of the three-dimensional coordinates of the left and right shoulders. Then, once the Z coordinate is determined, it is possible to calculate the X and Y coordinates of the three-dimensional coordinates of the left and right shoulders from the two-dimensional coordinates.

[0028] The three-dimensional length acquisition unit 28e calculates, for example, a three-dimensional length 34a (first three-dimensional length) from the calculated three-dimensional coordinates of the waist center 32 to the left shoulder skeleton point 30c1, and a three-dimensional length 34b (second three-dimensional length) from the three-dimensional coordinates of the waist center 32 to the right shoulder skeleton point 30c2.

[0029] The posture determination unit 28f estimates (determines) the posture of the posture estimation subject 18 using the calculated three-dimensional coordinates and the two-dimensional coordinates on the two-dimensional image. The posture difference determination unit 28f1 compares the three-dimensional length 34a and the three-dimensional length 34b calculated by the three-dimensional length acquisition unit 28e. If the difference is within a predetermined value as a result of the comparison by the posture difference determination unit 28f1, for example, if the difference between the three-dimensional length 34a and the three-dimensional length 34b is within ±2%, the normal posture determination unit 28f2 determines that the posture estimation subject 18 is normally seated in the front row seat 14.

[0030] When the normal posture determination unit 28f2 determines that the current posture of the posture estimation subject 18 is considered to be normal sitting, it stores the three-dimensional coordinate data of each part as normal sitting information in the storage unit N. For example, when the posture estimation subject 18 is determined to be normal sitting, the "waist coordinate (height)", "length from abdomen to shoulder", "shoulder width", etc. are saved, and the first stage process is temporarily terminated. In other words, it is assumed that the preparation for starting the second stage process is completed. Note that when the normal posture determination unit 28f2 determines that the current posture of the posture estimation subject 18 is not normal sitting, that is, when the difference between the three-dimensional length 34a and the three-dimensional length 34b is greater than ±2%. In this case, the above-mentioned process is executed again at the next execution timing of the first stage process. In other words, the first stage process is executed repeatedly until the posture estimation subject 18 changes its posture and the posture is considered to be normal sitting.

[0031] When the first stage processing is completed, that is, when the left and right three-dimensional lengths match with an error within a predetermined value and three-dimensional coordinates of the posture estimation subject 18 in a normally seated state can be acquired. In this case, the posture estimation device 26 is ready to execute the second stage processing. The second stage processing is a process for estimating the current posture state of the posture estimation subject 18 using the normally seated information acquired in the first stage processing.

[0032] In the posture estimation device 26, as in the previous process, the image acquisition unit 28a continuously acquires two-dimensional images M in order to estimate the current posture state of the posture estimation subject 18 present in the vehicle cabin 10a. Then, when multiple posture estimation subjects 18 exist in the two-dimensional images M acquired by the image acquisition unit 28a, the two-dimensional skeleton point detection unit 28b selects an area that may include one posture estimation subject 18 from among them according to a predetermined rule (for example, a predetermined priority order). Next, the two-dimensional skeleton point detection unit 28b detects (extracts) skeleton points (coordinates) on the two-dimensional image M using, for example, a skeleton point detection model using well-known artificial intelligence technology. Then, the distortion removal unit 28c removes lens distortion from the two-dimensional skeleton point coordinates detected by the two-dimensional skeleton point detection unit 28b.

[0033] Then, the three-dimensional coordinate acquisition unit 28d calculates three-dimensional coordinates for each frame of the two-dimensional image M acquired by the image acquisition unit 28a, and uses the three-dimensional coordinates to estimate the current posture of the posture estimation subject 18.

[0034] FIG. 7 is an exemplary and schematic explanatory diagram showing a reference body center index 38 and a reference shoulder width index 40 used in estimating a current posture in the posture estimation device 26. As shown in FIG. 7, when the posture estimation subject 18 is seated in a seat (for example, the front row seat 14), the height of the waist can be considered to be the same regardless of the posture. Therefore, the three-dimensional coordinate acquisition unit 28d calculates the three-dimensional X and Y coordinates of the waist center from the three-dimensional Y coordinate of the waist center and the two-dimensional coordinate of the waist center (two-dimensional image M), and acquires the three-dimensional coordinates (X, Y, Z) of the waist center 32. In addition, since the three-dimensional X coordinate of the abdomen center 36, which is the central position of the abdomen in the X direction, and the three-dimensional X coordinate of the waist center 32 are almost equal, the three-dimensional X coordinate of the waist center 32 is used as the three-dimensional X coordinate of the abdomen center 36. Then, the three-dimensional coordinate acquisition unit 28d uses the three-dimensional X coordinate of the abdomen center 36 and the two-dimensional image M to acquire the three-dimensional coordinate of the abdomen center 36.

[0035] Next, the current posture determination unit 28f3 of the posture determination unit 28f acquires a reference body center index 38 and a reference shoulder width index 40 that are references for posture estimation of the posture estimation subject 18. The reference body center index 38 is an index corresponding to the three-dimensional length from the three-dimensional coordinates of the abdomen center 36 of the posture estimation subject 18 to the shoulder height (the median value between the left shoulder skeleton point 30c1 and the right shoulder skeleton point 30c2, i.e., the shoulder width median position 30d). The reference shoulder width index 40 is an index corresponding to the shoulder width length, which is the length between the left shoulder skeleton point 30c1 and the right shoulder skeleton point 30c2.

[0036] The current posture determination unit 28f3 adjusts the position of the approximately T-shaped reference indicator composed of the acquired reference body center indicator 38 and reference shoulder width indicator 40 by displacing the reference indicator in three dimensions (rotational movement displacement, horizontal movement displacement, or a combination of these) so that the reference shoulder width indicator 40 matches the left and right coordinates of the shoulders (the coordinates of the left shoulder skeletal point 30c1 and the coordinates of the right shoulder skeletal point 30c2) in the two-dimensional image that is the estimation target acquired by the image acquisition unit 28a.

[0037] FIG. 8 is an exemplary schematic explanatory diagram showing a state in which the reference indices (reference body center index 38, reference shoulder width index 40) are aligned and adjusted to match the posture P to be estimated in the posture estimation device 26 (current posture determination section 28f3).

[0038] For example, in the case of posture P1, the posture estimation subject 18 is seated with a tilt toward the right (-X direction) with respect to the front row seat 14. That is, the body center line HC is tilted with respect to the seat center line SC. The current posture determination unit 28f3 rotates the reference indices (reference body center index 38, reference shoulder width index 40) in a two-dimensional image of the posture estimation subject 18, for example, around the abdominal center 36 so that the reference indices match the left and right coordinates of the shoulders of the posture estimation subject 18 (the coordinates of the left shoulder skeleton point 30c1 and the coordinates of the right shoulder skeleton point 30c2). As a result, the coordinates of the left shoulder skeleton point 30c1 and the coordinates of the right shoulder skeleton point 30c2 in the current posture of the posture estimation subject 18 are acquired, and further, the three-dimensional coordinates of the shoulder width center position 30d can be acquired. Therefore, the current posture determination unit 28f3 can estimate the posture of the posture estimation subject 18 by comparing each three-dimensional coordinate in the normal sitting posture with each three-dimensional coordinate in the current posture. For example, when the subject 18 is inclined to the right (-X direction) with respect to the normal sitting posture as in posture P1, the shoulder width center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 of the current posture move diagonally downward to the right with respect to the shoulder width center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 of the normal sitting posture. Conversely, when the current posture of the posture estimation subject 18 is inclined to the left (+X direction) with respect to the normal sitting posture, the shoulder width center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 of the current posture move diagonally downward to the left with respect to the shoulder width center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 of the normal sitting posture. In this way, the current posture of the posture estimation subject 18 can be estimated based on the coordinates of the shoulder width center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 in the current posture relative to the coordinates of the shoulder width center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 in the normal sitting posture.

[0039] Also, for example, in the case of posture P2, the posture estimation subject 18 is seated in a state where the body is twisted counterclockwise (right shoulder is pushed forward) with respect to the seat back line SB indicating the seat back 14d in a top view. In this case, the current posture determination unit 28f3 also rotates the reference indices (reference body center index 38, reference shoulder width index 40) in three dimensions (for example, rotates around the reference body center index 38) so that the reference indices (reference body center index 38, reference shoulder width index 40) match the left and right coordinates of the shoulders of the posture estimation subject 18 (the coordinates of the left shoulder skeleton point 30c1 and the coordinates of the right shoulder skeleton point 30c2) in the two-dimensional image capturing the posture estimation subject 18. As a result, the coordinates of the left shoulder skeleton point 30c1 and the coordinates of the right shoulder skeleton point 30c2 in the current posture of the posture estimation subject 18 are acquired, and further, the three-dimensional coordinates of the shoulder width center position 30d can be acquired. The current posture determination unit 28f3 estimates the posture of the posture estimation subject 18 by comparing the three-dimensional coordinates in the normal sitting posture. For example, when the body is twisted counterclockwise with respect to the normal sitting posture, as in posture P2, the shoulder width center position 30d and the right shoulder skeletal point 30c2 of the current posture move forward with respect to the shoulder width center position 30d and the right shoulder skeletal point 30c2 of the normal sitting posture. Also, the coordinate of the left shoulder skeletal point 30c1 moves backward with respect to the coordinate of the right shoulder skeletal point 30c2. In this way, the posture of the current posture estimation subject 18 can be estimated based on the position change of the coordinates of the shoulder width center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 of the current posture with respect to the coordinates of the shoulder width center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 of the normal sitting posture.

[0040] Moreover, in the case of posture P3, the posture estimation subject 18 is seated with a leftward shift in the lateral direction (+X direction) with respect to the front row seat 14. That is, the body center line HC is shifted to the left with respect to the seat center line SC. The current posture determination unit 28f3 slides the reference indicators (reference body center indicator 38, reference shoulder width indicator 40) leftward in three dimensions so that the reference indicators match the left and right coordinates of the shoulders of the posture estimation subject 18 (the coordinates of the left shoulder skeletal point 30c1 and the coordinates of the right shoulder skeletal point 30c2). As a result, the coordinates of the left shoulder skeletal point 30c1 and the coordinates of the right shoulder skeletal point 30c2 in the current posture of the posture estimation subject 18 are acquired, and further, the three-dimensional coordinates of the shoulder width center position 30d can be acquired. As in the other examples, the current posture determination unit 28f3 estimates the posture of the current posture estimation subject 18 by comparing each three-dimensional coordinate in the normal sitting posture with each three-dimensional coordinate in the current posture. For example, in the case of posture P3, when the posture is shifted to the left (+X direction) from the normal sitting posture, the shoulder width center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 of the current posture move leftward in the +X direction relative to the shoulder width center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 of the normal sitting posture. Conversely, when the current posture of the posture estimation subject 18 is shifted to the right (-X direction) from the normal sitting posture, each coordinate also moves in the -X direction. In this way, the posture of the current posture estimation subject 18 can be estimated based on the coordinates of the shoulder width center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 of the current posture relative to the coordinates of the shoulder width center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 of the normal sitting posture.

[0041] In the case of posture P4, the posture estimation subject 18 is seated in a forwardly shifted position relative to the front row seat 14. That is, as shown in posture P4, the shoulder width center position 30d and the abdominal center 36 are shifted downward. The current posture determination unit 28f3 slides the reference indicators (reference body center indicator 38, reference shoulder width indicator 40) downward in three dimensions so that the reference indicators match the left and right coordinates of the shoulders of the posture estimation subject 18 (the coordinates of the left shoulder skeleton point 30c1 and the coordinates of the right shoulder skeleton point 30c2). As a result, the coordinates of the left shoulder skeleton point 30c1 and the coordinates of the right shoulder skeleton point 30c2 in the current posture of the posture estimation subject 18 are acquired, and further, the three-dimensional coordinates of the shoulder width center position 30d can be acquired. As in the other examples, the current posture determination unit 28f3 estimates the posture of the current posture estimation subject 18 by comparing each three-dimensional coordinate in the normal sitting posture with each three-dimensional coordinate in the current posture. For example, in the case of posture P4, which is shifted forward from the normal sitting posture, the shoulder center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 of the current posture move downward relative to the shoulder center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 of the normal sitting posture. In this way, the current posture of the posture estimation subject 18 can be estimated based on the coordinates of the shoulder center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 of the current posture relative to the coordinates of the shoulder center position 30d of the normal sitting posture.

[0042] The current posture determination unit 28f3 may estimate the magnitude of posture displacement based on the amount of deviation of the coordinates of the shoulder width center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 in the current posture from the coordinates of the shoulder width center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2 in the normal sitting posture. For example, the estimation may be performed as a "large", "medium", or "small" inclination to the right, or a 10° inclination to the right. The sitting postures shown in FIG. 8 as postures P1 to P4 are only examples, and the current posture of the posture estimation subject 18 can be similarly estimated in other postures by comparing the coordinates of the shoulder width center position 30d, the left shoulder skeletal point 30c1, and the right shoulder skeletal point 30c2.

[0043] The flow of attitude estimation control in attitude estimation device 26 as described above will be described with reference to the flowcharts of FIGS.

[0044] First, the flow of the pre-stage process (normal seating posture estimation process) will be described with reference to FIG. 9. The image acquisition unit 28a of the posture estimation device 26 acquires two-dimensional images M captured by the in-vehicle infrared imaging unit 22 sequentially or at a predetermined interval (timing) (S100). Next, when there are multiple regions recognized as posture estimation subjects 18 in the two-dimensional image, the two-dimensional skeleton point detection unit 28b selects a region that may include one posture estimation subject 18 from among them according to a predetermined rule, and detects (extracts) skeleton points (coordinates) on the two-dimensional image using a skeleton point detection model using well-known artificial intelligence technology, etc., as described in FIG. 4 (S102). Then, the distortion removal unit 28c removes lens distortion from the two-dimensional skeleton point coordinates detected by the two-dimensional skeleton point detection unit 28b (S104).

[0045] Next, the three-dimensional coordinate acquisition unit 28d identifies (selects) one posture estimation subject 18 to be estimated from the two-dimensional image (S106), and acquires the three-dimensional coordinates of each skeleton point 30 (S108). Then, the three-dimensional length acquisition unit 28e calculates, for example, a three-dimensional length 34a from the three-dimensional coordinates of the waist center 32 to the left shoulder skeleton point 30c1, and a three-dimensional length 34b from the three-dimensional coordinates of the waist center 32 to the right shoulder skeleton point 30c2, as described in FIG. 6 (S110). Next, the posture difference determination unit 28f1 compares the three-dimensional length 34a and the three-dimensional length 34b calculated by the three-dimensional length acquisition unit 28e, and the normal posture determination unit 28f2 determines whether the posture estimation subject 18, which is the current processing target, is normally seated on a seat (for example, the front row seat 14) based on the comparison result of the posture difference determination unit 28f1 (S112). When it is determined that the posture estimation subject 18 currently being processed is not properly seated (No in S112), it is determined that proper seating is not established for the currently selected posture estimation subject 18 in the currently processed two-dimensional image. Therefore, in order to perform processing using another two-dimensional image, the posture determination unit 28f shifts the process to S100, causes the image acquisition unit 28a to acquire another two-dimensional image, and executes the steps from S102 onwards again.

[0046] In the process of S112, if the normal posture determination unit 28f2 determines that the currently selected posture estimation subject 18 is sitting normally (Yes in S112), the posture determination unit 28f stores the three-dimensional coordinate data of each skeletal part as normal sitting information in the storage unit N (S114). Then, the normal posture determination unit 28f2 determines whether posture estimation (determination of normal sitting) of all subjects who can be regarded as posture estimation subjects 18 in the currently processed two-dimensional image has been completed (S116). Then, if posture estimation (determination of normal sitting) of all subjects has been completed (Yes in S116), the posture estimation device 26 (posture determination unit 28f) temporarily ends this flow. That is, the previous stage process ends.

[0047] In S116, if posture estimation (determination of whether or not the subject is properly seated) has not been completed for all subjects (No in S116), the proper posture determination unit 28f2 proceeds to the processing of S106, identifies (selects) an area in the currently processed two-dimensional image where an unprocessed posture estimation subject 18 is deemed to exist, and continues to execute the processing from S108 onwards.

[0048] Next, the flow of the latter process will be described with reference to Fig. 10. The current posture determination unit 28f3 determines whether the former process is complete (S200). That is, the current posture determination unit 28f3 determines whether there is regular seating information for all posture estimation subjects 18 who are considered to be present in the vehicle interior 10a, and if not all regular seating information is available (the former process is not complete) (No in S200), this flow is temporarily terminated and the former process is waited for completion.

[0049] In the process of S200, if there is proper seating information for all posture estimation subjects 18 in the vehicle interior 10a (preceding process is completed) (Yes in S200), the image acquisition unit 28a acquires the latest two-dimensional image M captured by the in-vehicle infrared imaging unit 22 (S202). Next, if there are multiple image areas recognized as posture estimation subjects 18 in the two-dimensional image, the two-dimensional skeleton point detection unit 28b selects an area that may include one posture estimation subject 18 from among them according to a predetermined rule, and detects (extracts) skeleton points (coordinates) on the two-dimensional image using a skeleton point detection model using a well-known artificial intelligence technology or the like, as described in FIG. 4 (S204). Then, the distortion removal unit 28c removes lens distortion from the two-dimensional skeleton point coordinates detected by the two-dimensional skeleton point detection unit 28b (S206).

[0050] Next, the three-dimensional coordinate acquisition unit 28d identifies (selects) one of the posture estimation subjects 18 to be estimated from the two-dimensional image (S208), and acquires the three-dimensional coordinates of the abdomen center (S210). Then, the current posture determination unit 28f3 acquires the reference indices, that is, the reference body center index 38 and the reference shoulder width index 40, as described in Fig. 7. Then, the current posture determination unit 28f3 estimates the posture of the current posture estimation subject 18 based on the reference indices (the reference body center index 38 and the reference shoulder width index 40) (S212). 8, the current posture determination unit 28f3 acquires three-dimensional coordinates of the left and right shoulders of the current posture estimation subject 18 by, for example, rotating or sliding the reference indicators in three dimensions so that the reference indicators (reference body center indicator 38, reference shoulder width indicator 40) match the left and right coordinates of the shoulders of the posture estimation subject 18 (the coordinates of the left shoulder skeletal point 30c1 and the coordinates of the right shoulder skeletal point 30c2) in the two-dimensional image. Then, the current posture determination unit 28f3 estimates the posture of the current posture estimation subject 18 by comparing the acquired three-dimensional coordinates of the left and right shoulders (the three-dimensional coordinates of the left shoulder skeletal point 30c1 and the three-dimensional coordinates of the right shoulder skeletal point 30c2) and the three-dimensional coordinate of the abdominal center with the three-dimensional coordinates when properly seated.

[0051] The estimated posture of the posture estimation subject 18 is used, for example, for controlling the deployment pressure when an airbag is deployed. Furthermore, since posture estimation makes it possible to estimate the position of the head or other specific parts of the posture estimation subject 18 (passenger), this can be used, for example, for controlling the airflow of an air conditioning device. In addition, the estimated posture of the posture estimation subject 18 (passenger) can be used for various controls that can utilize the posture. Furthermore, the posture estimation device 26 of this embodiment can realize posture estimation easily and at low cost using a single imaging device.

[0052] Then, the current posture determination unit 28f3 determines whether or not posture estimation of all posture estimation subjects 18 that can be considered to be present in the vehicle interior 10a has been completed (S214). If not yet completed (No in S214), the posture determination unit 28f returns to the process of S202 and executes the processes from S204 onward again to continue posture estimation of other posture estimation subjects 18.

[0053] In the flowchart of Fig. 10, a new two-dimensional image is acquired when estimating the posture of another posture estimation subject 18. In this case, posture estimation of the posture estimation subject 18 can be performed based on the latest two-dimensional image. In another embodiment, posture estimation of the other posture estimation subject 18 may be performed using the same two-dimensional image. In this case, if the processing in S214 is No, the process proceeds to the processing in S208, which allows for simplification of the processing.

[0054] In the process of S214, when it is determined that the posture estimation of all posture estimation subjects 18 who are considered to be present in the vehicle interior 10a has been completed (Yes in S214), this flow is temporarily ended.

[0055] 9 and 10 show an example in which image regions recognized as posture estimation subjects 18 from a two-dimensional image are individually identified (selected) and the pre-processing and post-processing are performed. In another embodiment, the pre-processing and post-processing may be performed simultaneously (in parallel) for multiple or all posture estimation subjects 18 present in the two-dimensional image. In this case, the processing load on the CPU 28 increases, but postures of multiple posture estimation subjects 18 in the vehicle interior 10a can be estimated simultaneously, and the processing time can be shortened.

[0056] The posture estimation program for posture estimation realized by CPU 28 of posture estimation device 26 of this embodiment may be configured to be provided by being recorded in a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD) in the form of an installable or executable file.

[0057] Furthermore, the posture estimation program for executing the posture estimation process of this embodiment may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the posture estimation program executed in this embodiment may be configured to be provided or distributed via a network such as the Internet.

[0058] Although the present embodiment and the modified examples have been described above, the above-mentioned embodiment and modified examples are merely examples and are not intended to limit the scope of the invention. The above-mentioned novel embodiment and modified examples can be implemented in various forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. The above-mentioned embodiment and modified examples are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0059] 10...vehicle, 14...front row seat, 16...rear row seat, 22...infrared imaging unit (imaging device), 24...infrared irradiation unit, 26...posture estimation device, 28...CPU, 28a...image acquisition unit, 28b...two-dimensional skeletal point detection unit, 28c...distortion removal unit, 28d...three-dimensional coordinate acquisition unit, 28e...three-dimensional length acquisition unit, 28f...posture determination unit, 28f1...posture difference determination unit, 28f2...normal posture determination unit, 28f3...current posture determination unit, 30...skeletal point, 30a...lumbar skeletal point, 30b...abdominal skeletal point, 30c...shoulder skeletal point, 32...lumbar center, 34...three-dimensional length, 36...abdominal center, 38...reference body center index, 40...reference shoulder width index, CC...vehicle center line, SC...seat center line, HC...body center line, M...two-dimensional image.

Claims

1. an image acquisition unit that acquires a two-dimensional image of at least one posture estimation subject seated in a seat in the vehicle using a single imaging device provided in a vehicle cabin; a three-dimensional coordinate acquisition unit that acquires three-dimensional coordinates of at least a waist, an abdomen, and a shoulder of a central body portion of the posture estimation subject and three-dimensional coordinates of a left shoulder and a right shoulder based on the two-dimensional coordinates of the two-dimensional image; a three-dimensional length acquisition unit that acquires a first three-dimensional length determined by the three-dimensional coordinates of the waist and the three-dimensional coordinates of the left shoulder and a second three-dimensional length determined by the three-dimensional coordinates of the waist and the three-dimensional coordinates of the right shoulder; a posture determination unit that determines, based on the first three-dimensional length and the second three-dimensional length, whether the person to be estimated is in a normal seated posture in which a seat center line that passes through a center position in the left-right direction of the seat and is defined in a vertical direction coincides with a body center line that passes through a center position in the left-right direction of the person to be estimated; A posture estimation device comprising:

2. 2. The posture estimation device according to claim 1, wherein, when the posture determination unit determines that the posture is the normal sitting posture, the posture determination unit estimates the current posture of the posture estimation subject seated in the seat based on a difference from the normal sitting posture using a reference body center index corresponding to a three-dimensional length from the abdomen to the height of the shoulders in the body center and a reference shoulder width index corresponding to a shoulder width length between the left shoulder and the right shoulder.

3. 3. The posture estimation device according to claim 2, wherein the posture determination unit estimates the current posture by assuming that a height of the waist of the posture estimation subject from a seat surface is the same regardless of the seated posture, and that the positions of the waist and the abdomen are approximately the same.

Images