Position detection device, position detection program, position detection method, and human body monitoring system

Abstract

The present invention is provided with: a feature point detection unit (11) for detecting a target feature point in a captured image on the basis of a captured image captured by an imaging device (2); a search region determination unit (12) for determining a three-dimensional region on the basis of the position on the captured image of the target feature point detected by the feature point detection unit (11); a point group acquisition unit (13) for acquiring reflection point group information based on the reflected wave of a radio wave having been radiated toward a target region by a radio wave sensor (3) and reflected by a reflection object including a person in the target region; and a position identification unit (14) for detecting the three-dimensional position of the target feature point from three-dimensional position information of a reflection point detected in the three-dimensional region among a plurality of reflection points on the basis of the three-dimensional region and the reflection point group information.

Description

Position detection device, position detection program, position detection method, and human body monitoring system

[0001] The present disclosure relates to a position detection device, a position detection program, a position detection method, and a human body monitoring system.

[0002] Conventionally, there is known a technique for detecting the positions of feature points (hereinafter referred to as "target feature points") corresponding to body parts of a person based on an image capturing an area including the person (hereinafter referred to as "captured image"). Information regarding the positions of the detected target feature points (hereinafter referred to as "position information") is used, for example, to determine the person's physique. Here, if the distance from the imaging device to the person changes, the size of the person in the captured image capturing the person changes. Therefore, the positions of the target feature points detected in the captured image must be detected taking into account their depth positions. If the depth positions are not taken into account when determining the positions of the target feature points detected in the captured image, there is a possibility that the person's physique, etc., may not be accurately determined using the position information of the detected target feature points. For example, Patent Literature 1 discloses a physique estimation device for a vehicle that acquires the positions of multiple skeletal points of an occupant's body image included in an image captured by a camera that captures the interior of the vehicle as skeletal point coordinates in a two-dimensional coordinate system of the captured image, and has a front coordinate estimation unit that performs machine learning to estimate and output front coordinates that indicate the location of the skeletal points when the occupant is viewed from the front by inputting distance information that indicates the distances between multiple preset skeletal points and angle information that indicates the angles of skeletal lines connecting the multiple skeletal points with respect to a reference line, and determines the physique of the occupant from the estimated front coordinates.When determining the physique of the occupant, the physique estimation device acquires the fore-and-aft position and height of the seat base of the seat and the tilt angle of the seat back as seat position information and corrects the skeletal point coordinates to correction values ​​corresponding to the distance between the occupant and the camera.

[0003] Japanese Patent Application Laid-Open No. 2021-081836

[0004] Conventional technologies such as those disclosed in Patent Document 1 have difficulty distinguishing between different postures, including those with different depth positions, that form similar skeletal lines in a captured image, potentially resulting in erroneous physique determination. Therefore, there remains a problem in that it is not possible to accurately detect the positions of target feature points detected in a captured image while taking into account their depth positions, which is necessary for accurate physique determination. For example, with technologies such as those disclosed in Patent Document 1, when an occupant is imaged from the dashboard side of a vehicle, in the reference posture, the occupant's shoulder width direction is imaged horizontally relative to the imaging plane, and the occupant's torso is imaged perpendicularly relative to the imaging plane. However, if the occupant is seated closer to the front or to the side of the seat, but with a translational shift from the reference posture, the change in seating position cannot be accurately detected from skeletal line information. As a result, for example, there is a concern that the physique of an occupant sitting closer to the front may be perceived as larger. Even if seat information were utilized, the occupant sits farther away from the seat, making correction difficult using only seat information.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a position detection device that is capable of detecting the position, including the depth position, of target feature points detected from a captured image, even if the posture forms similar skeletal lines on the captured image.

[0006] The position detection device disclosed herein includes a feature point detection unit that detects target feature points corresponding to parts of a person's body in an image captured by an imaging device of a target area where a person may be present, based on an image captured by the imaging device; a search area determination unit that determines a three-dimensional area in real space in which the target feature points are estimated to exist, based on the positions on the image of the target feature points detected by the feature point detection unit; a point cloud acquisition unit that acquires reflection point cloud information regarding a plurality of reflection points, including three-dimensional position information of a plurality of reflection points of radio waves from reflecting objects, based on reflected waves of radio waves emitted by a radio wave sensor toward the target area and reflected by reflecting objects including people within the target area; and a position identification unit that identifies the three-dimensional position of the target feature point from the three-dimensional position information of a reflection point detected within the three-dimensional area among the plurality of reflection points, based on the three-dimensional area determined by the search area determination unit and the reflection point cloud information acquired by the point cloud acquisition unit, and detects the identified three-dimensional position as the three-dimensional position of the target feature point.

[0007] According to the present disclosure, the position detection device is configured as described above, so that it can detect the position, including the depth position, of target feature points detected from the captured image even if the posture forms similar skeletal lines on the captured image.

[0008] 4A and 4B are diagrams illustrating an example of the configuration of a position detection device according to embodiment 1.

[0023] FIG. 4B is a diagram illustrating an example of the installation position and orientation of an imaging device mounted on a vehicle, and an example of the installation position and orientation of a radio wave sensor mounted on the vehicle, in embodiment 1.

[0024] FIG. 4C is a diagram illustrating an example of the installation position and orientation of an imaging device mounted on a vehicle, and another example of the installation position and orientation of a radio wave sensor mounted on the vehicle, in embodiment 1.

[0025] FIG. 4D is a diagram illustrating an image captured by the imaging device. 5A is a diagram illustrating an example of a three-dimensional area determined by a search area determination unit based on the positions on a captured image of target feature points detected by a feature point detection unit in embodiment 1, where FIG. 5A is a diagram showing a captured image in which target feature points are captured, and FIGS. 5B, 5C, and 5D are diagrams showing the three-dimensional area determined by the search area determination unit. FIG. 5B is a flowchart illustrating the operation of a human body monitoring system according to embodiment 1. FIG. 7A is a diagram illustrating a captured image in which target feature points are captured, and FIGS. 7B, 7C, and 7D are diagrams showing the three-dimensional area determined by the search area determination unit in embodiment 1. 8A and 8B are diagrams for explaining an example of a method in which a position detection device, in embodiment 1, identifies the three-dimensional position of a certain target feature point based on the three-dimensional areas of other target feature points and the three-dimensional positions of the other detected target feature points when no reflection point is detected within the three-dimensional area of ​​the certain target feature point; FIG. 8A is a diagram showing an image in which the target feature point is captured, and FIGS. 8B, 8C, and 8D are diagrams showing the three-dimensional area initially determined by the search area determination unit.9A, 9B, 9C, and 9D are diagrams showing a three-dimensional area redetermined by the search area determination unit. 10A and 10B are diagrams showing an example of the hardware configuration of a position detection device according to embodiment 1. 10B are diagrams showing an example of the configuration of a position detection device according to embodiment 2. 10C are diagrams showing an example of the configuration of a position detection device according to embodiment 2. 10D are diagrams showing an example of the hardware configuration of a position detection device according to embodiment 1. 10E are diagrams showing an example of the configuration of a position detection device according to embodiment 2. 10F are diagrams showing an example of the hardware configuration of a position detection device according to embodiment 1. 10G are diagrams showing an example of the configuration of a position detection device according to embodiment 2. 10G are diagrams showing an example of the hardware configuration of a position detection device according to embodiment 1. 10H are diagrams showing an example of the configuration of a position detection device according to embodiment 2. 10H are diagrams showing an example of the hardware configuration of a position detection device according to embodiment 1. 10H are diagrams showing an example of the hardware configuration of a position detection device according to embodiment 1. 10H are diagrams showing an example of the hardware configuration of a position detection device according to embodiment 2 ...1. 10H are diagrams showing an example of the hardware configuration of a position detection device according to embodiment 1

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0010] Embodiment 1. A position detection device according to embodiment 1 identifies the three-dimensional positions of a person's body parts, such as the head, eyes, nose, neck, shoulders, or waist, based on an image captured by an imaging device (hereinafter referred to as the "captured image") and information sensed by a radio wave sensor, and detects the identified three-dimensional positions as the three-dimensional positions of the person's body parts. In embodiment 1, a person's body part is represented by a feature point associated with the person's body part. That is, the three-dimensional position of a person's body part is represented by the three-dimensional position of the feature point corresponding to the person's body part. Note that which point on the person's body part is designated as the feature point corresponding to the position of the person's body part is determined in advance. In embodiment 1, a feature point associated with the person's body part that represents the position of the person's body part is referred to as a "target feature point." In addition, in the following embodiment 1, as an example, the person whose body part the position detection device detects the three-dimensional position of is assumed to be an occupant sitting in the driver's seat or passenger seat of a vehicle. In the following embodiment 1, the occupant sitting in the driver's seat or passenger seat is also simply referred to as the "occupant." In the following first embodiment, as an example, it is assumed that the three-dimensional position detected by the position detection device is used to determine the physique of the occupant.

[0011] 1 is a diagram showing an example of the configuration of a position detection device 1 according to embodiment 1. The position detection device 1 is mounted on a vehicle 1000 and connected to an imaging device 2, a radio wave sensor 3, and a physique detection device 4. The position detection device 1 and the physique detection device 4 constitute a human body monitoring system 100.

[0012] The imaging device 2 is configured by a visible light camera or an infrared camera mounted on the vehicle 1000. The imaging device 2 may be shared with, for example, a so-called "Driver Monitoring System (DMS)." The imaging device 2 has an imaging angle of view that can capture an image of an area including an occupant seated in at least one of the driver's seat and the passenger seat (hereinafter referred to as the "front seat") of the vehicle 1000. The imaging angle of view is measured in degrees. The imaging device 2 outputs the captured image to the position detection device 1.

[0013] The radio wave sensor 3 is configured to be able to measure, for example, the distance and angle to a moving object, including an occupant. The radio wave sensor 3 senses the interior of the vehicle. Specifically, the radio wave sensor 3 emits radio waves toward the vehicle interior and acquires information (hereinafter referred to as "sensor information") regarding the distance and angle to a moving object, including an occupant, present in the vehicle interior (hereinafter referred to as "reflecting object") based on the reflected waves of the emitted radio waves reflected by the reflecting object. Based on the sensor information, the radio wave sensor 3 generates information regarding a plurality of reflection points (hereinafter referred to as "reflection point cloud information"), including three-dimensional position information of the plurality of reflection points of the radio waves reflected by the reflecting object. Note that the radio wave sensor 3 may acquire the information regarding the plurality of reflection points using a known signal processing technique. In the following first embodiment, the plurality of reflection points are also referred to as a "reflection point cloud." Note that in the first embodiment, the radio wave sensor 3 is assumed to be a general radio wave sensor. In general, the radio wave sensor 3 can acquire, for example, the distance to the moving object, the azimuth angle, the elevation angle, etc. If necessary, the radio wave sensor 3 may convert the distance, azimuth angle, elevation angle, etc. of the polar coordinate system into a Cartesian coordinate system to obtain the X (left-right position), Y (up-down position), and Z (front-rear position) Cartesian coordinates. Through these processes, the radio wave sensor 3 obtains a reflection point cloud of reflection points on a reflecting object as seen from the radio wave sensor 3. The radio wave sensor 3 can obtain information on the three-dimensional position of points where the occupant's body is moving, such as near the occupant's head, torso, or feet. The radio wave sensor 3 outputs reflection point cloud information generated based on the sensor information to the position detection device 1. Details of the reflection point cloud information will be described later.

[0014] FIG. 2 is a diagram showing an example of the installation position and orientation of the imaging device 2 mounted on the vehicle 1000 and an example of the installation position and orientation of the radio wave sensor 3 mounted on the vehicle 1000 in the first embodiment. Note that FIG. 2 illustrates the vehicle 1000 as viewed from the left side relative to the traveling direction of the vehicle 1000, with the front seats indicated as 501 and the rear seats indicated as 502. As shown in FIG. 2 , the imaging device 2 is installed, for example, on the dashboard in an orientation that allows it to capture an image of a real space where a passenger seated in the front seat is assumed to be present, i.e., a real space (target area) where target feature points are assumed to be present. In the first embodiment, the position of the imaging device 2 is represented by the point where the lens surface intersects with the imaging axis. In the first embodiment, the imaging axis refers to the optical axis of the imaging unit, i.e., the optical system of the imaging device 2. In FIG. 2 , the position of the imaging device 2 is shown as 2a for convenience.

[0015] The radio wave sensor 3 is installed, for example, on an overhead console in an orientation that allows it to sense the real space where a passenger seated in the front seat is assumed to be present, i.e., the real space (target area) where the target feature points are assumed to be present. In the first embodiment, the position of the radio wave sensor 3 is represented by the center of the radio wave sensor 3, which is determined in relation to the sensing range of the radio wave sensor 3. In Fig. 2, the position of the radio wave sensor 3 is denoted by 3a for convenience.

[0016] The imaging device 2 and the radio wave sensor 3 are installed such that the imaging range of the imaging device 2 (indicated by F101 in FIG. 2 ) and the sensing range of the radio wave sensor 3 (indicated by F102 in FIG. 2 ) overlap. In FIG. 2 , the overlapping area between the imaging range of the imaging device 2 and the sensing range of the radio wave sensor 3 is indicated by F103. In the first embodiment, the coordinate system in the three-dimensional space of the imaging device 2 and the coordinate system in the three-dimensional space of the radio wave sensor 3 are expressed by the same coordinate system. That is, the X-axis, Y-axis, and Z-axis in the coordinate system of the imaging device 2 are the same as the X-axis, Y-axis, and Z-axis in the coordinate system of the radio wave sensor 3. In the first embodiment, in the coordinate system in the three-dimensional space common to the imaging device 2 and the radio wave sensor 3, the coordinates of the position of the imaging device 2 are (0,0,0).

[0017] In the following first embodiment, as an example, the imaging device 2 and the radio wave sensor 3 are respectively installed at positions as shown in FIG. 3 . Similar to FIG. 2 , FIG. 3 illustrates the vehicle 1000 as seen from the left side relative to the traveling direction of the vehicle 1000, with the front seats indicated as 501 and the rear seats indicated as 502. Also, in FIG. 3 , an occupant is indicated as 201. For convenience, the imaging device 2 is located at the position indicated as 2a in FIG. 3 , and the radio wave sensor 3 is located at the position indicated as 3a in FIG. 3 . In FIG. 3 , the radio wave sensor 3 is installed on the overhead console, and the imaging device 2 is installed on the dashboard. The imaging device 2 and the radio wave sensor 3 have the same installation axes in the left-right, up-down, and front-to-back directions, and the same installation coordinates in the left-right and front-to-back directions. The imaging device 2 and the radio wave sensor 3 have different installation coordinates in the up-down direction. Specifically, the radio wave sensor 3 is installed α [m] above the imaging device 2.

[0018] The position detection device 1 identifies the three-dimensional position of the target feature point based on the captured image captured by the imaging device 2 and the reflection point cloud information generated by the radio wave sensor 3, and detects the identified three-dimensional position as the three-dimensional position of the target feature point. Upon detecting the three-dimensional position of the target feature point, the position detection device 1 outputs information relating to the three-dimensional position of the target feature point (hereinafter referred to as "position information") to the physique detection device 4. A detailed configuration example of the position detection device 1 will be described later.

[0019] The physique detection device 4 is mounted on the vehicle 1000 and determines the physique of the occupant based on the position information output from the position detection device 1. A detailed configuration example of the physique detection device 4 will be described later.

[0020] Information regarding the physique of the occupant determined by the physique detection device 4 (hereinafter referred to as "physique information") is used in various applications.

[0021] For example, the physique information is output to a collision safety device (not shown) mounted on the vehicle 1000. The collision safety device controls an airbag (not shown) or a seatbelt pretensioner (not shown) mounted on the vehicle 1000 in accordance with the occupant's physique determined by the physique detection device 4. The airbag, seatbelt pretensioner, etc. need to be controlled taking into account the occupant's physique. For example, the airbag needs to be controlled based on the distance between the occupant and the airbag corresponding to the occupant's physique when the airbag is activated so that the explosive force of the airbag does not injure the occupant. Furthermore, for example, the seatbelt pretensioner needs to be controlled taking into account the neck position corresponding to the occupant's physique when the seatbelt pretensioner is activated so that the seatbelt does not strangle the occupant. Therefore, the physique detection device 4 is required to appropriately determine the occupant's physique so that the collision safety device appropriately activates the control function of the airbag, seatbelt pretensioner, etc. That is, the position detection device 1 is required to accurately detect the three-dimensional positions of the body parts of the occupant, which are the basis for determining the physique of the occupant.

[0022] Furthermore, for example, the physique information may be output to an abandonment detection device (not shown) mounted on the vehicle 1000. The abandonment detection device outputs an alarm when it detects that an infant has been abandoned in the vehicle cabin based on the occupant's physique determined by the physique detection device 4. An infant must be detected as abandoned when only an infant is present in the vehicle cabin. For example, even if an infant is present in the vehicle cabin, if there is also an adult present, it cannot be said that an infant has been abandoned. The abandonment detection device must consider not only the presence or absence of a person in the vehicle cabin, but also the physique of the person present in the vehicle cabin. Therefore, in order for the abandonment detection device to properly detect an infant being abandoned, the physique detection device 4 is required to properly determine the physique of the occupant. In other words, the position detection device 1 is required to accurately detect the three-dimensional positions of the occupant's body parts, which are the basis for determining the occupant's physique.

[0023] The position detection device 1 according to the first embodiment makes it possible to detect the three-dimensional positions of the body parts of an occupant, i.e., the three-dimensional positions of target feature points corresponding to the body parts of the occupant, with high accuracy.

[0024] A description will be given of a position detection device 1 according to embodiment 1. As shown in Fig. 1, the position detection device 1 includes a feature point detection unit 11, a search area determination unit 12, a point cloud acquisition unit 13, and a position identification unit 14.

[0025] The feature point detection unit 11 acquires a captured image from the imaging device 2 and detects target feature points in the captured image based on the captured image captured by the imaging device 2. Here, the feature point detection unit 11 detects a target feature point in the captured image that corresponds to the occupant's head. In the first embodiment, the target feature point corresponding to the occupant's head is the center point of the occupant's eyes. Note that in the first embodiment, the center of the eyes does not necessarily have to be the exact center, but also includes "approximately the center." Here, the target feature point corresponding to the occupant's head is the center point of the occupant's eyes, but this is merely an example. For example, the target feature point corresponding to the occupant's head may be a point corresponding to the nose, etc., as long as a point that is assumed to be a representative point indicating the head is the target feature point corresponding to the occupant's head. The position of the occupant's head is used, for example, to estimate the occupant's sitting height. More specifically, the position of the occupant's head is used, for example, to determine the occupant's physique based on the occupant's sitting height. The position of the occupant's head is information useful for determining the occupant's physique. The feature point detection unit 11 detects one or more target feature points in the captured image. The feature point detection unit 11 can detect multiple target feature points in the captured image. The feature point detection unit 11 may detect target feature points in the captured image using a well-known technique for detecting human feature points from a captured image, and therefore a detailed description of how the feature point detection unit 11 detects target feature points will be omitted. For the target feature point corresponding to the occupant's head, the feature point detection unit 11 may, for example, detect both eyes in the captured image using a well-known method, and detect the center points of the detected eyes on the captured image as the target feature point corresponding to the occupant's head. The target feature point in the captured image is represented by coordinates on the captured image. The target feature point corresponding to the occupant's head in the captured image is derived from the average value of the coordinates indicating the positions of both eyes on the captured image. The feature point detection unit 11 outputs information relating to the detected target feature points (hereinafter referred to as “target feature point information”) to the search area determination unit 12 .

[0026] The search area determination unit 12 determines an area in real space (hereinafter referred to as a "three-dimensional area") in which the target feature points are estimated to exist, based on the positions in the captured image of the target feature points detected by the feature point detection unit 11. Note that the search area determination unit 12 determines a three-dimensional area for each target feature point detected by the feature point detection unit 11.

[0027] Here, an example of a method for determining a three-dimensional area by the search area determination unit 12 will be described. Here, as an example, a three-dimensional area in which a target feature point corresponding to a head detected by the feature point detection unit 11 is estimated to exist will be determined.

[0028] The search area determination unit 12 first calculates an equation representing a line indicating the direction from the imaging device 2 toward the three-dimensional position of the target feature point (hereinafter referred to as a "direction estimation line") based on the position of the target feature point on the captured image detected by the feature point detection unit 11. The search area determination unit 12 can calculate the equation representing the direction estimation line by calculating a "feature point position angle" indicating the angle of the direction from the imaging device 2 toward the three-dimensional position of the target feature point with respect to the direction of the imaging axis of the imaging device 2 based on the position of the target feature point on the captured image detected by the feature point detection unit 11. The feature point position angle calculated by the search area determination unit 12 will be described with reference to the drawings.

[0029] 4A and 4B are diagrams illustrating an example of the feature point position angles θcx and θcy calculated by the search area determination unit 12 and an example of a method for calculating the feature point position angles θcx and θcy by the search area determination unit 12 in embodiment 1. In Fig. 4, an occupant is indicated by 201, and the position of the occupant's head on the captured image, i.e., the position on the captured image of the target feature point corresponding to the occupant's head, is indicated by 202. Figs. 4A and 4B are diagrams illustrating an example of the imaging angle of view θx and feature point position angle θcx of the imaging device 2 relative to the horizontal direction, Fig. 4C is a diagram illustrating an example of the imaging angle of view θy and feature point position angle θcy of the imaging device 2 relative to the vertical direction, and Fig. 4D is a diagram illustrating a captured image captured by the imaging device 2.

[0030] As described above, the imaging device 2 has imaging angles of view θx and θy. Here, as shown in FIGS. 4A and 4B, θx is the imaging angle of view with respect to the horizontal direction (the direction along the imaginary plane defined by the x-axis and z-axis in FIG. 4). As shown in FIG. 4A, θcx is the feature point position angle with respect to the horizontal direction. While FIG. 4A is shown for ease of understanding θcx, here, for example, as shown in FIG. 4B, the feature point position angle θcx = 0 [degrees]. In other words, it is assumed that the horizontal direction of the line connecting the imaging device 2 and the target feature point coincides with the imaging axis (indicated by IA in FIG. 4) of the imaging device 2. As shown in FIG. 4C, θy is the imaging angle of view with respect to the vertical direction (the direction along the imaginary plane defined by the y-axis and z-axis in FIG. 4). As shown in FIG. 4D, θcy is the feature point position angle with respect to the vertical direction. Note that the units of the feature point position angles θcx and θcy are degrees.

[0031] When a target feature point is located at a position where the feature point position angles θcx and θcy are angles as shown in Fig. 4B and Fig. 4C, the captured image of the target feature point will look like the captured image indicated by 21 in Fig. 4D. The feature point detection unit 11 detects the target feature point on the captured image indicated by 21 in Fig. 4D. Here, the target feature point detected by the feature point detection unit 11 is represented by, for example, coordinates (Xc, Yc).

[0032] The search area determination unit 12 calculates feature point position angles θcx and θcy relative to the imaging axis of the imaging device 2 based on the position of the target feature point on the captured image, represented by coordinates (Xc, Yc) detected by the feature point detection unit 11. When calculating the feature point position angles θcx and θcy, the search area determination unit 12 assumes that the coordinates of a position in the captured image corresponding to the imaging axis of the imaging device 2, i.e., the position coordinates of the center of the captured image, are set to (0, 0), as shown in FIG. 4D . Specifically, the search area determination unit 12 pre-stores the values ​​of the imaging angle of view θx and θy, the value of the resolution in the first axis direction (x-axis direction in FIG. 4 ) in the captured image (indicated by m in FIG. 4D ), and the value of the resolution in the second axis direction (y-axis direction in FIG. 4 ) in the captured image (indicated by n in FIG. 4D ). The unit of resolution is pixels (px).

[0033] The search area determination unit 12 calculates the feature point position angles θcx and θcy using the following equations (1) and (2) based on the coordinates (Xc, Yc) of the position of the target feature point detected by the feature point detection unit 11. Xc: tan θcx = (m / 2): tan(θx / 2) ... (1) Yc: tan θcy = (n / 2): tan(θy / 2) ... (2)

[0034] It can be said that the three-dimensional position coordinates of the target feature point lie on the straight line of the following equation (3).

[0035] Here, the coordinates on the captured image are (Xc, Yc)=(0, Yc), and the feature point position angle θcx=0 (degrees).

[0036] The search area determination unit 12 calculates the feature point position angles in this way, and calculates an equation representing the orientation estimation line, such as the above equation (3), which is found from the calculated feature point position angles.

[0037] Note that, taking into consideration that the positions of the target feature points on the captured image or the feature point position angles fluctuate due to temporal fluctuations of the target feature points, the search area determination unit 12 may suppress fluctuations by time-series filtering, etc. For example, the search area determination unit 12 may calculate the feature point position angle and the orientation estimation line by averaging the positions of the target feature points in the captured image detected by the feature point detection unit 11 with those of target feature points over a predetermined period of time in the past, or may average the calculated feature point position angle with those calculated over a predetermined period of time in the past.

[0038] After calculating the feature point position angle and the orientation estimation line, the search area determination unit 12 next determines a region with a margin based on the angular direction of the target feature point, in other words, the orientation estimation line, as a three-dimensional region. More specifically, the search area determination unit 12 determines the three-dimensional region so that it satisfies at least one of the following characteristics: <Feature (1)> to <Feature (4)>. <Feature (1)> The three-dimensional region is narrower the closer the distance from the image capture device 2 and wider the farther the distance from the image capture device 2, based on the characteristic of the image capture device 2 that the field of view becomes wider the farther from the image capture device 2. <Feature (2)> The three-dimensional region is a region with a margin that takes into account the detection accuracy or stability of the target feature point detected from the captured image. <Feature (3)> The three-dimensional region is a region with a margin that takes into account the sensing accuracy or stability of the reflection point cloud by the radio wave sensor 3. <Feature (4)> The three-dimensional area is an area with a margin that takes into consideration disturbance factors such as the movement of the occupant.

[0039] When the search area determination unit 12 determines a three-dimensional area based on the orientation estimation line so as to have the above-described characteristics, the three-dimensional area is determined as shown in FIG. 5 . FIG. 5 is a diagram illustrating an example of a three-dimensional area determined by the search area determination unit 12 in the first embodiment based on the positions of target feature points detected by the feature point detection unit 11 in the captured image. Note that, again, as an example, the search area determination unit 12 determines a three-dimensional area in which a target feature point corresponding to a head detected by the feature point detection unit 11 is estimated to exist. However, regarding the positional relationship between the image capture device 2 and the target feature point, in the description of the example of the method for determining a three-dimensional area by the search area determination unit 12 using FIG. 4 , the feature point position angle θcx = 0 [degrees] was used. However, here, for ease of understanding of the three-dimensional area, the positional relationship between the image capture device 2 and the target feature point is assumed to be a feature point position angle θcx = 0 [degrees]. In other words, it is assumed that the horizontal directions of the orientation estimation line (indicated by "SH" in Fig. 5) and the imaging axis (indicated by IA in Fig. 5) of the imaging device 2 do not match. In Fig. 5, for example, it is assumed that the target feature point exists to the left of the optical axis of the imaging device 2 as viewed from the imaging device 2, as shown in Fig. 4A.

[0040] FIG. 5A shows a captured image (indicated by "21a" in FIG. 5A) in which the target feature point is captured. FIGS. 5B, 5C, and 5D are diagrams showing three-dimensional areas (indicated by "R" in FIGS. 5B, 5C, and 5D) determined by the search area determination unit 12. FIG. 5B shows the three-dimensional area when viewed from above the interior of the vehicle. FIG. 5C shows the three-dimensional area when viewed from the left side of the vehicle 1000, to the right of the line passing through the point between the driver's seat and the passenger seat in the vehicle width direction. FIG. 5D shows the three-dimensional area when viewed from near the dashboard. As in the installation positions shown in FIG. 3, the image capture device 2 and the radio wave sensor 3 are installed so that the installation axes in the left-right, up-down, and front-to-back directions are equal to the installation coordinates in the left-right and front-to-back directions. In FIGS. 5B, 5C, and 5D, 2a indicates the position of the image capture device 2. 5B, 5C, and 5D show simplified front seats 501 and rear seats 502. For convenience, Figures 5B, 5C, and 5D show radio wave sensors 3 and black circles representing reflection point clouds obtained by the radio wave sensors 3, but the search area determination unit 12 does not use the reflection point cloud information output from the radio wave sensors 3 to determine the three-dimensional area.

[0041] The search area determination unit 12 determines three-dimensional areas such as those shown in Figures 5B, 5C, and 5D based on the positions of target feature points in the captured image as shown in Figure 5A, for example. The three-dimensional areas shown in Figures 5B, 5C, and 5D are three-dimensional areas determined by the search area determination unit 12 so as to have the above-mentioned features (1) to (4). In Figure 5A, the occupant is indicated by 201, and the position of the occupant's head on the captured image, i.e., the position of the target feature point corresponding to the occupant's head on the captured image, is indicated by 202.

[0042] Here, the margin is set with the position of the image capture device 2 as a fulcrum and in a direction away from the orientation estimation line by a predetermined angle. That is, the search area determination unit 12 determines the three-dimensional area as a cone-shaped area, for example, with the position of the image capture device 2 as its vertex and the orientation estimation line calculated based on the positions of the target feature points in the captured image detected by the feature point detection unit 11 as a perpendicular line extending from the vertex. Note that this is merely an example. For example, the search area determination unit 12 may determine the three-dimensional area as a polygonal pyramid-shaped area, with the position of the image capture device 2 as its vertex and the orientation estimation line calculated based on the positions of the target feature points in the captured image detected by the feature point detection unit 11 as a perpendicular line extending from the vertex. Alternatively, the three-dimensional area may be expressed by combining multiple solids to have a shape that satisfies at least one of <Feature 1> to <Feature 4>. The margin may also be changed as appropriate depending on the state of the sensing or occupant. For example, the margin may be increased when the occupant's movement is large, or when the sensing error or distribution range is large, for example, when the distribution range of the reflection point group within a specified time is large.

[0043] The search area determination unit 12 may determine, as a three-dimensional area, an area with a margin based on the orientation estimation line. Note that the above-mentioned <Feature (1)> to <Feature (4)> are merely examples, and the search area determination unit 12 may determine a three-dimensional area so as to have other features. For example, the search area determination unit 12 may determine a three-dimensional area so as to have the following <Feature (5)> in addition to the above-mentioned <Feature (1)> to <Feature (4)>. <Feature (5)> The three-dimensional area has a size corresponding to the size of the part of the human body indicated by the target feature point.

[0044] The search area determination unit 12 may change the size of the margin depending on the size of the part of the occupant's body indicated by the target feature point so as to have the above-mentioned <Feature (5)>. For example, the search area determination unit 12 determines the three-dimensional area of ​​the target feature point corresponding to the head so that the three-dimensional area of ​​the target feature point corresponding to the head is larger than the three-dimensional area of ​​the target feature point corresponding to the right shoulder.

[0045] Furthermore, the search area determination unit 12 may limit the size of the three-dimensional area in the forward direction or the size of the three-dimensional area in the rearward direction, for example, so that the size corresponds to the size of the occupant's body part indicated by the target feature point (here, the head), so as to have the above-mentioned <Feature (5)>. The forward and rearward directions here refer to the forward and rearward directions relative to the traveling direction of the vehicle 1000. Figures 5B , 5C , and 5D illustrate an example in which the search area determination unit 12 limits the rearward size of the three-dimensional area taking into account the size of the head. For example, a maximum vertical distance (referred to as the occipital distance) from the installation position of the image capture device 2 to a plane along the rear surface of the head of the occupant in the driver's seat or passenger seat is pre-set and stored in the search area determination unit 12. The search area determination unit 12 pre-sets the rearward size of the three-dimensional area so that the area is up to the pre-set occipital distance.

[0046] Once the three-dimensional region is determined as described above, the search region determination unit 12 outputs information indicating the determined three-dimensional region (hereinafter referred to as "three-dimensional region information") to the position identification unit 14. The three-dimensional region information includes, for example, the coordinates of the vertices, i.e., the installation position of the image capture device 2, information on an equation indicating the orientation estimation line, information indicating a margin based on the orientation estimation line (for example, an angle extending from the position of the image capture device 2 to the orientation estimation line in a direction away from the orientation estimation line, with the position of the image capture device 2 as the fulcrum), and information indicating the vertical distance from the position of the image capture device 2 to the forward or backward end point of the three-dimensional region. Note that this is merely an example, and the three-dimensional region information may be information that allows the range of the three-dimensional region to be identified. Note that the search region determination unit 12 generates three-dimensional region information for each target feature point and outputs it to the position identification unit 14. At this time, the search area determination unit 12 associates the three-dimensional area information with information indicating the positions of the target feature points in the captured image detected by the feature point detection unit 11 and outputs the information to the position identification unit 14 .

[0047] The point cloud acquisition unit 13 acquires the reflection point cloud information from the radio wave sensor 3. The point cloud acquisition unit 13 outputs the acquired reflection point cloud information to the position identification unit 14.

[0048] The position identification unit 14 identifies the three-dimensional position of the target feature point from the three-dimensional position information of the reflection points detected within the three-dimensional area among the multiple reflection points, based on the three-dimensional area determined by the search area determination unit 12 and the reflection point cloud information acquired by the point cloud acquisition unit 13, and detects the identified three-dimensional position as the three-dimensional position of the target feature point. The position identification unit 14 detects the three-dimensional position of the target feature point using coordinates expressed in the same coordinate system as the coordinate system in the three-dimensional space of the image capture device 2 and the coordinate system in the three-dimensional space of the radio wave sensor 3. Note that the position identification unit 14 can identify the three-dimensional area within the vehicle cabin based on the three-dimensional area output from the search area determination unit 12. Furthermore, the position identification unit 14 can determine the three-dimensional position of each reflection point acquired by the radio wave sensor 3 based on the reflection point cloud information output from the point cloud acquisition unit 13.

[0049] In the first embodiment, it is assumed that the positional relationship between the imaging device 2 and the radio wave sensor 3 is known in advance. If the position identification unit 14 can identify a three-dimensional area, it can determine which of the reflection points has been detected within the three-dimensional area. The feature point detection unit 11 detects a target feature point in the captured image, thereby identifying the position of the target feature point within a two-dimensional plane, but this position information does not include information on the position in the depth direction. The position identification unit 14 supplements the position in the depth direction with the reflection point group information acquired from the radio wave sensor 3, identifies the three-dimensional position including the position of the target feature point in the depth direction, and detects the three-dimensional position of the target feature point.

[0050] Here, several examples will be given to explain specific methods by which the position identifying unit 14 detects the three-dimensional positions of target feature points, i.e., the three-dimensional positions of body parts of an occupant. Note that the methods described below are merely examples, and the position identifying unit 14 may detect the three-dimensional positions of target feature points using other methods. It is sufficient that the position identifying unit 14 uses reflection point group information to supplement the depth direction position of the target feature points detected as two-dimensional coordinates on the captured image by the feature point detection unit 11, thereby identifying the three-dimensional positions of the target feature points.

[0051] <Example of Identification Method (1)> For example, the position identification unit 14 determines a representative point within the three-dimensional region by statistically processing the three-dimensional positions of reflection points detected within the three-dimensional region, and identifies the three-dimensional position of the determined representative point as the three-dimensional position of the target feature point, thereby detecting the three-dimensional position of the target feature point. For example, the position identification unit 14 designates a point located at the average or median of the three-dimensional coordinates of the reflection points detected within the three-dimensional region as the representative point. For example, designating a point located at the average of the three-dimensional coordinates of the reflection points detected within the three-dimensional region as the representative point is effective when the detected reflection points appear within a certain range in a predetermined distribution, such as a normal distribution. On the other hand, if there is a bias, such as a large outlier, the average value will be influenced by the outlier and may increase or decrease. In such cases, it is effective to designate a point located at the median of the three-dimensional coordinates of the reflection points detected within the three-dimensional region as the representative point. The position identification unit 14 appropriately selects the type of statistical processing to be performed to determine the representative point depending on the distribution of the reflection points obtained. By determining a representative point within the three-dimensional region by performing statistical processing such as calculating the average or median of the coordinates of the three-dimensional positions of the reflection points detected within the three-dimensional region, the position identification unit 14 is expected to be able to stably detect the three-dimensional positions of the target feature points, even when some of the reflection points are temporarily undetected or erroneously detected, compared to the <Identification Method Example (2)> described below. Here, the representative point within the three-dimensional region does not necessarily have to be the average or median, and the position identification unit 14 may perform appropriate statistical processing depending on the application. For example, when estimating the three-dimensional position of a target feature point on the surface of the face, the position identification unit 14 may limit the three-dimensional positions of the reflection points detected within the three-dimensional region to those closest to the front of the human body and take the average, or estimate the reflection point detected furthest forward as the three-dimensional position of the target feature point on the surface of the face. In addition, the position identification unit 14 may use only one of the coordinates (x, y, or z) of the three-dimensional positions of the reflection points detected within the three-dimensional area as the coordinate to be used as the average value or the median value.For example, in a two-dimensional captured image, the horizontal and vertical directions can be estimated with high accuracy, but depth information is not available. Therefore, the position identification unit 14 may estimate the depth of the reflection point by calculating only the depth coordinate, i.e., only the coordinate of the dimension corresponding to the depth direction of the image capture device 2, as the depth distance by calculating the average or median of the coordinates of the reflection points detected in the three-dimensional region using the reflection point cloud information acquired from the radio wave sensor 3, and may use the intersection of a plane indicating the estimated depth distance and the orientation estimation line as a representative point in the three-dimensional region. As a result, even if there is no reflection point on the orientation estimation line of the occupant, the position identification unit 14 can accurately estimate the three-dimensional position of the target feature point by estimating the depth position of the target feature point from surrounding reflection points and further estimating the three-dimensional position of the target feature point based on the intersection of the plane indicating the depth distance and the orientation estimation line. The three-dimensional region information acquired by the position identification unit 14 from the search region determination unit 12 includes, for example, information on an equation indicating the orientation estimation line. The position identification unit 14 may also estimate a surface or line in three-dimensional space by fitting a reflection point cloud detected within a three-dimensional region, and use the intersection of the surface or line and the orientation estimation line as a representative point to estimate the three-dimensional position of the target feature point. For example, a method may be used in which the most likely line segment is derived from the reflection point cloud detected within a three-dimensional region using least-squares approximation or the like. Because the occupant's body is a continuous structure, the reflection point cloud based on the reflection point cloud information obtained from the radio wave sensor 3 should also be obtained so as to follow the occupant's body. However, some of the reflection points may not be detected (i.e., not detected) due to small movement or the influence of surrounding structures. Furthermore, unnecessary detections may also be obtained (i.e., false detections). By estimating a surface or line based on the shape of the occupant's body based on multiple reflection points, the position identification unit 14 can estimate the general shape of the occupant's body even if the radio wave sensor 3 fails to detect or falsely detects parts of the occupant's body. The position specifying unit 14 estimates the three-dimensional position of the target feature point using the intersection of this with the orientation estimation line as a representative point, thereby making it possible to accurately estimate the three-dimensional position of the target feature point. When specifying the three-dimensional position, a statistical process may be performed, followed by a process of offsetting the position.The reason for this is that when the radio wave sensor 3 is installed in front of the occupant, the reflection point of the radio wave sensor 3 is likely to be detected near the front face of the occupant, and when the radio wave sensor 3 is installed behind the occupant, the reflection point of the radio wave sensor 3 is likely to be detected near the rear face of the occupant. For example, when the radio wave sensor 3 is installed in a position that detects the rear side of the occupant and the radio wave sensor 3 detects a reflection point near the back of the occupant's head, if it is desired to estimate the coordinates of the front of the occupant's face, it is preferable to output the three-dimensional position after offsetting it forward by the size of the occupant's face. The offset amount may be set in advance taking into account the installation position of the radio wave sensor 3 or the position of the detection point of the radio wave sensor 3, or it may be adjusted based on the result of a physique determination (described later) detected up to the previous processing cycle as a real-time sensing result, for example, depending on whether the occupant is large or small.

[0052] <Example of Identification Method (2)> For example, the position identification unit 14 determines, among the reflection points detected within the three-dimensional region, the reflection point that is closest to the orientation estimation line as a representative point, and identifies the three-dimensional position of the determined representative point as the three-dimensional position of the target feature point, thereby detecting the three-dimensional position of the target feature point. Note that the three-dimensional region information that the position identification unit 14 acquires from the search region determination unit 12 includes, for example, information on an equation that indicates the orientation estimation line. By the position identification unit 14 determining, as the representative point, the reflection point that is closest to the orientation estimation line among the reflection points detected within the three-dimensional region, the position identification unit 14 can identify the three-dimensional position of the target feature point with a smaller amount of calculation.

[0053] <Example Identification Method (3)> For example, the position identification unit 14 acquires a solid that covers the group of reflection points present in the three-dimensional space based on the three-dimensional positions of the reflection points detected in the three-dimensional space. The position identification unit 14 then determines a point on the solid that is closest to the orientation estimation line as a representative point, and identifies the three-dimensional position of the determined representative point as the three-dimensional position of the target feature point, thereby detecting the three-dimensional position of the target feature point. An example of a solid that covers the group of reflection points is a convex hull. A convex hull is the smallest convex set that contains a given set. The position identification unit 14 may derive the convex hull using a known method. For example, the position identification unit 14 determines a representative point from the group of reflection points near the three-dimensional space based on the principle of the convex hull, and identifies the three-dimensional position of the determined representative point as the three-dimensional position of the target feature point. In particular, the position identification unit 14 acquires a convex hull present in the three-dimensional space based on the three-dimensional positions of the reflection points detected in the three-dimensional space, and identifies the point on the convex hull that is closest to the orientation estimation line as the representative point.

[0054] For example, if a sufficient number of reflection points are obtained within the three-dimensional region of the target feature point corresponding to the occupant's head, the position identification unit 14 can identify the three-dimensional position of the target feature point based on the principle of convex hulls. This allows the position closest to the orientation estimation line within the three-dimensional structure estimated to be the occupant's head represented by the reflection point cloud to be determined as the three-dimensional position of the target feature point. Even if there is no reflection point on the orientation estimation line where radio waves from the radio wave sensor 3 are reflected by a reflecting object, highly accurate three-dimensional position identification can be expected. Note that, although a convex hull is used here as a solid that covers the reflection point group, the solid that covers the reflection point group does not necessarily have to be a convex hull. It can also have any other shape that covers the reflection point group within the three-dimensional region and is suitable for identifying the three-dimensional position of the target feature point. An example of a shape that is suitable for identifying the three-dimensional position of the target feature point is a shape similar to the shape of the part of the occupant's body that the target feature point corresponds to.

[0055] <Example Identification Method (4)> For example, the position identification unit 14 uses a pre-generated three-dimensional pattern to determine, as a representative point, a point on the three-dimensional pattern that corresponds to a target feature point that is most highly correlated with the reflection point group in the three-dimensional region, and identifies the three-dimensional position of the determined representative point as the three-dimensional position of the target feature point, thereby detecting the three-dimensional position of the target feature point. The three-dimensional pattern is, for example, a virtual three-dimensional model of a standard head or human body. A three-dimensional model of a standard head or human body is generated in advance and stored in a location accessible to the position identification unit 14. For example, in the case of a head, a three-dimensional model of an adult head having a standard size or shape of an adult's head and a three-dimensional model of a child's head having a standard size or shape of a child's head are generated in advance based on statistical data, etc., and stored in a location accessible to the position identification unit 14. For example, in the case of a human body, a three-dimensional model of an adult human body and a three-dimensional model of a child's human body are similarly generated in advance based on statistical data, etc., and stored in a location accessible to the position identification unit 14. In detail, the position identification unit 14 places a virtual three-dimensional model that resembles a person (more specifically, for example, a head or a human body) within the three-dimensional area, changes the position and posture of the three-dimensional model so as to reduce the error between the shape of the reflection point group within the three-dimensional area and the shape of the three-dimensional model, and sets the point on the three-dimensional model that corresponds to the target feature point after the position and posture have been changed as the representative point.

[0056] The position identification unit 14 may determine the point with the highest correlation between the reflection point group in the three-dimensional region and the three-dimensional model, taking into account, for example, which surface of the occupant's body is irradiated with radio waves from the radio wave sensor 3. For example, when radio waves from the radio wave sensor 3 are irradiated to the front of the occupant's body, the position identification unit 14 changes the position and orientation of the three-dimensional model so as to reduce the error between the shape of the front portion of the three-dimensional model and the shape of the reflection point group in the three-dimensional region, and sets the point corresponding to the target feature point on the three-dimensional model after the position and orientation change as the representative point. Furthermore, when radio waves from the radio wave sensor 3 are irradiated to the rear of the occupant's body, the position identification unit 14 changes the position and orientation of the three-dimensional model so as to reduce the error between the shape of the rear portion of the three-dimensional model and the shape of the reflection point group in the three-dimensional region, and sets the point corresponding to the target feature point on the three-dimensional model after the position and orientation change as the representative point.

[0057] 1 , it is assumed that one radio wave sensor 3 is provided in the vehicle 1000, but this is merely an example, and a plurality of radio wave sensors 3 may be provided in the vehicle 1000, and the point cloud acquisition unit 13 of the position detection device 1 may acquire reflection point cloud information from the plurality of radio wave sensors 3. In this case, the position identification unit 14 may combine the reflection point cloud information acquired by the point cloud acquisition unit 13 from the plurality of radio wave sensors 3, and determine a representative point in the three-dimensional region for the combined reflection point cloud information using the above-mentioned methods such as <Example Identification Method (1)> to <Example Identification Method (4)>, and identify the three-dimensional position of the determined representative point as the three-dimensional position of the target feature point.

[0058] The position identification unit 14 outputs information indicating the three-dimensional positions of the detected target feature points (hereinafter referred to as "position information") to the physique detection device 4. The position information includes at least coordinates representing the three-dimensional positions of the target feature points. The position information may further include, for example, coordinates representing the positions of the target feature points in the captured image detected by the feature point detection unit 11. The position identification unit 14 generates position information for each target feature point and outputs it to the physique detection device 4.

[0059] A description will be given of a physique detection device 4 according to embodiment 1. As shown in FIG.

[0060] The position information acquisition unit 41 acquires position information from the position detection device 1. The position information acquisition unit 41 outputs the acquired position information to the physique determination unit .

[0061] The physique determination unit 42 determines the physique of the occupant based on the position information acquired by the position information acquisition unit 41. That is, the physique determination unit 42 determines the physique of the occupant based on the three-dimensional positions of the target feature points detected in the image captured by the imaging device 2 in the position detection device 1 using the reflection point cloud information acquired from the radio wave sensor 3, in other words, the three-dimensional positions of the occupant's body parts. The physique determination unit 42 determines the physique of each occupant. For example, areas corresponding to each seat in the vehicle cabin (e.g., the driver's seat, the passenger seat, and the rear seats) are set in advance. The physique determination unit 42 can identify which target feature points correspond to which body parts of which occupant, more specifically, which seats the occupant is sitting in, based on the areas corresponding to each seat and the three-dimensional positions of the target feature points based on the position information.

[0062] In the first embodiment, the physique determined by the physique determination unit 42 is, for example, one of "child," "small male," "small female," "average male," "average female," "large male," or "large female." Note that this is merely an example, and the definition of the physique determined by the physique determination unit 42 can be set as appropriate.

[0063] The physique determination unit 42 may determine the physique of an occupant using known technology for determining the physique of a person from information indicating the position of a person's body parts, but an example of how the physique determination unit 42 determines the physique of an occupant will be explained below.

[0064] The physique determination unit 42, for example, selects target feature points (hereinafter referred to as "physique determination feature points") to be used for determining the physique of the occupant from the target feature points, and determines the physique of the occupant based on the selected physique determination feature points. After selecting the physique determination feature points, the physique determination unit 42 converts the three-dimensional positions of the physique determination feature points detected by the position detection device 1 into positions for physique determination (hereinafter referred to as "physique determination positions"). The physique determination positions are expressed by coordinates in a three-dimensional coordinate system or by coordinates in a two-dimensional coordinate system. When the physique determination positions of the physique determination feature points are expressed by coordinates in a three-dimensional coordinate system, the physique determination positions are expressed by coordinates in the same coordinate system as the coordinate system in the three-dimensional space of the image capture device 2 and the coordinate system in the three-dimensional space of the radio wave sensor 3, for example. When the physique determination positions of the physique determination feature points are expressed by coordinates in a two-dimensional coordinate system, the physique determination positions are expressed by coordinates in the same coordinate system as the coordinate system in the captured image, for example.

[0065] For example, the physique determination unit 42 selects target feature points corresponding to the occupant's right and left shoulders as physique determination feature points, and converts the physique determination positions of the physique determination feature points (the occupant's right and left shoulders) into positions represented by coordinates in a three-dimensional coordinate system. It is also assumed that the position detection device 1 detects the three-dimensional position (x_right shoulder, y_right shoulder, z_right shoulder) of the target feature point corresponding to the occupant's right shoulder and the three-dimensional position (x_left shoulder, y_left shoulder, z_left shoulder) of the target feature point corresponding to the occupant's left shoulder. In this case, the physique determination unit 42 uses, for example, the three-dimensional positions of the target feature points corresponding to the occupant's right and left shoulders, detected by the position detection device 1, as the physique determination positions of the physique determination feature points corresponding to the occupant's right and left shoulders, respectively. That is, the physique determination position of the physique determination feature point corresponding to the occupant's right shoulder is (x_right shoulder, y_right shoulder, z_right shoulder), and the physique determination position corresponding to the occupant's left shoulder is (x_left shoulder, y_left shoulder, z_left shoulder).

[0066] For example, the physique determination unit 42 selects target feature points corresponding to the occupant's right shoulder and left shoulder as physique determination feature points, and converts the physique determination positions of the physique determination feature points (the occupant's right shoulder and left shoulder) into positions represented by coordinates in a two-dimensional coordinate system. Note that the position detection device 1 detects the three-dimensional position (x_right shoulder, y_right shoulder, z_right shoulder) of the target feature point corresponding to the occupant's right shoulder and the three-dimensional position (x_left shoulder, y_left shoulder, z_left shoulder) of the target feature point corresponding to the occupant's left shoulder. In this case, the physique determination unit 42 may convert, for example, the three-dimensional positions of the target feature points corresponding to the occupant's right shoulder and left shoulder detected by the position detection device 1 into coordinates in the two-dimensional coordinate system, and use these as the physique determination positions of the physique determination feature points. For example, if the three-dimensional positions of the target feature points corresponding to the occupant's right and left shoulders detected by the position detection device 1 are three-dimensional positions that are estimated to indicate that the occupant is sitting in a forward-leaning position, the physique determination unit 42 estimates the positions of the occupant's right and left shoulders in the captured image, assuming that the occupant is leaning against the back of the seat. A specific example of an estimation method is a method of estimating the positions (points at which coordinates) at which the right and left shoulders are imaged in the captured image when the z coordinates of the three-dimensional positions of the target feature points corresponding to the occupant's right and left shoulders are moved in the z direction in three-dimensional space so that they correspond to the z coordinates of the three-dimensional positions of the target feature points corresponding to the occupant's right and left shoulders if the occupant were leaning against the back of the seat. Note that this is merely an example, and the physique determination unit 42 may estimate the positions of the occupant's right and left shoulders in the captured image using known technology. The physique determination unit 42 determines the two-dimensional positions (x_px_right shoulder, y_px_right shoulder) and left shoulder (x_px_left shoulder, y_px_left shoulder) of the occupant in the estimated captured image as the physique determination positions of the physique determination feature points.

[0067] The physique determination unit 42 can perform physique determination by converting the three-dimensional positions of the physique-determining feature points detected by the position detection device 1 into physique-determining positions, which are two-dimensional positions on the captured image when all occupants are seated in the seats under the same conditions, in this case, the same seating conditions regardless of the occupant. That is, even if the physique determination is based on two-dimensional positions that do not include information on the depth direction position, the physique determination unit 42 can perform physique determination taking the depth direction position into consideration. In any case, as long as the physique determination unit 42 converts the selected physique-determining feature points into physique-determining positions that allow for determination of differences in physique based on the three-dimensional positions detected by the position detection device 1, the physique determination positions may be converted to any positions.

[0068] The physique determination unit 42 selects physique-determining feature points, converts the three-dimensional positions of the physique-determining feature points into physique-determining positions, sets an index capable of expressing differences in physique based on the converted physique-determining positions of the physique-determining feature points, and determines the physique of the occupant from the set index. An example of an index capable of expressing differences in physique is the distance between a plurality of physique-determining feature points (hereinafter referred to as "inter-feature point distance"). For example, the physique determination unit 42 calculates the inter-feature point distance between the physique-determining feature point corresponding to the occupant's right shoulder and the physique-determining feature point corresponding to the occupant's left shoulder, and uses the calculated inter-feature point distance to determine the occupant's physique. The inter-feature point distance between the physique-determining feature point corresponding to the occupant's right shoulder and the physique-determining feature point corresponding to the occupant's left shoulder corresponds to the occupant's shoulder width. This is merely an example. For example, the physique determination unit 42 may select physique determination feature points corresponding to the occupant's elbows (right elbow and left elbow) in addition to the above-mentioned physique determination feature points, convert the three-dimensional positions of the occupant's elbows into physique determination positions, and, in addition to the above-mentioned inter-feature point distances, use, for example, the inter-feature point distance between the physique determination feature point corresponding to the occupant's right shoulder and the physique determination feature point corresponding to the occupant's right elbow, or the inter-feature point distance between the physique determination feature point corresponding to the occupant's left shoulder and the physique determination feature point corresponding to the occupant's left elbow, in determining the occupant's physique. Which target feature points are to be used as physique determination feature points and which inter-feature point distances between physique determination feature points are to be used in determining the occupant's physique can be set as appropriate. Furthermore, the physique determination unit 42 may determine the occupant's physique using a plurality of inter-feature point distances.

[0069] For example, the physique determination unit 42 determines the physique of the occupant by comparing the inter-feature point distance with preset conditions (hereinafter referred to as "physique determination conditions"). For example, the physique determination conditions include the following: (Condition 1) Range of inter-feature point distance for a "small-sized man"; (Condition 2) Range of inter-feature point distance for a "small-sized woman"; (Condition 3) Range of inter-feature point distance for a "small-sized woman"; (Condition 4) Range of inter-feature point distance for a "standard-sized man"; (Condition 5) Range of inter-feature point distance for a "standard-sized woman"; (Condition 6) Range of inter-feature point distance for a "large-sized man"; and (Condition 7) Range of inter-feature point distance for a "large-sized woman". For example, if the calculated inter-feature point distance satisfies (Condition 4), the physique determination unit 42 determines that the occupant's physique is that of a "standard-sized man," and if the calculated inter-feature point distance satisfies (Condition 3), the physique determination unit 42 determines that the occupant's physique is that of a "standard-sized man." Also, for example, when the physique determination unit 42 determines the physique of an occupant using the distances between multiple feature points, if the combination of distances between multiple skeletal coordinate points satisfies the physique determination conditions, it determines that the physique of the occupant is the physique set by the physique determination conditions.

[0070] The physique determination conditions may be determined based on data obtained by actually observing a person with a reference physique using the image capture device 2 or the radio wave sensor 3, or may be determined based on statistical information about the human body, or the specifications of the image capture device 2 or the radio wave sensor 3 (e.g., how a person with a certain physique is observed depending on where they sit). The physique determination unit 42 may also estimate the occupant's physique using a trained model in machine learning (hereinafter referred to as a "machine learning model"). The machine learning model is, for example, a model that receives the distance between feature points as input and outputs information indicating the physique. The machine learning model is generated in advance and stored in a location that can be referenced by the physique determination unit 42. The position detection device 1 according to the first embodiment can detect the positions, including the depth positions, of target feature points detected from the captured image even when the occupant has a posture that forms similar skeletal lines in the captured image. This allows the physique determination unit 42 to accurately classify the physique even when the occupant has a posture that forms similar skeletal lines in the captured image. Furthermore, in the case of using only captured images, it is possible to consider a method of learning various posture changes, including those in the depth direction, in advance. However, in embodiment 1, the radio wave sensor 3 can supplement information in the depth direction, making it possible to reduce the amount of learning data for posture changes in the depth direction compared to learning posture changes, including those in the depth direction, using only captured images.

[0071] In the above example, the index capable of expressing a difference in physique was the distance between feature points. However, this is merely an example, and an index capable of expressing a difference in physique may be, for example, the area surrounding the feature points for physique determination. The index capable of expressing a difference in physique may be any index as long as it is associated with the actual difference in physique from the physique determination positions of the feature points for physique determination. As described above, information useful for determining an occupant's physique includes information on the position of the occupant's head. However, for example, in cases where the height of the seat varies due to the function of a seat lifter, it is effective to determine the occupant's physique using the distance between two points corresponding to the width or height of the body, which varies depending on the occupant's physique, such as the distance between the occupant's shoulders, the distance between the occupant's hips, the distance between the neck and one shoulder, or the distance between the head and hips. For example, the head position is effective when used to determine the occupant's physique when the height of the seat is fixed, i.e., when differences in physique appear in the seat height. The physique determination unit 42 may determine the physique of the occupant using, for example, the position of the head and the positions of both shoulders.

[0072] For example, if the distance from the image capture device 2 to the occupant changes depending on the seat position (front, back, up, down) or the reclining state (angle), the size of the occupant as seen in the captured image changes. Therefore, if the physique detection device 4 attempts to determine the occupant's physique from the two-dimensional positions of target feature points in the captured image detected from the captured image, the accuracy of determining the occupant's physique may be reduced. In contrast, the physique detection device 4 according to embodiment 1 can determine the occupant's physique taking into account the depth position from the image capture device 2 to the occupant based on the three-dimensional positions of target feature points detected by the position detection device 1 according to embodiment 1 taking into account the depth position from the image capture device 2 to the occupant, thereby enabling more accurate determination of the occupant's physique.

[0073] After determining the physique of the occupant, the physique determination unit 42 outputs the physique information to a device such as a collision safety device or an abandoned vehicle detection device.

[0074] The operation of the human body monitoring system 100 according to the first embodiment will now be described. FIG. 6 is a flowchart for explaining the operation of the human body monitoring system 100 according to the first embodiment. Of the processes shown in the flowchart of FIG. 6, steps ST1 to ST4 are performed by the position detection device 1 according to the first embodiment, and steps ST5 to ST6 are performed by the physique detection device 4 according to the first embodiment. For example, when the vehicle 1000 is powered on, the human body monitoring system 100 starts the operation shown in the flowchart of FIG. 6 and repeats the operation shown in the flowchart of FIG. 6 until the vehicle 1000 is powered off. While the timing of powering on or off the vehicle 1000 has been described as an example, for example, in the case of an abandoned vehicle detection device, the device may continue to operate for a certain period of time even after the vehicle 1000 is powered off, based on the power supply from the battery on the vehicle 1000, and the timing of powering on or off the vehicle 1000 is set appropriately by an application.

[0075] The feature point detection unit 11 acquires a captured image from the imaging device 2 and detects target feature points in the captured image based on the captured image captured by the imaging device 2 (step ST1). The feature point detection unit 11 outputs target feature point information to the search area determination unit 12.

[0076] The search area determination unit 12 determines a three-dimensional area (step ST2) based on the positions on the captured image of the target feature points detected by the feature point detection unit 11 in step ST1. The search area determination unit 12 outputs three-dimensional area information to the position identification unit 14. At this time, the search area determination unit 12 associates the three-dimensional area information with information indicating the positions of the target feature points on the captured image detected by the feature point detection unit 11, and outputs the three-dimensional area information to the position identification unit 14.

[0077] The point cloud acquisition unit 13 acquires the reflection point cloud information from the radio wave sensor 3 (step ST3). The point cloud acquisition unit 13 outputs the acquired reflection point cloud information to the position identification unit 14.

[0078] The position specifying unit 14 specifies the three-dimensional position of the target feature point from the three-dimensional position information of the reflection points detected within the three-dimensional area among the multiple reflection points, based on the three-dimensional area determined by the search area determining unit 12 in step ST2 and the reflection point cloud information acquired by the point cloud acquiring unit 13 in step ST3, and detects the specified three-dimensional position as the three-dimensional position of the target feature point (step ST4). The position specifying unit 14 outputs the position information to the physique detection device 4.

[0079] The position information acquisition unit 41 acquires the position information output from the position identification unit 14 of the position detection device 1 in step ST4 (step ST5). The position information acquisition unit 41 outputs the acquired position information to the physique determination unit 42.

[0080] The physique determination unit 42 determines the physique of the occupant based on the position information acquired by the position information acquisition unit 41 in step ST5 (step ST6). The physique determination unit 42 outputs the physique information to a device such as a collision safety device or an abandoned person detection device.

[0081] 6, the processing is performed in the order of steps ST1 to ST3, but this is merely an example. For example, the order of steps ST1 to ST2 and step ST3 may be reversed, or steps ST1 to ST2 and step ST3 may be performed in parallel.

[0082] In this way, the position detection device 1 detects target feature points in the captured image based on the captured image captured by the imaging device 2, and determines a three-dimensional area in real space in which the target feature points are estimated to exist based on the positions of the detected target feature points in the captured image. The position detection device 1 also acquires reflection point cloud information from the radio wave sensor 3. Based on the determined three-dimensional area and the reflection point cloud information, the position detection device 1 then identifies the three-dimensional position of the target feature point from the three-dimensional position information of a reflection point among multiple reflection points detected within the three-dimensional area, and detects the identified three-dimensional position as the three-dimensional position of the target feature point. Therefore, the position detection device 1 can detect the position, including the depth position, of the target feature points detected in the captured image even if the target is in a posture that forms similar skeletal lines in the captured image.

[0083] The position detection device 1 uses the captured image to limit a three-dimensional area within the vehicle interior, and identifies the three-dimensional position of the target feature point using the group of reflection points within the three-dimensional area obtained by the radio wave sensor 3. Even if the radio wave sensor 3 does not necessarily obtain a reflection point in the direction of the target feature point, the position detection device 1 can identify the three-dimensional position of the target feature point, including its depth position, by utilizing reflection points within the three-dimensional area surrounding the direction of the target feature point. Note that, for example, one reason why the radio wave sensor 3 may not obtain a reflection point is that the radio wave sensor 3 is designed to detect moving objects and is unable to detect stationary points.

[0084] Even if the position detection device 1 attempts to detect the three-dimensional position of a target feature point corresponding to a body part of an occupant using only the radio wave sensor 3 without using the imaging device 2, the radio wave sensor 3 alone cannot determine which body part of the occupant the obtained reflection point is based on, and therefore there is a possibility of erroneously detecting the three-dimensional position of the target feature point. For example, if the occupant has their hand raised, the radio wave sensor 3 may erroneously distinguish whether the obtained reflection point is based on, or is based on, the radio wave reflected by the occupant's raised hand. As a result, when detecting the three-dimensional position of the occupant's head, the position detection device 1 may erroneously detect the actual three-dimensional position of the occupant's hand as the three-dimensional position of the occupant's head. In contrast, the position detection device 1 according to the first embodiment uses the imaging device 2 to define a three-dimensional region around the position of a target feature point (e.g., a target feature point corresponding to the head) whose three-dimensional position is to be detected, based on the captured image captured by the imaging device 2. Therefore, the position detection device 1 can more accurately determine the reflection points obtained by the radio wave sensor 3 based on the reflected waves of radio waves reflected from the parts of the occupant's body that correspond to the target feature points whose three-dimensional positions are to be detected, and can more accurately identify the three-dimensional positions of the target feature points by utilizing the reflection point group information obtained from the radio wave sensor 3.

[0085] Furthermore, in general, the radio wave sensor 3 may detect reflection points in areas where no human body is present due to multipath. Furthermore, the radio wave sensor 3 may erroneously detect an object with movement other than a human body, such as a shaking seat, as being near the human body. The position detection device 1 according to the first embodiment uses the imaging device 2, and limits the three-dimensional area, for example, to the vicinity of the position of the occupant's head based on the captured image captured by the imaging device 2. Therefore, when detecting the three-dimensional position of a target feature point whose three-dimensional position is to be detected, the three-dimensional position of the target feature point can be detected without using reflection points that should not be used.

[0086] Furthermore, the size of the body part to which the target feature point corresponds varies depending on the part. For example, the face, shoulders (right or left shoulder), and waist (right or left waist) are different sizes. Therefore, it is thought that the size of the area in which the reflection point acquired by the radio wave sensor 3 exists also varies depending on the body part from which the reflection point was obtained. The position detection device 1 according to the first embodiment makes it possible to set the size of the three-dimensional area according to the size of the body part of the occupant indicated by the target feature point, and thereby detect the three-dimensional position of the target feature point by limiting the reflection points obtained around the body part of the occupant to which the target feature point corresponds, among many reflection points, and therefore it is possible to detect the three-dimensional position of the target feature point with high accuracy.

[0087] Furthermore, the physique detection device 4 acquires position information indicating the three-dimensional positions of the target feature points detected by the position detection device 1, and determines the physique of the occupant based on the acquired position information. By performing physique determination taking into account the depth position based on the three-dimensional positions detected by the position detection device 1, the physique detection device 4 can improve the accuracy of physique determination compared to, for example, a case where the physique of the occupant is determined only from the positions of the target feature points on the captured image.

[0088] In the above first embodiment, the position detection device 1 detects the three-dimensional position of a target feature point corresponding to, for example, the head of an occupant, and the method in which the position detection device 1, more specifically the search area determination unit 12 of the position detection device 1, determines a three-dimensional area of ​​the target feature point corresponding to the head of the occupant has been described. In other words, the three-dimensional area determination method has been described assuming that the search area determination unit 12 determines one three-dimensional area for one target feature point. This is merely an example, and the position detection device 1 can also determine one three-dimensional area for multiple target feature points, for example. The following description will be made with reference to the drawings.

[0089] 7 is a diagram illustrating an example of a three-dimensional area determined by the search area determination unit 12 in Embodiment 1 based on the positions on the captured image of target feature points detected by the feature point detection unit 11, and is a diagram illustrating an example of a three-dimensional area when the search area determination unit 12 determines one three-dimensional area for multiple target feature points. Here, as an example, it is assumed that the search area determination unit 12 has determined three-dimensional areas in which target feature points corresponding to both shoulders, i.e., the right shoulder and the left shoulder, detected by the feature point detection unit 11, are estimated to exist. Regarding the positional relationship between the imaging device 2 and the target feature points, the feature point position angle θcx is not 0 degrees, as in the example of the method for determining a three-dimensional area by the search area determination unit 12 described above with reference to FIG. 5 .

[0090] FIG. 7A shows a captured image (indicated by "21b" in FIG. 7) in which the target feature point is captured. FIGS. 7B, 7C, and 7D are diagrams showing three-dimensional areas (indicated by "R" in FIGS. 7B, 7C, and 7D) determined by the search area determination unit 12. FIG. 7B shows the three-dimensional area when viewed from above the vehicle interior. FIG. 7C shows the three-dimensional area when viewed from the left side of the vehicle interior, to the right of the line passing through the point between the driver's seat and the passenger seat in the vehicle width direction of the vehicle 1000, relative to the traveling direction. FIG. 7D shows the three-dimensional area when viewed from near the dashboard. Note that, as in the installation positions shown in FIG. 3, the image capture device 2 and the radio wave sensor 3 are installed so that the installation axes in the left-right, up-down, and front-to-back directions and the installation coordinates in the left-right and front-to-back directions are equal to each other. In FIGS. 7B, 7C, and 7D, 2a indicates the installation position of the image capture device 2. 7B, 7C, and 7D show simplified views of the front seat 501 and the rear seat 502. For convenience, Figures 7B, 7C, and 7D show black circles representing the radio wave sensor 3 and the reflection point clouds obtained by the radio wave sensor 3, but the search area determination unit 12 does not use the reflection point cloud information output from the radio wave sensor 3 to determine the three-dimensional area.

[0091] The search area determination unit 12 determines three-dimensional areas as shown in Figures 7B, 7C, and 7D based on the positions of two target feature points in the captured image as shown in Figure 7A, i.e., the position of the target feature point corresponding to the right shoulder in the captured image and the position of the target feature point corresponding to the left shoulder in the captured image. The three-dimensional areas shown in Figures 7B, 7C, and 7D are three-dimensional areas determined by the search area determination unit 12 so as to have, for example, the above-mentioned features (1) to (5). In Figure 7A, the occupant is indicated by 201, the position of the occupant's right shoulder, i.e., the position of the target feature point corresponding to the occupant's right shoulder in the captured image, is indicated by 202a, and the position of the occupant's left shoulder, i.e., the position of the target feature point corresponding to the occupant's left shoulder in the captured image, is indicated by 202b.

[0092] As shown in Figures 7B, 7C, and 7D, the search area determination unit 12 can determine one three-dimensional area for two target feature points, one corresponding to the right shoulder and the other corresponding to the left shoulder. Using the orientation estimation line of the target feature point corresponding to the right shoulder (referred to as the "right shoulder orientation estimation line") or the orientation estimation line of the target feature point corresponding to the left shoulder (referred to as the "left shoulder orientation estimation line") as a reference, the search area determination unit 12 determines, as a three-dimensional area, an area indicated by the angle formed by the right shoulder orientation estimation line and the left shoulder orientation estimation line, with margins in the horizontal and vertical directions. In Figures 7B, 7C, and 7D, the right shoulder orientation estimation line is indicated by "SR," and the left shoulder orientation estimation line is indicated by "SL." In addition, in Figures 7B and 7C, θcx １ , θcy １ are the feature point position angles θcx, θcy of the target feature point corresponding to the right shoulder, and θcx ２ , θcy ２are the feature point position angles θcx, θcy of the target feature point corresponding to the left shoulder. For example, the search area determination unit 12 determines a three-dimensional area having a horizontal range that ranges from an angle obtained by extending the right shoulder direction estimation line in the opposite direction from the left shoulder direction estimation line by an angle that indicates a preset margin horizontally from the right shoulder direction estimation line and the left shoulder direction estimation line to an angle obtained by extending the left shoulder direction estimation line in the opposite direction from the right shoulder direction estimation line by an angle that indicates a preset margin horizontally from the right shoulder direction estimation line and the left shoulder direction estimation line, and a vertical range that ranges from an angle obtained by extending the right shoulder direction estimation line or the left shoulder direction estimation line vertically upward from the right shoulder direction estimation line or the left shoulder direction estimation line by an angle that indicates a preset margin vertically downward from the right shoulder direction estimation line or the left shoulder direction estimation line.

[0093] For example, the left and right shoulders are estimated to move in tandem based on the characteristics of a human body shape. The search area determination unit 12 may determine a single three-dimensional area for multiple target feature points corresponding to multiple body parts, such as the left and right shoulders, that are estimated to move in tandem. That is, the search area determination unit 12 may determine the three-dimensional area of ​​the target feature points by taking into account the relationship between the body parts of the occupant indicated by the target feature points and the body parts of the occupant indicated by other target feature points. This allows the position detection device 1 to identify the three-dimensional positions of multiple target feature points corresponding to the body parts of the occupant that are estimated to move in tandem with each other from the group of reflection points detected within a single three-dimensional area. As a result, the position detection device 1 can prevent the detection of three-dimensional positions of the target feature points that are estimated to be impossible for the body parts corresponding to the target feature points based on the structure of the human body, and can estimate the three-dimensional position of the occupant even when some reflection points are missing. In this case, the position identification unit 14 in the position detection device 1 identifies the three-dimensional position of each target feature point by, for example, <Identification Method Example (1)> of the above-mentioned identification method examples. In this case, the position identification unit 14 first estimates the depth coordinate of the target feature point, and estimates the three-dimensional position of the target feature point based on the intersection of the direction estimation line and a line indicating the estimated depth distance of the target feature point. Note that this is just one example, and the position identification unit 14 may also use other methods, such as the above-mentioned <Identification Method Example (1)> to <Identification Method Example (4)>.

[0094] Furthermore, in the above-described first embodiment, as a method for preventing the position detection device 1 from detecting, as the three-dimensional position of a target feature point, a three-dimensional position that is estimated to be impossible given the structure of the human body as the three-dimensional position of the body part to which the target feature point corresponds, for example, the position detection device 1 may detect the three-dimensional position of a certain target feature point, and then take into consideration the structure of the human body, in other words, the relationship between the body part of the occupant indicated by a certain target feature point and the body parts of the occupant indicated by other target feature points, determine three-dimensional regions of other target feature points that are estimated to move in conjunction with the certain target feature point, and detect the three-dimensional positions of the other target feature points. For example, in the position detection device 1, the search area determination unit 12 determines the three-dimensional area of ​​a target feature point corresponding to the right shoulder (referred to as a "right shoulder target feature point"), and the position identification unit 14 detects the three-dimensional position of the right shoulder target feature point based on the three-dimensional area of ​​the right shoulder target feature point and the reflection point cloud information. After that, the search area determination unit 12 determines the three-dimensional area of ​​the left shoulder target feature point, taking into account the relationship with the detected right shoulder target feature point, so as to avoid including an impossible position as the three-dimensional position of the left shoulder target feature point that is estimated to move in conjunction with the right shoulder target feature point. This makes it possible for the position identification unit 14, when identifying the three-dimensional position of the left shoulder target feature point, to reduce the possibility of identifying the three-dimensional position of the left shoulder target feature point based on an unnecessary reflection point, i.e., a reflection point detected at a position that is an impossible three-dimensional position for the left shoulder target feature point. Furthermore, for example, when the position identification unit 14 identifies the three-dimensional position of the left shoulder target feature point, it may take into consideration the three-dimensional position of the right shoulder target feature point that has already been detected, and may avoid identifying, as the three-dimensional position of the left shoulder target feature point, a position that is impossible as the three-dimensional position of the left shoulder target feature point that is estimated to move in conjunction with the right shoulder target feature point.

[0095] In this way, in the position detection device 1, the position identification unit 14 can identify the three-dimensional position of the target feature point by taking into consideration the relationship of the part of the occupant's body indicated by the target feature point with the part of the occupant's body indicated by other target feature points. This allows the position detection device 1 to prevent the three-dimensional position of the target feature point from being detected based on unnecessary reflection points among the reflection points detected within the three-dimensional area, thereby increasing the detection accuracy of the three-dimensional position of the target feature point.

[0096] Furthermore, in the above-described first embodiment, when attempting to identify the three-dimensional position of a certain target feature point, if there is no reflection point detected within the three-dimensional region, the position detection device 1 may take into consideration the relationship between the part of the occupant's body indicated by the certain target feature point and the part of the occupant's body indicated by other target feature points, and may redetermine the three-dimensional region of the certain target feature point based on the three-dimensional regions of other target feature points, for example, other target feature points located around the certain target feature point on the captured image, and the three-dimensional positions of the other detected target feature points, thereby identifying the three-dimensional position of the certain target feature point.

[0097] For example, a specific example will be described using drawings, with one target feature point being a target feature point corresponding to the right shoulder (referred to as a "right shoulder target feature point") and another target feature point being a target feature point corresponding to the left shoulder (referred to as a "left shoulder target feature point").

[0098] 8 and 9 are diagrams illustrating an example of a method in which the position detection device 1, in accordance with Embodiment 1, specifies the three-dimensional position of a certain target feature point based on the three-dimensional areas of the other target feature points and the three-dimensional positions of the detected other target feature points when no reflection point is detected within the three-dimensional area of ​​the certain target feature point. Here, as described above, as an example, the certain target feature point is a right shoulder target feature point and the other target feature point is a left shoulder target feature point, and the position detection device 1 specifies the three-dimensional position of the right shoulder target feature point. Regarding the positional relationship between the imaging device 2 and the target feature point, the feature point position angle θcx is not 0 degrees, as in the example of the method for determining a three-dimensional area by the search area determination unit 12 described above with reference to FIG. 5 .

[0099] Fig. 8A shows a captured image (indicated by "21c" in Fig. 8) in which a target feature point is captured. Fig. 9A shows a captured image (indicated by "21d" in Fig. 9) in which a target feature point is captured. Figs. 8B, 8C, and 8D are diagrams showing the three-dimensional region initially determined by the search region determination unit 12 (indicated by "R1" in Figs. 8B, 8C, and 8D), and Figs. 9B, 9C, and 9D are diagrams showing the three-dimensional region redetermined by the search region determination unit 12 (indicated by "R2" in Figs. 9B, 9C, and 9D). 8B and 9B are diagrams showing a three-dimensional area when the interior of the vehicle is viewed from above, and Fig. 8C and Fig. 9C are diagrams showing a three-dimensional area when the interior of the vehicle is viewed from the left side on the right side of the straight line passing through the point between the driver's seat and the passenger seat in the vehicle width direction of the vehicle 1000, with respect to the traveling direction, and Fig. 8D and Fig. 9D are diagrams showing a three-dimensional area when the interior of the vehicle is viewed from near the dashboard. １ , θcy １ are the feature point position angles θcx, θcy of the right shoulder target feature point, and θcx ２ , θcy ２ are the feature point position angles θcx, θcy of the left shoulder target feature point.

[0100] As shown in Figure 3, the image capture device 2 and the radio wave sensor 3 are installed so that the installation axes in the left-right, up-down, and front-to-back directions and the installation coordinates in the left-right and front-to-back directions are equal to each other. In Figures 8B, 8C, 8D, 9B, 9C, and 9D, 2a indicates the position of the image capture device 2. Also, in Figures 8B, 8C, 8D, 9B, 9C, and 9D, the front seat 501 and the rear seat 502 are shown in simplified form. For convenience, Figures 8B, 8C, 8D, 9B, 9C, and 9D show the radio wave sensor 3 and black circles representing the reflection point clouds obtained by the radio wave sensor 3, but the search area determination unit 12 does not use the reflection point cloud information output from the radio wave sensor 3 to determine the three-dimensional area. Also, in Figure 8A or Figure 9A, the occupant is indicated by 201, the position of the occupant's right shoulder on the captured image, i.e., the position of the occupant's right shoulder target feature point on the captured image, is indicated by 202a, and the position of the occupant's left shoulder on the captured image, i.e., the position of the occupant's left shoulder target feature point on the captured image, is indicated by 202b.

[0101] For example, in the position detection device 1, the search area determination unit 12 determines the three-dimensional area of ​​the right shoulder target feature point based on the position of the right shoulder target feature point on the captured image detected by the feature point detection unit 11. Then, the position identification unit 14 attempts to identify the three-dimensional position of the right shoulder target feature point from the three-dimensional position information of the reflection points detected within the three-dimensional area, based on the three-dimensional area of ​​the right shoulder target feature point and the reflection point group information. However, suppose that no detected reflection points exist within the three-dimensional area of ​​the right shoulder target feature point (see FIG. 8 ).

[0102] In this case, for example, the position identification unit 14 outputs to the search area determination unit 12 an instruction to redetermine the three-dimensional area of ​​the right shoulder target feature point (hereinafter referred to as a "three-dimensional area redetermining instruction"), together with information capable of identifying the left shoulder target feature point, which is another target feature point, and information indicating the three-dimensional position of the detected left shoulder target feature point. The position identification unit 14 may also output information on a reflection point detected within the three-dimensional area of ​​the left shoulder target feature point. With regard to the left shoulder target feature point, the position identification unit 14 considers that it has been able to detect the three-dimensional position of the left shoulder target feature point because a reflection point detected within that three-dimensional area exists. Note that the arrow from the position identification unit 14 to the search area determination unit 12 is omitted from FIG. 1.

[0103] When a three-dimensional area redetermining instruction is output from the position specifying unit 14, the search area determination unit 12 redetermines the three-dimensional area of ​​the right shoulder target feature point. For example, the search area determination unit 12 redetermines the three-dimensional area of ​​the right shoulder target feature point so that it includes the three-dimensional area of ​​the left shoulder target feature point. For example, the search area determination unit 12 determines one three-dimensional area for the right shoulder target feature point and the left shoulder target feature point, and redetermines this as the three-dimensional area of ​​the right shoulder target feature point (see FIG. 9 ). The method of determining one three-dimensional area for multiple target feature points has already been explained, so redundant explanation will be omitted. Furthermore, for example, the search area determination unit 12 may redetermine, as the three-dimensional area of ​​the right shoulder target feature point, an area with a margin based on the orientation estimation line of the right shoulder target feature point, taking into account a reflection point detected within the three-dimensional area of ​​the left shoulder target feature point, and including the reflection point. The search area determination unit 12 outputs three-dimensional area information indicating the three-dimensional area of ​​the redetermined right shoulder target feature point to the position identification unit 14.

[0104] The position identification unit 14 sets reflection points within the three-dimensional region of the right shoulder target feature point before redetermining, for example, by linearly interpolating the reflection points detected within the three-dimensional region of the redetermined right shoulder target feature point, based on the three-dimensional region information and reflection point group information output from the search region determination unit 12. Within the three-dimensional region of the redetermined right shoulder target feature point, reflection points detected within the three-dimensional region of the left shoulder target feature point originally determined by the search region determination unit 12, as well as reflection points resulting from reflection of radio waves from parts of the occupant's body from the left shoulder to the chest, are detected. The position identification unit 14 uses these reflection points to set reflection points that did not exist within the three-dimensional region of the right shoulder target feature point before redetermining. Then, the position identification unit 14 identifies the three-dimensional position of the right shoulder target feature point using, for example, the above-mentioned methods such as <Example Identification Method (1)> to <Example Identification Method (4)> based on the reflection points within the set three-dimensional region of the right shoulder target feature point before it was redetermined, and detects the identified three-dimensional position as the three-dimensional position of the right shoulder target feature point.

[0105] Note that the output of a three-dimensional area redetermination instruction from the position identification unit 14 to the search area determination unit 12, the redetermination of the three-dimensional area by the search area determination unit 12, and the detection of a certain target feature point (right shoulder target feature point) based on the three-dimensional area redeterminated by the position identification unit 14 and the three-dimensional position of another target feature point that has already been detected (left shoulder target feature point) as described above are performed, for example, in the position identification process of step ST4 in the flowchart of FIG. 6 .

[0106] In this way, even if no reflection point is detected in a region in real space (i.e., a three-dimensional region) where a certain target feature point is estimated to exist, the position detection device 1 can detect the three-dimensional position of the certain target feature point based on the three-dimensional regions of other target feature points and the three-dimensional positions of the detected other target feature points. As a result, the position detection device 1 can improve the detection accuracy of the three-dimensional position of the target feature point. Note that the above example describes a case where no reflection point is detected in the three-dimensional region of the certain target feature point, but this is merely an example. For example, the position detection device 1 may be capable of detecting the three-dimensional position of a certain target feature point based on the three-dimensional regions of other target feature points and the three-dimensional positions of the detected other target feature points when a reflection point is detected in the three-dimensional region of the certain target feature point but the reflection point is unstable, or when a large number of reflection points are detected in the three-dimensional region. Examples of unstable reflection points include an extremely small number of reflection points, reflection points being scattered randomly within the three-dimensional region, or reflection points being detected and not being detected, resulting in unstable detection in the time direction.

[0107] In the first embodiment described above, the imaging device 2 and the radio wave sensor 3 are respectively positioned as shown in FIG. 3 , but this is merely an example. It is sufficient that the imaging device 2 is installed so as to be able to capture an image of a target area where target feature points are expected to exist, and the radio wave sensor 3 is installed so as to be able to sense the target area where target feature points are expected to exist, and the positional relationship between the imaging device 2 and the radio wave sensor 3 is known in advance. However, by installing the imaging device 2 and the radio wave sensor 3 so that their orientations and positions match, the amount of calculation required for processing when the position detection device 1 identifies the three-dimensional position of the target feature point can be reduced. In other words, when the imaging device 2 and the radio wave sensor 3 are installed so that their orientations and positions match, it can be said that the imaging device 2 and the radio wave sensor 3 are installed in optimal positions that minimize the amount of calculation required for processing when the position detection device 1 identifies the three-dimensional position of the target feature point.

[0108] In the first embodiment, the phrase "the imaging device 2 and the radio wave sensor 3 are installed so that their positions coincide with each other" refers to the fact that the imaging device 2 and the radio wave sensor 3 are installed so that their horizontal (X-axis) positions, vertical (Y-axis) positions, depth (Z-axis) positions, and installation inclination in three-dimensional space coincide with each other. Here, in the optimal installation position of the imaging device 2 and the radio wave sensor 3 that can minimize the amount of computation required for the processing to identify the three-dimensional position of the target feature point in the position detection device 1, the positions of the imaging device 2 and the radio wave sensor 3 do not need to coincide perfectly, but rather only need to coincide approximately within a predetermined tolerance. Furthermore, in the optimal installation position of the imaging device 2 and the radio wave sensor 3 that can minimize the amount of computation required for the processing to identify the three-dimensional position of the target feature point in the position detection device 1, the orientation of the imaging device 2 and the radio wave sensor 3 do not need to coincide perfectly, but only need to coincide approximately within a predetermined tolerance. Note that in FIGS. 2 and 3 , the positions of the imaging device 2 and the radio wave sensor 3 do not coincide.

[0109] In the first embodiment described above, the imaging device 2 and the radio wave sensor 3 are each installed in the center of the dashboard of the first-row passenger or near the overhead console in the vehicle cabin, but this is merely an example. The imaging device 2 and the radio wave sensor 3 may be installed anywhere as long as they can sense the occupant to be detected. For example, the radio wave sensor 3 may be installed between the first and second-row passengers or on the ceiling of the second-row passengers. Multiple imaging devices 2 may be installed in the vehicle cabin, and multiple radio wave sensors 3 may be installed in the vehicle cabin. The imaging device 2 and the radio wave sensor 3 may be installed so that the imaging range of the imaging device 2 and the sensing range of the radio wave sensor 3 overlap.

[0110] In the first embodiment described above, the occupant whose three-dimensional position of the target feature point is to be detected is the occupant sitting in the driver's seat or the passenger seat of the vehicle 1000, but this is merely an example. The occupant whose three-dimensional position of the target feature point is to be identified may be an occupant sitting in the rear seat. The position detection device 1 can also detect the three-dimensional position of the target feature point corresponding to a body part of the occupant sitting in the rear seat.

[0111] In the first embodiment, the person for whom the position detection device 1 detects the three-dimensional positions of the target feature points is an occupant of the vehicle 1000. However, this is merely an example. The position detection device 1 can detect the three-dimensional positions of target feature points corresponding to the body parts of the occupant in various vehicles. Furthermore, the position detection device 1 can detect the three-dimensional positions corresponding to the body parts of a person present in real space, not limited to vehicle occupants, such as a person present in a room. For example, the position detection device 1 may be applied as a position detection device that detects the three-dimensional positions of target feature points corresponding to the body parts of a person present in a building. The physique detection device 4 may determine the physique of a person present in the building based on the three-dimensional positions of the target feature points corresponding to the body parts of a person present in the building detected by the position detection device 1. For example, if the building is a kindergarten, the physique detection device 4 determines the physique of the person present in the kindergarten. The physique information output as a result of the physique detection device 4 determining the physique of a person present in the kindergarten can be used, for example, in a system that detects the disappearance of a small child from a room and outputs an alarm. Since the position detection device 1 can detect the positions, including the depth positions, of target feature points detected from the captured image, the system can accurately perform the infant monitoring detection function.

[0112] In the first embodiment, the three-dimensional position detected by the position detection device 1 is used to determine the physique of an occupant. However, this is merely an example, and the three-dimensional position detected by the position detection device 1 may be used in an application other than determining the physique of an occupant. For example, the three-dimensional position detected by the position detection device 1 may be used in an application for estimating a person's posture. For example, the position detection device 1 may output position information to an alarm device (not shown), which may then estimate the person's posture based on the position information to detect a person who has fallen and output an alarm. The three-dimensional position detected by the position detection device 1 may also be useful for saving lives. Furthermore, for example, the position detection device 1 may output position information to an occupant behavior detection device (not shown), which may then estimate the posture of the occupant of the vehicle 1000 based on the position information to detect the occupant's behavior, such as the occupant attempting to exit the vehicle. Furthermore, for example, an alarm device can estimate the position of a person's head based on the position information, detect that the person's head may hit some kind of obstacle, and warn the person, and the three-dimensional position detected by the position detection device 1 can be used to monitor the situation in which a person is placed.

[0113] In the first embodiment described above, the position detection device 1 is an on-board device mounted on the vehicle 1000, and the feature point detection unit 11, search area determination unit 12, point cloud acquisition unit 13, and position identification unit 14 are provided in the on-board device. However, this is merely an example. For example, some of the feature point detection unit 11, search area determination unit 12, point cloud acquisition unit 13, and position identification unit 14 may be mounted in the on-board device, and the others may be provided in a server connected to the on-board device via a network, thereby forming a system with the on-board device and the server. Furthermore, the feature point detection unit 11, search area determination unit 12, point cloud acquisition unit 13, and position identification unit 14 may all be provided in the server.

[0114] In the first embodiment, the physique detection device 4 is an in-vehicle device mounted on the vehicle 1000, and the position information acquisition unit 41 and the physique determination unit 42 are provided in the in-vehicle device, but this is merely an example. Some of the position information acquisition unit 41 and the physique determination unit 42 may be provided in the in-vehicle device and the rest may be provided in the server, or the position information acquisition unit 41 and the physique determination unit 42 may be provided in the server.

[0115] 10A and 10B are diagrams illustrating an example of the hardware configuration of the position detection device 1 according to the first embodiment. In the first embodiment, the functions of the feature point detection unit 11, the search area determination unit 12, the point cloud acquisition unit 13, and the position identification unit 14 are realized by a processing circuit 1001. That is, the position detection device 1 includes the processing circuit 1001 for performing control to detect the three-dimensional positions of the target feature points by supplementing the depth direction positions of the target feature points detected based on the captured image captured by the imaging device 2 with the reflection point cloud information generated by the radio wave sensor 3. The processing circuit 1001 may be dedicated hardware as shown in FIG. 10A or a processor 1004 that executes a program stored in memory as shown in FIG. 10B.

[0116] If the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0117] When the processing circuit is a processor 1004, the functions of the feature point detection unit 11, search area determination unit 12, point cloud acquisition unit 13, and position identification unit 14 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 1005. The processor 1004 executes the functions of the feature point detection unit 11, search area determination unit 12, point cloud acquisition unit 13, and position identification unit 14 by reading and executing the program stored in memory 1005. In other words, the position detection device 1 includes memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST4 of FIG. 6 described above. It can also be said that the program stored in memory 1005 causes a computer to execute the processing procedures or methods of the feature point detection unit 11, search area determination unit 12, point cloud acquisition unit 13, and position identification unit 14. Here, the memory 1005 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

[0118] It is also possible to realize some of the functions of the feature point detection unit 11, the search area determination unit 12, the point cloud acquisition unit 13, and the position identification unit 14 with dedicated hardware and some with software or firmware. For example, the function of the point cloud acquisition unit 13 can be realized by a processing circuit 1001 as dedicated hardware, and the functions of the feature point detection unit 11, the search area determination unit 12, and the position identification unit 14 can be realized by the processor 1004 reading and executing programs stored in the memory 1005.

[0119] The position detection device 1 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as an imaging device 2, a radio wave sensor 3, or a physique detection device 4.

[0120] 10A and 10B show an example of the hardware configuration of the physique detection device 4 according to the first embodiment. In the first embodiment, the functions of the position information acquisition unit 41 and the physique determination unit 42 are realized by a processing circuit 1001. That is, the physique detection device 4 includes the processing circuit 1001 for controlling the determination of a person's physique based on the three-dimensional positions of target feature points detected by the position detection device 1.

[0121] When the processing circuit is the processor 1004, the functions of the position information acquisition unit 41 and the physique determination unit 42 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 1005. The processor 1004 executes the functions of the position information acquisition unit 41 and the physique determination unit 42 by reading and executing the program stored in the memory 1005. That is, the physique detection device 4 includes the memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST5 to ST6 of FIG. 6 described above. It can also be said that the program stored in the memory 1005 causes a computer to execute the processing procedures or methods of the position information acquisition unit 41 and the physique determination unit 42.

[0122] The functions of the position information acquisition unit 41 and the physique determination unit 42 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the function of the position information acquisition unit 41 may be realized by a processing circuit 1001 as dedicated hardware, and the function of the physique determination unit 42 may be realized by a processor 1004 reading and executing a program stored in a memory 1005. The physique detection device 4 also includes an input interface unit 1002 and an output interface unit 1003 that perform wired or wireless communication with devices such as the position detection device 1.

[0123] As described above, according to the first embodiment, the position detection device 1 is configured to include: a feature point detection unit 11 that detects, based on an image captured by the imaging device 2 of a target area in which a person may be present, target feature points that correspond to parts of a person's body in the captured image; a search area determination unit 12 that determines a three-dimensional area in real space in which the target feature points are estimated to exist, based on the positions on the captured image of the target feature points detected by the feature point detection unit 11; a point cloud acquisition unit 13 that acquires reflection point cloud information regarding a plurality of reflection points, including three-dimensional position information of a plurality of reflection points of radio waves from reflecting objects, based on reflected waves of radio waves emitted by the radio wave sensor 3 toward the target area and reflected by reflecting objects including people within the target area; and a position identification unit 14 that identifies the three-dimensional position of the target feature point from the three-dimensional position information of a reflection point detected within the three-dimensional area among the plurality of reflection points, based on the three-dimensional area determined by the search area determination unit 12 and the reflection point cloud information acquired by the point cloud acquisition unit 13, and detects the identified three-dimensional position as the three-dimensional position of the target feature point. Therefore, the position detection device 1 can detect the position, including the depth position, of the target feature point detected from the captured image even if the person is in a posture that forms a similar skeletal line in the captured image. The position detection device 1 detects the three-dimensional position of the target feature point by effectively utilizing reflection point cloud information regarding a plurality of reflection points based on reflected waves of radio waves emitted by the radio wave sensor 3 and reflected by a reflective object, which are detected in a direction in which the three-dimensional position of the target feature point exists, which is estimated based on the position of the target feature point detected in the captured image captured by the imaging device 2. The position detection device 1 can detect the three-dimensional position of the target feature point with high accuracy based on the reflection points of radio waves on parts of the person's body detected by the radio wave sensor 3, even if the person is not in a posture that is aligned with the seat, for example, if the person is sitting away from the seat back.

[0124] Furthermore, according to the first embodiment, the position identifying unit 14 in the position detection device 1 determines a representative point within the three-dimensional region based on the three-dimensional positions of the reflection points detected within the three-dimensional region, and identifies the three-dimensional position of the determined representative point as the three-dimensional position of the target feature point. Therefore, the position detection device 1 can detect the position, including the depth position, of the target feature point detected from the captured image even if the posture forms similar skeletal lines on the captured image.

[0125] Furthermore, according to the first embodiment, in the position detection device 1, the search area determination unit 12 sets the size of the three-dimensional area to a size corresponding to the size of the part of the human body indicated by the target feature point. The position detection device 1 can detect the three-dimensional position of the target feature point by limiting the use of reflection points detected in a direction in which the three-dimensional position of the target feature point exists, which is estimated based on the position of the target feature point detected in the image captured by the imaging device 2, out of a large number of reflection point groups that can be detected by the radio wave sensor 3, to points that are estimated to be reflection points in the part of the body indicated by the target feature point, thereby enabling more accurate detection of the three-dimensional position of the target feature point.

[0126] Furthermore, according to the first embodiment, in the position detection device 1, the search area determination unit 12 determines the three-dimensional area of ​​the target feature point in consideration of the relationship of the human body part indicated by the target feature point with the human body parts indicated by other target feature points, and the position identification unit 14 is configured to identify the three-dimensional position of the target feature point based on the three-dimensional area of ​​the target feature point determined by the search area determination unit 12 in consideration of the relationship of the human body part indicated by the target feature point with the human body parts indicated by other target feature points, and on the reflection point cloud information acquired by the point cloud acquisition unit 13. Therefore, for reflection points detected within the three-dimensional area, the position detection device 1 can prevent the three-dimensional position of the target feature point from being detected based on unnecessary reflection points, thereby improving the detection accuracy of the three-dimensional position of the target feature point. Furthermore, when no reflection point is detected in a region in real space (i.e., a three-dimensional region) where a certain target feature point is estimated to exist, or when a large number of reflection points are detected in the three-dimensional region, the position detection device 1 can detect the three-dimensional position of the target feature point based on the three-dimensional regions of other target feature points and the three-dimensional positions of the detected other target feature points, thereby improving the detection accuracy of the three-dimensional position of the target feature point. Furthermore, even when a reflection point is detected in the three-dimensional region of a certain target feature point but the reflection point is unstable, the position detection device 1 can detect the three-dimensional position of the target feature point based on the three-dimensional regions of the other target feature points and the three-dimensional positions of the detected other target feature points, thereby improving the detection accuracy of the three-dimensional position of the target feature point.

[0127] Furthermore, according to the first embodiment, the human body monitoring system 100 includes a position information acquisition unit 41 that acquires position information indicating the three-dimensional positions of target feature points detected by the position detection device 1 as described above, and a physique determination unit 42 that determines the physique of a person based on the position information acquired by the position information acquisition unit 41. Therefore, the human body monitoring system 100 can improve the accuracy of determining the physique of a person compared to, for example, a case in which the physique of a person is determined only from the positions of target feature points in a captured image that are detected in the captured image.

[0128] Furthermore, according to the first embodiment, in the human body monitoring system 100, the physique determination unit 42 selects physique-determining feature points from among the target feature points, calculates physique-determining positions of the physique-determining feature points based on the three-dimensional positions of the physique-determining feature points detected by the position detection device 1, and determines the physique of the person based on the calculated physique-determining positions. Therefore, the human body monitoring system 100 can improve the accuracy of determining a person's physique compared to, for example, determining a person's physique only from the positions of target feature points in a captured image that are detected in the captured image.

[0129] Embodiment 2 In embodiment 2, an embodiment will be described in which the position detection device has a function of determining a search area by also using information about the seat, a function of timing the image capture of the image capture device and the radio wave irradiation of the radio wave sensor, a function of performing time series processing on the reflection points detected by the radio wave sensor, and a function of performing time series processing on the three-dimensional positions of the detected target feature points, and the physique detection device has a function of determining whether or not physique judgment needs to be performed.

[0130] In the following second embodiment, as an example, the person whose body part is to be detected by the position detection device is assumed to be a vehicle occupant sitting in the driver's seat or passenger seat of a vehicle, and the imaging device and radio wave sensor are assumed to be installed in the positions as described in the first embodiment with reference to FIG.

[0131] FIG. 11 is a diagram illustrating a configuration example of a position detection device 1a according to a second embodiment. In the configuration example of the position detection device 1a shown in FIG. 11, components similar to those of the position detection device 1 according to the first embodiment described using FIG. 1 in the first embodiment are denoted by the same reference numerals, and redundant description will be omitted. The position detection device 1a according to the second embodiment differs from the position detection device 1 according to the first embodiment described using FIG. 1 in that it includes a synchronization control unit 15, a seat information acquisition unit 16, a time-series point cloud processing unit 17, and a time-series position identification unit 18. Furthermore, the position detection device 1a according to the second embodiment is connected to a seat sensor 5 in addition to an imaging device 2 and a radio wave sensor 3. The seat sensor 5 is mounted on a vehicle 1000a. The position detection device 1a and a physique detection device 4a constitute a human body monitoring system 100a. In the configuration example of the physique detection device 4a shown in FIG. 11, components similar to those of the physique detection device 4 according to the first embodiment described using FIG. 1 in the first embodiment are denoted by the same reference numerals, and redundant description will be omitted. The physique detection device 4 a according to the second embodiment differs from the physique detection device 4 according to the first embodiment described with reference to FIG. 1 in that it includes a posture determination unit 43 .

[0132] The seat sensor 5 detects the seat position (front, back, up, down), reclining state (reclining angle), etc. of the seat in the vehicle cabin. The seat sensor 5 outputs information relating to the detected seat position, reclining state, etc. of the seat (hereinafter referred to as "seat information") to the position detection device 1.

[0133] The synchronization control unit 15 synchronizes the timing at which the imaging device 2 captures an image with the timing at which the radio wave sensor 3 emits radio waves. The closer the timing at which the imaging device 2 captures an image and the timing at which the radio wave sensor 3 emits radio waves and receives the reflected waves from the radio wave reflected by a reflective object, the more likely it is that the imaging device 2 and the radio wave sensor 3 will capture the same movement of the target feature point. By synchronizing the timing at which the imaging device 2 captures an image with the timing at which the radio wave sensor 3 emits radio waves, the position detection device 1 can improve the accuracy of associating the target feature point in the image acquired from the imaging device 2 with the reflection point group based on the reflection point group information acquired from the radio wave sensor 3, and can detect the three-dimensional position of the target feature point with higher accuracy. Note that one possible method of synchronization control by the synchronization control unit 15 is to send a timing synchronization signal to both the imaging device 2 and the radio wave sensor 3, for example.

[0134] The sheet information acquisition unit 16 acquires sheet information from the sheet sensor 5. The sheet information acquisition unit 16 outputs the acquired sheet information to the search area determination unit 12.

[0135] In the second embodiment, the search area determination unit 12 limits the range of the three-dimensional area based on the seat information acquired by the seat information acquisition unit 16. Specifically, for example, the search area determination unit 12 determines the seat position based on the seat information and limits the size of the three-dimensional area in the forward direction or the size of the three-dimensional area in the rearward direction based on the seat position. The forward and rearward directions here refer to the forward and rearward directions relative to the traveling direction of the vehicle 1000. For example, it is assumed that a reflection point group detected in an area behind the seat back is not a reflection point based on the radio waves emitted by the radio wave sensor 3 reflected by a body part of the occupant sitting in that seat. By limiting the range of the three-dimensional area based on the seat position, the search area determination unit 12 can exclude unnecessary reflection points when the position identification unit 14 identifies the three-dimensional position of the target feature point. This enables the position detection device 1a to detect the three-dimensional position of the target feature point with higher accuracy.

[0136] The time series point cloud processing unit 17 performs time series processing (hereinafter referred to as "time series point cloud processing") on time series reflection point cloud information for a predetermined period (hereinafter referred to as "point cloud acquisition period") L (L≧1) based on the reflection point cloud information acquired by the point cloud acquisition unit 13. In the second embodiment, the time series point cloud processing unit 17 accumulates the reflection point cloud information output from the point cloud acquisition unit 13 in a time series manner, for example, in an internal buffer. The time series point cloud processing unit 17 may delete reflection point cloud information older than a predetermined period from the accumulated reflection point cloud information so as not to accumulate reflection point cloud information for more than the predetermined period. In the second embodiment, the time series point cloud processing performed by the time series point cloud processing unit 17 is assumed to be, for example, a process of increasing the number of reflection points based on the time series reflection point cloud information, a process of stabilizing the number of reflection points by averaging the reflection point clouds, or a process of removing unnecessary reflection points. Increasing the number of reflection points means, for example, adding reflection points detected by the radio wave sensor 3 in the previous processing cycle to reflection points detected in the current processing cycle. For example, if the radio wave sensor 3 operates at a predetermined processing cycle (e.g., every 100 ms), and 10 reflection points were detected in the previous processing cycle and 3 reflection points were detected in the current processing cycle, the position identification unit 14 would perform processing using a total of 13 reflection points from the current processing cycle and the previous processing cycle. If the position detection device 1a did not include the time-series point cloud processing unit 17, the position identification unit 14 would process only the three reflection points detected in the current processing cycle. In contrast, if the position detection device 1a includes the time-series point cloud processing unit 17, the position identification unit 14 would perform processing using a total of 13 reflection points from the current processing cycle and the previous processing cycle. This allows the position identification unit 14 to more accurately estimate the three-dimensional position of the target feature point. Reflection point group averaging involves averaging or interpolating reflection points detected at similar locations in a time series, and is intended to leave only representative reflection points among similar reflection points in a time series, or to interpolate reflection points detected at nearly the same coordinates over multiple periods. This makes it possible to reduce the amount of data compared to simply leaving all the time series information, or to realize processing that is tolerant to temporary non-detection of reflection points.Furthermore, an unnecessary reflection point is, for example, a reflection point that is detected at a certain timing in a position that is clearly different from that at other timings when the reflection points are viewed in time series.

[0137] The time-series point cloud processing unit 17 combines the time-series reflection point cloud information for the point cloud acquisition cycle after the time-series point cloud processing, and outputs the combined reflection point cloud information (hereinafter referred to as "reflection point cloud information after time-series point cloud processing") to the position specifying unit 14. In the second embodiment, the point cloud acquisition unit 13 outputs the acquired reflection point cloud information to the time-series point cloud processing unit 17.

[0138] Furthermore, in the second embodiment, the position identifying unit 14 identifies the three-dimensional position of the target feature point based on the three-dimensional area determined by the search area determining unit 12 and the time-series reflection point cloud information for the point cloud acquisition cycle after the time-series point cloud processing has been performed by the time-series point cloud processing unit 17, more specifically, the reflection point cloud information after the time-series point cloud processing. The specific method for identifying the three-dimensional position of the target feature point by the position identifying unit 14 has already been explained in the first embodiment, and therefore a duplicate explanation will be omitted.

[0139] Here, the point cloud acquisition period L can be set as appropriate. For example, the longer the point cloud acquisition period L is set and the more reflection point clouds are accumulated, the more the time series point cloud processing unit 17 can perform time series point cloud processing on reflection point cloud information that includes non-detection or over-detection of reflection points, interpolate the reflection points, and output stable post-time series point cloud processing reflection point cloud information to the position identification unit 14. As a result, the position identification unit 14 can identify the three-dimensional position of the target feature point based on the stable post-time series point cloud processing reflection point cloud information in which the reflection points are interpolated. On the other hand, if the point cloud acquisition period L is shortened, the time series point cloud processing unit 17 can output the post-time series point cloud processing reflection point cloud information to the position identification unit 14 at an earlier timing, thereby improving the real-time performance of the position identification unit 14 in identifying the three-dimensional position of the target feature point. The point cloud acquisition period L is determined by an administrator or the like, taking into consideration, for example, the detection accuracy of the three-dimensional positions of the target feature points required in the human body monitoring system 100a, or the delay time required to detect the three-dimensional positions of the target feature points.

[0140] The time-series position identification unit 18 acquires position information indicating the three-dimensional position of the target feature point identified by the position identification unit 14, and performs time-series processing (hereinafter referred to as "time-series position identification processing") on the three-dimensional position based on the time-series position information. In the second embodiment, the time-series position identification unit 18 chronologically stores the position information output from the position identification unit 14 in, for example, an internal buffer. The time-series position identification unit 18 may delete position information older than a predetermined period from the stored position information so as not to store position information for more than the predetermined period. In the second embodiment, the time-series position identification processing performed by the time-series position identification unit 18 is assumed to be, for example, a three-dimensional position filtering process based on the time-series position information, such as temporarily not detecting the three-dimensional position of the target feature point or suppressing fluctuations in the three-dimensional position of the target feature point due to noise, etc. "Suppressing temporary non-detection" means, for example, that even if the three-dimensional position is not detected in a certain processing cycle, interpolation is performed based on the detection results of a previous processing cycle and output. The period of time to be interpolated and output is appropriately set depending on the period of time during which temporary non-detection occurs or the time allowed by the application. Suppressing fluctuations in the three-dimensional position means, more specifically, suppressing fluctuations in the coordinate values ​​representing the three-dimensional position, and suppressing fluctuations in the coordinate values ​​means suppressing fluctuations due to temporary outliers by, for example, smoothing the time-series three-dimensional positions of past processing cycles. The time-series position identification unit 18 performs time-series position identification processing of the three-dimensional positions of the target feature points detected by the position identification unit 14, using, for example, a tracking filter such as a known Kalman filter.

[0141] The time-series position specifying unit 18 outputs the position information after the time-series position specifying process to the physique detection device 4a. In the second embodiment, the position specifying unit 14 outputs the position information to the time-series position specifying unit 18. In the second embodiment, the position information acquiring unit 41 of the physique detection device 4a acquires the position information output from the time-series position specifying unit 18 of the position detection device 1a.

[0142] Furthermore, in the second embodiment, in the position detection device 1a, in addition to the functions of the position identification unit 14 described in the first embodiment, the position identification unit 14 may have a function of determining whether or not the detection results of the target feature points in the captured image by the feature point detection unit 11 and the reflection point cloud information acquired by the point cloud acquisition unit 13 satisfy predetermined conditions (hereinafter referred to as "position detection conditions"), and detecting the three-dimensional positions of the target feature points when the detection results and the reflection point cloud information satisfy the position detection conditions. The position detection conditions include conditions for the position identification unit 14 to function normally. The position detection conditions are set in advance by an administrator or the like and stored in a location that can be referenced by the position identification unit 14. For example, the position detection conditions include conditions such as "the target feature points are detected in the captured image," "the number of reflection points indicated by the reflection point cloud information is not extremely small," "the number of reflection points indicated by the reflection point cloud information is not extremely large," or "the movement of the target feature points in the captured image is not drastic." The position specifying unit 14 may, for example, acquire the detection result of the target feature point in the captured image by the feature point detection unit 11 via the search area determination unit 12, or may acquire the detection result of the target feature point in the captured image by the feature point detection unit 11 directly from the feature point detection unit 11. Note that in Fig. 11, the arrow from the feature point detection unit 11 to the position specifying unit 14 is omitted.

[0143] For example, if the detection result of the target feature point in the captured image by the feature point detection unit 11 or the reflection point cloud information acquired by the point cloud acquisition unit 13 does not satisfy the position detection conditions, the position identification unit 14 does not detect the three-dimensional position of the target feature point. For example, if the target feature point is not detected in the captured image, if the number of reflection points indicated by the reflection point cloud information is extremely small, if the number of reflection points indicated by the reflection point cloud information is extremely large, or if the target feature point moves vigorously in the captured image, the position identification unit 14 does not detect the three-dimensional position of the target feature point, thereby making it possible for the position detection device 1a to not output an abnormal three-dimensional position.

[0144] The posture determination unit 43 estimates the posture of the occupant based on the position information acquired from the position detection device 1a by the position information acquisition unit 41. Note that the posture determination unit 43 may estimate the posture of the occupant, for example, based on whether or not the posture is out of alignment, or may estimate which of preset posture types the occupant falls into, such as a reference posture, a forward leaning posture, a sideways posture, or a leaning back posture, or may estimate the posture based on the three-dimensional position of a target feature point corresponding to a predetermined body part, such as the head.

[0145] The posture determination unit 43 estimates the posture of the occupant using, for example, a machine learning model. The machine learning model is, for example, a model that receives position information as input and outputs information indicating the posture. The machine learning model is generated in advance and stored in a location that can be referenced by the posture determination unit 43. The posture determination unit 43 estimates the posture of the occupant by inputting the position information acquired by the position information acquisition unit 41 into the machine learning model to obtain information indicating the posture.

[0146] The posture determination unit 43 may estimate the posture of the occupant based on rules in accordance with conditions for estimating the posture of the occupant (hereinafter referred to as "posture estimation conditions") that are set in advance by an administrator or the like. The posture estimation conditions are stored in a location that can be referenced by the posture determination unit 43. For example, the posture estimation conditions may include a condition that "if the head is extremely low, or if the head is close to the window, or if the head position is shifted from the center of the seat, it is estimated that the posture is poor."

[0147] After estimating the occupant's posture, the posture determination unit 43 determines whether the estimated occupant's posture is suitable for physical type determination. It should be noted that what postures are suitable for physical type determination and what postures are not suitable for physical type determination are predetermined. For example, if the posture determination unit 43 estimates the occupant's posture as a posture in which the three-dimensional positions of target feature points corresponding to predetermined body parts (e.g., head and shoulders) are located close to an airbag, the posture determination unit 43 determines that the occupant's posture is not suitable for physical type determination. For example, if the head and shoulders are located close to the airbag, controlling the deployment of the airbag may pose a risk of harming the occupant, even if the control is appropriate for the occupant's physical type. Therefore, if the posture determination unit 43 estimates the occupant's posture as a posture in which the three-dimensional positions of target feature points corresponding to predetermined body parts (e.g., head and shoulders) are located close to the airbag, the posture determination unit 43 determines that the posture is not suitable for physical type determination.

[0148] When the posture determination unit 43 determines that the estimated posture of the occupant is suitable for physique determination, it outputs the position information acquired by the position information acquisition unit 41 to the physique determination unit 42. When the posture determination unit 43 determines that the estimated posture of the occupant is not suitable for physique determination, it does not output the position information to the physique determination unit 42. As a result, in the physique detection device 4a, when the physique determination unit 42 determines that the posture of the occupant estimated based on the three-dimensional positions of the target feature points detected by the position detection device 1a is not suitable for physique determination, that is, when physique determination of the occupant based on the three-dimensional positions of the target feature points detected by the position detection device 1a should not be performed, it is possible not to perform physique determination of the occupant. Note that in the second embodiment, the physique determination unit 42 performs physique determination of the occupant based on the position information output from the posture determination unit 43.

[0149] Here, the posture determination unit 43 determines whether the occupant's posture is suitable for physical type determination, but this is merely an example. For example, when the position information acquisition unit 41 estimates the occupant's posture based on the position information acquired from the position detection device 1a, the posture determination unit 43 may output information about the estimated occupant's posture to the physical type determination unit 42 together with the position information output from the position information acquisition unit 41. The physical type determination unit 42 may then determine whether the occupant's posture is suitable for physical type determination based on the information about the occupant's posture output from the posture determination unit 43, and perform a physical type determination on the occupant if it determines that the occupant's posture is suitable for physical type determination. Furthermore, in the above-described method, the posture determination unit 43 determines whether the occupant's posture is suitable for physical type determination. However, the posture determination unit 43 may also determine whether the occupant's posture is suitable for airbag control, and determine whether to perform airbag control if the occupant's posture is not suitable for airbag control.

[0150] The operation of the human body monitoring system 100a according to the second embodiment will now be described. FIG. 12 is a flowchart for explaining the operation of the human body monitoring system 100a according to the second embodiment. Of the processes shown in the flowchart of FIG. 12, steps ST11 to ST17 are performed by the position detection device 1a according to the second embodiment, and steps ST18 to ST21 are performed by the physique detection device 4a according to the second embodiment. For example, when the vehicle 1000a is powered on, the human body monitoring system 100a starts the operation shown in the flowchart of FIG. 12 and repeats the operation shown in the flowchart of FIG. 12 until the vehicle 1000a is powered off. While the timing of powering on or off the vehicle 1000 has been described as an example, for example, in the case of an abandoned vehicle detection device, the device may continue to operate for a certain period of time even after the vehicle 1000 is powered off, based on the power supply from the battery of the vehicle 1000, and the timing of powering on or off the vehicle 1000 is appropriately set by an application.

[0151] The synchronization control unit 15 synchronizes the timing at which the imaging device 2 captures an image with the timing at which the radio wave sensor 3 emits radio waves (step ST11). For example, the synchronization control unit 15 sends a timing synchronization signal to both the imaging device 2 and the radio wave sensor 3.

[0152] The sheet information acquiring unit 16 acquires sheet information from the sheet sensor 5 (step ST12). The sheet information acquiring unit 16 outputs the acquired sheet information to the search area determining unit 12.

[0153] The feature point detection unit 11 acquires the captured image from the imaging device 2 and detects target feature points in the captured image based on the captured image captured by the imaging device 2 (step ST13). The feature point detection unit 11 outputs target feature point information to the search area determination unit 12.

[0154] The search area determination unit 12 determines a three-dimensional area based on the sheet information acquired by the sheet information acquisition unit 16 in step ST12 and the positions on the captured image of the target feature points detected by the feature point detection unit 11 in step ST13 (step ST14). Specifically, the search area determination unit 12 determines a three-dimensional area based on the positions on the captured image of the target feature points detected by the feature point detection unit 11, and limits the range of the three-dimensional area based on the sheet information acquired by the sheet information acquisition unit 16. The search area determination unit 12 outputs the three-dimensional area information to the position identification unit 14. At this time, the search area determination unit 12 associates the three-dimensional area information with information indicating the positions of the target feature points on the captured image detected by the feature point detection unit 11, and outputs the three-dimensional area information to the position identification unit 14.

[0155] The point cloud acquisition unit 13 acquires the reflection point cloud information from the radio wave sensor 3 (step ST15). The point cloud acquisition unit 13 outputs the acquired reflection point cloud information to the position identification unit 14.

[0156] The time-series point cloud processing unit 17 performs time-series point cloud processing on the time-series reflection point cloud information for the point cloud acquisition period L based on the reflection point cloud information acquired by the point cloud acquisition unit 13 in step ST15 (step ST16). The time-series point cloud processing unit 17 combines the time-series reflection point cloud information for the point cloud acquisition period L after the time-series point cloud processing, and outputs the combined reflection point cloud information after time-series point cloud processing to the position identification unit 14.

[0157] The position identification unit 14 identifies the three-dimensional position of the target feature point based on the three-dimensional area determined by the search area determination unit 12 in step ST14 and the time-series reflection point cloud information for the point cloud acquisition cycle after the time-series point cloud processing unit 17 has performed time-series point cloud processing in step ST16, more specifically, the reflection point cloud information after time-series point cloud processing, and detects the identified three-dimensional position as the three-dimensional position of the target feature point (step ST17). The position identification unit 14 outputs the position information to the time-series position identification unit 18.

[0158] In addition, in step ST17, the position identification unit 14 may determine whether or not the detection result of the target feature point in the captured image by the feature point detection unit 11 and the reflection point cloud information acquired by the point cloud acquisition unit 13 satisfy the position detection conditions, and if the detection result and the reflection point cloud information satisfy the position detection conditions, detect the three-dimensional position of the target feature point.

[0159] The time-series position specifying unit 18 acquires position information indicating the three-dimensional position of the target feature point specified by the position specifying unit 14 in step ST17, and performs a time-series position specifying process on the three-dimensional position based on the time-series position information (step ST18). The time-series position specifying unit 18 outputs the position information after the time-series position specifying process to the physique detection device 4a. In the second embodiment, the position specifying unit 14 outputs the position information to the time-series position specifying unit 18.

[0160] The position information acquisition unit 41 acquires the position information output from the position identification unit 14 of the position detection device 1a in step ST18 (step ST19). The position information acquisition unit 41 outputs the acquired position information to the physique determination unit 42.

[0161] The posture determination unit 43 estimates the posture of the occupant based on the position information acquired by the position information acquisition unit 41 from the position detection device 1a in step ST19. After estimating the posture of the occupant, the posture determination unit 43 determines whether the estimated posture of the occupant is suitable for physique determination (step ST20). If the posture determination unit 43 determines that the estimated posture of the occupant is suitable for physique determination, it outputs the position information acquired by the position information acquisition unit 41 to the physique determination unit 42. On the other hand, if the posture determination unit 43 determines that the estimated posture of the occupant is not suitable for physique determination, it does not output the position information to the physique determination unit 42. In this case, the operation of the human body monitoring system 100a skips the processing of step ST21.

[0162] The physique determination unit 42 determines the physique of the occupant based on the position information output from the posture determination unit 43 in step ST20 (step ST21). The physique determination unit 42 outputs the physique information to a device such as a collision safety device or an abandoned person detection device.

[0163] In step ST20, when the position information acquisition unit 41 estimates the occupant's posture based on the position information acquired from the position detection device 1a, the posture determination unit 43 may output information on the estimated occupant's posture to the physique determination unit 42 together with the position information output from the position information acquisition unit 41. Then, in step ST21, the physique determination unit 42 may determine whether the occupant's posture is suitable for physique determination based on the information on the occupant's posture output from the posture determination unit 43, and may perform physique determination of the occupant if it is determined that the occupant's posture is suitable for physique determination.

[0164] 12, the processing is performed in the order of steps ST12 to ST16, but this is merely an example. For example, the order in which steps ST12 to ST14 and steps ST15 to ST16 are performed may be reversed, or the order in which steps ST12 to ST14 and steps ST15 to ST16 are performed may be reversed. Also, the order in which steps ST12 and ST13 are performed may be reversed, or the order in which steps ST12 and ST13 are performed may be reversed.

[0165] 11 , human body monitoring system 100a is configured to include, in addition to human body monitoring system 100 according to embodiment 1, synchronization control unit 15, seat information acquisition unit 16, time-series point cloud processing unit 17, time-series position identification unit 18, and posture determination unit 43. However, this is merely an example. These components are added or deleted as appropriate, taking into consideration the required accuracy of detecting the three-dimensional positions of target feature points for human body monitoring system 100a, the computational load, and the like.

[0166] In the second embodiment, the position detection device 1a can also determine one three-dimensional region for a plurality of target feature points.

[0167] Furthermore, in the above-described second embodiment, the position detection device 1a may, for example, detect the three-dimensional position of a certain target feature point, and then take into consideration the structure of the human body, in other words, the relationship between the part of the occupant's body indicated by the certain target feature point and the part of the occupant's body indicated by the other target feature points, to determine the three-dimensional regions of other target feature points that are estimated to move in conjunction with the certain target feature point, and detect the three-dimensional positions of the other target feature points.

[0168] Furthermore, in the second embodiment described above, when attempting to identify the three-dimensional position of a certain target feature point, if there is no reflection point detected within the three-dimensional region, the position detection device 1a may redetermine the three-dimensional region of the certain target feature point based on the three-dimensional regions of other target feature points, for example, other target feature points located around the certain target feature point on the captured image, and the three-dimensional positions of the other detected target feature points, and identify the three-dimensional position of the certain target feature point.

[0169] In the second embodiment described above, the image capture device 2 and the radio wave sensor 3 are respectively set at the positions shown in Fig. 3, but this is merely an example. It is sufficient that the image capture device 2 is installed so as to be able to capture an image of a target area where target feature points are assumed to exist, and the radio wave sensor 3 is installed so as to be able to sense the target area where target feature points are assumed to exist, and it is sufficient that the positional relationship between the image capture device 2 and the radio wave sensor 3 is known in advance.

[0170] Furthermore, in the above-described second embodiment, one imaging device 2 and one radio wave sensor 3 are installed in the vehicle cabin, but this is merely an example. Multiple imaging devices 2 may be installed in the vehicle cabin, and multiple radio wave sensors 3 may be installed in the vehicle cabin. The imaging devices 2 and the radio wave sensors 3 may be installed so that the imaging range of the imaging device 2 and the sensing range of the radio wave sensor 3 overlap. Furthermore, in the above-described second embodiment, one seat sensor 5 is installed in the vehicle cabin, but this is merely an example. Multiple seat sensors 5 may be installed in the vehicle cabin.

[0171] In the second embodiment described above, the occupant whose three-dimensional position of the target feature point is to be detected is the occupant sitting in the driver's seat or the passenger seat of the vehicle 1000a, but this is merely an example. The occupant whose three-dimensional position of the target feature point is to be identified may be an occupant sitting in the rear seat. The position detection device 1a can also detect the three-dimensional position of the target feature point corresponding to a body part of the occupant sitting in the rear seat.

[0172] In the second embodiment described above, the person for whom the position detection device 1a detects the three-dimensional positions of the target feature points is the occupant of the vehicle 1000a, but this is merely an example. The position detection device 1a can detect the three-dimensional positions of target feature points corresponding to body parts of the occupant in various vehicles. Furthermore, the position detection device 1a can detect the three-dimensional positions corresponding to body parts of a person present in real space, such as a person present in a room, without being limited to a vehicle occupant.

[0173] In the second embodiment, the three-dimensional position detected by the position detection device 1a is used to determine the physique of an occupant. However, this is merely an example, and the three-dimensional position detected by the position detection device 1a may be used for applications other than determining the physique of an occupant. For example, the human body monitoring system 100a may not include the physique determination unit 42, but may include the position detection device 1a, a position information acquisition unit 41 that acquires position information indicating the three-dimensional positions of target feature points detected by the position detection device 1a, and a posture determination unit 43 that estimates the posture of a person based on the position information acquired by the position information acquisition unit 41. The posture determination unit 43 may be included in, for example, an alarm device, an occupant behavior detection device, or a physique detection device 4a. The posture determination unit 43 can use the three-dimensional position detected by the position detection device 1a to help save lives, for example, by acquiring position information output from the position detection device 1a via the position information acquisition unit 41 and estimating the posture of the person based on the position information to detect a person who has collapsed and output an alarm. Furthermore, the posture determination unit 43 can also use the three-dimensional position detected by the position detection device 1a to help monitor the situation of a person, for example, by estimating the posture of the occupant of the vehicle 1000a based on the position information, detecting that the occupant is about to exit the vehicle. Furthermore, the posture determination unit 43 can also use the three-dimensional position detected by the position detection device 1a to help monitor the situation of a person, for example, by estimating the position of the person's head based on the position information, detecting that the person's head may hit an obstacle, and issuing a warning to the person. The human body monitoring system 100 according to the first embodiment may also include the posture determination unit 43 instead of the physique determination unit 42.

[0174] In the second embodiment, the position detection device 1a is an in-vehicle device mounted on the vehicle 1000a, and the feature point detection unit 11, the search area determination unit 12, the point cloud acquisition unit 13, the position identification unit 14, the synchronization control unit 15, the seat information acquisition unit 16, the time-series point cloud processing unit 17, and the time-series position identification unit 18 are provided in the in-vehicle device. However, this is merely an example. For example, the position detection device 1a may be an in-vehicle device mounted on the vehicle 1000a, and some of the feature point detection unit 11, the search area determination unit 12, the point cloud acquisition unit 13, the position identification unit 14, the synchronization control unit 15, the seat information acquisition unit 16, the time-series point cloud processing unit 17, and the time-series position identification unit 18 may be mounted in the in-vehicle device, and the rest may be provided in a server connected to the in-vehicle device via a network, thereby constituting a system with the in-vehicle device and the server. In addition, the position detection device 1a may be an on-board device mounted on the vehicle 1000, and the feature point detection unit 11, search area determination unit 12, point cloud acquisition unit 13, position identification unit 14, synchronization control unit 15, seat information acquisition unit 16, time series point cloud processing unit 17, and time series position identification unit 18 may all be provided on a server.

[0175] In the second embodiment, the physique detection device 4a is an in-vehicle device mounted on the vehicle 1000a, and the position information acquisition unit 41, the physique determination unit 42, and the posture determination unit 43 are provided in the in-vehicle device, but this is merely an example. Some of the position information acquisition unit 41, the physique determination unit 42, and the posture determination unit 43 may be provided in the in-vehicle device and the rest may be provided in the server, or the position information acquisition unit 41, the physique determination unit 42, and the posture determination unit 43 may be provided in the server.

[0176] 10A and 10B show an example of a hardware configuration of the position detection device 1a according to the second embodiment. In the first embodiment, the functions of the feature point detection unit 11, the search area determination unit 12, the point cloud acquisition unit 13, the position identification unit 14, the synchronization control unit 15, the sheet information acquisition unit 16, the time-series point cloud processing unit 17, and the time-series position identification unit 18 are realized by a processing circuit 1001. That is, the position detection device 1a includes the processing circuit 1001 for performing control to detect the three-dimensional positions of the target feature points by supplementing the depth positions of the target feature points in the captured images, which are detected based on the captured images captured by the imaging device 2, with the reflection point cloud information generated by the radio wave sensor 3.

[0177] When the processing circuit is a processor 1004, the functions of the feature point detection unit 11, search area determination unit 12, point cloud acquisition unit 13, position identification unit 14, synchronization control unit 15, sheet information acquisition unit 16, time-series point cloud processing unit 17, and time-series position identification unit 18 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 1005. The processor 1004 reads and executes the program stored in memory 1005 to perform the functions of the feature point detection unit 11, search area determination unit 12, point cloud acquisition unit 13, position identification unit 14, synchronization control unit 15, sheet information acquisition unit 16, time-series point cloud processing unit 17, and time-series position identification unit 18. In other words, the position detection device 1a includes a memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST11 to ST18 of FIG. 12 described above. In addition, the program stored in memory 1005 can also be said to cause the computer to execute the processing procedures or methods of the feature point detection unit 11, search area determination unit 12, point cloud acquisition unit 13, position identification unit 14, synchronization control unit 15, sheet information acquisition unit 16, time series point cloud processing unit 17, and time series position identification unit 18.

[0178] It is to be noted that the functions of the feature point detection unit 11, the search area determination unit 12, the point cloud acquisition unit 13, the position identification unit 14, the synchronization control unit 15, the sheet information acquisition unit 16, the time-series point cloud processing unit 17, and the time-series position identification unit 18 may be partially implemented by dedicated hardware and partially implemented by software or firmware. For example, the function of the point cloud acquisition unit 13 may be implemented by a processing circuit 1001 as dedicated hardware, and the functions of the feature point detection unit 11, the search area determination unit 12, the position identification unit 14, the synchronization control unit 15, the sheet information acquisition unit 16, the time-series point cloud processing unit 17, and the time-series position identification unit 18 may be implemented by the processor 1004 reading and executing programs stored in the memory 1005.

[0179] The position detection device 1 a also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as the imaging device 2 , the radio wave sensor 3 , or the physique detection device 4 .

[0180] 10A and 10B show an example of the hardware configuration of the physique detection device 4a according to the second embodiment. In the second embodiment, the functions of the position information acquisition unit 41, the physique determination unit 42, and the posture determination unit 43 are realized by a processing circuit 1001. That is, the physique detection device 4a includes the processing circuit 1001 for controlling the determination of a person's physique based on the three-dimensional positions of target feature points detected by the position detection device 1a.

[0181] When the processing circuit is a processor 1004, the functions of the position information acquisition unit 41, the physique determination unit 42, and the posture determination unit 43 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 1005. The processor 1004 executes the functions of the position information acquisition unit 41, the physique determination unit 42, and the posture determination unit 43 by reading and executing the program stored in memory 1005. That is, the physique detection device 4a includes memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST19 to ST21 of FIG. 1 described above. It can also be said that the program stored in memory 1005 causes a computer to execute the processing procedures or methods of the position information acquisition unit 41, the physique determination unit 42, and the posture determination unit 43.

[0182] It is also possible to realize some of the functions of the position information acquisition unit 41, the physique determination unit 42, and the posture determination unit 43 with dedicated hardware and some with software or firmware. For example, the function of the position information acquisition unit 41 can be realized by a processing circuit 1001 as dedicated hardware, and the functions of the physique determination unit 42 and the posture determination unit 43 can be realized by a processor 1004 reading and executing a program stored in a memory 1005. The physique detection device 4a also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as the position detection device 1a.

[0183] As described above, according to the second embodiment, the position detection device 1a includes, in addition to the configuration of the position detection device 1 according to the first embodiment, a time series point cloud processing unit 17 that performs time series point cloud processing on the time series reflection point cloud information for the point cloud acquisition period, which is acquired by the point cloud acquisition unit 13 and is based on reflected waves of radio waves emitted by the radio wave sensor 3 toward a target area and reflected by a reflective object, and the position identification unit 14 identifies the three-dimensional position of the target feature point based on the three-dimensional area and the time series reflection point cloud information for the point cloud acquisition period after the time series point cloud processing has been performed by the time series point cloud processing unit 17. Therefore, the position detection device 1a can detect the position, including the depth position, of the target feature point detected from the captured image even if the target is in a posture that forms a similar skeletal line on the captured image. The position detection device 1a detects the three-dimensional position of the target feature point by effectively utilizing reflection point cloud information regarding multiple reflection points based on reflected waves of radio waves emitted by the radio wave sensor 3 reflected by a reflective object, detected in a direction in which the three-dimensional position of the target feature point exists, the three-dimensional position being estimated based on the position of the target feature point detected in the captured image captured by the imaging device 2. The position detection device 1a can accurately detect the three-dimensional position of the target feature point based on the reflection points of radio waves detected by the radio wave sensor 3 on parts of the person's body, even if the person is not in a seated position, such as sitting away from the seat back. Furthermore, if the point cloud acquisition period is set long, the position detection device 1a can identify the three-dimensional position of the target feature point based on the reflection point cloud information after time-series point cloud processing, in which the reflection points are interpolated and stable. Furthermore, if the point cloud acquisition period is set short, the position detection device 1a can improve the real-time accuracy of identifying the three-dimensional position of the target feature point.

[0184] Furthermore, according to the second embodiment, the position detection device 1a is configured to include a time-series position identification unit 18 that acquires position information indicating the three-dimensional position of the target feature point identified by the position identification unit 14 and performs time-series position identification processing on the three-dimensional position based on the time-series position information. By applying time-series filtering to the detected three-dimensional position, the position detection device 1a can suppress fluctuations in the detected three-dimensional position due to temporary non-detection, noise, multipath inherent to the radio wave sensor 3, and the like, thereby improving the detection accuracy of the three positions of the target feature point.

[0185] Furthermore, according to the second embodiment, in the position detection device 1a, the position identification unit 14 is configured to identify the three-dimensional position of the target feature point when the detection result of the target feature point in the captured image by the feature point detection unit 11 and the reflection point cloud information acquired by the point cloud acquisition unit 13 satisfy the position detection conditions. Therefore, in the position detection device 1a, for example, when the target feature point is not detected in the captured image, when the number of reflection points indicated in the reflection point cloud information is extremely small, when the number of reflection points indicated in the reflection point cloud information is extremely large, or when the target feature point moves vigorously in the captured image, the position identification unit 14 does not detect the three-dimensional position of the target feature point, thereby preventing the output of an abnormal three-dimensional position.

[0186] Furthermore, in the second embodiment, a person is seated in a seat, and the position detection device 1a includes a seat information acquisition unit 16 that acquires seat information related to the position of the seat where the person is seated, and the search area determination unit 12 is configured to limit the range of the three-dimensional area based on the seat information acquired by the seat information acquisition unit 16. Therefore, when a person is seated in a seat, as in the case of the vehicle 1000a, for example, the position detection device 1a can utilize the seat position to limit the area where the person may be present, thereby improving the detection accuracy of the three-dimensional position of the target feature point.

[0187] Furthermore, according to the second embodiment, the physique detection device 4a includes a posture determination unit 43 that estimates the posture of the person based on the position information acquired by the position information acquisition unit 41, and the physique determination unit 42 is configured to determine the physique of the person based on the position information when the posture of the person estimated by the posture determination unit 43 is a posture suitable for measuring the physique of the person. Therefore, the position detection device 1a can improve the accuracy of determining the physique of the occupant, for example, compared to when the physique of the occupant is determined only from the positions of target feature points on the captured image.

[0188] In addition, the present disclosure allows for free combination of the respective embodiments, modification of any of the components of the respective embodiments, or omission of any of the components of the respective embodiments.

[0189] The position detection device of the present disclosure can detect the position, including the depth position, of target feature points detected from a captured image, even if the posture forms similar skeletal lines on the captured image.

[0190] 1, 1a Position detection device, 11 Feature point detection unit, 12 Search area determination unit, 13 Point cloud acquisition unit, 14 Position identification unit, 15 Synchronization control unit, 16 Seat information acquisition unit, 17 Time series point cloud processing unit, 18 Time series position identification unit, 2 Imaging device, 3 Radio wave sensor, 4, 4a Body size detection device, 41 Position information acquisition unit, 42 Body size determination unit, 43 Posture determination unit, 5 Seat sensor, 100, 100a Human body monitoring system, 1000, 1000a Vehicle, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory.

Claims

1. A position detection device comprising: a feature point detection unit that detects, based on an image captured by an imaging device of a target area where a person may be present, target feature points corresponding to parts of the person's body in the captured image; a search area determination unit that determines a three-dimensional area in real space where the target feature points are presumed to exist, based on the positions of the target feature points detected on the captured image by the feature point detection unit; a point cloud acquisition unit that acquires reflection point cloud information regarding a plurality of reflection points, including three-dimensional position information of the plurality of reflection points of radio waves from reflecting objects within the target area, based on radio waves emitted by a radio wave sensor toward the target area and reflected by the reflecting objects, including the person; and a position identification unit that identifies the three-dimensional position of the target feature point from the three-dimensional position information of the reflection point detected within the three-dimensional area among the plurality of reflection points, based on the three-dimensional area determined by the search area determination unit and the reflection point cloud information acquired by the point cloud acquisition unit, and detects the identified three-dimensional position as the three-dimensional position of the target feature point.

2. The position detection device according to claim 1, characterized in that the search area determination unit determines the three-dimensional area to be an area having the shape of a cone or a polygonal pyramid, with the position of the imaging device as its vertex and a straight line extending from the vertex indicating the direction from the imaging device to the three-dimensional position of the target feature point calculated based on the position of the target feature point on the captured image detected by the feature point detection unit.

3. The position detection device according to claim 1, characterized in that the position identification unit determines a representative point within the three-dimensional area based on the three-dimensional positions of the reflection points detected within the three-dimensional area, and identifies the three-dimensional position of the determined representative point as the three-dimensional position of the target feature point.

4. The position detection device according to claim 3, characterized in that the position identification unit determines the representative point to be a point located at the average or median value of the coordinates of the three-dimensional positions of the reflection points detected within the three-dimensional area.

5. The position detection device according to claim 3, characterized in that the position identification unit uses the coordinates of the three-dimensional position of the reflection point detected within the three-dimensional area to estimate coordinates of a dimension corresponding to at least the depth direction of the imaging device as a depth distance, and determines the representative point to be the intersection of a plane indicating the estimated depth distance and a straight line indicating the direction from the imaging device toward the three-dimensional position of the target feature point.

6. The position detection device according to claim 3, characterized in that the position identification unit determines, as the representative point, the reflection point detected within the three-dimensional area that is closest to a straight line indicating the direction from the imaging device toward the three-dimensional position of the target feature point, which is calculated based on the position of the target feature point on the captured image detected by the feature point detection unit.

7. The position detection device according to claim 3, characterized in that the position identification unit acquires a solid that covers the group of reflection points present in the three-dimensional area based on the three-dimensional positions of the reflection points detected within the three-dimensional area, and determines as the representative point the point on the solid that is closest to a straight line indicating the direction from the imaging device toward the three-dimensional position of the target feature point, calculated based on the position on the captured image of the target feature point detected by the feature point detection unit.

8. The position detection device according to claim 3, characterized in that the position identification unit places a virtual three-dimensional model that resembles the person within the three-dimensional area, changes the position and orientation of the three-dimensional model so as to reduce the error between the shape of the point cloud of the reflection points within the three-dimensional area and the shape of the three-dimensional model, and sets the point on the three-dimensional model after the position and orientation have been changed that corresponds to the target feature point as the representative point.

9. The position detection device according to claim 1, characterized in that the search area determination unit determines the size of the three-dimensional area according to the size of the part of the body of the person indicated by the target feature point.

10. The position detection device according to claim 1, characterized in that the search area determination unit determines the three-dimensional area of ​​the target feature point by taking into consideration the relationship between the part of the body of the person indicated by the target feature point and the part of the body of the person indicated by other target feature points, and the position identification unit identifies the three-dimensional position of the target feature point based on the three-dimensional area of ​​the target feature point determined by the search area determination unit by taking into consideration the relationship between the part of the body of the person indicated by the target feature point and the part of the body of the person indicated by other target feature points, and the reflection point cloud information acquired by the point cloud acquisition unit.

11. A position detection device according to claim 1, further comprising a synchronization control unit that synchronizes the timing at which the imaging device captures the image with the timing at which the radio wave sensor emits the radio waves.

12. The position detection device according to claim 1, further comprising a time series point cloud processing unit that performs time series point cloud processing on the reflection point cloud information acquired by the point cloud acquisition unit, which is based on the reflected waves of the radio waves emitted by the radio wave sensor towards the target area and reflected by the reflective object, in a time series corresponding to a point cloud acquisition period; and the position identification unit identifies the three-dimensional position of the target feature point based on the three-dimensional area and the reflection point cloud information in a time series corresponding to the point cloud acquisition period after the time series point cloud processing unit has performed the time series point cloud processing.

13. The position detection device according to claim 1, further comprising a time-series position identification unit that acquires position information indicating the three-dimensional position of the target feature point identified by the position identification unit, and performs time-series position identification processing on the three-dimensional position based on the time-series position information.

14. The position detection device according to claim 1, characterized in that the position identification unit identifies the three-dimensional position of the target feature point when the detection result of the target feature point in the captured image by the feature point detection unit and the reflection point cloud information acquired by the point cloud acquisition unit satisfy the position detection conditions.

15. The position detection device according to claim 1, characterized in that the person is seated in a seat, and the device is equipped with a seat information acquisition unit that acquires seat information regarding the position of the seat where the person is seated, and the search area determination unit limits the range of the three-dimensional area based on the seat information acquired by the seat information acquisition unit.

16. A position detection program that causes a computer to function as: a feature point detection unit that detects, based on an image captured by an imaging device of a target area where a person may be present, target feature points corresponding to parts of the person's body in the captured image; a search area determination unit that determines a three-dimensional area in real space in which the target feature points are presumed to exist, based on the positions of the target feature points detected on the captured image by the feature point detection unit; a point cloud acquisition unit that acquires reflection point cloud information regarding a plurality of reflection points, including three-dimensional position information of the plurality of reflection points of radio waves from reflecting objects, based on the reflected waves of radio waves emitted by a radio wave sensor toward the target area and reflected by the reflecting objects, including the person, within the target area; and a position identification unit that identifies the three-dimensional position of the target feature point from the three-dimensional position information of the reflection point detected within the three-dimensional area among the plurality of reflection points, based on the three-dimensional area determined by the search area determination unit and the reflection point cloud information acquired by the point cloud acquisition unit, and detects the identified three-dimensional position as the three-dimensional position of the target feature point.

17. A position detection method comprising the steps of: a feature point detection unit detecting, based on an image captured by an imaging device of a target area where a person may be present, target feature points corresponding to parts of the person's body in the captured image; a search area determination unit determining, based on the position of the target feature points on the captured image detected by the feature point detection unit, a three-dimensional area in real space where the target feature points are estimated to exist; a point cloud acquisition unit acquiring, based on radio waves emitted by a radio wave sensor toward the target area and reflected by reflecting objects including the person within the target area, reflection point cloud information related to a plurality of reflection points including three-dimensional position information of the plurality of reflection points of the radio waves from the reflecting objects; and a position identification unit identifying, based on the three-dimensional area determined by the search area determination unit and the reflection point cloud information acquired by the point cloud acquisition unit, the three-dimensional position of the target feature point from the three-dimensional position information of the reflection point detected within the three-dimensional area among the plurality of reflection points, and detecting the identified three-dimensional position as the three-dimensional position of the target feature point.

18. A human body monitoring system comprising: a position detection device according to any one of claims 1 to 15; a position information acquisition unit that acquires position information indicating the three-dimensional position of the target feature point detected by the position detection device; and a physique determination unit that determines the physique of the person based on the position information acquired by the position information acquisition unit.

19. The human body monitoring system of claim 18, wherein the physique determination unit selects a feature point for physique determination from the target feature points, calculates a physique determination position of the feature point for physique determination based on the three-dimensional position of the feature point for physique determination detected by the position detection device, and determines the physique of the person based on the calculated physique determination position.

20. A human body monitoring system as described in claim 18, further comprising a posture determination unit that estimates the posture of the person based on the position information acquired by the position information acquisition unit, and wherein the physique determination unit determines the physique of the person based on the position information when the posture of the person estimated by the posture determination unit is a posture suitable for determining the physique of the person.

21. A human body monitoring system comprising: a position detection device according to any one of claims 1 to 15; a position information acquisition unit that acquires position information indicating the three-dimensional position of the target feature point detected by the position detection device; and a posture determination unit that estimates the posture of the person based on the position information acquired by the position information acquisition unit.

Images