Position estimation device, method, system, program product, and movement sensing device

Abstract

The invention relates to a position estimation device, method, system, program product, and movement sensing device. An underfoot position estimation device (4) is provided with: a detection unit (21) that detects, from an image generated by an imaging unit (2) that captures an image of a predetermined region, a person region in which a person is presented within the predetermined region; and an estimation unit (22) that estimates, as the underfoot position of the person, the point of intersection between the end edge of the person region and a line that tends from the reference point in the person region to the vanishing point in the image, thereby accurately estimating the underfoot position of the person presented in the image.

Description

TECHNICAL FIELD

[0001] The present application relates to a foot position estimation device, a foot position estimation method, a foot position estimation system, a foot position estimation program, and a movement sensing device that estimate a position of a foot of a person appearing in an image. BACKGROUND

[0002] A technique of estimating a posture of a person appearing in an image is proposed (see Japanese Patent Application Publication No. 2015-79339). In this technique, a posture of a person is estimated based on a posture evaluation formula according to a feature quantity calculated for a person region in an input image. At this time, in this technique, a lowermost point of the person region is estimated as a position of a foot of the person.

[0003] As a camera for monitoring use, there are cases where a camera having a wide-angle lens is used as a photographing optical system, and there are cases where a camera having a fisheye lens is used as a photographing optical system according to the situation so as to include as wide a region as possible in a photographing range. In such a case, an object appearing in an image is sometimes largely distorted due to a distortion aberration of the photographing optical system. Therefore, in the above-described technique, it is sometimes not possible to accurately estimate a position of a foot of a person appearing in an image. SUMMARY

[0004] Therefore, an object of the present application is to provide a foot position estimation device that can accurately estimate a position of a foot of a person appearing in an image.

[0005] As one embodiment of the present application, a foot position estimation device is provided. The foot position estimation device has a detection section that detects a person region in which a person appears within a prescribed region from an image generated by a photographing section configured to photograph the prescribed region, and an estimation section that estimates, as a position of a foot of the person, an intersection of a line that trends toward a vanishing point in the image from a reference point within the person region and an end edge of the person region.

[0006] In one embodiment, the estimation section corrects the position of the foot of the person in such a manner that a position on the reference point side of the line that trends toward the vanishing point from the reference point by a correction distance from the intersection is set as the position of the foot of the person, wherein the correction distance is set in advance according to a distance between the vanishing point and the reference point.

[0007] In one embodiment, in a case where the vanishing point exists within the person region, the estimation section estimates, as the position of the foot of the person, a position obtained by dividing the distance between the reference point and the vanishing point in accordance with a ratio of the distance between the reference point and the vanishing point to the distance between the intersection and the reference point, wherein the intersection is an intersection of an end edge of the person region on the reference point side of the line that trends toward the vanishing point and the line.

[0008] In one embodiment, the detection section detects a person region from each of a plurality of images obtained in time series over a prescribed period, and the estimation section estimates a foot-under position of a person in each of the plurality of images. Also, the foot-under position estimation device further has a tracking section that tracks the detected person in the plurality of images and obtains a movement trajectory of the foot-under position of the person over the prescribed period, and a determination section that determines that the detected person moved over the prescribed period when a distance between the foot-under position of the detected person at the start of the prescribed period and the foot-under position of the detected person at the end of the prescribed period is greater than or equal to a first threshold value and a length of the movement trajectory from the start to the end of the prescribed period is greater than or equal to a second threshold value.

[0009] In one embodiment, the detection section detects a person region from each of a plurality of images obtained at different photographing timings, and the estimation section records the foot-under position estimated from each of the plurality of images in a storage section. Also, the foot-under position estimation device further has a determination section that determines a position where there is an abnormality based on a distribution of the foot-under positions stored in the storage section.

[0010] According to another embodiment, a foot-under position estimation method is provided. The foot-under position estimation method includes detecting a person region in which a person is present within a prescribed region from an image generated by a photographing section configured to photograph the prescribed region, and estimating an intersection of a line that trends from a reference point within the person region to a vanishing point in the image and an end edge of the person region as a foot-under position of the person.

[0011] According to still another embodiment, a foot-under position estimation system is provided. The foot-under position estimation system has a photographing section configured to photograph a prescribed region, and a foot-under position estimation device that estimates a foot-under position of a person located within the prescribed region. Also, the foot-under position estimation device has a detection section that detects a person region in which a person is present within the prescribed region from an image generated by the photographing section, and an estimation section that estimates an intersection of a line that trends from a reference point within the person region to a vanishing point in the image and an end edge of the person region as a foot-under position of the person.

[0012] According to still another embodiment, a foot-under position estimation computer program product is provided. The foot-under position estimation computer program product includes instructions for causing a computer to perform the following actions: detecting a person region in which a person is present within a prescribed region from an image generated by a photographing section configured to photograph the prescribed region, and estimating an intersection of a line that trends from a reference point within the person region to a vanishing point in the image and an end edge of the person region as a foot-under position of the person.

[0013] According to still another embodiment, a movement sensing device is provided. The movement sensing device has: a detection section that detects a person within a prescribed region from each of a plurality of images in time series generated by a photographing section provided to photograph the prescribed region, and estimates a position of a prescribed part of the person on one or more images on which the person is presented; a tracking section that tracks the detected person in the one or more images, and calculates a movement trajectory of the position of the prescribed part of the person over a prescribed period; and a determination section that determines that the detected person moved within the prescribed period when a distance between the position of the prescribed part of the detected person at the start of the prescribed period and the position of the prescribed part of the detected person at the end of the prescribed period is greater than or equal to a first threshold value, and a length of the movement trajectory from the start to the end of the prescribed period is greater than or equal to a second threshold value.

[0014] The underfoot position estimation device of the present disclosure has an effect of accurately estimating an underfoot position of a person presented in an image. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a schematic configuration diagram of a system related to movement determination in which the underfoot position estimation device is installed according to an embodiment.

[0016] Figure 2 is a hardware configuration diagram of the underfoot position estimation device.

[0017] Figure 3 is a functional block diagram of a processor associated with movement sensing processing including underfoot position estimation.

[0018] Figure 4A is a schematic explanatory diagram of a summary related to estimation of an underfoot position of a passenger.

[0019] Figure 4B is a schematic explanatory diagram of a summary related to estimation of an underfoot position of a passenger.

[0020] Figure 5 is a schematic explanatory diagram of a summary related to estimation of an underfoot position of a passenger according to a modified example.

[0021] Figure 6A is a schematic explanatory diagram of a summary of movement sensing determination.

[0022] Figure 6B is a schematic explanatory diagram of a summary of movement sensing determination.

[0023] Figure 7 is an action flowchart of movement sensing processing including underfoot position estimation.

[0024] Figure 8is a functional block diagram of a processor of a modification example regarding determination of an abnormal region based on a result of foot position estimation. DETAILED DESCRIPTION

[0025] Hereinafter, a foot position estimation device, a foot position estimation method executed on the foot position estimation device, a foot position estimation computer program, and a foot position estimation system will be described with reference to the drawings. The foot position estimation device detects a person region in which a person present in a prescribed region is presented from an image generated by an imaging section, and estimates a foot position of the person as an intersection of a line from a reference point in the person region to a vanishing point in the image and an end edge of the person region. Thus, even in a case where a camera that presents a large distortion of an object in an image due to distortion aberration or the like is used as the imaging section, the foot position estimation device can accurately estimate the foot position of the person.

[0026] Hereinafter, an example in which the foot position estimation device is utilized in a system for sensing movement of a passenger who is boarding a vehicle will be described. The passenger is one example of a person who is a sensing object of movement. Further, the foot is one example of a prescribed portion. However, the foot position estimation device is not limited to this example, and can be used for sensing movement of a person present in a prescribed region inside a moving object such as a rail vehicle, which can be boarded by a passenger or an occupant, or sensing movement of a person present in a prescribed region of any building or facility.

[0027] Figure 1 is a schematic configuration diagram of a system 10 related to movement determination, which is installed with a foot position estimation device according to one embodiment. The system 10 installed with the foot position estimation device is mounted on a vehicle 1. The vehicle 1 is a vehicle such as a bus, which can be boarded by a plurality of passengers, and in which there is a vehicle interior space in which the passengers can stand and move. The system 10 has a camera 2, a notification device 3, and a foot position estimation device 4.

[0028] The camera 2 is one example of an imaging section, and for example, has a wide-angle lens or a fish-eye lens as an imaging optical system, is mounted so as to face downward near a ceiling in a vehicle interior of the vehicle 1, so that a camera 2 imaging range includes an entire vehicle interior region in which passengers in the vehicle 1 can stay. The vehicle interior region is one example of a prescribed region that is imaged by the imaging section. The camera 2 generates an image that presents the vehicle interior region at each prescribed imaging period (for example, 1 / 30 to 1 / 10 seconds). The camera 2 outputs the generated image to the foot position estimation device 4 via a vehicle interior network at each time of image generation.

[0029] The notification device 3 is a device capable of giving a prescribed notification to a passenger staying in the vehicle 1, such as a speaker, a buzzer, or a display device, and is installed in the vehicle 1. The notification device 3 outputs a prescribed notification, such as a voice indicating a notice to the passenger to move, or a message corresponding to the notification, to the passenger in the vehicle 1, in accordance with a notification signal from the underfoot position estimation device 4.

[0030] The underfoot position estimation device 4 performs a movement sensing process including an underfoot position estimation process, based on the image generated by the camera 2.

[0031] Figure 2 is a hardware configuration diagram of the underfoot position estimation device 4. As shown in Figure 2 , the underfoot position estimation device 4 has a communication interface 11, a memory 12, and a processor 13. The communication interface 11, the memory 12, and the processor 13 can be configured as separate circuits, respectively, or can be integrally configured as one integrated circuit.

[0032] The communication interface 11 has an interface circuit for connecting the underfoot position estimation device 4 to the in-vehicle network. Also, the communication interface 11 delivers the image received from the camera 2 to the processor 13. Further, the communication interface 11 outputs a notification signal accepted from the processor 13 to the notification device 3.

[0033] The memory 12 is an example of a storage unit, such as a volatile semiconductor memory and a non-volatile semiconductor memory. Also, the memory 12 stores various programs and various data used in the movement sensing process including the underfoot position estimation process performed by the processor 13 of the underfoot position estimation device 4. For example, the memory 12 stores parameters for determining a classifier used for detection of a passenger, threshold values for determination of movement sensing, positions of vanishing points, and positions and ranges of in-vehicle regions appearing in an image, and the like. Also, the memory 12 temporarily stores the image accepted from the camera 2 and various data generated in the middle of the movement sensing process.

[0034] The processor 13 has one or a plurality of CPUs (Central Processing Units) and peripheral circuits thereof. The processor 13 can also have other arithmetic circuits such as a logic operation unit, a numerical operation unit, or a graphics processing unit. Also, the processor 13 performs the movement sensing process.

[0035] Figure 3is a functional block diagram of the processor 13 related to the movement sensing processing including the underfoot position estimation processing. The processor 13 has a detection section 21, an estimation section 22, a tracking section 23, a determination section 24, and a notification processing section 25. These sections possessed by the processor 13 are, for example, functional modules realized by a computer program acting on the processor 13. Alternatively, these sections possessed by the processor 13 can also be dedicated arithmetic circuits provided to the processor 13. Further, the processing performed by the detection section 21 and the estimation section 22 among these sections possessed by the processor 13 corresponds to the underfoot position estimation processing.

[0036] The detection section 21 detects a passenger located within the in-vehicle region from each of the plurality of images generated by the camera 2 in time series. In the present embodiment, the detection section 21 detects a passenger from the latest image obtained by the camera 2 at every prescribed period. The detection section 21 performs the same processing for each image, and thus, hereinafter, the processing for one image will be described.

[0037] In the present embodiment, the detection section 21 detects, from the image, a person region in which each passenger is present, for each passenger.

[0038] The detection section 21 detects a passenger by inputting an image received by the camera 2 from the underfoot position estimation device 4 to a classifier that is previously learned in a manner of detecting a passenger's torso and head. As such a classifier, a classifier based on a so-called deep neural network (DNN) is used. For example, as the classifier, a DNN having an architecture of a convolutional neural network (CNN) such as a single shot multibox detector (SSD) or YOLO or a DNN having an attention mechanism such as a vision transformer (ViT) is used. Alternatively, further, as the classifier, a classifier based on another machine learning method such as AdaBoost can also be used. The classifier is previously learned in accordance with a prescribed learning method such as a backpropagation method using many training images including an image in which a passenger to be detected is present.

[0039] The classifier outputs a reliability indicating the accuracy of the passenger being presented for each region on the input image. Then, the detection unit 21 detects a region in which the reliability is equal to or greater than a predetermined detection threshold as a person region. Also, in a case where a plurality of person regions overlap, the detection unit 21 prevents one passenger from being detected multiple times by performing Non-Maximum Suppression (NMS) or Soft NMS (Soft Non-Maximum Suppression). That is, the detection unit 21 calculates Intersection over Union (IoU) for a plurality of person regions that overlap each other, and in a case where the IoU is equal to or greater than a predetermined threshold, the detection unit 21 deletes a person region other than a person region in which the reliability is the greatest. Alternatively, the greater the IoU, the more the detection unit 21 reduces the reliability, and a person region in which the reduced reliability is less than a predetermined detection threshold is deleted.

[0040] The detection unit 21 notifies the estimation unit 22 of the position and the range of the person region in which the detected passenger is presented for each passenger.

[0041] The estimation unit 22 estimates the underfoot position of the detected passenger for each passenger. In the present embodiment, the estimation unit 22 estimates an intersection of a line that trends toward a vanishing point in the image from a reference point within the person region and any one end edge of the person region as the underfoot position of the person. In the present embodiment, the camera 2 is mounted so as to face downward from the ceiling in the vehicle interior, and thus a straight line obtained by extending the center line of an upright passenger toward the underfoot side of the passenger is estimated as a line that trends toward the vanishing point on the image. Here, the line that trends toward the vanishing point from the reference point within the person region becomes a line obtained by approximating the center line, and thus the underfoot of the passenger is estimated to be at an intersection of the line and the end edge of the person region.

[0042] Note that the horizontal position of the reference point is set, for example, at the midpoint of the horizontal direction of the person region. Further, the vertical position of the reference point is set at a position that is a distance obtained by multiplying the length of the person region in the vertical direction by a prescribed coefficient a (for example, 0.5 to 0.6) greater than 0 and less than 1, from the end edge of the person region farther from the vanishing point, on the side of the end edge of the person region closer to the vanishing point. Here, the greater the ratio of the length of the person region in the vertical direction to the length in the horizontal direction, the greater the value of the coefficient a can be set. Also, in the case where the angle formed by the line connecting the vanishing point and the center of gravity of the person region and the horizontal direction is less than 45°, the horizontal direction and the vertical direction in the above description can be exchanged to set the position of the reference point. In this way, by adjusting the position of the reference point according to the shape of the person region, the angle difference between the line from the reference point tending toward the vanishing point and the center line of the passenger appearing in the person region can be made small, and thus the position of the feet can be more accurately estimated.

[0043] Further, depending on the position of the passenger, the vanishing point can sometimes be included in the person region. In such a case, the passenger is in a position close to vertically below the camera 2, and thus the possibility that the feet of the passenger are hidden from view by other parts of the passenger's body is high. That is, the possibility that the feet of the passenger are in a position inside the outer edge of the person region is high. Therefore, the estimation unit 22 estimates as the position of the feet of the passenger the position obtained by dividing the distance between the reference point and the vanishing point in the straight line connecting the reference point and the vanishing point in the ratio of the distance from the reference point to the vanishing point to the distance between the intersection point and the reference point, where the intersection point is the intersection of the straight line and the end edge of the person region on the side closer to the vanishing point. Also, in the case where the distance between the reference point and the vanishing point is sufficiently small, that is, in the case where the distance is below a prescribed same determination threshold (for example, several pixels), the estimation unit 22 can also take the vanishing point itself as the position of the feet of the passenger.

[0044] Figure 4A and Figure 4B are explanatory diagrams of the outline of the estimation of the position of the feet of the passenger. In the example shown in Figure 4A , the vanishing point 401 is present below the person region 410. Therefore, the intersection 413 of the straight line 412 connecting the reference point 411 in the person region 410 and the vanishing point 401 and the lower end of the person region 410 is estimated as the position of the feet of the passenger appearing in the person region 410.

[0045] In Figure 4BIn the example shown, the vanishing point 401 is present within the person region 420. Therefore, on a straight line 422 connecting the reference point 421 within the person region 420 and the vanishing point 401, the ratio r (=d2 / dl) of the distance d2 from the reference point 421 to the vanishing point 401 to the distance dl between the intersection point 424 and the reference point 421, where the intersection point 424 is the intersection point of the straight line 422 and the end side 423 of the person region 420 on the side closer to the vanishing point 401, is calculated. Then, the position 425 obtained by internally dividing the distance between the reference point 421 and the vanishing point 401 in accordance with the ratio r is presumed to be the position under the passenger's feet that appears in the person region 420.

[0046] Further, even in the case where the vanishing point is present outside the person region, depending on the characteristics of the photographing optical system of the camera 2, the position under the passenger's feet on the image is sometimes located within the person region. Therefore, according to the modification, the presumption unit 22 can also correct the position under the passenger's feet in such a manner that the position on the reference point side by a correction distance from the intersection point along the line connecting the reference point and the vanishing point is set as the position under the passenger's feet, where the correction distance is set in advance in accordance with the distance between the reference point and the vanishing point within the person region. In this case, the relationship between the distance between the reference point and the vanishing point and the correction distance from the intersection point to the actual position under the passenger's feet is investigated in advance through experiments. A reference table showing the relationship between the distance between the reference point and the vanishing point and the correction distance, which is produced on the basis of the results of the experiments, is stored in the memory 12 in advance. Then, the presumption unit 22 can obtain the correction distance corresponding to the distance between the reference point and the vanishing point by referring to the reference table.

[0047] Figure 5 is a diagram for explaining the outline of the presumption of the position under the passenger's feet according to the modification. In this example, a correction distance d2 corresponding to the distance dl between a reference point 511 within a person region 510 and a vanishing point 501 is set along a straight line 512 connecting the reference point 511 and the vanishing point 501. Then, a position 514 within the person region 510 located on the reference point 511 side closer to the intersection point 513 of the straight line 512 and the end side of the person region 510 by the correction distance d2 is presumed to be the position under the passenger's feet.

[0048] The presumption unit 22 notifies the tracking unit 23 and the determination unit 24 of the presumed positions under the feet of the respective passengers detected from the image, the position, and the range of the person region.

[0049] The tracking unit 23 tracks the passenger detected in one or more images in which the passenger appears among a plurality of images generated by the camera 2 in time series. The tracking unit 23 associates the person regions of the same passenger with each other throughout the plurality of images with respect to the respective passengers detected throughout the plurality of images.

[0050] The tracking section 23 applies a prescribed tracking method such as KLT tracking or ByteTrack to each of the person regions in the latest image. Thereby, the tracking section 23 detects the person region in the latest image in the previously obtained image (hereinafter referred to as a past image), and establishes association with the person region of the same passenger in the tracking. At each time when the estimation result on the foot position in the latest image is notified from the estimation section 22, the tracking section 23 tracks each of the passengers by repeatedly performing the above processing. Then, the tracking section 23 labels a unique identification number (hereinafter referred to as a passenger ID) to each of the passengers in the tracking, and links a line in which the foot positions specified for the passenger in the tracking are sequentially connected in time as a movement trajectory of the foot positions of the passenger. For each of the person regions in the person regions detected from the latest image, for which no corresponding person region is established in any of the person regions in which the passengers in the tracking are presented in the past images, the tracking section 23 assumes that the passenger presented in the person region is a passenger who newly enters the in-vehicle region and newly starts the tracking. In contrast, for any of the passengers in the tracking, in a case where no association is established with any of the person regions in the latest image with respect to the person region of the passenger, the tracking section 23 assumes that the passenger in the tracking exits from the in-vehicle region and ends the tracking.

[0051] The determination section 24 calculates, for each of one or more passengers detected, the distance between the foot position of the passenger at the start of a prescribed period (for example, several seconds) and the foot position of the passenger at the end of the prescribed period and the length of the movement trajectory in the period, on the basis of the tracking result obtained by the tracking section 23. Then, in a case where the distance between the foot positions at the start and the end is equal to or greater than a first threshold value and the length of the movement trajectory is equal to or greater than a second threshold value, the determination section 24 determines that the passenger moved in the prescribed period.

[0052] The prescribed period can be set to any of the periods in which the passenger is tracked. For example, at each time when the tracking result obtained by the tracking section 23 is updated, the determination section 24 can set the time of the update as the end of the prescribed period, and set the timing at which the prescribed period is retroactively counted from the time of the update as the start of the prescribed period. Further, the determination section 24 can be configured to set the start and the end of the prescribed period as described above in a movement prohibition period in which the movement of the passenger who is riding the vehicle 1 is prohibited. Note that the movement prohibition period can be set to a period in which the door of the vehicle 1 is closed or a period in which the vehicle 1 is moving. Therefore, the determination section 24 can receive information related to the opening and closing of the door or information related to the running state of the vehicle 1 from an electronic control unit that controls the door or the running of the vehicle 1, and set the movement prohibition period on the basis of the received information.

[0053] Figure 6A and Figure 6B are explanatory diagrams of a summary of the movement sensing determination. In Figure 6A the example shown, the distance dl between the passenger's foot position Ps at the start of the prescribed period and the passenger's foot position Pe at the end of the prescribed period is greater than or equal to the first threshold value Thl. Also, the length d2 along the movement locus 601 of the passenger's foot position from the start to the end of the prescribed period is greater than or equal to the second threshold value Th2. Therefore, in this example, the passenger's movement is sensed.

[0054] In Figure 6B the example shown, the length d3 along the movement locus 611 of the passenger's foot position from the passenger's foot position Ps at the start of the prescribed period to the passenger's foot position Pe at the end is also greater than or equal to the second threshold value Th2. However, in this example, the distance d4 between the foot position Ps at the start of the prescribed period and the foot position Pe at the end of the prescribed period is less than the first threshold value Thl. Therefore, in this example, the passenger's movement is not sensed.

[0055] Note that the determination section 24 calculates the sum of the distances between the foot positions at two consecutive time points on the movement locus in the prescribed period as the movement locus of the foot position from the start to the end of the prescribed period. Alternatively, the determination section 24 can calculate the sum of the distance between the foot position at the start of the prescribed period and the foot position at a certain time point in the prescribed period and the sum of the distance between the foot position at the certain time point and the foot position at the end of the prescribed period as the length of the movement locus of the foot position from the start to the end of the prescribed period. Note that the certain time point can be set to, for example, a time point halfway between the start and the end of the prescribed period or a time point at which the foot position is farthest from the foot position at the start or the end of the prescribed period.

[0056] Further, the aspect ratio of each pixel of the camera 2 is not always 1: 1. Therefore, in such a case, the determination section 24 can calculate the distance between two points on the movement locus for which the distance is calculated by multiplying at least one of the horizontal distance and the vertical distance by a correction coefficient corresponding to the reciprocal of the aspect ratio of the pixel.

[0057] Note that the determination section 24 can perform the above-described processing each time an image is obtained by the camera 2 in the tracking of the passenger, and sense the movement of the passenger only when the above-described conditions for the movement sensing are satisfied a plurality of times in succession.

[0058] Further, the determination section 24 can be configured to not sense the movement of the passenger when the position of the passenger's foot at the end of the prescribed period is outside the vehicle 1. This is because, in this case, the passenger is highly likely to have gotten off the vehicle 1, and thus there is no point in sensing the movement of the passenger.

[0059] When the movement of any of the passengers in the tracking is sensed, the determination section 24 notifies the notification processing section 25 of the sensing result.

[0060] When the movement of any of the passengers is notified from the determination section 24, the notification processing section 25 outputs a notification signal to the notification device 3 via the communication interface 11, which alerts each passenger in the vehicle to refrain from moving. Alternatively, the notification processing section 25 can output a movement sensing signal to an electronic control unit that controls the running of the vehicle 1 via the communication interface 11, which indicates that the movement of the passenger is sensed. When the movement sensing signal is received while the vehicle 1 is stopped, the electronic control unit can maintain the state in which the vehicle 1 is stopped until the movement sensing signal is no longer received. Alternatively, when the movement sensing signal is received while the vehicle 1 is moving, the electronic control unit can decelerate the vehicle 1 at a deceleration that does not cause a passenger standing in the vehicle to fall, and can also stop the vehicle 1 on the shoulder.

[0061] Figure 7 is an action flowchart of the movement sensing processing including the foot position estimation processing. The processor 13 executes the movement sensing processing in accordance with the action flowchart shown below.

[0062] The detection section 21 detects a person region in which a passenger is present from the image generated by the camera 2 (step S101). The estimation section 22 estimates the position of the passenger's foot based on a line connecting a reference point and a vanishing point within the person region (step S102). The tracking section 23 decides a movement trajectory of the position of the foot by tracking the detected passenger (step S103).

[0063] The determination section 24 determines whether or not the distance dl between the position of the foot at the start of the prescribed period and the position of the foot at the end of the prescribed period is greater than or equal to the first threshold value Thl (step S104). In the case where the distance dl is greater than or equal to the first threshold value Thl (step S104 - Yes), the determination section 24 determines whether or not the length d2 of the movement trajectory of the position of the foot from the start of the prescribed period to the end is greater than or equal to the second threshold value Th2 (step S105). In the case where the length d2 of the movement trajectory is greater than or equal to the second threshold value Th2 (step S105 - Yes), the determination section 24 senses the movement of the passenger. Then, the notification processing section 25 notifies the passengers in the vehicle of a notice of the attention alert for movement prohibition via the notification device 3 (step S106). Thereafter, the processor 13 ends the movement sensing processing.

[0064] Further, in a case where the distance dl is smaller than the first threshold value Thl in step S104 (step S104 - No), or in a case where the length d2 of the movement trajectory is smaller than the second threshold value Th2 in step S105 (step S105 - No), the processor 13 does not sense the movement of the passenger, and ends the movement sensing processing. Note that, in a case where a plurality of passengers are sensed in steps S101 to S103, and tracking is being performed, the processor 13 executes the processing of steps S104 to S105 for each passenger, and executes the processing of step S106 when movement is sensed for any one of the passengers.

[0065] As described above, the foot position estimation device estimates the intersection of the line that trends from the reference point within the person region to the vanishing point in the image and the end edge of the person region as the foot position of the person present in the person region. Thereby, even in a case where a camera that has a large distortion of an object present in an image due to a distortion aberration or the like is used as the imaging unit, the foot position estimation device can accurately estimate the foot position of the person. Further, even without using a processing that has a large amount of calculation such as the posture estimation model, the foot position estimation device can accurately estimate the foot position of the person.

[0066] Note that, in the vehicle 1, there are sometimes regions where passengers cannot pass due to the presence of objects such as armrests. Therefore, according to the modification, a region on the image corresponding to the region where passengers cannot pass (hereinafter referred to as the impassable region) is stored in advance in the storage 12. Then, the determination section 24 determines whether the movement trajectory of the passenger under tracking crosses the impassable region. In a case where the movement trajectory crosses the impassable region, the tracking of the passenger can fail. Therefore, the determination section 24 does not sense movement with respect to the passenger who becomes such a movement trajectory. Further, when the movement speed of the passenger indicated by the movement trajectory is too fast, the likelihood of the tracking of the passenger failing is high. Therefore, in a case where the movement speed calculated from the distance between any two consecutive points on the movement trajectory and the time difference exceeds an upper limit speed, the determination section 24 does not sense movement with respect to the passenger who becomes such a movement trajectory.

[0067] Further, a passenger in which the movement distance between two consecutive time points among the passengers under tracking gradually becomes longer is assumed to be accelerating. Therefore, with respect to such a passenger, the tracking section 23 can correct the predicted position of the person region in the next frame in a manner that the predicted position is further away from the detected position of the person region in the latest image within the nearest prescribed period along the direction of movement of the passenger on the image.

[0068] Further, the detection section 21 can also change the threshold for the IoU in the NMS processing or the Soft NMS processing according to the position on the image. For example, in the case where a fish-eye lens is used as the photographing optical system of the camera 2, the closer to the periphery of the image, the smaller the subject is rendered. Therefore, the closer to the periphery of the image, the smaller the detection section 21 can make the threshold for the IoU. Likewise, the closer to the periphery of the image, the smaller the tracking section 23 can make the threshold for the IoU in the next frame of the person region on the predicted position of the person region in the tracking and the person region detected from the image of the next frame in the case where the ByteTrack is used as the tracking method. Alternatively, the processor 13 can also be configured to execute the movement sensing processing according to the above-described embodiment or the modified example after performing a preprocessing of correcting the distortion caused by the distortion aberration of the photographing optical system of the camera 2 or the like on each image obtained by the camera 2.

[0069] Note that according to the modified example of the movement sensing device, the detection section 21 can also estimate the position of the torso of the person instead of estimating the position of the feet of the person. The torso is another example of the prescribed part. In this case, the classifier used by the detection section 21 for the detection of the person can be previously learned in such a manner that only the torso is included in the person region. Then, in this case, the detection section 21 sets the center of gravity of the person region as the position of the torso. Further, in this case, the tracking section 23 can set a line connecting the positions of the center of gravity of the person region in the images in which the person is detected in chronological order as the movement trajectory of the torso of the detected person. Then, the determination section 24 can determine whether or not the person has moved based on the movement trajectory of the torso as in the above-described embodiment. In this modified example, not only the distance between the positions of the torso at the start time and the end time of the prescribed period is referred to, but also the length along the movement trajectory is referred to, and thus, the movement of the detected person can be accurately sensed.

[0070] The foot-under position estimation device according to the above-described embodiment or modification example can also be used for other purposes than movement sensing. For example, the processor 13 can also determine whether the estimated foot-under position of the occupant is included in a corresponding region on the image that corresponds to a prescribed region within the vehicle 1. Then, in a case where the foot-under position is included in the corresponding region, the processor 13 can also determine that the occupant is located within the prescribed region. Note that the prescribed region can be set, for example, as a region in the vicinity of a door opening of the vehicle 1, such as a region in which the occupant is prohibited from entering when the door is opened and closed. In this case, in a case where it is determined that the occupant has entered the prescribed region, the processor 13 can notify a control unit that controls opening and closing of the door of the vehicle 1 via the communication interface 11 that the occupant is located in the vicinity of the door. After this notification, the processor 13 can also notify the control unit via the communication interface 11 that the occupant has disappeared from the vicinity of the door in a case where it is determined that the occupant is outside the prescribed region for any occupant. Then, the control unit can be set so as not to perform opening and closing of the door during a period from when the occupant is notified of being located in the vicinity of the door to when the occupant is notified of having disappeared from the vicinity of the door.

[0071] Further, the processor 13 can determine a section in which the vehicle is crowded based on the estimated foot-under positions of the respective occupants. In this case, a section in which the occupants can stay within the vehicle 1 is divided into a plurality of sub-sections. Then, a corresponding block on the image is set for each sub-section. The position and range of the corresponding block of each sub-section can be stored in advance in the storage 12. The processor 13 determines a block that includes the estimated foot-under position of each of the occupants detected from the latest image, and thereby counts the number of occupants whose foot-under positions are included in the block for each block. Then, the processor 13 determines, for each block, the number of occupants whose foot-under positions are included in the block as the number of occupants located in a sub-section corresponding to the block. Then, the processor 13 determines a sub-section in which the number of occupants becomes a prescribed threshold as a crowded sub-section. Thus, the processor 13 can determine a crowded sub-section and a non-crowded sub-section within the vehicle.

[0072] Further, the foot-under position estimation device can be used to determine a region in which there is an abnormality based on a distribution of the estimated foot-under positions of the respective occupants.

[0073] Figure 8is a functional block diagram of a processor according to a modification regarding determination of an abnormal region based on a result of estimation of a position under feet. The processor 13 according to the modification has a detection section 21, an estimation section 22, and a determination section 26. These sections possessed by the processor 13 are functional modules realized by a computer program acting on the processor 13, for example. Alternatively, these sections possessed by the processor 13 can be dedicated arithmetic circuits provided to the processor 13. Hereinafter, portions of the sections possessed by the processor 13 that are different from those of the above-described embodiment are described.

[0074] The detection section 21 detects a person region from each of a plurality of images generated by the camera 2 at mutually different imaging timings. Then, the estimation section 22 stores, for each of the plurality of images, a position under feet of an occupant estimated from a person region detected from the image together with the imaging timing of the image in the memory 12.

[0075] The determination section 26 determines an abnormal region based on a distribution of positions under feet of the occupant within a recent prescribed period stored in the memory 12. For example, in a region in the vehicle 1 in which the occupant can stay but in which the occupant is not present for a period of a certain degree or more, there can be an abnormality in which the occupant avoids the region. Therefore, the determination section 26 determines a region in which the occupant is not present for the recent prescribed period as an abnormal region. Note that the prescribed period can be set to several tens of minutes to several hours, for example.

[0076] Therefore, as with the above-described modification, an interval in which the occupant can stay in the vehicle 1 is divided into a plurality of subintervals. Then, a corresponding block on an image is set for each subinterval. The position and range of the corresponding block of each subinterval can be stored in the memory 12 in advance. The determination section 26 determines a block including a position under feet of each of the occupants detected from each image stored in the memory 12 for which the imaging timing is within the recent prescribed period. Thus, the determination section 26 counts the number of positions under feet included in each block for each block for positions under feet. Then, the determination section 26 determines a subinterval corresponding to a block for which the number of positions under feet is less than a prescribed detection threshold (for example, 1 to 3) as an abnormal region.

[0077] When the abnormal region is determined, the determination unit 26 transmits information indicating the position and range of the determined region and the identification information of the vehicle 1 to a server (not shown) that manages the vehicle 1 via the communication interface 11 and a wireless communication terminal (not shown) mounted on the vehicle 1. Alternatively, the determination unit 26 can also notify a control unit that controls the travel of the vehicle 1 of information indicating the position and range of the determined region via the communication interface 11. In this case, when notified that the abnormal region is determined, the control unit can cause the vehicle 1 to travel to a place where the vehicle 1 is maintained after causing the occupant of the vehicle 1 to get off at a prescribed place.

[0078] According to this modification, the foot-under position estimation device can determine the abnormal region using the estimation results of the foot-under positions of the respective occupants.

[0079] The computer program that causes the computer to execute the processing executed in the processor 13 of the foot-under position estimation device 4 according to the above-described embodiment or modification can also be distributed as a computer program product, for example, recorded in a computer-readable recording medium such as an optical recording medium or a magnetic recording medium.

[0080] As described above, various changes can be made by those skilled in the art within the scope of the present application according to the mode of implementation.

Claims

1. A foot position estimation device, comprising: a detection section that detects a person region in which a person is present within a prescribed region from an image generated by a photographing section configured to photograph the prescribed region; and an estimation section that estimates, as a foot position of the person, an intersection of a line that trends from a reference point within the person region to a vanishing point in the image and an end edge of the person region.

2. The foot position estimation device according to claim 1, wherein the estimation section corrects the foot position in a manner that sets a position on the reference point side along the line by a correction distance from the intersection, the correction distance being set in advance in accordance with a distance between the vanishing point and the reference point.

3. The foot position estimation device according to claim 1, wherein in a case where the vanishing point is present within the person region, the estimation section estimates, as the foot position, a position obtained by dividing the distance between the reference point and the vanishing point in accordance with a ratio of the distance between the intersection and the reference point to the distance between the reference point and the vanishing point, the intersection being an intersection of the line and the end edge of the person region that is closer to the vanishing point on the line.

4. The foot position estimation device according to any one of claims 1 to 3, wherein the detection section detects the person region from each of a plurality of the images obtained in time series within a prescribed period, the estimation section estimates the foot position of the person in each of the plurality of the images, the foot position estimation device further comprises: a tracking section that tracks the detected person in the plurality of the images, and obtains a movement trajectory of the foot position of the person within the prescribed period; and a determination section that determines that the detected person moves within the prescribed period in a case where a distance between the foot position of the detected person at the start of the prescribed period and the foot position of the detected person at the end of the prescribed period is greater than or equal to a first threshold value, and a length of the movement trajectory from the start to the end of the prescribed period is greater than or equal to a second threshold value.

5. The foot position estimation device according to any one of claims 1 to 3, wherein the detection section detects the person region from each of a plurality of the images that differ in photographing timing, the estimation section records the foot position estimated from each of the plurality of the images in a storage section, the foot position estimation device further comprises a determination section that determines a position where there is an abnormality based on a distribution of the foot positions stored in the storage section.

6. A foot position estimation method, comprising: detecting a person region in which a person is present within a prescribed region from an image generated by a photographing section configured to photograph the prescribed region; and estimating, as a foot position of the person, an intersection of a line that trends from a reference point within the person region to a vanishing point in the image and an end edge of the person region.

7. A foot position estimation system, comprising: ​ a photographing section configured to photograph a prescribed area; and a foot position estimation device configured to estimate a foot position of a person present in the prescribed area, the foot position estimation device includes: a detection section configured to detect a person region in which the person is present in the prescribed area from an image generated by the photographing section; and an estimation section configured to estimate, as the foot position of the person, an intersection of a line that trends from a reference point in the person region toward a vanishing point in the image and an end edge of the person region.

8. A computer program product for foot position estimation, comprising instructions for causing a computer to perform the following actions: detecting a person region in which a person is present in a prescribed area from an image generated by a photographing section configured to photograph the prescribed area; and estimating, as the foot position of the person, an intersection of a line that trends from a reference point in the person region toward a vanishing point in the image and an end edge of the person region.

9. A movement sensing device, comprising: a detection section configured to detect a person in the prescribed area from each of a plurality of images in time series generated by a photographing section configured to photograph the prescribed area, and to estimate a position of a prescribed part of the detected person on one or more images on which the detected person is present; a tracking section configured to track the detected person in the one or more images, and to calculate a movement trajectory of the position of the prescribed part of the person over a prescribed period; and a determination section configured to determine that the detected person moved during the prescribed period when a distance between the position of the prescribed part of the detected person at the start of the prescribed period and the position of the prescribed part of the detected person at the end of the prescribed period is greater than or equal to a first threshold value, and a length of the movement trajectory from the start to the end of the prescribed period is greater than or equal to a second threshold value.

Images