Foot position estimation device, foot position estimation method, foot position estimation system, foot position estimation computer program, and movement detection device

Abstract

To provide a foot position estimation device capable of accurately estimating a foot position of a person in an image.SOLUTION: A foot position estimation device 4 comprises: a detection unit 21 for detecting a person area showing a person within a predetermined area from an image generated by an image capturing unit 2 provided so as to capture images of the predetermined area; and an estimation unit 22 for estimating an intersection of a line extending from a reference point in the person area toward a vanishing point in the image with an edge of the person area as a foot position of the person. Such an arrangement enables accurate estimation of the foot position of a person in an image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a foot position estimation device, a foot position estimation method, a foot position estimation system, a computer program for foot position estimation, and a movement detection device that estimate the foot positions of a person depicted in an image. [Background technology]

[0002] A technology has been proposed for estimating the posture of a person depicted in an image (see Patent Document 1). In this technology, the posture of the person is estimated based on a posture evaluation formula from feature amounts calculated for the person area in the input image. In this case, this technology estimates the lowest point of the person area as the position of the person's feet. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-79339 Summary of the Invention [Problem to be solved by the invention]

[0004] Cameras used for surveillance applications often have a wide-angle lens, or in some cases, a fisheye lens, as their imaging optical system to capture as wide an area as possible. In such cases, the objects depicted in the image may be significantly distorted due to distortion aberrations in the imaging optical system. As a result, the above-mentioned technology may not be able to accurately estimate the foot positions of people depicted in the image.

[0005] Therefore, an object of the present invention is to provide a foot position estimation device that can accurately estimate the foot positions of a person depicted in an image. [Means for solving the problem]

[0006] One aspect of the present invention provides a foot position estimation device that includes a detection unit that detects a person area showing a person within a predetermined area from an image generated by an imaging unit that is configured to capture the predetermined area, and an estimation unit that estimates, as the person's foot position, an intersection point between a line extending from a reference point within the person area toward a vanishing point in the image and an edge of the person area.

[0007] In one embodiment, the estimation unit corrects the position of the person's feet by a correction distance that is preset according to the distance between the vanishing point and the reference point, so that the position on the reference point side along the line from the intersection point to the vanishing point is the position of the person's feet.

[0008] In one embodiment, when the vanishing point is located within the person area, the estimation unit estimates the position of the person's feet to be the position obtained by dividing the distance between the reference point and the vanishing point by the ratio of the distance between the reference point and the vanishing point to the distance between the intersection of the edge of the person area closest to the vanishing point on the line from the reference point to the vanishing point and that line.

[0009] In one embodiment, the detection unit detects a person area from each of a plurality of images acquired in time series over a predetermined period of time, and the estimation unit estimates the foot position of the person in each of the plurality of images. The foot position estimation device further includes a tracking unit that tracks the person detected in the plurality of images to obtain a movement trajectory of the person's foot position over the predetermined period, and a determination unit that determines that the detected person has moved over the predetermined period if the distance between the detected person's foot position at the start of the predetermined period and the detected person's foot position at the end of the predetermined period is equal to or greater than a first threshold and the length of the movement trajectory from the start to the end of the predetermined period is equal to or greater than a second threshold.

[0010] In one embodiment, the detection unit detects a person area from each of a plurality of images captured at different times, and the estimation unit records the foot positions estimated from each of the plurality of images in the storage unit. The foot position estimation device further includes an identification unit that identifies a position where some abnormality exists based on the distribution of foot positions stored in the storage unit.

[0011] According to another embodiment, there is provided a foot position estimation method, which includes detecting a person area depicting a person within a predetermined area from an image generated by an imaging unit configured to capture the predetermined area, and estimating, as the position of the person's feet, an intersection point between a line extending from a reference point within the person area toward a vanishing point in the image and an edge of the person area.

[0012] According to yet another embodiment, there is provided a foot position estimation system. This foot position estimation system includes an imaging unit configured to capture an image of a predetermined area, and a foot position estimation device configured to estimate the foot position of a person located within the predetermined area. The foot position estimation device includes a detection unit configured to detect a person area representing a person within the predetermined area from an image generated by the imaging unit, and an estimation unit configured to estimate, as the person's foot position, an intersection point between a line extending from a reference point within the person area toward a vanishing point in the image and an edge of the person area.

[0013] According to yet another embodiment, there is provided a computer program for estimating foot positions, which includes instructions for causing a computer to detect a person area depicting a person within a predetermined area from an image generated by an imaging unit configured to capture the predetermined area, and estimate, as the person's feet position, an intersection point between a line extending from a reference point within the person area toward a vanishing point in the image and an edge of the person area.

[0014] According to yet another embodiment, there is provided a movement detection device including: a detection unit that detects a person within a predetermined area from each of a plurality of time-series images generated by an imaging unit configured to capture images of the predetermined area, and estimates a position of a predetermined part of the person in one or more images in which the detected person is depicted; a tracking unit that tracks the person detected in the one or more images to obtain a movement trajectory of the position of the predetermined part of the person over a predetermined period of time; and a determination unit that determines that the detected person has moved over the predetermined period of time if the distance between the position of the predetermined part of the detected person at the start of the predetermined period and the position of the predetermined part of the detected person at the end of the predetermined period of time is equal to or greater than a first threshold value and the length of the movement trajectory from the start to the end of the predetermined period of time is equal to or greater than a second threshold value. [Effects of the Invention]

[0015] The foot position estimation device according to the present disclosure has the effect of being able to accurately estimate the foot positions of a person depicted in an image. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a schematic configuration diagram of a system relating to movement determination in which a foot position estimation device according to one embodiment is implemented. [Figure 2] FIG. 2 is a hardware configuration diagram of the foot position estimation device. [Figure 3] FIG. 2 is a functional block diagram of a processor related to movement detection processing including foot position estimation. [Figure 4] 10(a) and 10(b) are explanatory diagrams each showing an outline of estimation of the foot position of a passenger. [Figure 5] FIG. 10 is an explanatory diagram of an outline of estimation of the foot positions of a passenger according to a modified example. [Figure 6] 10A and 10B are explanatory diagrams each showing an outline of movement detection determination. [Figure 7] 10 is an operational flowchart of a movement detection process including foot position estimation. [Figure 8]FIG. 10 is a functional block diagram of a processor according to a modified example that identifies an area where an abnormality exists based on the result of foot position estimation. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, a foot position estimation device, a foot position estimation method executed on the foot position estimation device, a computer program for foot position estimation, and a foot position estimation system will be described with reference to the drawings. This foot position estimation device detects a person area depicting a person located within a predetermined area from an image generated by an imaging unit, and estimates the person's feet as the intersection of a line extending from a reference point within the person area toward a vanishing point in the image with an edge of the person area. This makes it possible for this foot position estimation device to accurately estimate the person's foot position even when a camera is used as the imaging unit, which causes significant distortion of objects depicted in the image due to distortion aberrations, etc.

[0018] An example in which the foot position estimation device is used in a system for detecting the movement of passengers on board a vehicle will be described below. A passenger is an example of a person whose movement is to be detected. Furthermore, feet are an example of a predetermined body part. However, the foot position estimation device is not limited to this example, and may also be used to detect the movement of a person within a predetermined area provided inside a moving object that passengers or crew members can board, such as a railway vehicle, or within a predetermined area provided in some kind of building or facility.

[0019] FIG. 1 is a schematic diagram of a system 10 related to movement determination in which a foot position estimation device according to one embodiment is implemented. The system 10 in which the foot position estimation device is implemented is mounted on a vehicle 1. The vehicle 1 is a vehicle such as a bus that can accommodate multiple passengers and has enough interior space for the passengers to stand and move around. The system 10 has a camera 2, a notification device 3, and a foot position estimation device 4.

[0020] Camera 2 is an example of an imaging unit, and has, for example, a wide-angle lens or a fisheye lens as an imaging optical system so that the imaging range of camera 2 includes the entire interior area of ​​vehicle 1 where passengers can be located, and is attached facing downward near the ceiling of vehicle 1. The interior area is an example of a predetermined area imaged by the imaging unit. Camera 2 generates an image representing the interior area at predetermined imaging intervals (for example, 1 / 30 to 1 / 10 seconds). Every time camera 2 generates an image, it outputs the generated image to foot position estimation device 4 via the in-vehicle network.

[0021] The notification device 3 is a device that can issue a predetermined notification to passengers staying inside the vehicle 1, and has, for example, a speaker, a buzzer, or a display device, and is attached inside the vehicle 1. The notification device 3 outputs a predetermined notification in accordance with the notification signal from the foot position estimation device 4, for example, a sound representing a notification that alerts passengers inside the vehicle 1 to the movement of the passengers, or displays a message corresponding to the notification.

[0022] The foot position estimation device 4 executes a movement detection process including a foot position estimation process based on the image generated by the camera 2.

[0023] Fig. 2 is a hardware configuration diagram of the foot position estimation device 4. As shown in Fig. 2, the foot position estimation device 4 has a communication interface 11, a memory 12, and a processor 13. The communication interface 11, the memory 12, and the processor 13 may each be configured as separate circuits, or may be configured integrally as a single integrated circuit.

[0024] The communication interface 11 has an interface circuit for connecting the foot position estimation device 4 to the in-vehicle network. The communication interface 11 passes the image received from the camera 2 to the processor 13. The communication interface 11 also outputs the notification signal received from the processor 13 to the notification device 3.

[0025] The memory 12 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 12 stores various programs and various data used in the movement detection process, including the foot position estimation process, executed by the processor 13 of the foot position estimation device 4. For example, the memory 12 stores parameters for identifying a classifier used to detect an occupant, a threshold for determining movement detection, the position of a vanishing point, and the position and range of the vehicle interior area represented on the image. Furthermore, the memory 12 temporarily stores images received from the camera 2 and various data generated during the movement detection process.

[0026] The processor 13 includes one or more central processing units (CPUs) and their peripheral circuits. The processor 13 may further include other arithmetic circuits such as a logic unit, a numerical calculation unit, or a graphics processing unit. The processor 13 executes a movement detection process.

[0027] 3 is a functional block diagram of the processor 13 related to the movement detection process including the foot position estimation process. The processor 13 has a detection unit 21, an estimation unit 22, a tracking unit 23, a determination unit 24, and a notification processing unit 25. Each of these units in the processor 13 is, for example, a functional module realized by a computer program running on the processor 13. Alternatively, each of these units in the processor 13 may be a dedicated arithmetic circuit provided in the processor 13. Furthermore, among these units in the processor 13, the processes executed by the detection unit 21 and the estimation unit 22 correspond to the foot position estimation process.

[0028] The detection unit 21 detects passengers located within the vehicle interior area from each of a plurality of time-series images generated by the camera 2. In this embodiment, the detection unit 21 detects passengers from the latest image obtained by the camera 2 at predetermined intervals. The detection unit 21 only needs to perform the same processing on each image, so the processing on one image will be described below.

[0029] In this embodiment, the detection unit 21 detects, for each passenger, a person region in which that passenger is depicted from the image.

[0030] The detection unit 21 detects passengers by inputting images received by the foot position estimation device 4 from the camera 2 into a classifier that has been trained in advance to detect the trunk and head of the passenger. Such a classifier is based on a so-called deep neural network (DNN). For example, the classifier may be a DNN with a convolutional neural network (CNN)-type architecture such as Single Shot MultiBox Detector or YOLO, or a DNN with an attention mechanism such as Vision Transformer. Alternatively, the classifier may be a classifier based on another machine learning method such as AdaBoost. The classifier is trained in advance according to a predetermined learning method such as backpropagation using a large number of training images including images of the passenger to be detected.

[0031] The classifier outputs confidence levels for various regions on the input image, indicating the likelihood that a passenger is represented. The detection unit 21 then detects regions with confidence levels equal to or greater than a predetermined detection threshold as human regions. Furthermore, when multiple human regions overlap, the detection unit 21 performs non-maximum suppression (NMS) or soft NMS to prevent multiple detections of a single passenger. That is, the detection unit 21 calculates the intersection over union (IoU) of multiple overlapping human regions, and if the IoU is equal to or greater than a predetermined threshold, removes human regions other than the human region with the highest confidence level. Alternatively, the detection unit 21 reduces the confidence level as the IoU increases, and removes human regions whose reduced confidence level falls below the predetermined detection threshold.

[0032] The detection unit 21 notifies the estimation unit 22 of the position and range of the person area in which each detected passenger is represented.

[0033] The estimation unit 22 estimates the foot position of each detected passenger. In this embodiment, the estimation unit 22 estimates the position of the person's feet as the intersection of a line extending from a reference point within the person area to a vanishing point in the image and one of the edge sides of the person area. In this embodiment, the camera 2 is attached facing downward from the ceiling of the vehicle, so a straight line extending from the midline of an upright passenger toward the passenger's feet is estimated to extend toward the vanishing point on the image. Here, the line extending from the reference point within the person area to the vanishing point approximates the midline, so the passenger's feet are estimated to be at the intersection of that line and an edge side of the person area.

[0034] The horizontal position of the reference point is set, for example, to the horizontal midpoint of the person area. The vertical position of the reference point is set to a position on the edge of the person area closer to the vanishing point, the edge farthest from the vanishing point, from either the top or bottom edge of the person area, a distance calculated by multiplying the vertical length of the person area by a predetermined coefficient α (e.g., 0.5 to 0.6) greater than 0 and less than 1. The coefficient α may be set to a larger value as the ratio of the vertical length to the horizontal length of the person area increases. Furthermore, if the angle between the line connecting the vanishing point and the center of gravity of the person area and the horizontal direction is less than 45°, the horizontal and vertical directions in the above description may be reversed to set the position of the reference point. Adjusting the position of the reference point in this way according to the shape of the person area reduces the angular difference between the line extending from the reference point to the vanishing point and the midline of the passenger depicted in the person area, enabling more accurate estimation of the foot position.

[0035] Furthermore, depending on the passenger's position, the vanishing point may be included in the person area. In such a case, since the passenger is located close to vertically below camera 2, it is highly likely that the passenger's feet are hidden by other parts of the passenger's body and are not visible. In other words, it is highly likely that the passenger's feet are located inside the outer edge of the person area. Therefore, the estimation unit 22 estimates the position of the passenger's feet as a position obtained by dividing the distance between the reference point and the vanishing point internally using the ratio of the distance from the reference point to the distance between the reference point and the vanishing point on a line connecting the reference point and the vanishing point to the distance between the reference point and the intersection of the line and the edge of the person area closer to the vanishing point. Furthermore, if the distance between the reference point and the vanishing point is sufficiently small, i.e., if the distance is equal to or less than a predetermined identity determination threshold (e.g., a few pixels), the estimation unit 22 may determine the vanishing point itself as the position of the passenger's feet.

[0036] 4(a) and 4(b) are explanatory diagrams each showing an overview of estimating the position of a passenger's feet. In the example shown in Fig. 4(a), a vanishing point 401 exists below a person area 410. Therefore, an intersection 413 between a line 412 connecting a reference point 411 in the person area 410 with the vanishing point 401 and the bottom edge of the person area 410 is estimated as the position of the passenger's feet shown in the person area 410.

[0037] 4(b), a vanishing point 401 exists within a person area 420. Therefore, on a line 422 connecting a reference point 421 within the person area 420 to the vanishing point 401, a ratio r (=d2 / d1) is calculated of a distance d2 from the reference point 421 to the vanishing point 401, to a distance d1 between the reference point 421 and an intersection 424 of the line 422 and an edge 423 of the person area 420 closer to the vanishing point 401. Then, a position 425 obtained by dividing the distance between the reference point 421 and the vanishing point 401 by the ratio r is estimated as the position of the feet of the passenger depicted in the person area 420.

[0038] Furthermore, even if a vanishing point exists outside the person area, the position of the passenger's feet in the image may be located within the person area depending on the characteristics of the imaging optical system of camera 2. Therefore, according to a modified example, the estimation unit 22 may correct the position of the passenger's feet by a correction distance that is preset according to the distance between the reference point and the vanishing point within the person area, so that the position of the passenger's feet is located on the side of the intersection point along the line connecting the reference point and the vanishing point, away from the reference point. In this case, the relationship between the distance between the reference point and the vanishing point and the correction distance from the intersection point to the actual foot position is experimentally investigated in advance. A reference table that represents the relationship between the distance between the reference point and the vanishing point and the correction distance, created based on the experimental results, is stored in advance in memory 12. Then, by referring to the reference table, the estimation unit 22 can determine the correction distance corresponding to the distance between the reference point and the vanishing point.

[0039] 5 is an explanatory diagram outlining the estimation of the position of a passenger's feet according to this modified example. In this example, a correction distance d2 is set along a straight line 512 connecting a reference point 511 in a person area 510 with a vanishing point 501, according to a distance d1 between the reference point 511 and the vanishing point 501. Then, a position 514 in the person area 510 that is closer to the reference point 511 along the straight line 512 than an intersection 513 between the line 512 and the edge of the person area 510 by the correction distance d2 is estimated as the position of the passenger's feet.

[0040] The estimation unit 22 notifies the tracking unit 23 and the determination unit 24 of the estimated position of the feet of each passenger detected from the image, and the position and range of the person area.

[0041] The tracking unit 23 tracks the detected passenger in one or more images in which the passenger is depicted, among a plurality of time-series images generated by the camera 2. For each passenger detected across a plurality of images, the tracking unit 23 associates person regions of the same passenger across the plurality of images.

[0042] The tracking unit 23 applies a predetermined tracking method, such as KLT tracking or ByteTrack, to each person region in the latest image. As a result, the tracking unit 23 associates each person region in the latest image with a person region of the same passenger detected and currently being tracked in a previously acquired image (hereinafter referred to as a past image). Each time the estimation unit 22 notifies the tracking unit 23 of the foot position estimation result for the latest image, the tracking unit 23 repeats the above process to track each individual passenger. The tracking unit 23 assigns a unique identification number (hereinafter referred to as a passenger ID) to each passenger being tracked, and defines a line connecting the foot positions specified for the passenger being tracked in chronological order as the movement trajectory of the passenger's foot position. For a person region detected in the latest image that cannot be associated with any person region representing the passenger being tracked in a past image, the tracking unit 23 considers the passenger represented in the person region to have newly entered the vehicle interior and begins new tracking. Conversely, if the person area for any of the passengers being tracked in the past images is not associated with any of the person areas in the latest images, the tracking unit 23 determines that the passenger being tracked has exited the vehicle interior area and terminates tracking.

[0043] The determination unit 24 determines the distance between the foot position of the passenger at the start of a predetermined period (e.g., a few seconds) and the foot position of the passenger at the end of the predetermined period, and the length of the movement trajectory during that period, for each of the one or more detected passengers, based on the tracking results by the tracking unit 23. Then, the determination unit 24 determines that the passenger has moved during the predetermined period if the distance between the foot positions at the start and end is equal to or greater than a first threshold and the length of the movement trajectory is equal to or greater than a second threshold.

[0044] The predetermined period can be any period during which the passenger is being tracked. For example, each time the tracking result by the tracking unit 23 is updated, the determination unit 24 may set the time of the update as the end of the predetermined period and the time preceding the update by a predetermined period as the start of the predetermined period. The determination unit 24 may also set the start and end of the predetermined period as described above within a movement prohibition period during which movement of passengers aboard the vehicle 1 is prohibited. The movement prohibition period may be a period during which the door at the entrance to the vehicle 1 is closed or a period during which the vehicle 1 is moving. Therefore, the determination unit 24 may receive information regarding the opening and closing of the doors or information regarding the traveling state of the vehicle 1 from an electronic control unit that controls the doors or traveling of the vehicle 1, and set the movement prohibition period based on the received information.

[0045] 6(a) and 6(b) are explanatory diagrams each showing an outline of movement detection and determination. In the example shown in FIG. 6(a), the distance d1 between the passenger's foot position Ps at the start of a predetermined period and the passenger's foot position Pe at the end of the predetermined period is equal to or greater than a first threshold value Th1. Furthermore, the length d2 along the movement trajectory 601 of the passenger's foot position from the start to the end of the predetermined period is equal to or greater than a second threshold value Th2. Therefore, in this example, the movement of the passenger is detected.

[0046] 6(b), the length d3 along the movement trajectory 611 from the passenger's foot position Ps at the start of the predetermined period to the passenger's foot position Pe at the end of the predetermined period is equal to or greater than the second threshold value Th2. However, in this example, the distance d4 between the passenger's foot position Ps at the start of the predetermined period and the passenger's foot position Pe at the end of the predetermined period is less than the first threshold value Th1. Therefore, in this example, the movement of the passenger is not detected.

[0047] The determination unit 24 calculates the sum of the distances between the foot positions at two consecutive points on the movement trajectory during the predetermined period as the length along the movement trajectory of the foot positions from the start to the end of the predetermined period. Alternatively, the determination unit 24 may calculate the length of the movement trajectory of the foot positions from the start to the end of the predetermined period as the sum of the distance between the foot position at the start of the predetermined period and the foot position at a specific point in time during the predetermined period, and the distance between the foot position at the specific point in time and the foot position at the end of the predetermined period. The specific point in time may be, for example, the midpoint between the start and end of the predetermined period, or the point at which the foot position is farthest from the foot position at the start or end of the predetermined period.

[0048] Furthermore, the aspect ratio of each pixel of camera 2 may not be 1:1. In such a case, determination unit 24 may calculate the distance between two points on the movement trajectory for which the distance is to be calculated by multiplying at least one of the horizontal distance and the vertical distance between the two points by a correction coefficient equivalent to the reciprocal of the aspect ratio of the pixels.

[0049] In addition, while tracking a passenger, the determination unit 24 may perform the above processing each time an image is obtained by the camera 2, and detect the movement of the passenger only if the above movement detection conditions are met multiple times in a row.

[0050] Furthermore, the determination unit 24 may be configured not to detect the movement of a passenger if the foot position of the passenger at the end of the predetermined period is outside the vehicle 1. In this case, it is highly likely that the passenger has disembarked from the vehicle 1, and therefore there is no point in detecting the movement of the passenger.

[0051] When the determination unit 24 detects the movement of any passenger being tracked, it notifies the notification processing unit 25 of the detection result.

[0052] When the notification processing unit 25 is notified by the determination unit 24 that movement of any passenger has been detected, it outputs a notification signal to the notification device 3 via the communication interface 11, warning each passenger in the vehicle not to move. Alternatively, the notification processing unit 25 may output a movement detection signal, indicating that movement of a passenger has been detected, to an electronic control unit that controls the traveling of the vehicle 1, via the communication interface 11. When the electronic control unit receives the movement detection signal while the vehicle 1 is stopped, it may cause the vehicle 1 to continue to be stopped until the movement detection signal is no longer received. Alternatively, when the electronic control unit receives the movement detection signal while the vehicle 1 is moving, it may decelerate the vehicle 1 at a deceleration that will not cause a passenger standing inside the vehicle to fall, and may further cause the vehicle 1 to stop on the shoulder of the road.

[0053] 7 is an operational flowchart of the movement detection process including the foot position estimation process. The processor 13 executes the movement detection process in accordance with the operational flowchart shown below.

[0054] The detection unit 21 detects a person area representing a passenger from an image generated by the camera 2 (step S101). The estimation unit 22 estimates the position of the passenger's feet based on a line connecting a reference point and a vanishing point in the person area (step S102). The tracking unit 23 tracks the detected passenger and obtains a movement trajectory of the passenger's feet (step S103).

[0055] The determination unit 24 determines whether the distance d1 between the foot positions at the start and end of a predetermined period during tracking is equal to or greater than a first threshold Th1 (step S104). If the distance d1 is equal to or greater than the first threshold Th1 (step S104-Yes), the determination unit 24 determines whether the length d2 of the movement trajectory of the foot position from the start to the end of the predetermined period is equal to or greater than a second threshold Th2 (step S105). If the length d2 of the movement trajectory is equal to or greater than the second threshold Th2 (step S105-Yes), the determination unit 24 detects the movement of the passenger. Then, the notification processing unit 25 notifies the passengers in the vehicle via the notification device 3 of a warning to not move (step S106). Thereafter, the processor 13 ends the movement detection process.

[0056] Furthermore, if the distance d1 is less than the first threshold Th1 in step S104 (step S104-No), or if the length d2 of the movement trajectory is less than the second threshold Th2 in step S105 (step S105-No), the processor 13 ends the movement detection process without detecting the movement of the passenger. Note that, if multiple passengers are detected in steps S101 to S103 and are being tracked, the processor 13 executes the processes of steps S104 to S105 for each passenger, and when the movement of any passenger is detected, executes the process of step S106.

[0057] As explained above, this foot position estimation device estimates the foot position of a person depicted in the person area by estimating the intersection of a line extending from a reference point within the person area toward a vanishing point in the image with the edge of the person area. This allows this foot position estimation device to accurately estimate the foot position of a person even when a camera with significant distortion of objects depicted in the image due to distortion aberration or the like is used as the imaging unit. Furthermore, this foot position estimation device can accurately estimate the foot position of a person without using computationally intensive processing such as a posture estimation model.

[0058] Note that there may be areas inside the vehicle 1 where passengers cannot pass due to the presence of objects such as handrails. Therefore, according to a modified example, areas on the image corresponding to areas inside the vehicle where passengers cannot pass (hereinafter referred to as impassable areas) are pre-stored in the memory 12. The determination unit 24 then determines whether the movement trajectory of the passenger being tracked crosses the impassable area. If the movement trajectory crosses the impassable area, there is a possibility that tracking of the passenger has failed. Therefore, the determination unit 24 does not detect the movement of a passenger whose movement trajectory is such a trajectory. Furthermore, if the movement speed of the passenger indicated on the movement trajectory is too fast, there is a high possibility that tracking of the passenger has failed. Therefore, if the movement speed calculated from the distance and time difference between any two consecutive points on the movement trajectory exceeds the upper limit speed, the determination unit 24 does not detect the movement of a passenger whose movement trajectory is such a trajectory.

[0059] Furthermore, if a passenger is being tracked and the distance he or she moves between two consecutive points in time gradually increases, it is assumed that he or she is accelerating. Therefore, for such a passenger, the tracking unit 23 may correct the predicted position of the person area in the next frame so that the predicted position of the person area in the next frame moves further away from the detected position of the person area in the most recent image during the most recent predetermined period along the direction of movement of the passenger on the image.

[0060] Furthermore, the detection unit 21 may change the threshold for IoU in the NMS processing or Soft NMS processing depending on the position on the image. For example, when a fisheye lens is used as the imaging optical system of the camera 2, the subject appears smaller as the subject approaches the periphery of the image. Therefore, the detection unit 21 may also decrease the threshold for IoU as the subject approaches the periphery of the image. Similarly, when ByteTrack is used as the tracking method, the tracking unit 23 may decrease the threshold for IoU between the predicted position in the next frame of the person region of the passenger being tracked and the person region detected from the image of the next frame as the subject approaches the periphery of the image. Alternatively, the processor 13 may perform preprocessing on each image obtained by the camera 2 to correct distortion due to distortion aberration of the imaging optical system of the camera 2, and then perform the movement detection processing according to the above embodiment or modification.

[0061] According to a modified example of the movement detection device, the detection unit 21 may estimate the position of the trunk of the person instead of estimating the position of the person's feet. The trunk is another example of a predetermined body part. In this case, the classifier used by the detection unit 21 to detect the person may be trained in advance so that only the trunk is included in the person region. In this case, the detection unit 21 determines the position of the trunk as the center of gravity of the person region. In addition, in this case, the tracking unit 23 may determine, as a line connecting the positions of the centers of gravity of the person regions in each image in which the person is detected in chronological order, as the movement trajectory of the detected person's trunk. Then, the determination unit 24 may determine whether the person has moved based on the movement trajectory of the trunk, as in the above embodiment. In this modified example, the movement of the detected person can be accurately detected because not only the distance between the positions of the trunk at the start and end of the predetermined period but also the length along the movement trajectory is referenced.

[0062] The foot position estimation device according to the above embodiment or modification may be used for purposes other than movement detection. For example, the processor 13 may determine whether the estimated foot position of the occupant is included in a corresponding area on the image that corresponds to a predetermined area within the vehicle 1. If the foot position is included in the corresponding area, the processor 13 may determine that the occupant is located within the predetermined area. The predetermined area may be, for example, an area near the entrance / exit of the vehicle 1, into which the occupant is prohibited when the door is opened or closed. In this case, if the processor 13 determines that the occupant has entered the predetermined area, the processor 13 may notify the control unit that controls the opening and closing of the doors of the vehicle 1 via the communication interface 11 that the occupant is near the door. After the notification, if the processor 13 determines that any occupant is outside the predetermined area, the processor 13 may notify the control unit via the communication interface 11 that the occupant has disappeared from near the door. The control unit may then refrain from opening or closing the door from the time it is notified that an occupant is near the door until it is notified that the occupant has left the area near the door.

[0063] The processor 13 may also identify crowded areas in the vehicle 1 based on the estimated foot positions of each occupant. In this case, the area in the vehicle 1 where occupants can be located is divided into multiple sub-areas. A corresponding block on the image is then set for each sub-area. The position and range of the block corresponding to each sub-area may be stored in the memory 12 in advance. The processor 13 identifies blocks containing the estimated foot positions of each occupant detected from the latest image, and counts the number of occupants whose foot positions are contained in each block. The processor 13 then identifies the number of occupants whose foot positions are contained in each block as the number of occupants located in the sub-area corresponding to that block. The processor 13 then identifies a sub-area where the number of occupants reaches a predetermined threshold as a crowded sub-area. This allows the processor 13 to identify crowded and uncrowded sub-areas in the vehicle.

[0064] Furthermore, the foot position estimation device may be used to identify an area where some abnormality exists based on the distribution of the estimated foot positions of each occupant.

[0065] 8 is a functional block diagram of a processor according to a modified example that identifies an area where an abnormality exists based on the results of foot position estimation. The processor 13 according to this modified example has a detection unit 21, an estimation unit 22, and an identification unit 26. Each of these units in the processor 13 is, for example, a functional module realized by a computer program running on the processor 13. Alternatively, each of these units in the processor 13 may be a dedicated arithmetic circuit provided in the processor 13. Below, we will explain the parts of the units in the processor 13 that are different from the above embodiment.

[0066] The detection unit 21 detects a person area from each of a plurality of images generated by the camera 2 capturing images at different capture times. The estimation unit 22 then stores in the memory 12 the foot position of the occupant estimated from the person area detected from each of the plurality of images together with the capture time of the image.

[0067] The identification unit 26 identifies an area where an abnormality exists based on the distribution of occupant foot positions over a recent predetermined period stored in the memory 12. For example, in an area where an occupant is not present for a certain period of time even though it is possible for an occupant to be present inside the vehicle 1, there is a possibility that some abnormality exists that causes the occupant to avoid that area. Therefore, the identification unit 26 identifies such an area where an occupant has not been present for a recent predetermined period of time as an area where an abnormality exists. The predetermined period may be, for example, several tens of minutes to several hours.

[0068] Therefore, similar to the above-described modified example, the area in the vehicle 1 where occupants can stay is divided into a plurality of sub-areas. Then, a corresponding block on the image is set for each sub-area. The position and range of the block corresponding to each sub-area may be stored in advance in the memory 12. The identification unit 26 identifies blocks including the foot positions of each occupant detected from each image stored in the memory 12, the image being captured within the most recent predetermined period. Thereby, the identification unit 26 counts the number of foot positions included in each block. Then, the identification unit 26 identifies a sub-area corresponding to a block in which the number of foot positions is less than a predetermined detection threshold (e.g., 1 to 3) as an area where some abnormality exists.

[0069] When an area where an abnormality exists is identified, the identification unit 26 transmits information indicating the position and range of the identified area and identification information of the vehicle 1 to a server (not shown) that manages the vehicle 1 via the communication interface 11 and a wireless communication terminal (not shown) mounted on the vehicle 1. Alternatively, the identification unit 26 may notify a control unit that controls the driving of the vehicle 1 of the information indicating the position and range of the identified area via the communication interface 11. In this case, when the control unit is notified that an area where an abnormality exists has been identified, the control unit may have the occupants of the vehicle 1 disembark at a predetermined location and then drive the vehicle 1 to a location where maintenance of the vehicle 1 will be performed.

[0070] According to this modification, the foot position estimating device can identify an area where some abnormality exists by utilizing the estimation results of the foot positions of each occupant.

[0071] A computer program that causes a computer to execute the processing executed by the processor 13 of the foot position estimation device 4 according to the above embodiment or modification may be recorded on a recording medium such as an optical recording medium or a magnetic recording medium and distributed.

[0072] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]

[0073] REFERENCE SIGNS LIST 1 vehicle, 2 camera, 3 notification device, 4 foot position estimation device, 10 system, 11 communication interface, 12 memory, 13 processor, 21 detection unit, 22 estimation unit, 23 tracking unit, 24 determination unit, 25 notification processing unit, 26 identification unit

Claims

1. a detection unit that detects a person area depicting a person within a predetermined area from an image generated by an imaging unit that is provided to capture the predetermined area; an estimation unit that estimates an intersection point between a line extending from a reference point within the person area toward a vanishing point in the image and an edge of the person area as a foot position of the person; A foot position estimation device having the above configuration.

2. 2. The foot position estimation device according to claim 1, wherein the estimation unit corrects the foot position so that the foot position is a position along the line from the intersection point toward the reference point by a correction distance that is preset according to the distance between the vanishing point and the reference.

3. 2. The foot position estimation device according to claim 1, wherein, when the vanishing point is present within the person area, the estimation unit estimates as the foot position a position obtained by dividing the distance between the reference point and the vanishing point by a ratio of the distance between the reference point and the vanishing point to the distance between the intersection of the edge of the person area closer to the vanishing point on the line and the line and the reference point.

4. the detection unit detects the person region from each of the plurality of images obtained in time series over a predetermined period of time; the estimation unit estimates a foot position of the person in each of the plurality of images; a tracking unit that tracks the person detected in the plurality of images to obtain a movement trajectory of the foot position of the person during the predetermined period; a determination unit that determines that the detected person has moved during the predetermined period if a distance between the foot position of the detected person at the start of the predetermined period and the foot position of the detected person at the end of the predetermined period is equal to or greater than a first threshold and a length of the movement trajectory from the start to the end of the predetermined period is equal to or greater than a second threshold; The foot position estimation device according to any one of claims 1 to 3, further comprising:

5. the detection unit detects the person area from each of the plurality of images captured at different times; the estimation unit records the foot positions estimated from each of the plurality of images in a storage unit; 4. The foot position estimation device according to claim 1, further comprising an identification unit that identifies a position where some abnormality exists based on the distribution of the foot positions stored in the storage unit.

6. detecting a person area depicting a person within a predetermined area from an image generated by an imaging unit provided to capture the predetermined area; an intersection of a line extending from a reference point within the person area to a vanishing point in the image with an edge of the person area is estimated as the foot position of the person; A foot position estimation method including:

7. an imaging unit provided to capture an image of a predetermined area; a foot position estimation device that estimates the foot positions of a person located within the predetermined area; and The foot position estimation device a detection unit that detects a person area representing a person within the predetermined area from the image generated by the imaging unit; an estimation unit that estimates an intersection point between a line extending from a reference point within the person area toward a vanishing point in the image and an edge of the person area as a foot position of the person; A foot position estimation system having the above configuration.

8. detecting a person area depicting a person within a predetermined area from an image generated by an imaging unit provided to capture the predetermined area; an intersection of a line extending from a reference point within the person area to a vanishing point in the image with an edge of the person area is estimated as the foot position of the person; A computer program for estimating foot position that causes a computer to execute the above.

9. a detection unit that detects a person within a predetermined area from each of a plurality of time-series images generated by an imaging unit that is configured to capture the predetermined area, and estimates a position of a predetermined part of the person on one or more images in which the detected person is depicted; a tracking unit that tracks the person detected in the one or more images to obtain a movement trajectory of the position of the predetermined part of the person for a predetermined period of time; a determination unit that determines that the detected person has moved during the predetermined period if a distance between a position of the predetermined part of the detected person at the start of the predetermined period and a position of the predetermined part of the detected person at the end of the predetermined period is equal to or greater than a first threshold and a length of the movement trajectory from the start to the end of the predetermined period is equal to or greater than a second threshold; A movement detection device having the following.

Images