Vehicle departure readiness determination device
The vehicle departure determination device uses person and skeleton detection with movement thresholds to accurately assess passenger movement, addressing inaccuracies in existing systems and enhancing safety.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- TOYOTA JIDOSHA KK
- Filing Date
- 2023-08-31
- Publication Date
- 2026-05-19
AI Technical Summary
Existing systems inaccurately determine vehicle departure readiness due to misdetection of passenger skeletons, leading to potential safety hazards.
A vehicle departure determination device that combines person detection, skeleton detection, and movement calculation to accurately assess passenger movement, using thresholds to permit or prohibit vehicle departure based on both methods.
Enhances the accuracy of determining vehicle departure readiness by minimizing false negatives from skeleton misdetection, thereby ensuring passenger safety.
Smart Images
Figure 0007861738000001 
Figure 0007861738000002 
Figure 0007861738000003
Abstract
Description
Technical Field
[0001] The present disclosure relates to an apparatus for determining whether a vehicle can be started.
Background Art
[0002] A technique is known in which a person is photographed by a camera, a skeleton of the person is detected from an image generated by the photographing, and a posture of the person is determined.
[0003] In order to ensure the safety of passengers in a vehicle such as a bus, a skeleton detection technique may be used to determine whether the vehicle can be started. Specifically, a passenger in the vehicle is photographed by a camera. Before the vehicle starts, a skeleton of the photographed passenger is detected, and based on a movement amount of the detected skeleton, it is determined whether the vehicle can be started. For example, if it is determined that a passenger is in the middle of a standing-up motion or if it is determined that a passenger is in the middle of a sitting-down motion, the vehicle start is prohibited from the viewpoint of safety.
[0004] Patent Document 1 describes an apparatus that detects a person inside a vehicle by a sensor and determines the safety inside the vehicle according to the detection result.
Prior Art Documents
Patent Documents
[0005]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0006] By the way, if it is determined whether the vehicle can be started only based on skeleton detection, when a misdetection of the skeleton occurs, it is impossible to accurately determine whether the vehicle can be started.
[0007] The purpose of this disclosure is to accurately determine whether a vehicle can depart, compared to the case where only the skeletons of the vehicle's passengers are detected to determine whether the vehicle can depart. [Means for solving the problem]
[0008] One aspect of the present disclosure includes: a person detection unit for detecting a person inside a vehicle; a skeleton detection unit for detecting the skeleton of a person detected by the person detection unit; a calculation unit for calculating the amount of movement of the skeleton detected by the skeleton detection unit; and a determination unit that prohibits the departure of the vehicle if the amount of movement of the skeleton is greater than or equal to a first threshold, and determines whether or not the vehicle may depart based on the detection result by the person detection unit if the amount of movement of the skeleton is less than the first threshold. The calculation unit further calculates the amount of movement of the person detected by the person detection unit, and the determination unit prohibits the departure of the vehicle if the amount of movement of the skeleton is less than the first threshold and the amount of movement of the person is equal to or greater than the second threshold, and permits the departure of the vehicle if the amount of movement of the skeleton is less than the first threshold and the amount of movement of the person is less than the second threshold. This is a vehicle departure determination device characterized by the following: [Effects of the Invention]
[0009] According to this disclosure, it is possible to determine whether or not the vehicle can depart more accurately compared to the case where only the skeletons of the vehicle's passengers are detected to determine whether or not the vehicle can depart. [Brief explanation of the drawing]
[0010] [Figure 1] This is a block diagram showing the configuration of the determination device according to the embodiment. [Figure 2] This is a diagram showing an image of the interior of a car. [Figure 3] This is a diagram to explain human detection and skeleton detection. [Figure 4] This is a diagram illustrating the movement of detected individuals. [Modes for carrying out the invention]
[0011] Referring to Figure 1, the configuration of the determination device 10 according to this embodiment will be described. Figure 1 is a block diagram showing the configuration of the determination device 10.
[0012] The determination device 10 is an example of a device for determining whether a vehicle 12 can depart. When a vehicle 12, such as a bus, is stopped, the determination device 10 analyzes an image representing the inside of the vehicle 12 to determine whether the vehicle 12 can depart.
[0013] A camera 14 is installed inside the vehicle 12. The camera 14 takes pictures of the inside of the vehicle 12 and generates images. For example, a fisheye lens is used for the camera 14. The images are output to the determination device 10. The determination device 10 analyzes the images to determine whether or not the vehicle 12 is allowed to depart.
[0014] The determination device 10 includes a human detection unit 16, a skeleton detection unit 18, a calculation unit 20, a determination unit 22, and a control unit 24.
[0015] The person detection unit 16 detects people inside the vehicle 12 by analyzing images captured by the camera 14. Known techniques are used for detecting people. For example, the person detection unit 16 sets bounding boxes (regions surrounding objects represented in the image) in the image and performs classification of the class of objects represented within the bounding boxes. The person detection unit 16 detects people inside the vehicle 12 by determining whether or not the object represented within the bounding box is a person.
[0016] The skeleton detection unit 18 detects the human skeleton represented in the image by analyzing the image captured by the camera 14. The skeleton detection unit 18 detects the human skeleton detected by the human detection unit 16. Known techniques such as Open Pose are used as the technique for detecting the skeleton. For example, the skeleton detection unit 18 extracts features from the image, estimates the positions that appear to be joint points of the human body based on the features, and estimates the affinity between the joint points.
[0017] The calculation unit 20 calculates the amount of movement of the person detected by the person detection unit 16. For example, the calculation unit 20 calculates the amount of movement of the bounding box surrounding the same person as the amount of movement of the person. Specifically, the calculation unit 20 associates an ID with each bounding box. The calculation unit 20 calculates the amount of movement of the bounding boxes associated with the same ID as the amount of movement of the person surrounded by the bounding box.
[0018] In addition, the calculation unit 20 calculates the amount of movement of the skeleton detected by the skeleton detection unit 18. That is, the calculation unit 20 calculates the amount of movement of each joint point of the human body. As a technique for calculating the amount of movement of the skeleton, a known technique such as optical flow is used. Optical flow is a technique for detecting the movement of an object between frames and displaying the speed of the movement as a vector. For example, the block matching method is used as the optical flow. The block matching method is a technique for detecting the movement of an object by dividing an image into a plurality of regions and detecting regions with high similarity between frames.
[0019] The determination unit 22 determines whether the vehicle 12 can start based on the detection result of the person detection unit 16 and the detection result of the skeleton detection unit 18 when the vehicle 12 is stopped.
[0020] For example, when the amount of movement of the skeleton detected by the skeleton detection unit 18 is equal to or greater than the first threshold value, the determination unit 22 determines whether the vehicle 12 can start based on the detected amount of movement of the skeleton. Specifically, when the detected amount of movement of the skeleton is equal to or greater than the first threshold value, the determination unit 22 prohibits the vehicle 12 from starting. The first threshold value is set in advance.
[0021] When the amount of movement of the skeleton is equal to or greater than the first threshold value, there is a possibility that the passenger is moving. Even if the passenger is not moving, there is a possibility that the passenger is in the process of standing up or sitting down at the same place. If the vehicle 12 starts in such a situation, the passenger may fall. To prevent this, when the amount of movement of the skeleton is equal to or greater than the first threshold value, the determination unit 22 prohibits the vehicle 12 from starting.
[0022] When the detected movement amount of the skeleton is less than the first threshold value, the determination unit 22 determines whether the vehicle 12 can depart based on the detection result by the person detection unit 16. Specifically, when the detected movement amount of the person is greater than or equal to the second threshold value, the determination unit 22 prohibits the departure of the vehicle 12. The second threshold value is set in advance.
[0023] Even when the passenger is in the middle of moving, standing up, or sitting, due to misdetection of the skeleton, the detected movement amount of the skeleton may be less than the first threshold value. In this case, the determination unit 22 determines the presence or absence of passenger movement by a person detection method different from the skeleton detection method. When the movement amount of the passenger is greater than or equal to the second threshold value, since there is a possibility that the passenger is moving, the determination unit 22 prohibits the departure of the vehicle 12. By doing so, even when misdetection of the skeleton occurs, the departure of the vehicle 12 can be prohibited.
[0024] When the movement amount of the skeleton is less than the first threshold value and the movement amount of the person is less than the second threshold value, the determination unit 22 permits the departure of the vehicle 12. When the movement amounts calculated by different detection methods are less than the threshold values, since the possibility that the person is moving is low, the determination unit 22 permits the departure of the vehicle 12.
[0025] [[ID=,13]]The control unit 24 controls the departure of the vehicle 12 based on the determination result of the determination unit 22. When the determination unit 22 permits the departure of the vehicle 12 and the driver instructs departure, the control unit 24 causes the vehicle 12 to depart. When the determination unit 22 prohibits the departure of the vehicle 12, even if the driver instructs departure, the control unit 24 does not cause the vehicle 12 to depart.
[0026] When the departure of the vehicle 12 is prohibited, the control unit 24 may generate a warning sound from a speaker provided in the vehicle 12, or may display a warning message on a display provided in the vehicle 12.
[0027] The following will explain specific examples with reference to Figures 2 to 4. Figure 2 is a diagram showing an image representing the interior of a vehicle. Figure 3 is a diagram to explain person detection and skeleton detection. Figure 4 is a diagram to explain the amount of movement of the detected person.
[0028] Image 26 shown in Figure 2 is an image taken by camera 14. For example, vehicle 12 is a bus, and the interior 28 and automatic door 30 of the bus are shown in image 26. Passengers A to D are on board vehicle 12 and are shown in image 26. Camera 14 is a fisheye lens, and a wide area of the interior 28 is shown in image 26.
[0029] Refer to Figure 3 to explain human detection and skeleton detection. Figures 3(a) to (c) show images of passenger A. Figure 3(a) shows an image taken at time t1. Figure 3(b) shows an image taken at time t2. Figure 3(c) shows an image taken at time t3. Time t2 is later than time t1, and time t3 is later than time t2.
[0030] The person detection unit 16 sets a bounding box 32 in the area where passenger A is represented in image 26, and determines that the object within the bounding box 32 is passenger A, who is a person. When passenger A moves their hands or feet, the size of the area occupied by passenger A changes, and as shown in Figures 3(b) and (c), the size of the bounding box 32 also changes.
[0031] The skeletal detection unit 18 detects joint points S1 to S17 of passenger A shown in image 26. Joint point S1 is the center of the face, joint point S2 is the joint of the right ear, joint point S3 is the joint of the left ear. Joint point S4 is the joint of the neck. Joint point S5 is the joint of the right shoulder, joint point S6 is the joint of the right elbow, joint point S7 is the joint of the right wrist. Joint point S8 is the joint of the left shoulder, joint point S9 is the joint of the left elbow, and joint point S10 is the joint of the left wrist. Joint point S11 is the center of the waist. Joint point S12 is the joint on the right side of the waist, joint point S13 is the joint of the right knee, and joint point S14 is the joint of the right ankle. Articular point S15 is the articular point on the left side of the hip, articular point S16 is the articular point of the left knee, and articular point S17 is the articular point of the left ankle.
[0032] The optical flow 34 shown in Figure 3(b) is the optical flow at time t2, and represents the velocity of passenger A's movement between time t1 and time t2 as a vector.
[0033] The optical flow 36 shown in Figure 3(c) is the optical flow at time t3, and represents the velocity of passenger A's movement between time t2 and time t3 as a vector.
[0034] The arrows in optical flows 34 and 36 are vectors representing the velocity of movement. Between time t1 and time t2, passenger A raises both arms. Therefore, in optical flow 34, an upward-pointing vector is shown in the part corresponding to both arms. Between time t2 and time t3, passenger A raises both arms even further. Therefore, in optical flow 36, an upward-pointing vector is shown in the part corresponding to both arms. The parts where vectors are shown are hatched according to the velocity of movement.
[0035] For example, if the amount of movement of at least one of the joint points S1 to S17 is greater than or equal to a first threshold, the determination unit 22 prohibits the departure of the vehicle 12. In this case, if the amount of movement of at least one of the passengers A to D is greater than or equal to the first threshold, the determination unit 22 prohibits the departure of the vehicle 12. The determination unit 22 may also prohibit the departure of the vehicle 12 if the number of passengers whose amount of movement of at least one joint point is greater than or equal to the first threshold is greater than or equal to a threshold.
[0036] The determination unit 22 may prohibit the departure of the vehicle 12 if the amount of movement of a specific skeletal structure is greater than or equal to a first threshold. For example, the specific skeletal structure may be the waist or shoulders. The determination unit 22 prohibits the departure of the vehicle 12 if the amount of movement of the joint points constituting the waist or shoulders is greater than or equal to a first threshold. If the waist or shoulders are moving, the passenger may be performing a standing or sitting motion, so the determination unit 22 prohibits the departure of the vehicle 12 to prevent falls.
[0037] If the amount of movement of multiple joint points among all joint points S1 to S17 (for example, the amount of movement of a number of joint points equal to or greater than a threshold) is equal to or greater than a first threshold, the determination unit 22 may prohibit the departure of the vehicle 12.
[0038] The calculation unit 20 may calculate the amount of movement of each skeleton by applying different weighting processes to each skeleton. Specifically, the calculation unit 20 applies a weight to the amount of movement of each joint point according to that joint point. For example, the movement of the waist and shoulders is suitable for determining whether a person is standing up or sitting down. In other words, the movement of the waist and shoulders is suitable for determining safety. Therefore, the calculation unit 20 makes the weight of the waist joint point and the shoulder joint point heavier than the weight of the other joint points. For example, the calculation unit 20 makes the weighting coefficient for the waist joint point and the shoulder joint point larger than the weighting coefficient for the other joint points. The calculation unit 20 applies a weighting process to the amount of movement of each joint point, and the determination unit 22 determines whether the vehicle 12 can start based on the comparison result between the weighted amount of movement and the first threshold. In this way, the determination of whether the vehicle 12 can start can be made by giving importance to the amount of movement of skeletons suitable for safety determination.
[0039] Referring to Figure 4, the amount of human movement detected by the human detection unit 16 will be explained.
[0040] Figure 4(a) shows images of passengers A, B, and C taken by camera 14 at time t4. Bounding box 38 is set for passenger A's image. Bounding box 40 is set for passenger B's image. Bounding box 42 is set for passenger C's image.
[0041] Figure 4(b) shows images of passengers A, B, and C taken by camera 14 at time t5. Time t5 is later than time t4. Bounding box 44 is set for passenger A's image. Bounding box 46 is set for passenger B's image. Bounding box 48 is set for passenger C's image. In Figure 4(b), bounding boxes 38, 40, and 42 are shown as dashed lines.
[0042] Between time t4 and time t5, passengers A, B, and C move, and therefore each passenger's bounding box also moves.
[0043] The calculation unit 20 calculates the difference in the position of each bounding box as the amount of passenger movement. In the example shown in Figure 4(b), the calculation unit 20 calculates the difference between the position of bounding box 38 and the position of bounding box 44 as the amount of passenger A's movement between time t4 and time t5. The calculation unit 20 calculates the difference between the position of bounding box 40 and the position of bounding box 46 as the amount of passenger B's movement between time t4 and time t5. The calculation unit 20 calculates the difference between the position of bounding box 42 and the position of bounding box 48 as the amount of passenger C's movement between time t4 and time t5.
[0044] For example, if the amount of skeletal movement of all passengers is less than the first threshold, the determination unit 22 determines whether or not the vehicle 12 can depart based on the amount of passenger movement.
[0045] The determination unit 22 prohibits the departure of the vehicle 12 if the movement of at least one person is equal to or greater than the second threshold. In the example shown in Figure 4(b), the movement of passengers A and B is equal to or greater than the second threshold, while the movement of passenger C is less than the second threshold. Therefore, the determination unit 22 prohibits the departure of the vehicle 12. The determination unit 22 may also prohibit the departure of the vehicle 12 if the movement of multiple passengers is equal to or greater than the second threshold.
[0046] As described above, according to this embodiment, if the amount of skeletal movement is greater than or equal to the first threshold, there is a possibility that a person is moving, and therefore the departure of the vehicle 12 is prohibited. Even if a person is moving, the amount of skeletal movement may not be greater than or equal to the first threshold due to false detection of the skeleton. In this case, the presence or absence of human movement is determined based on the amount of human movement detected by the human detection unit 16. This makes it possible to determine whether or not to depart more accurately compared to the case where departure is determined solely by skeleton detection.
[0047] The determination unit 22 may determine whether or not the vehicle 12 can depart at each unit time interval. If the number of consecutive times it is determined to be safe (i.e., the number of consecutive times it is determined that the amount of skeletal movement is less than the first threshold and the amount of human movement is less than the second threshold) is equal to or greater than a preset threshold, the determination unit 22 permits the vehicle 12 to depart. Otherwise, the determination unit 22 prohibits the vehicle 12 from departing. The length of the unit time interval is preset.
[0048] If the length of time during which it is determined to be safe for an extended period (i.e., the length of time during which the amount of skeletal movement is less than the first threshold and the amount of human movement is less than the second threshold for an extended period) is equal to or greater than a preset threshold, the determination unit 22 may permit the departure of the vehicle 12. Otherwise, the determination unit 22 prohibits the departure of the vehicle 12.
[0049] All or part of the determination device 10 can be implemented using hardware resources such as a processor and electronic circuits, and devices such as memory may be used as needed in its implementation. Alternatively, all or part of the determination device 10 may be implemented by a computer. In other words, all or part of the determination device 10 may be implemented through the cooperation of hardware resources such as a CPU (Central Processing Unit) and memory provided by a computer, and software (programs) that define the operation of the CPU, etc. This program is stored in the storage device of the determination device 10 via a recording medium such as a CD or DVD, or via a communication path such as a network. As another example, all or part of the determination device 10 may be implemented by a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), or FPGA (Field Programmable Gate Array), etc. Of course, a GPU (Graphics Processing Unit), etc., may also be used.
[0050] The functions of the determination device 10 may be implemented by a single device or by multiple devices.
[0051] The determination device 10 may be installed in the vehicle 12 or in an external device such as a server. If the determination device 10 is installed in an external device, the determination device 10 communicates with the vehicle 12 via a communication path such as the internet to perform person detection, skeleton detection, and control of the vehicle 12's departure. [Explanation of symbols]
[0052] 10 Judgment device, 12 Vehicle, 14 Camera, 16 Person detection unit, 18 Skeleton detection unit, 20 Calculation unit, 22 Judgment unit, 24 Control unit.
Claims
[Claim 1] A human detection unit that detects people inside the vehicle, A skeleton detection unit that detects the skeleton of a person detected by the person detection unit, A calculation unit that calculates the amount of skeleton movement detected by the skeleton detection unit, If the amount of movement of the frame is greater than or equal to a first threshold, the vehicle's departure is prohibited; if the amount of movement of the frame is less than the first threshold, the determination unit determines whether or not the vehicle's departure is permitted based on the detection result by the human detection unit. It has, The calculation unit further calculates the amount of movement of the person detected by the person detection unit, The determination unit prohibits the departure of the vehicle if the amount of movement of the skeleton is less than the first threshold and the amount of movement of the person is equal to or greater than the second threshold, and permits the departure of the vehicle if the amount of movement of the skeleton is less than the first threshold and the amount of movement of the person is less than the second threshold. A vehicle departure determination device characterized by the following: