Propriety determination device for vehicle departure

Abstract

To more accurately determine propriety of departure of a vehicle as compared with a case of determining the propriety of the departure of the vehicle by detecting only the skeleton of a passenger of the vehicle.SOLUTION: A person detection unit 16 detects a person inside a vehicle 12. A skeleton detection unit 18 detects a skeleton of the person detected by the person detection unit 16. A calculation unit 20 calculates a movement amount of the skeleton detected by the skeleton detection unit 18. When the movement amount of the skeleton is equal to or greater than a first threshold value, a determination unit 22 prohibits departure of the vehicle 12. When the movement amount of the skeleton is less than the first threshold value, the determination unit 22 determines whether the vehicle 12 is allowed to depart on the basis of a detection result of the person detection unit 16.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an apparatus for determining whether a vehicle is ready to depart. [Background technology]

[0002] 2. Description of the Related Art There is known a technique for photographing a person with a camera, detecting the person's skeleton from an image generated by the photograph, and determining the posture of the person.

[0003] In order to ensure the safety of passengers in vehicles such as buses, skeletal detection technology is sometimes used to determine whether the vehicle can depart. Specifically, passengers in the vehicle are photographed by a camera. Before the vehicle departs, the photographed passenger skeletons are detected, and whether the vehicle can depart is determined based on the amount of movement of the detected skeleton. For example, if it is determined that a passenger is in the middle of standing up or sitting down, the vehicle is prohibited from departing from the viewpoint of safety.

[0004] Patent Document 1 describes an apparatus that detects people inside a vehicle using a sensor and determines the safety inside the vehicle based on the detection result. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-199123 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, if a determination is made as to whether or not a vehicle can depart based solely on frame detection, in the event of an erroneous frame detection, it is not possible to accurately determine whether or not the vehicle can depart.

[0007] An object of the present disclosure is to accurately determine whether or not a vehicle can depart, compared to a case in which only the skeletons of vehicle passengers are detected to determine whether or not the vehicle can depart. [Means for solving the problem]

[0008] One aspect of the present disclosure is a vehicle departure feasibility determination device comprising a human detection unit that detects a person inside a vehicle, a skeleton detection unit that detects the skeleton of a person detected by the human detection unit, a calculation unit that calculates the amount of skeleton movement detected by the skeleton detection unit, and a determination unit that prohibits the vehicle from departing if the amount of skeleton movement is greater than or equal to a first threshold, and determines whether the vehicle can depart based on the detection result by the human detection unit if the amount of skeleton movement is less than the first threshold. Effect of the Invention

[0009] According to the present disclosure, it is possible to more accurately determine whether or not a vehicle can depart, compared to a case in which only the skeletons of vehicle passengers are detected to determine whether or not the vehicle can depart. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a configuration of a determination device according to an embodiment; [Diagram 2] FIG. 1 is a diagram showing an image showing the interior of a vehicle. [Diagram 3] FIG. 1 is a diagram for explaining human detection and skeletal detection. [Figure 4] FIG. 11 is a diagram for explaining the amount of movement of a detected person. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The configuration of a determination device 10 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the determination device 10.

[0012] The determination device 10 is an example of a device for determining whether or not the vehicle 12 is permitted to depart. The determination device 10 determines whether or not the vehicle 12 is permitted to depart by analyzing an image showing the interior of the vehicle 12 when the vehicle 12, such as a bus, is stopped.

[0013] A camera 14 is provided inside the vehicle 12. The camera 14 takes an image of the inside of the vehicle 12 and generates an image. For example, a fisheye lens is used as the camera 14. The image is output to the determination device 10. The determination device 10 analyzes the image to determine whether or not the vehicle 12 is allowed to depart.

[0014] The determination device 10 includes a human detection unit 16, a skeleton detection unit 18, a calculation unit 20, a determination unit 22, and a control unit 24.

[0015] The human detection unit 16 detects a person in the vehicle 12 by analyzing the image captured by the camera 14. A known technology is used to detect a person. For example, the human detection unit 16 sets a bounding box (a region surrounding an object shown in the image) in the image and performs class classification of the object shown in the bounding box. The human detection unit 16 detects a person in the vehicle 12 by determining whether the object shown in the bounding box is a person or not.

[0016] The skeleton detection unit 18 detects the skeleton of a person depicted in an image by analyzing the image captured by the camera 14. The skeleton detection unit 18 detects the skeleton of a person detected by the human detection unit 16. A known technique such as Open Pose is used as a technique for detecting a skeleton. For example, the skeleton detection unit 18 extracts feature amounts from the image, estimates positions likely to be joint points of a human body based on the feature amounts, and estimates the association (affinity) between the joint points.

[0017] The calculation unit 20 calculates the amount of movement of a person detected by the person detection unit 16. For example, the calculation unit 20 calculates the amount of movement of a bounding box surrounding the same person as the amount of movement of the person. Specifically, the calculation unit 20 associates an ID with each bounding box. The calculation unit 20 calculates the amount of movement of the bounding box associated with the same ID as the amount of movement of the person surrounded by the bounding box.

[0018] Furthermore, the calculation unit 20 calculates the amount of movement of the skeleton detected by the skeleton detection unit 18. That is, the calculation unit 20 calculates the amount of movement of each joint point of the human body. A known technique such as optical flow is used to calculate the amount of movement of the skeleton. Optical flow is a technique for detecting the movement of an object between frames and displaying the speed of the movement as a vector. For example, a block matching method is used as the optical flow. The block matching method is a technique for detecting the movement of an object by dividing an image into multiple regions and detecting regions with high similarity between frames.

[0019] When the vehicle 12 is stopped, the determination unit 22 determines whether or not the vehicle 12 can depart based on the detection results of the human detection unit 16 and the detection results of the skeleton detection unit 18.

[0020] For example, when the amount of movement of the skeleton detected by the skeleton detection unit 18 is equal to or greater than a first threshold, the determination unit 22 determines whether or not the vehicle 12 is allowed to depart based on the amount of movement of the skeleton detected. Specifically, when the amount of movement of the skeleton detected is equal to or greater than a first threshold, the determination unit 22 prohibits the vehicle 12 from departing. The first threshold is set in advance.

[0021] When the amount of movement of the skeleton is equal to or greater than the first threshold, the passenger may be moving. Even if the passenger is not moving, the passenger may be standing up or sitting down in the same place. If the vehicle 12 departs in such a situation, the passenger may fall. To prevent this, when the amount of movement of the skeleton is equal to or greater than the first threshold, the determination unit 22 prohibits the vehicle 12 from departing.

[0022] When the detected amount of movement of the skeleton is less than the first threshold, the determination unit 22 determines whether or not the vehicle 12 is allowed to depart based on the detection result by the human detection unit 16. Specifically, when the detected amount of movement of the person is equal to or greater than the second threshold, the determination unit 22 prohibits the vehicle 12 from departing. The second threshold is set in advance.

[0023] Even when a passenger is moving, standing up, or sitting, the amount of skeleton movement may be less than the first threshold due to skeletal erroneous detection. In this case, the determination unit 22 determines whether or not the passenger has moved using a human detection method different from the skeleton detection method. If the amount of passenger movement is equal to or greater than the second threshold, the determination unit 22 prohibits the departure of the vehicle 12 since there is a possibility that the passenger is moving. In this way, even if skeletal erroneous detection occurs, the departure of the vehicle 12 can be prohibited.

[0024] When the amount of movement of the skeleton is less than the first threshold and the amount of movement of the person is less than the second threshold, the determination unit 22 permits the departure of the vehicle 12. When the amounts of movement calculated by the different detection methods are less than the thresholds, the possibility that a person is moving is low, so the determination unit 22 permits the departure of the vehicle 12.

[0025] The control unit 24 controls the departure of the vehicle 12 based on the determination result of the determination unit 22. When the determination unit 22 permits the departure of the vehicle 12, and the driver instructs the vehicle to depart, the control unit 24 causes the vehicle 12 to depart. When the determination unit 22 prohibits the vehicle 12 from depart, the control unit 24 does not cause the vehicle 12 to depart even if the driver instructs the vehicle to depart.

[0026] If the departure of vehicle 12 is prohibited, control unit 24 may generate a warning sound from a speaker provided in vehicle 12, or may display a warning message on a display provided in vehicle 12.

[0027] A specific example will be described below with reference to Fig. 2 to Fig. 4. Fig. 2 is a diagram showing an image of the interior of a vehicle. Fig. 3 is a diagram for explaining human detection and skeleton detection. Fig. 4 is a diagram for explaining the amount of movement of a detected person.

[0028] 2 is an image captured by the camera 14. For example, the vehicle 12 is a bus, and the interior 28 and automatic doors 30 of the bus are shown in the image 26. Passengers A to D are aboard the vehicle 12 and are shown in the image 26. The camera 14 has a fisheye lens, and a wide range of the interior 28 is captured in the image 26.

[0029] Human detection and skeleton detection will be described with reference to Fig. 3. Fig. 3(a) to (c) show images of passenger A. Fig. 3(a) shows an image taken at time t1. Fig. 3(b) shows an image taken at time t2. Fig. 3(c) shows an image taken at time t3. Time t2 is a time later than time t1, and time t3 is a time later than time t2.

[0030] The human detection unit 16 sets a bounding box 32 in the area in the image 26 where passenger A is depicted, and determines that the object within the bounding box 32 is passenger A. When passenger A moves his or her hands or feet, the size of the area occupied by passenger A changes, and so the size of the bounding box 32 also changes, as shown in Figures 3(b) and (c).

[0031] The skeleton detection unit 18 detects joint points S1 to S17 of the passenger A shown in the image 26. The joint point S1 is the center of the face, the joint point S2 is the joint point of the right ear, and the joint point S3 is the joint point of the left ear. The joint point S4 is the joint point of the neck. The joint point S5 is the joint point of the right shoulder, the joint point S6 is the joint point of the right elbow, and the joint point S7 is the joint point of the right wrist. The joint point S8 is the joint point of the left shoulder, the joint point S9 is the joint point of the left elbow, and the joint point S10 is the joint point of the left wrist. The joint point S11 is the center of the waist. The joint point S12 is the joint point on the right side of the waist, the joint point S13 is the joint point of the right knee, and the joint point S14 is the joint point of the right ankle. The joint point S15 is a left hip joint point, the joint point S16 is a left knee joint point, and the joint point S17 is a left ankle joint point.

[0032] The optical flow 34 shown in FIG. 3(b) is the optical flow at time t2, and represents the speed of the movement of passenger A from time t1 to time t2 as a vector.

[0033] The optical flow 36 shown in FIG. 3(c) is the optical flow at time t3, and represents the speed of the movement of passenger A from time t2 to time t3 as a vector.

[0034] The arrows in the optical flows 34 and 36 are vectors that indicate the speed of movement. Between time t1 and time t2, passenger A raises both arms. Therefore, in the optical flow 34, an upward-pointing vector is shown in the portion corresponding to both arms. Between time t2 and time t3, passenger A raises both arms again. Therefore, in the optical flow 36, an upward-pointing vector is shown in the portion corresponding to both arms. The portion showing the vector is hatched according to the speed of movement.

[0035] For example, when the amount of movement of at least one of all the joint points S1 to S17 is equal to or greater than the first threshold, the determination unit 22 prohibits the departure of the vehicle 12. In this case, when the amount of movement of at least one joint point of even one of the passengers A to D is equal to or greater than the first threshold, the determination unit 22 prohibits the departure of the vehicle 12. When the number of passengers whose amount of movement of at least one joint point is equal to or greater than the first threshold becomes equal to or greater than a threshold, the determination unit 22 may prohibit the departure of the vehicle 12.

[0036] When the amount of movement of a specific skeleton is equal to or greater than a first threshold, the determination unit 22 may prohibit the departure of the vehicle 12. For example, the specific skeleton is a waist or a shoulder. When the amount of movement of a joint point constituting the waist or a shoulder is equal to or greater than a first threshold, the determination unit 22 prohibits the departure of the vehicle 12. If the waist or shoulder is moving, the passenger may be standing up or sitting down, so in order to prevent the passenger from falling over, the determination unit 22 prohibits the departure of the vehicle 12.

[0037] When the movement amounts of a plurality of joint points among all the joint points S1 to S17 (for example, the movement amounts of a number of joint points equal to or greater than a threshold value) are equal to or greater than a first threshold value, the determination unit 22 may prohibit the vehicle 12 from starting.

[0038] The calculation unit 20 may calculate the movement amount of each skeleton by performing a different weighting process for each skeleton. Specifically, the calculation unit 20 performs weighting for each joint point on the movement amount of the joint point according to the joint point. For example, the movement of the waist and shoulders is a skeleton suitable for determining whether or not a person has stood up or sat down. In other words, the movement of the waist and shoulders is a skeleton suitable for determining safety. Therefore, the calculation unit 20 weights the waist joint point and the shoulder joint point heavier than the weights for the other joint points. For example, the calculation unit 20 makes the weighting coefficient for the waist joint point and the shoulder joint point larger than the weighting coefficient for the other joint points. The calculation unit 20 performs weighting process on the movement amount of the joint point for each joint point, and the determination unit 22 determines whether or not the vehicle 12 can depart based on the result of comparing the movement amount after the weighting process with the first threshold value. In this way, it is possible to determine whether or not the vehicle 12 can depart by placing importance on the movement amount of the skeleton suitable for safety determination.

[0039] The amount of movement of a person detected by the person detection unit 16 will be described with reference to FIG.

[0040] 4(a) shows images of passengers A, B, and C captured by camera 14 at time t4. A bounding box 38 has been established in the image of passenger A. A bounding box 40 has been established in the image of passenger B. A bounding box 42 has been established in the image of passenger C.

[0041] FIG. 4(b) shows images of passengers A, B, and C captured by camera 14 at time t5. Time t5 is a time after time t4. A bounding box 44 has been established in the image of passenger A. A bounding box 46 has been established in the image of passenger B. A bounding box 48 has been established in the image of passenger C. In FIG. 4(b), bounding boxes 38, 40, and 42 are shown with dashed lines.

[0042] Between time t4 and time t5, passengers A, B, and C move, so the bounding boxes of each passenger also move.

[0043] The calculation unit 20 calculates the difference in the position of each bounding box as the movement amount of the passenger. In the example shown in FIG. 4(b), the calculation unit 20 calculates the difference between the position of the bounding box 38 and the position of the bounding box 44 as the movement amount of passenger A between time t4 and time t5. The calculation unit 20 calculates the difference between the position of the bounding box 40 and the position of the bounding box 46 as the movement amount of passenger B between time t4 and time t5. The calculation unit 20 calculates the difference between the position of the bounding box 42 and the position of the bounding box 48 as the movement amount of passenger C between time t4 and time t5.

[0044] For example, when the amount of movement of the skeletons of all passengers is less than the first threshold value, the determination unit 22 determines whether or not the vehicle 12 is allowed to depart based on the amount of movement of the passengers.

[0045] When the amount of movement of at least one person is equal to or greater than the second threshold, the determination unit 22 prohibits the departure of the vehicle 12. In the example shown in FIG. 4(b), the amount of movement of passengers A and B is equal to or greater than the second threshold, and the amount of movement of passenger C is less than the second threshold. Therefore, the determination unit 22 prohibits the departure of the vehicle 12. When the amount of movement of multiple passengers is equal to or greater than the second threshold, the determination unit 22 may prohibit the departure of the vehicle 12.

[0046] As described above, according to this embodiment, when the amount of movement of the skeleton is equal to or greater than the first threshold, there is a possibility that a person is moving, and therefore the departure of the vehicle 12 is prohibited. Even if a person is moving, the amount of movement of the skeleton may not be equal to or greater than the first threshold due to erroneous detection of the skeleton. In this case, the presence or absence of movement of a person is determined based on the amount of movement of the person detected by the person detection unit 16. In this way, the possibility of departure can be determined more accurately than when the possibility of departure is determined based only on skeleton detection.

[0047] The determination unit 22 may determine whether or not the vehicle 12 is allowed to depart for each unit time. When the number of times it is consecutively determined to be safe (i.e., the number of times it is consecutively determined that the amount of movement of the skeleton is less than the first threshold and the amount of movement of the person is less than the second threshold) is equal to or greater than a preset threshold, the determination unit 22 permits the vehicle 12 to depart. In other cases, the determination unit 22 prohibits the vehicle 12 from depart. The length of the unit time is preset.

[0048] When the length of time during which it is continuously determined to be safe (i.e., the length of time during which it is continuously determined that the amount of movement of the skeleton is less than the first threshold and the amount of movement of the person is less than the second threshold) is equal to or greater than a preset threshold, the determination unit 22 may permit the departure of the vehicle 12. In other cases, the determination unit 22 prohibits the departure of the vehicle 12.

[0049] All or part of the determination device 10 can be realized by using hardware resources such as a processor and an electronic circuit, and a device such as a memory may be used as necessary in the realization. All or part of the determination device 10 may be realized by, for example, a computer. That is, all or part of the determination device 10 may be realized by cooperation between hardware resources such as a CPU (Central Processing Unit) and memory provided in a computer and software (program) that specifies the operation of the CPU, etc. The program is stored in the storage device of the determination device 10 via a recording medium such as a CD or DVD, or via a communication path such as a network. As another example, all or part of the determination device 10 may be realized by a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like. Of course, a GPU (Graphics Processing Unit) or the like may be used.

[0050] The functions of the determination device 10 may be realized by a single device or by multiple devices.

[0051] The determination device 10 may be provided in the vehicle 12, or may be provided in an external device such as a server. When the determination device 10 is provided in the external device, the determination device 10 performs human detection, skeleton detection, and control of the departure of the vehicle 12 by communicating with the vehicle 12 via a communication path such as the Internet. [Explanation of symbols]

[0052] 10 Determination device, 12 vehicle, 14 camera, 16 person detection unit, 18 skeleton detection unit, 20 calculation unit, 22 determination unit, 24 control unit.

Claims

1. A human detection unit that detects a human inside the vehicle; a skeleton detection unit that detects a skeleton of a person detected by the human detection unit; a calculation unit that calculates a movement amount of a skeleton detected by the skeleton detection unit; a determination unit that prohibits the vehicle from departing when the amount of movement of the skeleton is equal to or greater than a first threshold, and that determines whether or not the vehicle can depart based on a detection result by the human detection unit when the amount of movement of the skeleton is less than the first threshold; A vehicle departure permission determination device comprising:

2. 2. The vehicle departure permission determination device according to claim 1, The calculation unit further calculates a movement amount of the person detected by the person detection unit, the determination unit prohibits the vehicle from departing when the amount of movement of the skeleton is less than the first threshold and the amount of movement of the person is equal to or greater than a second threshold, and permits the vehicle to depart when the amount of movement of the skeleton is less than the first threshold and the amount of movement of the person is less than the second threshold. A vehicle departure permission determination device comprising:

Images