In-car monitoring device and safety status monitoring control program

Abstract

To monitor, specifically, the safety of standing passengers, in addition to the posture determination of crew members seated or standing.SOLUTION: An embodiment of the present invention can verify the determination of a provisional determination condition, by performing a confirmed determination (a determination based on the angles or relative positional relations of the specific skeleton points of a skeleton model) in addition to the provisional determination (a determination of whether or not the specific skeleton point of the skeleton model exists in a specific area). From this result, it can be accurately determined whether a passenger 14 is seated, standing alone, or holding a strap 26 with a standing posture.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an in-vehicle monitoring device and a safety situation monitoring control program. [Background technology]

[0002] In vehicles such as public buses, where both seated and standing passengers are present, the crew is obligated to monitor the interior of the vehicle to prevent situations in which passengers lose their balance. In the case of public buses, the driver must monitor the interior of the vehicle while driving, which places a greater burden on the driver than when there is a dedicated monitor.

[0003] It has been proposed to automate the process of monitoring the inside of a vehicle. As shown in Patent Documents 1 to 5, passenger monitoring mainly involves determining the posture of a passenger based on whether or not a part of the passenger or their skeleton coordinates are within a predetermined area.

[0004] Patent Document 1 describes monitoring the posture of passengers using a camera. Patent Document 2 describes determining whether a passenger is holding onto a pole inside the vehicle based on skeletal information of the passenger. Patent Document 3 describes determining whether a passenger's head is within a predetermined area. Patent Document 4 describes determining a passenger's posture pattern based on skeletal information. Patent Document 5 describes a technology for determining a passenger's state (for example, whether they are seated or not) based on skeletal information. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2022-036795 [Patent Document 2] International Publication No. 2022 / 137541 [Patent Document 3] Japanese Patent Publication No. 2022-176764 [Patent Document 4] International Publication No. 2023 / 095196 [Patent Document 5] Japanese Patent Publication No. 2022-150512 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the accuracy of the determination is low only if the technology for determining the posture is based on whether a part of the passenger or the skeletal coordinates are within a predetermined area. In particular, it is difficult to determine whether a standing passenger is reliably holding on to a support member such as a strap, handrail, or assist grip, which may hinder the determination of safety.

[0007] In consideration of the above, the present invention aims to provide an in-vehicle monitoring device and a safety situation monitoring control program that can not only determine the posture of occupants, including seated and standing passengers, but also monitor the safety of standing passengers in particular. [Means for solving the problem]

[0008] The in-vehicle monitoring device of the present invention comprises a camera capable of capturing images of the range of movement of passengers in a vehicle equipped with seats on which passengers sit and support members for passengers to hold on to while standing up, a generation unit that generates a skeletal model of the passenger identified from the image captured by the camera, and a judgment unit that determines safety when the first and second conditions are met, with a first condition being that a specific skeletal point of the skeletal model generated by the generation unit is located within a predetermined specific area that identifies the position of the seat and the support member, and a second condition being that a further condition for the passenger's skeletal point is met.

[0009] According to the present invention, the generation unit generates a skeletal model of the identified passenger from an image captured by a camera. The determination unit determines safety as a first condition that a predetermined skeletal point of the skeletal model generated by the generation unit is present within a predetermined specific area that identifies the positions of the seat and support member, and determines as a second condition that a further condition for the passenger's skeletal point is satisfied, when the first and second conditions are satisfied.

[0010] This makes it possible to determine the posture of passengers, including whether they are seated or standing, and in particular to monitor the safety of standing passengers.

[0011] In the present invention, the judgment unit is characterized by comprising a provisional judgment unit that provisionally determines the safety of the passengers when the first condition is met, and a final judgment unit that provisionally determines the safety of the passengers when the provisional judgment unit has made a determination and the second condition is met.

[0012] The provisional determination unit provisionally determines safety on the condition (first condition) that a predetermined skeleton point of a skeleton model exists within a predetermined specific area that specifies the positions of the seat and the support member.

[0013] When the provisional determination unit provisionally determines the safety of the passenger, the final determination unit determines whether the passenger is safe or unsafe on the condition that a further condition for the skeleton point of the passenger is satisfied (second condition).

[0014] In the present invention, when the provisional determination unit determines that the passenger is seated in a seat within the specific area and that safety is provisionally determined, the final determination unit determines that the passenger is seated in a seat that is within the specific area and that safety is provisionally determined, and then finalizes the safety if it determines that a predetermined skeletal angle of the lower body of the seated passenger is equal to or less than a threshold value.

[0015] When the provisional determination unit determines that a passenger is seated in the seat and that safety is provisionally determined, the final determination unit determines that safety is determined when it determines that a predetermined skeletal angle in the lower body of the seated passenger is equal to or less than a threshold value.

[0016] In the present invention, when the provisional determination unit determines that the passenger is standing while holding on to the support member present in the specific area and that the passenger is safe, the final determination unit determines that the passenger's hands are higher than the shoulders based on the relative positional relationship between two specific skeletal points on the upper body of the passenger who is standing while holding on to the support member, and then finalizes the passenger's safety.

[0017] When the provisional determination unit determines that the passenger is standing by holding on to the support member and that the passenger is safe, the final determination unit determines that the passenger's hands are higher than the shoulders based on the relative positional relationship between two specific skeletal points on the upper body of the passenger who is standing by holding on to the support member and then finalizes the passenger's safety.

[0018] The present invention is characterized in that the support member is a strap that can be held by the passenger in a standing position to ensure the safety.

[0019] In the present invention, the specific area for identifying the position of the strap includes a support member that supports the strap.

[0020] The present invention is characterized in that the determination result by the final determination unit is notified to a user outside the vehicle.

[0021] Normally, by notifying the driver, the driver can receive the notification and check the status of passengers without having to check mirrors, etc. On the other hand, by notifying a user outside the vehicle of the safety status, the safety status of multiple vehicles can be managed collectively.

[0022] The present invention is characterized in that passengers identified from images captured by the camera are classified by age, and a determination is made by the provisional determination unit for passengers within a predetermined age range.

[0023] For example, in the case of adults, the trunk is stable and can withstand slight shaking (sudden acceleration, sudden deceleration, etc.) without having to hold on to something, so if the ages of the people being judged are limited to those classified as children or elderly, the burden of the safety situation judgment process can be reduced.

[0024] In the present invention, the vehicle is an autonomous vehicle that is managed and operated by an autonomous driving management center, and the judgment result is notified to the autonomous driving management center, which, upon receiving the notification of the judgment result, instructs a change in the driving conditions of the vehicle.

[0025] In addition to controlling autonomous driving, the Autonomous Driving Management Center will be able to implement safer autonomous driving control by understanding the safety status of passengers.

[0026] A safety situation monitoring control program according to the present invention is characterized in that it causes a computer to operate as the generating unit, the provisional determining unit, and the final determining unit of the in-vehicle monitoring device. [Effects of the Invention]

[0027] As described above, the present invention can not only determine the posture of passengers, including whether they are seated or standing, but also monitor the safety of standing passengers in particular. [Brief explanation of the drawings]

[0028] [Figure 1] 1 is a side view of a public bus equipped with an in-vehicle monitoring device according to an embodiment of the present invention. [Figure 2] 1 is a schematic diagram of an in-vehicle monitoring device according to an embodiment of the present invention, which is mounted on a public bus; [Figure 3] FIG. 2 is a diagram of a skeleton model generated in the present embodiment. [Figure 4] FIG. 1 is a front view of an image captured by a surveillance camera. [Figure 5] 4 is a safety situation monitoring control routine in a safety situation monitoring control unit according to the present embodiment. [Figure 6] 6 is a control flowchart showing details of a determination processing subroutine executed in step 116 of FIG. 5. [Figure 7] This is an example of monitoring using an in-vehicle monitoring device according to this embodiment, where (A) is an image captured by a monitoring camera showing the presence of three passengers in different riding positions, (B) is an enlarged view of a seated passenger in Figure 7(A), (C) is an enlarged view of a passenger in a standing position (without holding on to the handrail) in Figure 7(A), (D) is an enlarged view of a passenger in a standing position (with holding on to the handrail) in Figure 7(A), and (E) is an enlarged view of a passenger standing by holding on to the handrail in a modified example. [Figure 8]1 is a schematic diagram showing the state of information communication between an automatically driven vehicle and an automatic driving management center that manages the vehicle. DETAILED DESCRIPTION OF THE INVENTION

[0029] 1 is a side view of a public bus 12 equipped with an in-vehicle monitoring device 10 (see FIG. 2) according to this embodiment. Note that the public bus 12 is just one example, and the in-vehicle monitoring device 10 is applicable to any vehicle that carries passengers 14. However, a public bus is used as an example of a vehicle in which passengers 14 seated in seats 16 and standing passengers 14 coexist.

[0030] The public bus 12 has a driver's seat 18 installed at the front of the vehicle, and seats 16 for passengers 14 to sit in are arranged behind the driver's seat 18.

[0031] The public bus 12 is provided with passenger boarding and alighting doors 20, and passengers 14 board and alight through the boarding and alighting doors 20. Note that although FIG. 1 illustrates boarding and alighting doors 20 only at the front of the vehicle, there are public buses that have boarding and alighting doors at the center or rear of the vehicle, and the number of boarding and alighting doors is not limited.

[0032] The floor surface 22 on which the seats 16 are arranged is stepped due to the vehicle body structure, with the steps rising towards the rear of the vehicle, making the safety of passengers 14 when moving lower than on a flat surface. Even if the surface were flat, the driver 24 seated in the driver's seat 18 would be obligated to monitor the interior of the vehicle using a rearview mirror (not shown) while driving to ensure that no situation occurs that could cause passengers 14 to lose their balance. If the floor surface 22 has steps, it is even more necessary to pay attention to the interior of the vehicle.

[0033] Here, the public bus 12 is equipped with support members such as hand straps 26, support bars 26A that support the hand straps 26, and handrail bars 28 to prevent standing passengers 14 from losing their balance. By holding onto the support members (standing up while holding onto them), standing passengers 14 can maintain their posture against swaying while the bus is moving, compared to passengers 14 who are not holding onto them. The determination of whether or not a passenger is holding onto a support member will be described later. Note that the support members may also be assist grips provided at the upper corners of the backrests of seats.

[0034] That is, from the viewpoint of safety monitoring, passengers 14 seated in seats 16 and passengers 14 standing up are evaluated as "safe," while passengers 14 not standing up are evaluated as "unsafe."

[0035] Surveillance cameras 30 constituting the in-vehicle monitoring device 10 are attached to the front and rear of the passenger bus 12. The surveillance cameras 30 are designed to capture the range within the vehicle in which passengers 14 may move without creating any blind spots.

[0036] In this embodiment, the image is captured using two surveillance cameras 30, but for example, one surveillance camera may be installed in front of the vehicle, or three or more surveillance cameras may be installed in any necessary location, not just in front and rear of the vehicle, in order to reduce blind spots.

[0037] (In-vehicle monitoring device 10) FIG. 2 is a schematic diagram of an in-vehicle monitoring device 10 according to this embodiment, which is mounted on a public bus 12 (see FIG. 1).

[0038] The in-vehicle monitoring device 10 includes n monitoring cameras 30 (n is a positive integer, and in this embodiment, n=2) and a safety situation monitoring control unit 32.

[0039] When installing multiple surveillance cameras 30, the surveillance cameras 30 do not need to be of the same model or specifications; it is sufficient to use a surveillance camera 30 with the optimal specifications (magnification, zoom range, wide-angle range, aperture, shutter speed, swivel angle, etc.) in the right place.

[0040] Furthermore, the surveillance camera 30 is only required to be able to grasp the outline of the passenger 14, and is not limited to an image based on so-called density, and may be an image based on temperature differences caused by infrared rays, or the like.

[0041] The surveillance camera 30 is connected to an image capture unit 34 of a safety situation monitoring control unit 32 .

[0042] The image capturing unit 34 is connected to the passenger identification unit 36 ​​and sends the captured image information to the passenger identification unit 36 ​​.

[0043] The passenger identification unit 36 ​​uses bounding box technology to identify the positions of the passengers 14 (see dotted rectangular frame A in FIG. 4), assigns an identification code to each passenger 14, and sends it to the tracking target determination unit 38.

[0044] The follow-up target determination unit 38 is connected to the movement follow-up execution unit 40, and follows up the movement of the passenger 14 determined by the follow-up target determination unit 38 using bounding box technology.

[0045] Here, the movement following execution unit 40 is connected to an alighting determination unit 42. The alighting determination unit 42 determines whether the passenger 14 being followed by the movement following execution unit 40 has alighted, and feeds back to the following target determination unit 38 the identification code of the passenger 14 whose alighting has been confirmed.

[0046] As a result, the follow-up target determination unit 38 can add the passengers 14 who have boarded the vehicle and delete the passengers 14 who have disembarked, thereby determining the passengers 14 to be followed.

[0047] The movement tracking execution unit 40 is connected to a skeletal model generation unit 44. The skeletal model generation unit 44 is connected to a determination instruction unit 46. When a trigger for a determination instruction is input to the determination instruction unit 46, the determination instruction unit 46 instructs the skeletal model generation unit 44 to generate a skeletal model of each passenger 14 that is currently being followed.

[0048] The trigger for the judgment instruction may be, for example, periodically (trigger 1), when the acceleration of the public bus 12 changes by more than a predetermined threshold (trigger 2), or when the driving history information of the public bus 12 (starting, stopping, accelerating, decelerating, curves) is detected (trigger 3).

[0049] As an example, the skeletal model is generated using a posture estimation AI. The posture estimation AI is a technology that visualizes a person's posture by detecting feature points such as the face and joints of the person through clothing. As shown in Figure 3, the posture of the person (passenger 14) is expressed by connecting skeletal points No. 1 to No. 16. Note that while conventional methods required time and effort, such as attaching markers to the body surface or using inertial sensors, posture estimation AI makes it possible to detect skeletal information simply by taking a picture with a surveillance camera 30.

[0050] The skeleton model generation unit 44 is connected to a determination processing unit 48, and uses each skeleton model to determine the safety (safe / unsafe) of each passenger 14. Details of the determination processing will be described later.

[0051] The determination processing unit 48 is connected to a determination result output unit 50, which outputs the determination result of the determination processing unit 48 to a predetermined device. The predetermined device displays the positions of passengers 14 in a safe state and passengers 14 in an unsafe state as simple images (for example, color-coded circle images) on a monitor or the like installed in the driver's seat 18. This allows the driver 24 to check the display on the monitor with less eye movement (while maintaining forward visibility for driving) than when checking using a mirror or the like, and to understand the status of the passengers 14.

[0052] (Determination process) FIG. 4 shows an image captured by the surveillance camera 30, with predetermined areas E0 to E10 (see the solid-line frames in FIG. 4) set in advance. The areas E0 to E10 may be set automatically using image recognition. Note that in FIG. 4, area E10 is set for the handrail bar 28, but the handrail bar image itself may be set as area E10. Furthermore, the number of areas is not limited to E0 to E10, and varies depending on the vehicle to which the in-vehicle monitoring device 10 is applied.

[0053] The reason why the support bar 26A is included in the strap area is that, for example, if a passenger 14 who is taller than the average height gets on the train, it may be difficult for him / her to hold on to the strap 26 at a height that corresponds to the average height, and he / she may end up holding on to the support bar 26A located above it.

[0054] Areas E0 to E10 indicate the area where the face and upper body of passenger 14 will be positioned when seated in seat 16, the straps 26, the support bars 26A that support the straps 26, and the areas where the hands of passenger 14 will be positioned as they hold onto the handrail bars 28. In the following, the area including the straps 26 and the support bars 26A will be collectively referred to as the strap area.

[0055] "Temporary judgment condition (first condition)" In this embodiment, the provisional determination condition is to determine whether a specific skeleton point (No. 1 to No. 16) of the skeleton model exists in a specific region (E0 to E10 shown in FIG. 4) (see Determination 1 and Determination 2 below).

[0056] (Determination 1) It is determined whether or not the person is seated by determining whether or not all of the skeleton points of the nose and both shoulders (No. 1 to No. 6 shown in FIG. 3) are included in the seating area (E0 to E7 shown in FIG. 4). In this embodiment, compared to when only one skeleton point is used to determine the seating area, three points (the nose and both shoulders) are used, which can reduce erroneous determinations.

[0057] (Decision 2) Determine whether the wrist skeletal point (No. 9 or No. 10 shown in Figure 3) is included in the strap area (E8 or E9 shown in Figure 4) or the handrail area (E10 shown in Figure 4), and determine whether the standing passenger 14 is holding on to the strap 26, support bar 26A, or handrail 28.

[0058] "Confirmed judgment condition (second condition)" Here, under the provisional determination conditions, a situation may arise in which it is uncertain whether the occupant is sitting or holding on to the strap 26.

[0059] Therefore, in this embodiment, the judgment of the provisional judgment condition is verified based on the angle and relative positional relationship of specific skeleton points (No. 1 to No. 16) of the skeleton model as a judgment condition for confirming the provisional judgment that is executed when the provisional judgment condition is met (see Judgment 3 and Judgment 4 below).

[0060] (Decision 3) If the condition is met in the above-mentioned Decision 1, the system further determines whether the angle of a predetermined skeletal point in the lower body, for example, the angle of either the knee or the waist (No. 11 to No. 14), is equal to or less than a threshold value, and confirms the seating. In this embodiment, the system checks whether the angle of the knee or the waist is equal to or less than a threshold value, that is, whether the knee or the waist is bent. Note that the angle determination may be performed for both the knee and the waist. This prevents the passenger 14 from being mistakenly determined to be safely seated when the passenger 14 is not seated properly.

[0061] In addition, in terms of accuracy of judgment, rather than seeking accuracy in the judgment, it may be acceptable to shift the judgment criteria in the direction of judging a situation as unsafe in situations where it is unclear whether it is safe or unsafe, from the perspective of alerting passengers 14 who are in an unsafe state.

[0062] (Decision 4) If the condition is met in the above-mentioned Decision 2, it is further determined whether or not the hands are positioned higher than the shoulders from the relative positional relationship of two specific skeletal points on the upper body, for example, whether or not the position of the wrist skeletal points (Nos. 9 and 10) is higher than the shoulder skeletal points (Nos. 5 and 6), and it is confirmed that the person is standing while holding on to the strap 26. According to Decision 4, it is possible to reduce erroneous determinations caused by, for example, the wrist being captured in another strap area due to the angle of view of the monitoring camera 30, compared to when it is determined in Decision 2 that the person is holding on to the strap 26 simply from the wrist skeletal points.

[0063] In this embodiment, the height determination of the positions of the wrist skeleton points (No. 9 and No. 10) and the shoulder skeleton points (No. 5 and No. 6) is performed on two-dimensional coordinates, but the height position may also be determined by analyzing the image captured by the surveillance camera 30 three-dimensionally (depth analysis).

[0064] In the above, the judgment criteria etc. are set mainly based on the gripping state of the strap 26, but the judgment specifications are similar for the support bar 26A and the handrail bar 28, and the judgment criteria values ​​etc. can be adjusted as necessary.

[0065] That is, there are cases where it is necessary to change the positional relationship of the skeleton points when making a grip determination between handrail bar 28 (extending in the height direction) and strap 26. For example, an example of a grip determination for handrail bar 28 will be shown below.

[0066] (Example of grasping judgment of handrail bar 28) One possible cause of the false detection is that the hand of the passenger 14 standing near the handrail 28 is simply within the area E10.

[0067] Therefore, as a provisional determination, it is determined whether or not the skeleton point of the wrist falls within the area E10 of the handrail bar 28.

[0068] Next, as a confirmation judgment, it is determined whether the wrist skeletal point is between the shoulder skeletal point and the waist skeletal point in the height direction. For example, in the state of Figure 7(E), the wrist skeletal point is between the shoulder skeletal point and the waist skeletal point in the height direction of the passenger 14, so it is judged to be safe. This is because the natural grip position for the handrail bar 28 is slightly lower than shoulder height, and when gripping the handrail bar 28, it is unlikely that the pole will be gripped at a position higher than the shoulders. When the hands are lowered, the wrists are normally lower than the waist.

[0069] The operation of this embodiment will be described below with reference to the flowcharts of FIGS.

[0070] FIG. 5 shows a safety situation monitoring control routine in the safety situation monitoring control unit 32.

[0071] In step 100, it is determined whether or not a determination start trigger has been input. The trigger may be periodic (trigger 1), acceleration change (trigger 2), or driving history information.

[0072] If there is no input of a trigger such as trigger 1 to trigger 3 and a negative determination is made in step 100, the process proceeds to step 120 to determine whether or not the operation of the public bus 12 has ended. If a negative determination is made in step 120, the process returns to step 100, and if a positive determination is made, this routine ends.

[0073] When a trigger such as trigger 1 to trigger 3 is input and a positive determination is made in step 100, the process proceeds to step 102 to determine whether or not there is a new passenger 14. When a positive determination is made in step 102, the process proceeds to step 104 to assign an identification code to each passenger 14, and then the process proceeds to step 106 to start tracking their movements. That is, for example, a bounding box (see dotted rectangular frame A in FIG. 4) is set for each passenger 14, tracking of their movements within the vehicle begins, and the process proceeds to step 108. On the other hand, when a negative determination is made in step 102, the process proceeds to step 108.

[0074] In step 108, it is determined whether any passengers 14 have disembarked. If the determination in step 108 is affirmative, the process proceeds to step 110, where the identification symbols of the disembarking passengers 14 are erased, movement tracking is terminated, and the process proceeds to step 112. On the other hand, if the determination in step 108 is negative, the process proceeds to step 112.

[0075] In step 112, a loop process of the following steps 114 and 116 is started in a predetermined order for the number of passengers 14 currently on board.

[0076] (Loop processing steps 114 and 116) In step 114, a skeleton model of the selected passenger 14 is generated (skeleton points are identified), and then the process proceeds to step 116, where a determination process (subroutine in FIG. 6, described later) is executed.

[0077] In the next step 118, when the determination process for all passengers 14 on board has been completed, the loop ends and the process proceeds to step 120. In step 120, it is determined whether or not the operation of the bus 12 has ended. If the determination in step 120 is negative, the process returns to step 100, and if the determination is positive, the routine ends.

[0078] FIG. 6 is a control flowchart showing the details of the determination processing subroutine executed in step 116 of FIG.

[0079] In step 150, as the first provisional judgment condition, it is determined whether or not the skeleton points of the nose and both shoulders (No. 1 to No. 6 shown in FIG. 3) are all included in the seating area (E0 to E7 shown in FIG. 4), and it is determined whether or not the person is seated.

[0080] If the determination in step 150 is affirmative, there is a high probability that the passenger 14 is seated, but there is also the possibility that the passenger 14 has simply bent over and their face is in the seating area (E0 to E7 shown in FIG. 4). Therefore, if the determination in step 150 is affirmative, the process proceeds to step 152, where, as determination 3 of the determination conditions, it is determined whether or not the angle of either the knees or the waist (No. 11 to No. 14) is equal to or less than a threshold value. If the determination is affirmative, the process proceeds to step 154, where the seating is confirmed. With this confirmation of seating, the passenger 14 is determined to be "safe," and the process proceeds to step 166, where the determination result is output and the process returns.

[0081] On the other hand, if a negative judgment is made in step 150 or a negative judgment is made in step 152, it is determined that the passenger 14 is not seated, and the process proceeds to step 156, where a provisional judgment is made that the passenger is in a standing position (unsafe), and the process proceeds to step 158.

[0082] In addition, taking into consideration a situation where a seated passenger 14 temporarily stands up and sits down again (hereinafter referred to as a temporary unsafe state), the temporary unsafe state may not be instantly determined to be an unsafe state, but may be determined to be an unsafe state if the temporary unsafe state continues for a predetermined period of time (predetermined image frames).

[0083] In step 158, as judgment 2 of the provisional judgment condition, it is determined whether the wrist skeleton point (No. 9 or No. 10 shown in Figure 3) is included in the strap area (E8 or E9 shown in Figure 4) and whether the standing passenger 14 is holding on to the strap.

[0084] If the determination in step 158 is affirmative, there is a high probability that the passenger 14 is standing but holding onto the strap 26. However, depending on the installation position and angle of the monitoring camera 30, the wrist may appear to be within the area in the image even though it is not actually within the area, and it may be determined that the arm is within the strap area (E8 or E9 shown in FIG. 5). Therefore, if the determination in step 158 is affirmative, the process proceeds to step 160, where, as determination 4 of the confirmation determination, it is determined whether the position of the wrist skeleton point (No. 9 and No. 10 shown in FIG. 3) is higher than the shoulder skeleton (No. 5 and No. 6 shown in FIG. 3). If the determination is affirmative, the process proceeds to step 162, where it is determined that the passenger 14 is standing and holding onto the strap 26. By determining that the passenger 14 is standing and holding onto the strap 26, it is determined that the passenger 14 is "safe," and the process proceeds to step 166, where the determination result is output, and return.

[0085] In the above step 160, the angle determination setting is specialized for the strap 26, but in parallel, determination may be made by setting numerical values ​​for the support bar 26A or the handrail bar 28.

[0086] On the other hand, if a negative judgment is made in step 158 or a negative judgment is made in step 160, it is determined that the passenger 14 is standing and not holding onto the strap 26, and the process proceeds to step 164, where it is determined that the passenger is not holding onto the strap (unsafe), and the process proceeds to step 165, where the judgment result is output, and the process returns.

[0087] (Example) FIG. 7(A) is an image captured by the monitoring camera 30, showing the state in which three passengers 14 with different riding styles are present in the interior image of the vehicle shown in FIG.

[0088] Figure 7(B) shows passenger 14 seated in seat 16, and since skeleton points No. 1 to No. 6 are included in a specific area E0 (decision 1 is established), and the angle of the knee (No. 11) is below the threshold value (decision 3), the seating is confirmed and the seat is judged to be "stable."

[0089] FIG. 7C shows a passenger 14 in a standing position (determination 1 and determination 2 fail), and the passenger is determined to be "unstable."

[0090] Figure 7(D) shows passenger 14 standing (judgment 1 is not established, judgment 2 is established), and since the position of the wrist skeleton point (No. 10) is higher than the shoulder skeleton point (No. 6), it is determined that the passenger is standing while holding onto the strap 26, and the result is a judgment of "stable."

[0091] In this embodiment, the result of passenger 14 posture determination is output to a monitor or the like that notifies the driver 24, but the passenger 14 may also be notified directly using an audio device or display device.

[0092] 8, when the target vehicle 12A is traveling under automatic driving control, the automatic driving management center 52 that remotely manages the automatic driving of the vehicle 12A may be notified. In response to this notification, the automatic driving management center 52 may output a driving instruction to the automatic driving control system of the vehicle 12A. The driving instruction may, for example, increase the speed or increase the curve entry speed if only safe passengers 14 are present, or conversely, decrease the speed or decrease the curve entry speed if unsafe passengers 14 are present.

[0093] Furthermore, in this embodiment, the skeleton points used for seating determination are the face and both shoulders, but this is not limited to the face and both shoulders, and the positions of other skeleton points or the relative positional relationships between other skeleton points may also be used as the determination targets.

[0094] Furthermore, in this embodiment, a safety / unsafety judgment is performed for all passengers 14, but passengers 14 may be classified into children (e.g., under 12 years old), adults (13 to 59 years old), elderly (60 years old or older), etc., and the safety / unsafety judgment may be specialized for children and elderly. Adults are physically strong and can maintain a well-balanced standing posture without holding on to something, so to reduce the processing load, it is possible to select those to be judged. In this case, classification can be performed by facial recognition based on images captured by the surveillance camera 30.

[0095] Furthermore, in this embodiment (particularly, the judgment processing subroutine of Figure 6), a judgment is made as to whether or not a standing passenger 14 is holding on to a strap 26, but this may be replaced with holding on to another support member such as a handrail.

[0096] Furthermore, for example, in decision 4, it was determined whether the positions of the wrist skeleton points (Nos. 9 and 10) were higher than the shoulder skeleton points (Nos. 5 and 6). However, depending on the height of the passenger 14, even if the positions of the wrist skeleton points (Nos. 9 and 10) are lower than the shoulder skeleton points (Nos. 5 and 6), there is a possibility that the passenger 14 may still be holding on to the strap 26. Therefore, correction may be made based on the height of the passenger 14 being judged. That is, after correction, it is determined that safety is maintained even if, for example, the height of the wrist is the same as or a certain amount lower than the height of the shoulder.

[0097] Furthermore, in addition to the height of the passenger 14, the positional relationship between the wrist skeleton points (No. 9 and No. 10) and the shoulder skeleton points (No. 5 and No. 6) may be displaced depending on the position and angle of the monitoring camera 30. For this reason, when multiple monitoring cameras 30 are used, the correction value may be changed for each monitoring camera.

[0098] Furthermore, a seated passenger 14 may be holding a child, and if the child is also followed by a bounding box, it is possible to confirm that the child is seated (two faces may be present within the region).

[0099] As explained above, in this embodiment, in addition to the provisional judgment (judgment as to whether or not a specific skeleton point (No. 1 to No. 16) of the skeleton model is present in a specific region (E0 to E10)), a definitive judgment (judgment based on the angle and relative positional relationship of the specific skeleton point (No. 1 to No. 16) of the skeleton model) is executed to verify the judgment of the provisional judgment condition. This makes it possible to accurately judge whether the passenger 14 is sitting, standing, or holding on to the strap 26 while standing. [Explanation of symbols]

[0100] 10 In-car monitoring device 12 Bus 14 passengers 16 seats 18 Driver's seat 20 Entrance / Exit 22 Floor surface 24 Driver 26 Hanging strap 28 Handrail 30 Surveillance cameras (cameras) 32 Safety Status Monitoring and Control Unit 34 Image capture unit 36 Passenger Identification Department 38 Tracking target determination unit 40 Movement tracking execution unit 42 Exit Judgment Department 44 Skeleton model generation unit (generation unit) 46 Judgment instruction section 48 Judgment processing unit (provisional judgment unit, final judgment unit) 50 Judgment result output unit

Claims

1. a camera capable of photographing the range of movement of passengers in a vehicle equipped with seats on which passengers sit and support members for passengers to hold on to and stand up; a generation unit that generates a skeletal model of the identified passenger from the image captured by the camera; a determination unit that determines safety when a first condition is met that a predetermined skeleton point of the skeleton model generated by the generation unit is within a predetermined specific area that identifies the positions of the seat and the support member, and a second condition is met that a further condition for the skeleton point of the passenger is met; and An in-vehicle monitoring device having the same.

2. The determination unit:

2. The in-vehicle monitoring device according to claim 1, further comprising: a provisional determination unit that provisionally determines the safety of the passengers when the first condition is met; and a final determination unit that provisionally determines the safety of the passengers when the provisional determination unit has made the determination and the second condition is met.

3. 3. The in-vehicle monitoring device according to claim 2, wherein when the provisional determination unit provisionally determines that the passenger is seated in a seat within the specific area and that safety is assured, the final determination unit finalizes the safety if it determines that a predetermined skeletal angle of the lower body of the seated passenger is equal to or less than a threshold value.

4. 3. The in-vehicle monitoring device according to claim 2, wherein when the provisional determination unit determines that the passenger is standing by holding on to the support member present in the specific area and that the passenger is safe, the final determination unit finalizes the safety if it determines that the hands of the passenger are higher than the shoulders based on the relative positional relationship of two specific skeletal points on the upper body of the passenger who is standing by holding on to the support member.

5. 5. The in-car monitoring device according to claim 4, wherein the support member is a strap that can be held by the passenger in a standing position to ensure the safety.

6. The in-car monitoring device according to claim 5 , wherein the specific area for identifying the position of the strap includes a support member for supporting the strap.

7. 3. The in-vehicle monitoring device according to claim 2, wherein the determination result by the determination unit is notified to a user outside the vehicle.

8. 3. The in-vehicle monitoring device according to claim 2, wherein passengers identified from the image captured by the camera are classified by age, and the provisional determination unit performs determination on passengers within a predetermined age range.

9. The vehicle is an autonomous vehicle that runs under the management of an autonomous driving management center, and the determination result is notified to the autonomous driving management center, The in-vehicle monitoring device according to claim 2 , wherein the automatic driving management center receives notification of the determination result and instructs a change in the driving conditions of the vehicle.

10. Computer, A safety situation monitoring control program that causes the in-vehicle monitoring device according to any one of claims 1 to 9 to operate as the generating unit and the determining unit.

Images