In-vehicle monitoring system and shared vehicle

Abstract

To enable requesting manner observation corresponding a congestion state in a vehicle.SOLUTION: A seat state estimation section 37 estimates presence / absence of occurrence of a baggage occupation seat occupied with baggage and presence / absence of occurrence of an overlapping-use seat used by a person sitting over a plurality of seats based on positions of the baggage and persons. A congestion rate calculation section 64 obtains a congestion rate in a cabin based on the recognized persons and baggage. A warning propriety determination section 65 gives a command to output warning to a share-ride vehicle 10 when it is estimated that at least one of the baggage occupation seat and the overlapping-use seat occurs and also when the obtained congestion rate exceeds a predetermined congestion threshold.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] This specification discloses a shared vehicle and an in-vehicle monitoring system for monitoring the interior of the shared vehicle.

Background Art

[0002] As an in-vehicle monitoring system, for example, Patent Document 1 discloses a system for estimating nuisance makers in a train. In this system, a camera is provided in the vehicle. When a person who has suffered a nuisance (the victim) reports to the system using a communication terminal, an image around the transmission position of the report signal is captured by the camera. From this captured image, face recognition is performed on the people present around the person who has suffered the nuisance, and it is stored in the face database in the system.

[0003] Also, in Patent Document 2, when an in-vehicle unit receives a notification from a notifier in the vehicle who has discovered the occupation of a seat by a large piece of luggage, the in-vehicle unit checks whether the seat is occupied using a camera in the vehicle interior. Then, when it is confirmed from the camera image that the seat is occupied by the luggage, the in-vehicle unit causes a message requesting cancellation to be displayed on the in-vehicle display.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] Incidentally, an act such as placing luggage on an empty seat may be permitted when the vehicle interior is sufficiently empty. Thus, the allowable degree of so-called in-vehicle manners varies depending on the congestion situation inside the vehicle. Therefore, this specification discloses an in-vehicle monitoring system and a shared vehicle that can request compliance with manners according to the congestion situation inside the vehicle.

Means for Solving the Problems

[0006] The in-vehicle monitoring system disclosed in this specification includes a shared vehicle and an operation management device that manages the operation of the shared vehicle. The shared vehicle includes an imager, an image recognition unit, and a seat state estimation unit. The imager is provided on the ceiling of the vehicle interior. The image recognition unit recognizes the people and luggage included in the in-vehicle image captured by the imager, and further estimates whether the recognized people are sitting or standing. The seat state estimation unit estimates the presence or absence of a luggage-occupied seat occupied by the recognized luggage and the presence or absence of an overlapping seated seat occupied by the seated people across a plurality of seats based on the positions of the recognized luggage and people. The operation management device includes a congestion rate calculation unit and a warning permission determination unit. The congestion rate calculation unit obtains the congestion rate in the vehicle interior based on the recognized people and luggage. The warning permission determination unit instructs the shared vehicle to output a warning when at least one of the luggage-occupied seat and the overlapping seated seat is estimated to occur and the obtained congestion rate exceeds a predetermined congestion threshold.

[0007] According to the above configuration, when at least one of the luggage-occupied seat and the overlapping seated seat occurs, the presence or absence of warning output is determined based on the congestion situation inside the vehicle.

[0008] Also, in the above configuration, the image recognition unit may obtain the occupied areas of the recognized people and luggage. Further, the congestion rate calculation unit may obtain the congestion rate based on the total sum of the occupied areas of the recognized people and luggage and the vehicle interior floor area.

[0009] According to the above configuration, for example, compared with the case of obtaining the congestion rate based on the number of people or the number of luggage, it is possible to calculate a highly accurate congestion rate that reflects the physical differences of people and the sizes of luggage.

[0010] Also, in the above configuration, the warning availability determination unit may execute a reservation status confirmation process when the congestion rate is equal to or lower than the congestion threshold. In this reservation status confirmation process, it is determined whether or not to withhold the output of a warning according to the ride reservation information for the shared vehicle.

[0011] According to the above configuration, when the vehicle interior is empty, the availability of warning output is determined based on the ride reservation status.

[0012] Also, in the above configuration, the shared vehicle may travel on a specified route and be able to stop at a bus stop provided along the specified route. In this case, as the ride reservation information, a planned boarding stop, which is the stop where the passenger wishes to board, and an assigned seat where the passenger plans to sit are set. The operation management device includes a storage unit that stores the ride reservation information. The warning availability determination unit, as a reservation status confirmation process, withholds the output of a warning by the shared vehicle when ride reservation information that satisfies both a stop condition, where the next scheduled stop of the shared vehicle is set as the planned boarding stop, and an assigned seat condition, where the assigned seat where the passenger plans to sit corresponds to at least one of a luggage - occupied seat and an overlapping - seating seat, is not stored in the storage unit.

[0013] According to the above configuration, even when at least one of a luggage - occupied seat and an overlapping - seating seat occurs, if the vehicle interior is empty and the luggage - occupied seat and the overlapping - seating seat do not correspond to the assigned seats of the passengers boarding at the next scheduled stop, the output of the warning is withheld.

[0014] In addition, the shared vehicle disclosed in this specification includes an imager, an image recognition unit, a seat state estimation unit, a congestion rate calculation unit, and a warning permission determination unit. The imager is provided on the ceiling of the passenger compartment. The image recognition unit recognizes the people and luggage included in the in-vehicle image captured by the imager, and further estimates whether the recognized people are sitting or standing. The seat state estimation unit estimates the presence or absence of luggage-occupied seats occupied by luggage and the presence or absence of overlapping seats occupied by people sitting across multiple seats based on the positions of the recognized luggage and people. The congestion rate calculation unit obtains the congestion rate in the passenger compartment based on the recognized people and luggage. The warning permission determination unit causes the warning device to output a warning when at least one of the luggage-occupied seats and the overlapping seats is estimated to exist and the obtained congestion rate exceeds a predetermined congestion threshold value.

Effect of the Invention

[0015] According to the in-vehicle monitoring system and the shared vehicle disclosed in this specification, it is possible to request compliance with manners according to the congestion situation inside the vehicle.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The shapes, materials, numbers, and numerical values described below are examples for explanation and can be appropriately changed according to the specifications of the in-vehicle monitoring system. Also, in all the drawings below, the same reference numerals are assigned to equivalent elements.

[0018] <Transportation Service> FIG. 1 exemplifies the outline of the transportation service provided by the shared vehicle 10 and the operation management device 50 according to the present embodiment. In this transportation service, the shared vehicle 10 travels on a specified route 90 which is the operation route, and an unspecified number of users are transported. The operation management device 50 manages the operations of a plurality of shared vehicles 10-1 to 10-8. Also, as part of the operation management, the in-vehicle monitoring system according to the present embodiment is configured by the shared vehicle 10 and the operation management device 50.

[0019] In the transportation service exemplified in FIG. 1, the specified route 90 is, for example, a circular route. The shared vehicle 10 circulates in one direction as shown by the arrow on the specified route 90. Further, the shared vehicle 10 can stop at stops ST1 to ST3 provided along the specified route 90.

[0020] In addition, a garage 92 is provided so as to be connected to the specified route 90. In FIG. 1, shared vehicles 10-5 to 10-8 waiting in the garage 92 are illustrated. As connection points to the garage 92, a collection point Pout and an input point Pin are provided on the specified route 90.

[0021] In addition, on the specified route 90, an operation schedule update point Pu for sending each operation schedule to the shared vehicles 10-1 to 10-4 in operation is provided. At the schedule update point Pu, from the operation management device 50, an operation schedule for one round starting from the operation schedule update point Pu of the shared vehicle 10 is provided to the shared vehicle 10 passing through the point.

[0022] As will be described later, in the in-vehicle monitoring system according to the present embodiment, an event suspected of a manners violation is detected in the vehicle interior of the shared vehicle 10 in operation. Specifically, the presence or absence of a luggage-occupied seat, which is a seat occupied by luggage, and the presence or absence of an overlapping seat, which occurs when one passenger sits across multiple seats, are determined by the shared vehicle 10.

[0023] When it is determined that at least one of the luggage-occupied seat and the overlapping seat has occurred, the operation management device 50 determines whether to output a warning to prompt the elimination of the luggage-occupied seat and the overlapping seat based on the congestion rate of the shared vehicle 10 in which the luggage-occupied seat and the overlapping seat have occurred and the reservation status.

[0024] Specifically, when the vehicle interior is empty and the reserved seat for a passenger whose next stop of the shared vehicle 10 is the boarding place does not correspond to either the luggage-occupied seat or the overlapping seat, the output of the warning is postponed. That is, when the luggage-occupied seat and the overlapping seat occur in a manner that does not substantially cause trouble to other passengers, the output of the warning is withheld. Thus, according to the in-vehicle monitoring system according to the present embodiment, it is possible to flexibly call for compliance with boarding manners according to the boarding situation such as the congestion rate and the reservation status.

[0025] <Overall Configuration of In-Vehicle Monitoring System> Fig. 2 illustrates the hardware configuration of the in-vehicle monitoring system according to this embodiment. Fig. 3 illustrates the functional blocks of the in-vehicle monitoring system. As described above, the in-vehicle monitoring system includes the shared vehicle 10 and the operation management device 50. The shared vehicle 10 and the operation management device 50 can communicate with each other using communication means such as the Internet 95.

[0026] <Operation management device> Referring to Fig. 1, the operation management device 50 is installed, for example, in a management company that provides a transportation service using a plurality of shared vehicles 10. The operation management device 50 is composed of, for example, a computer (electronic computing device). Referring to Fig. 2, as its hardware configuration, the operation management device 50 includes an input / output controller 51 that controls the input / output of data. The operation management device 50 also includes a CPU 52, an input unit 53, a display unit 54, a ROM 55, a RAM 56, a hard disk drive 57 (HDD), and a clock 59. Note that instead of the hard disk drive 57, a storage device such as an SSD (Solid State Drive) may be used. These components are connected to an internal bus 58.

[0027] At least one of the ROM 55 and the hard disk drive 57, which are storage devices, stores a program for performing operation management. When the above program is executed by the CPU 52 etc. of the operation management device 50, functional blocks as illustrated in Fig. 3 are formed in the operation management device 50. Also, when the CPU 52 reads and executes a non-transitory storage medium such as a DVD storing the program, functional blocks as illustrated in Fig. 3 are formed in the operation management device 50.

[0028] That is, the operation management device 50 includes an operation schedule storage unit 66 and a dynamic map storage unit 67 as a storage unit. The operation management device 50 also includes an operation schedule creation unit 61, an operation route creation unit 62, a boarding / reservation setting unit 63, a congestion rate calculation unit 64, and a warning permission determination unit 65 as a functional unit.

[0029] The dynamic map storage unit 67 stores dynamic map data which is map data. The dynamic map is a three-dimensional map, and for example, the position and shape (three-dimensional shape) of a roadway are stored. The three-dimensional shape of a roadway includes, for example, gradient, width, etc. Also, the positions of lanes, crosswalks, stop lines, etc. drawn on the roadway are stored in the dynamic map. In addition, the positions and shapes (three-dimensional shapes) of structures such as bus stops, buildings, traffic signals, etc. around the road are stored in the dynamic map. Furthermore, the position and shape of a parking lot are also stored in the dynamic map.

[0030] For example, in the dynamic map, a geographic coordinate system including latitude and longitude is used. When the shared vehicle 10 performs an automated driving operation, the operation route creation unit 62 extracts dynamic map data from the dynamic map storage unit 67 and creates operation map data including the driving route and the positions of bus stops.

[0031] The operation schedule creation unit 61 creates an operation schedule (in other words, an operation timetable) for the shared vehicle 10. For example, when receiving operation map data from the operation route creation unit 62, the operation schedule creation unit 61 creates an operation schedule including the arrival scheduled times and departure scheduled times of each of the bus stops ST1 to ST3 based on the driving route, the rated speed of the shared vehicle 10, and the standard stop times at the bus stops ST1 to ST3.

[0032] FIG. 4 illustrates an operation schedule table provided to the shared vehicle 10. In the operation schedule table, in addition to the arrival scheduled times and departure scheduled times of each of the bus stops ST1 to ST3, the passing time of the operation schedule update point Pu (see FIG. 1), the scheduled passing time of the collection point Pout, and the scheduled passing time of the input point Pin are also set.

[0033] Furthermore, the operation schedule table is provided with columns for recording the actual passing, arrival, and departure times of each point and a column for inputting the congestion rate. In addition, the operation schedule table is provided with a column for recording the presence or absence of boarding reservations and alighting reservations and the designated seat numbers.

[0034] As described above, as this operation schedule, a schedule for one round of the specified route 90 starting from the operation schedule update point Pu is created. The created operation schedule is provided to the shared vehicle 10 when the shared vehicle 10 passes through the operation schedule update point Pu. The provided operation schedule is stored in the operation schedule storage unit 35 in the shared vehicle 10.

[0035] Referring to FIG. 3, the boarding / alighting reservation setting unit 63 receives boarding reservation information input from the mobile terminal (e.g., smartphone) of the passenger wishing to board and stores it in the operation schedule storage unit 66. The boarding reservation information includes the boarding scheduled stop, which is the stop where the passenger wishing to board is scheduled to board, the alighting scheduled stop, which is the stop where the passenger is scheduled to alight, and the designated seat number where the passenger is scheduled to sit.

[0036] For example, referring to FIG. 4, in this operation schedule, boarding reservation information for boarding at the stop ST2 and sitting in the designated seat 42B is set. Also, in this operation schedule, boarding reservation information for boarding at the stop ST3 and sitting in the designated seats 42D and 42E, and alighting reservation information for the passenger sitting in the designated seat 42B to alight at the stop ST3 are set. In this way, the boarding reservation information set by the boarding / alighting reservation setting unit 63 is stored in the operation schedule storage unit 66 via the operation schedule creation unit 61.

[0037] The congestion rate calculation unit 64 obtains the congestion rate of the passenger compartment 40 (see FIG. 6) based on the people and luggage recognized by the in-vehicle image recognition unit 36 of the shared vehicle 10.

[0038] For example, the congestion rate calculation unit 64 obtains the congestion rate of the passenger compartment 40 based on the number of people and the number of luggage recognized by the in-vehicle image recognition unit 36. For example, the congestion rate calculation unit 64 counts the total number of people in the passenger compartment 40 after replacing two pieces of luggage with one person, and obtains the congestion rate from the total number of people and the capacity of the passenger compartment 40.

[0039] As another method for calculating the congestion rate, the congestion rate calculation unit 64 obtains the sum ΣSb of the areas Sb of the bounding boxes, which are the rectangular frames in FIG. 8 described later, of the passengers (persons) and luggage recognized by the vehicle interior image recognition unit 36. This sum ΣSb is treated as the total occupied area of all passengers and all luggage in the vehicle compartment 40.

[0040] Furthermore, the congestion rate calculation unit 64 obtains the congestion rate in the vehicle compartment 40 based on the total area sum ΣSb and the floor area Sa of the vehicle compartment 40. Specifically, the congestion rate calculation unit 64 obtains the ratio (ΣSb / Sa)×100[%] of the total occupied area ΣSb of the passengers and luggage to the vehicle compartment floor area Sa as the congestion rate A. The obtained congestion rate A is sent to the warning determination unit 65.

[0041] The warning determination unit 65 determines whether to output a warning to prompt the elimination of luggage-occupied seats and overlapping seats based on the congestion rate A and the ride reservation status. The details of this determination process will be described later. That is, when it is estimated that at least one of a luggage-occupied seat and an overlapping seat occurs and the obtained congestion rate A exceeds a predetermined congestion threshold, the warning determination unit 65 causes the in-vehicle speaker 45 (see FIG. 6), which is a warning device, and the in-vehicle display 47B to output a warning.

[0042] <Shared vehicle> FIG. 5 illustrates the appearance of the shared vehicle 10, and FIG. 6 illustrates the state inside the vehicle compartment 40 of the shared vehicle 10. In FIGS. 5 and 6, the vehicle body front-rear direction is indicated by an axis represented by the symbol FR, and the vehicle width direction is indicated by an axis represented by the symbol RW (Right Width). Also, the vehicle height direction is indicated by an axis represented by the symbol UP.

[0043] Referring to FIG. 5, the shared vehicle 10 is used as a shared bus, and a pair of doors 41, 41 serving as boarding and alighting openings are provided on the left side. Also, an out-vehicle display 47A is provided in front of the vehicle. The out-vehicle display 47A is also called a so-called digital signage and is composed of a liquid crystal display or an LED display.

[0044] Referring to FIG. 6, inside the passenger compartment 40, a plurality of passenger seats are provided along its wall surface. For example, the seats include reserved seats 42A to 42F that can be reserved in advance and unreserved free seats 49. Also, for the support of standing passengers, a plurality of handholds 44 are provided on the ceiling. Further, a handrail 46 is provided on the side wall of the passenger compartment 40.

[0045] Above the doors 41, 41, an in-vehicle display 47B is provided. Similar to the out-vehicle display 47A, the in-vehicle display 47B is composed of a liquid crystal display or an LED display. As will be described later, the in-vehicle display 47B can display a warning message to prompt the elimination of occupied seats with luggage and double-seated passengers. Note that this warning message can also be output as voice guidance from an in-vehicle speaker 45 provided on the ceiling of the passenger compartment 40.

[0046] In the central part of the ceiling of the passenger compartment 40, an in-vehicle camera 18, which is an imaging device, is provided. For example, the in-vehicle camera 18 is provided at the center in the vehicle width direction and the center in the vehicle longitudinal direction of the passenger compartment 40. For example, the in-vehicle camera 18 includes an image sensor such as a CMOS sensor or a CCD sensor.

[0047] For example, the in-vehicle camera 18 may be a so-called 360° camera and can image the entire passenger space of the passenger compartment 40. For example, the in-vehicle camera 18 includes the entire floor 43 of the passenger compartment 40 in its field of view.

[0048] The in-vehicle image captured by the in-vehicle camera 18 is, for example, an overhead view image as shown in FIG. 7. According to such an in-vehicle image, the occupied areas on the floor 43 of the passengers 101A, 101B and the luggage 102A, 102B can be obtained.

[0049] <Automatic Driving Control Mechanism of Shared Vehicle> Referring to FIGS. 2 and 3, the shared vehicle 10 is, for example, an autonomous vehicle equipped with an automatic driving function. For example, when the shared vehicle 10 travels on a specified route 90 (see FIG. 1) in providing a transportation service, based on the standards by the Society of Automotive Engineers (SAE) in the United States, the shared vehicle 10 can perform automatic driving at level 4 or level 5.

[0050] The shared vehicle 10 is used as a shared bus that stops at stops ST1 to ST3 for passengers to board and alight while automatically driving on a specified route 90. For example, the shared vehicle 10 is an electric vehicle that uses a rotary electric machine 17 (motor) as a drive source and a battery (not shown) as a power source. The shared vehicle 10 also includes, as a driving control mechanism, a brake mechanism 14A, a steering mechanism 14B, and an inverter 14C that controls the output of the rotary electric machine 17.

[0051] Furthermore, the shared vehicle 10 includes an external camera 11A, a rider unit 11B, a proximity sensor 12, a positioning unit 13, a clock 15, and a control unit 20 as mechanisms for acquiring the position of the own vehicle and grasping the surrounding situation. For example, sensor units are provided on the front, rear, and both side surfaces of the shared vehicle 10. The sensor unit is configured to include the external camera 11A and the rider unit 11B.

[0052] The rider unit 11B is a sensor unit for automatic driving, and is a distance measuring unit capable of measuring the distance between an object around the own vehicle and the own vehicle. In the rider unit 11B, a technique of measuring the distance to surrounding objects using a rider (LiDAR, Light Detection and Ranging), that is, laser light, is used.

[0053] The rider unit 11B is, for example, a solid-state type unit, and an emitter that irradiates infrared laser light outward from the vehicle, a receiver that receives the reflected light, and a lens and a mirror that control the irradiation angle of the laser light are mounted on a semiconductor substrate. The rider unit 11B can obtain three-dimensional point cloud data about the surrounding environment of the shared vehicle 10.

[0054] The external camera 11A captures the same field of view as the rider unit 11B. The external camera 11A includes an image sensor such as a CMOS sensor or a CCD sensor. The proximity sensor 12 is, for example, an infrared sensor, and is provided, for example, at the four corners of the shared vehicle 10 in a plan view. For example, when the shared vehicle 10 arrives at the boarding location, the proximity sensor 12 detects a protrusion such as a curb of the sidewalk. By this detection, accurate control to bring the shared vehicle 10 close to the curb and stop it becomes possible. The positioning unit 13 is a system that performs positioning by artificial satellites, and for example, a global navigation satellite system is used.

[0055] The control unit 20 may be, for example, an electronic control unit (ECU) of the shared vehicle 10, and is composed of a computer (electronic computing machine). Referring to FIG. 2, as its hardware configuration, the control unit 20 includes an input / output controller 21 that controls data input / output. Further, the control unit 20 includes, as an arithmetic unit, a CPU 22, a GPU 23 (Graphics Processing Unit), and a DLA 24 (Deep Learning Accelerators). Further, the control unit 20 includes, as a storage unit, a ROM 25, a RAM 26, and a hard disk drive 27 (HDD). Note that instead of the hard disk drive 27, a storage device such as an SSD (Solid State Drive) may be used. These components are connected to an internal bus 28.

[0056] At least one of the ROM 25 and the hard disk drive 27, which are storage devices, stores a program for performing automatic driving control of the shared vehicle 10. When the above program is executed by the CPU 22 etc. of the control unit 20, function blocks as illustrated in FIG. 3 are formed in the control unit 20. Alternatively, also when the CPU 22 reads and executes a non-transitory storage medium such as a DVD storing the program, function blocks as illustrated in FIG. 3 are formed in the control unit 20.

[0057] That is, as a functional block, the control unit 20 includes a scan data analysis unit 30, a self-position estimation unit 31, an autonomous driving control unit 32, a driving guidance unit 33, a vehicle interior image recognition unit 36, and a seat state estimation unit 37. Further, as a storage unit, the control unit 20 includes a dynamic map storage unit 34 and a driving schedule storage unit 35.

[0058] The scan data analysis unit 30 acquires a captured image captured by the vehicle exterior camera 11A. The scan data analysis unit 30 performs image recognition on the acquired captured image using a known deep learning method. By this image recognition, object detection in the captured image and recognition of its attributes (such as vehicles, pedestrians, and structures) are performed.

[0059] Also, the scan data analysis unit 30 acquires three-dimensional point cloud data from the lidar unit 11B. Further, the scan data analysis unit 30 creates peripheral data by superimposing the coordinates of the captured image that has undergone image recognition and the three-dimensional point cloud data. With the peripheral data, it is possible to detect what kind of attribute objects are and how far they are separated from the shared vehicle 10. This peripheral data is sent to the autonomous driving control unit 32.

[0060] The self-position estimation unit 31 acquires self-position information (latitude, longitude) from the positioning unit 13. For example, the self-position estimation unit 31 acquires self-position information from artificial satellites. The self-position information (own vehicle position information) obtained in this way is sent to the autonomous driving control unit 32.

[0061] The dynamic map storage unit 34 stores the driving route map data created by the driving route creation unit 62 of the operation management device 50. This driving route map data includes dynamic map data. Also, the driving schedule storage unit 35 stores the driving schedule (see FIG. 4) created by the driving schedule creation unit 61 of the operation management device 50.

[0062] The autonomous driving control unit 32 performs driving control of the shared vehicle 10 based on the operation route map data stored in the dynamic map storage unit 34, the self-position information (own vehicle position information) transmitted from the self-position estimation unit 31, and the peripheral data transmitted from the scan data analysis unit 30. When arriving at the stops ST1 to ST3, the shared vehicle 10 waits at the stops ST1 to ST3 until the departure time set in the operation schedule.

[0063] <Estimation of Occupied Seats and Double-Occupied Seats> The in-vehicle image recognition unit 36 recognizes persons (passengers) and luggage included in the in-vehicle image captured by the in-vehicle camera 18, and further estimates whether the recognized persons are sitting or standing. For example, the in-vehicle image recognition unit 36 implements SSD (Single Shot Multibox Detector) using supervised learning as an image recognition algorithm.

[0064] Since the SSD algorithm is a known technology, it will be briefly described here. In the SSD algorithm, position estimation and class estimation are performed within the captured image. That is, two types of estimations, namely, where in the image an object is located and what the attribute (class) of that object is, are performed simultaneously, that is, in a single operation (Single Shot) using a neural network.

[0065] In the SSD algorithm, when detecting persons and luggage, instead of extracting the boundary lines between the persons and luggage and the outside, the boundaries of the persons and luggage are defined by "bounding boxes" in the form of rectangular frames as illustrated in FIG. 8. For example, in the SSD algorithm, for position estimation, a plurality of (for example, nearly 10,000) rectangular frames called default boxes with different sizes and shapes are fitted onto the image, and it is calculated which of these frames contains the object without excess or deficiency. Further, each bounding box displays the attribute (class) of the object captured by that box.

[0066] The in-vehicle image recognition unit 36 assigns values of "human" for a person and "baggage" for an object as attributes. Further, the in-vehicle image recognition unit 36 assigns values of "standing" for a standing position and "seated" for a seated position as sub-attributes of the person "human". That is, for the recognized person, it is estimated whether the person is in a seated position or a standing position.

[0067] For example, Fig. 8 shows the result of image recognition processing by the SSD algorithm of the in-vehicle image recognition unit 36 for the in-vehicle image (overhead view image) of Fig. 7. According to the example of Fig. 8, a bounding box 111A to which an attribute value of "human,standing" is assigned is set for the standing person 101A. Also, a bounding box 111B to which an attribute value of "human,seated" is assigned is set for the seated person 101B. Further, bounding boxes 112A and 112B to which an attribute value of the class "baggage" is assigned are set for the baggage 102A and 102B.

[0068] In order to enable such image recognition, for example, the teacher data for training the SSD algorithm includes a pair of teacher data sets in which a standing person image is used as input data and the class "person,standing (human,standing)" and the position parameters of the bounding box surrounding the person are used as output data. Also, this teacher data includes a pair of teacher data sets in which a seated person image is used as input data and the class "person,seated (human,seated)" and the position parameters of the bounding box surrounding the person are used as output data. Further, the teacher data includes a pair of teacher data sets in which a baggage image is used as input data and the class "baggage" and the position parameters of the bounding box surrounding the baggage are used as output data.

[0069] Position parameters are given to each bounding box. That is, the center coordinates C[Cx,Cy] of the box, the box width W, and the box height H in the imaging image plane coordinates are given as the position parameters of the bounding box.

[0070] The area Sb of the bounding box can be obtained from the product of the box width W and the box height H of the bounding box. In the in-vehicle monitoring system according to this specification, the area Sb of the bounding box is used as the occupied area of a person or luggage. That is, the in-vehicle image recognition unit 36 recognizes the image regions of persons (passengers) and luggage included in the in-vehicle image captured by the in-vehicle camera 18, and obtains the occupied areas of the recognized persons and luggage.

[0071] The seat state estimation unit 37 estimates the presence or absence of a seat occupied by luggage and the occurrence of overlapping seats. A seat occupied by luggage refers to a seat occupied by luggage. An overlapping seat occurs when a person in a single seat straddles multiple seats.

[0072] When the viewing angle and magnification of the in-vehicle camera 18 are fixed values, the positions of the seats in the captured image within the in-vehicle camera 18 are fixed. For example, FIG. 9 shows a captured image including assigned seats 42A to 42F, and the seat regions of each of these assigned seats 42A to 42F in the captured image plane are determined at fixed positions.

[0073] The seat state estimation unit 37 obtains the overlapping area between the bounding box 112B and each seat region from the position parameters (center coordinates C [Cx, Cy] of the box, box width W, and box height H) of the bounding box 112B of the luggage 102B recognized by the in-vehicle image recognition unit 36 and the coordinate information of the seat regions of each of the assigned seats 42A to 42F in the image plane.

[0074] For example, as shown in FIG. 9, the seat state estimation unit 37 obtains the overlapping area So between the bounding box 112B and the seat area 142B of the assigned seat 42B. Further, the overlapping rate L [%], which is the ratio of the overlapping area So to the area Sb of the bounding box 112B, is obtained. When the overlapping rate L is equal to or greater than a predetermined overlapping threshold value Lth, for example, 70% or more, the seat state estimation unit 37 estimates that the assigned seat 42B has become an occupied seat by the luggage 102B. Information (for example, the assigned seat number) of the assigned seat 42B estimated to be an occupied seat is transmitted to the warning determination unit 65 of the operation management device 50.

[0075] In addition, the seat state estimation unit 37 estimates the presence or absence of double-occupied seats. Referring to FIG. 10, from the position parameters (the center coordinates C [Cx, Cy] of the box, the box width W, and the box height H) of the bounding box 111B of the seated person 101B recognized by the vehicle interior image recognition unit 36 and the coordinate information of the seat areas in the image plane of each of the assigned seats 42A to 42F, the seat state estimation unit 37 obtains the overlapping area between the bounding box 111B and each seat area.

[0076] For example, as shown in FIG. 10, the overlapping areas So_B and So_C between the bounding box 111B and the seat areas 142B and 142C of the surrounding assigned seats 42B and 42C are obtained respectively. Further, the overlapping rates L_B and L_C [%], which are the ratios of the overlapping areas So_B and So_C to the area Sb of the bounding box 112B, are obtained.

[0077] When both of the overlapping rates L_B and L_C are equal to or greater than a predetermined seating threshold value Lths, for example, 30% or more, the seat state estimation unit 37 estimates that the assigned seats 42B and 42C have become double-occupied seats. Information (for example, the assigned seat number) of the assigned seats 42B and 42C estimated to be double-occupied seats is transmitted to the warning determination unit 65 of the operation management device 50.

[0078] <Warning determination flow> FIG. 11 illustrates a warning permission determination flow in the operation management system according to the present embodiment. In this flow, the execution entity of each step is indicated by the shared vehicle 10 (V) or the operation management device 50 (S).

[0079] For example, the flow of FIG. 11 is activated at the timing when the shared vehicle 10 departs from each bus stop ST1 to ST3. In the following, the flow executed while the shared vehicle 10 departs from the bus stop ST1 (7:23) and heads to the next scheduled bus stop ST2, as shown in the operation schedule illustrated in FIG. 12, will be described.

[0080] First, the operation plan unit 33 (see FIG. 3) sets the values of the luggage flag and the seat flag in a storage unit (not shown) to 0 (S10). As will be described later, the content of the warning message in the vehicle is set according to the values of these flags.

[0081] Next, the in-vehicle image recognition unit 36 acquires the in-vehicle image captured by the in-vehicle camera 18 (S12), and performs image recognition of the people (passengers) and luggage in the in-vehicle image as illustrated in FIG. 8 (S14). Further, the in-vehicle image recognition unit 36 estimates whether a person is standing or sitting.

[0082] In addition, the in-vehicle image recognition unit 36 transmits the position parameters of the bounding box (the center coordinates C [Cx, Cy] of the box, the box width W, and the box height H) surrounding the recognized luggage and people to the congestion rate calculation unit 64 of the operation management device 50.

[0083] Next, the seat state estimation unit 37 determines the presence or absence of a luggage-occupied seat for the seats 42A to 42F and 49 in the vehicle compartment 40 using the above-described overlap threshold value Lth (S16). If it is estimated that a luggage-occupied seat has occurred, the operation plan unit 33 sets the value of the luggage flag to 1 (S18). In addition, the information (such as the seat number) of the luggage-occupied seat is transmitted to the warning permission determination unit 65 of the operation management device 50.

[0084] Furthermore, the seat state estimation unit 37 determines whether or not there is an overlapping seat for the seats 42A to 42F and 49 in the passenger compartment 40 by using the above-described seating threshold value Lths (S19). When it is estimated that an overlapping seat has occurred, the operation guidance unit 33 sets the value of the seat flag to 1 (S22). Also, information on the overlapping seat (such as the seat number) is transmitted to the warning determination unit 65 of the operation management device 50.

[0085] Return to step S16. When it is estimated by the seat state estimation unit 37 that there is no occupied seat by luggage, the presence or absence of an overlapping seat is determined in the same manner as in step S19 (S20). When it is estimated by the seat state estimation unit 37 that there is no overlapping seat, since neither an occupied seat by luggage nor an overlapping seat has occurred and the in-vehicle manners are maintained well, the warning determination flow ends. On the other hand, when it is estimated in step S20 that there is an overlapping seat, the process proceeds to step S22 described above, and the value of the seat flag is set to 1.

[0086] The position parameters of the bounding box (center coordinates C [Cx, Cy] of the box, box width W, and box height H) surrounding the recognized luggage and persons are transmitted from the passenger compartment image recognition unit 36 to the congestion rate calculation unit 64 of the operation management device 50. The congestion rate calculation unit 64 obtains the total sum ΣSb of the areas Sb of the bounding boxes of the persons and luggage (S24).

[0087] The congestion rate calculation unit 64 obtains the congestion rate A [%], which is the ratio of the total sum ΣSb of the areas to the passenger compartment floor area Sa (S26). Here, referring to FIG. 8, a part of the bounding boxes 111A and 111B of the persons 101A and 101B overlaps with the bounding boxes 112A and 112B of the luggage 102A and 102B. Therefore, the total sum ΣSb of the areas may exceed the passenger compartment floor area Sa, and the congestion rate A may take a value exceeding 100%.

[0088] Even in such a case, since it is clear that the cause lies in the partial overlap of the bounding boxes, the congestion rate calculation unit 64 does not treat the congestion rate A as an error value even if it exceeds 100%. The obtained congestion rate A is transmitted to the warning determination unit 65. Also, the congestion rate A is recorded in the operation schedule stored in the operation schedule storage unit 66.

[0089] The warning determination unit 65 determines whether the congestion rate A exceeds a predetermined congestion threshold K1 [%] (S28). The congestion threshold K1 is a threshold for determining whether the passenger compartment 40 is in a congested state, and for example, K1 = 70 [%].

[0090] When the congestion rate A > the congestion threshold K1, the warning determination unit 65 sends a warning notice to the operation guidance unit 33 of the shared vehicle 10 (S30). In other words, the warning determination unit 65 notifies the shared vehicle 10 that a warning is to be output.

[0091] In response to this, the operation guidance unit 33 refers to the luggage flag and the seat flag. Then, it outputs a warning announcement according to the flag with a set value of 1 (S34). For example, when the set value of the luggage flag is 1, the operation guidance unit 33 causes an in-vehicle display 47B (see FIG. 6) and an in-vehicle speaker 45 that function as warning devices to output a message to move the luggage from the seat to the lap or the like. Also, when the set value of the seat flag is 1, the operation guidance unit 33 causes the in-vehicle display 47B (see FIG. 6) and the in-vehicle speaker 45 to output a message to use the seat in a packed state.

[0092] Returning to step S28, when the congestion rate A ≤ the congestion threshold K1, that is, when the vehicle interior is in a normal state with available space, the warning determination unit 65 executes a reservation status confirmation process to determine whether to suspend the output of a warning according to the ride reservation information (S32).

[0093] The warning determination unit 65 refers to the operation schedule stored in the operation schedule storage unit 66 to check whether a reservation for boarding at the next stop ST2 as the scheduled boarding stop is set (stop condition). Further, when such a boarding reservation is set, the warning determination unit 65 checks whether the reserved seat for the scheduled seating in the boarding reservation is one of the luggage-occupied seat and the double-seated seat (designated seat condition).

[0094] When a reservation for boarding at the next stop ST2 as the scheduled boarding stop is not set (stop condition = invalid), it is clear that the passenger who wishes to reserve a seat with a luggage-occupied seat or a double-seated seat as the designated seat is not at the next stop ST2 where the train is scheduled to stop. Also, since the interior of the train is empty as described above, the warning determination unit 65 withholds the output of the warning. That is, the luggage-occupied seat and the double-seated seat are permitted.

[0095] Also, even when a reservation for boarding at the next stop ST2 as the scheduled boarding stop is set, if a reservation for boarding with a luggage-occupied seat or a double-seated seat as the designated seat is not set (designated seat condition = invalid), the warning determination unit 65 withholds the output of the warning.

[0096] On the other hand, when both the stop condition and the designated seat condition are satisfied, that is, when a reservation for boarding at the next stop ST2 as the scheduled boarding stop is set and a reservation for boarding with a luggage-occupied seat or a double-seated seat as the designated seat for the scheduled seating is set, the warning determination unit 65 notifies the operation guidance unit 33 of the output of the warning (S30).

[0097] In this way, in the warning determination flow according to the present embodiment, even for an act that is regarded as a manner violation during congestion, it is possible to flexibly call for compliance with the boarding manner, such as withholding the issuance of a warning, according to the boarding situation such as the congestion rate and the reservation status.

[0098] <Another example of the vehicle interior monitoring system> FIG. 13 shows another example of the in-vehicle monitoring system illustrated in FIG. 3. In FIG. 13, the congestion rate calculation unit 64 and the warning permission determination unit 65, which were provided in the operation management device 50 in FIG. 3, are provided in the control unit 20 of the shared vehicle 10. Other configurations are the same as those in FIG. 3.

[0099] Functional blocks such as those in FIG. 13 are formed by the CPU 22 (see FIG. 2) executing a program for performing automatic driving control of the shared vehicle 10, in the same manner as in FIG. 3. This program is stored in at least one of the ROM 25 and the hard disk drive 27, which are storage devices, or in a non-transitory storage medium such as a DVD.

[0100] In such a configuration, the warning permission determination flow in FIG. 11 is executed independently by the shared vehicle 10. That is, the execution subject of each step in FIG. 11 is replaced with the shared vehicle 10 (V).

Explanation of Reference Numerals

[0101] 10 Shared vehicle, 18 In-vehicle camera (imaging device), 20 Control unit, 33 Route guidance unit, 35 Operation schedule storage unit, 36 In-vehicle image recognition unit, 37 Seat state estimation unit, 40 Passenger compartment, 42A - 42F Seats (reserved seats), 43 Floor of the passenger compartment, 45 In-vehicle speaker (warning device), 47B In-vehicle display (warning device), 50 Operation management device, 61 Operation schedule creation unit, 62 Route creation unit, 63 Boarding / reserving setting unit, 64 Congestion rate calculation unit, 65 Warning permission determination unit, 66 Operation schedule storage unit, 67 Dynamic map storage unit, 90 Prescribed route, 101A, 101B Persons (passengers), 102A, 102B Luggage, 111A, 111B, 112A, 112B Bounding boxes, 142B, 142C Seat areas.

Claims

1. A vehicle interior monitoring system comprising a shared vehicle and an operation management device for managing the operation of the shared vehicle, wherein the shared vehicle includes an imager provided on the ceiling of the vehicle interior, an image recognition unit that recognizes persons and luggage included in an image of the vehicle interior captured by the imager, and further estimates whether the recognized person is sitting or standing, a seat state estimation unit that estimates the presence or absence of a luggage-occupied seat occupied by luggage and the presence or absence of an overlapping seat occupied by a person sitting across a plurality of seats based on the positions of the recognized luggage and persons, and the operation management device includes a congestion rate calculation unit that obtains a congestion rate in the vehicle interior based on the recognized persons and luggage, a warning permission determination unit that commands the shared vehicle to output a warning when at least one of the luggage-occupied seat and the overlapping seat is estimated to occur and the obtained congestion rate exceeds a predetermined congestion threshold, and when the congestion rate is equal to or less than the congestion threshold, the warning permission determination unit executes a reservation status confirmation process for determining whether to suspend the output of the warning according to the boarding reservation information for the shared vehicle, the shared vehicle travels on a specified route and can stop at a stop provided along the specified route, as the boarding reservation information, a boarding scheduled stop, which is a stop where a boarding applicant plans to board, and a reserved seat where the applicant plans to sit are set, and the operation management device includes a storage unit that stores the boarding reservation information, as the reservation status confirmation process, the warning permission determination unit suspends the output of the warning by the shared vehicle when the boarding reservation information that satisfies both a stop condition that sets the next scheduled stop of the shared vehicle as the boarding scheduled stop and a specified seat condition that the specified seat corresponds to at least one of the luggage-occupied seat and the overlapping seat is not stored in the storage unit, a vehicle interior monitoring system.

2. The vehicle interior monitoring system according to claim 1, wherein the image recognition unit obtains the occupied areas of the recognized persons and luggage, and the congestion rate calculation unit obtains the congestion rate based on the total of the occupied areas of the recognized persons and luggage and the floor area of the vehicle interior. A vehicle interior monitoring system.

3. an imager provided on the ceiling of the vehicle interior, an image recognition unit that recognizes persons and luggage included in an image of the vehicle interior captured by the imager, and further estimates whether the recognized person is sitting or standing, A seat state estimation unit that estimates the presence or absence of a luggage-occupied seat occupied by luggage and the presence or absence of an overlapping seat occupied by a person across a plurality of seats based on the recognized positions of the luggage and the person; A congestion rate calculation unit that obtains a congestion rate in the passenger compartment based on the recognized person and luggage; A warning permission determination unit that causes a warning device to output a warning when at least one of the luggage-occupied seat and the overlapping seat is estimated to exist and the obtained congestion rate exceeds a predetermined congestion threshold; A storage unit that stores ride reservation information; A shared vehicle that travels along a specified route and can stop at a stop provided along the specified route; When the congestion rate is equal to or less than the congestion threshold, the warning permission determination unit executes a reservation status confirmation process for determining whether to suspend the output of the warning according to the ride reservation information; As the ride reservation information, a scheduled stop for boarding, which is a stop where a passenger wishes to board, and a reserved seat for seating are set, and the ride reservation information is stored in the storage unit; As the reservation status confirmation process, the warning permission determination unit suspends the output of the warning when the ride reservation information that satisfies both a stop condition with the next scheduled stop as the scheduled stop for boarding and a specified seat condition where the specified seat corresponds to at least one of the luggage-occupied seat and the overlapping seat is not stored in the storage unit; Shared vehicle.

Images