Passenger monitoring devices, operation management systems

Abstract

To provide a passenger watching device capable of executing a real-time notification corresponding to a behavior or a state of a passenger in an unmanned passenger vehicle, and an operation management system.SOLUTION: An in-vehicle monitoring ECU comprises a function for outputting a specific message corresponding to a content of a detected action in addition to center cooperation processing for outputting from an intra-vehicle speaker voices of an operator acquired by a microphone of a control center in a case where a specific action of a passenger is detected. The in-vehicle monitoring ECU calculates a predictive recovery time Trc that is a prediction value of a time until communication is recovered, based on a radio wave intensity map indicating a radio wave intensity for each point in a case where it is impossible to communicate with the control center. In a case where the predictive recovery time is a relatively short value such as several seconds, for example, the communication recovery with the center is awaited without outputting the specific message, and as soon as the communication is recovered, connection to the operator is performed. In a case where the predictive recovery time Trc is equal to or greater than a predetermined time, on the other hand, the specific message corresponding to the action content is output.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present disclosure relates to a technique for alerting passengers to the risk of falling or the like in a vehicle that transports passengers, such as a bus. [Background technology]

[0002] Patent Document 1 discloses a configuration that uses an in-car camera and an ultrasonic sensor to detect the possibility of a passenger falling and notifies the driver using an in-car display. Note that the states in which a passenger may fall include a standing state and a state in which the passenger is not holding on to a strap. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-62414 A Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 is based on the premise that a crew member such as a driver or a security officer is on board the vehicle. In other words, Patent Document 1 does not assume application to unmanned passenger vehicles, which are passenger vehicles without a crew member on board, such as unmanned buses and robot taxis. Unmanned passenger vehicles are vehicles that run using technologies such as automatic driving and remote control. Even in unmanned passenger vehicles, a configuration is required that can make announcements such as warnings about unsafe behavior / states of passengers who are at risk of falling (hereinafter, unsafe behavior, etc.).

[0005] One possible configuration for alerting passengers to unsafe behavior, etc., in an unmanned passenger vehicle is one in which a passenger monitoring device detects unsafe behavior, etc., of passengers based on video footage from an in-vehicle camera and notifies an operator at a control center of the detection. According to this possible configuration, the operator can make an announcement according to the actual situation based on the notification from the passenger monitoring device, after checking the video footage from the in-vehicle camera transmitted wirelessly from the vehicle. The passenger monitoring device here refers to a device that is installed in a vehicle and has the function of detecting unsafe behavior, etc., of passengers. The passenger monitoring device can be understood as a device on the vehicle side that constitutes the operation management system.

[0006] However, as the unmanned passenger vehicle travels, the wireless communication environment changes dynamically. Therefore, the communication state between the passenger monitoring device and the control center is not always good. There may be a state in which the passenger monitoring device and the control center cannot communicate with each other, or a state in which the communication delay exceeds a predetermined value. If a communication problem occurs between the passenger monitoring device and the control center, it becomes difficult to make real-time announcements about unsafe behavior of passengers, etc.

[0007] The present disclosure has been made based on the above considerations or points of view, and one of its objectives is to provide a passenger monitoring device and operation management system that can issue alerts in real time based on passenger behavior or status in passenger vehicles where no crew is present. [Means for solving the problem]

[0008] The passenger monitoring device disclosed herein is a passenger monitoring device used in a passenger vehicle configured to be capable of operating under the management of a control center without a crew member on board, and configured to be capable of wireless communication with the control center, and includes a report processing unit (F3) that transmits to the control center an interior image of the vehicle, which is an image taken by a camera that captures the interior of the vehicle and is mounted on the passenger vehicle, a specific behavior detection unit (F1) that detects that a passenger is performing a specific behavior based on an input signal from a sensor mounted on the passenger vehicle, an alert data storage unit (M1) that stores data for outputting a voice message corresponding to the content of the specific behavior, and a detection unit (M2) that outputs a notification data corresponding to the content of the specific behavior based on the detection of the performance of the specific behavior. the passenger vehicle includes a response unit (F4) that performs a response process corresponding to the specified specific action received from the passenger vehicle, and a communication status acquisition unit (F2) that acquires a current communication status with the control center from a wireless communication device mounted on the passenger vehicle, and the response unit is configured to selectively, in parallel, or sequentially perform at least two types of processes as the response process: outputting a prescribed message corresponding to the specific action by voice from an in-vehicle speaker (18) that is a speaker installed in the passenger vehicle using data stored in the notification data storage unit; and acquiring voice data transmitted from the control center based on an in-vehicle image and outputting a voice corresponding to the voice data from the in-vehicle speaker. The response unit further comprises: Change the response process to be executed based on the communication status with the control center when the execution of a specific action is detected. When the performance of a specific action is detected and a problem occurs in communication with the control center, an estimated recovery time, which is the time until the communication quality with the control center reaches a predetermined required level or higher, is calculated based on communication quality map data indicating the wireless communication quality for each point stored in a specified storage medium (M2), and if the estimated recovery time is shorter than a predetermined allowable waiting time, the output of the specified message is put on hold and the device is configured to wait for the reception of voice data from the control center.

[0009] The response unit of the passenger monitoring device is configured to be capable of outputting a prescribed message corresponding to a specific action being performed by a passenger, in addition to processing to output a voice acquired from the control center. With such a configuration, a flexible response can be implemented to a specific action of a passenger according to the communication state with the control center. For example, when communication with the control center is possible, a center cooperation process is implemented, whereas when communication is not possible, a prescribed message is output. Thus, with the above configuration, even if a problem occurs in communication with the control center, a notification according to the behavior or state of a passenger can be implemented in real time in the unmanned passenger vehicle.

[0010] Further, a traffic management system for achieving the above object includes a control center (2) using at least one processor (231) for managing the operation of a passenger vehicle configured to be able to operate without a crew member on board, and a passenger monitoring device (30) used in the passenger vehicle and configured to be able to wirelessly communicate with the control center, the passenger monitoring device including a report processing unit (F3) for transmitting to the control center an in-vehicle image which is an image taken by a camera for capturing an image of the interior of the vehicle and mounted on the passenger vehicle, a specific behavior detection unit (F1) for detecting that a passenger is performing a specific behavior based on an input signal from a sensor mounted on the passenger vehicle, a notification data storage unit (M1) in which data for outputting a voice message corresponding to the content of the specific behavior is stored, a response unit (F4) for performing a response process corresponding to the detected specific behavior based on the detection of the performance of the specific behavior, a communication status acquisition unit (F2) for acquiring a current communication status with the control center from a wireless communication device mounted on the passenger vehicle, and a notification unit (M3) for detecting the performance of the specific behavior. a prescribed message output unit (F42) that uses data stored in the notification data storage unit to output a prescribed message corresponding to the specific action from an in-car speaker (18) that is a speaker installed in the passenger car based on the detection of the performance of the specific action, by voice, from the in-car speaker (18); and a center linkage unit (F43) that acquires voice data for an in-car announcement for the specific action from the control center based on the detection of the performance of the specific action, and outputs a voice corresponding to the voice data from the in-car speaker. When there is no problem in communication with the control center, the center linkage unit operates with priority over the prescribed message output unit, and a processor (231) of the control center is configured to perform the following operations based on the detection of the performance of the specific action: displaying an image for the operator to confirm the situation on an operator's display (221), acquiring the operator's voice in response to the specific action via a microphone (222), and transmitting the voice data corresponding to the operator's voice to the passenger monitoring device. The response unit is configured to, when the performance of the specific action is detected and a problem occurs in communication with the control center, calculate an expected recovery time, which is the time until the communication quality with the control center becomes equal to or higher than a predetermined required level, based on communication quality map data indicating the wireless communication quality for each point stored in a predetermined storage medium (M2), and, when the expected recovery time is shorter than a predetermined allowable waiting time, suspend output of the specified message and wait for reception of voice data from the control center. .

[0011] In the above operation management system, the passenger monitoring device is configured to output a prescribed message corresponding to a specific action being performed by a passenger, in addition to the center cooperation process that outputs a voice based on an instruction from the control center. With this configuration, for example, when communication with the control center is impossible, it is possible to take emergency measures such as outputting a prescribed message. In other words, even if a problem occurs in communication between the passenger monitoring device and the control center, it is possible to issue a notification in real time according to the behavior or state of the passenger in the unmanned passenger vehicle.

[0012] In addition, the symbols in parentheses in the claims indicate a correspondence with the specific means described in the embodiments described below as one aspect, and do not limit the technical scope of the present disclosure. [Brief description of the drawings]

[0013] [Figure 1] 1 is a diagram illustrating an overall view of a traffic monitoring system. [Diagram 2] FIG. 1 is a block diagram showing an example of a configuration of an in-vehicle system. [Diagram 3] FIG. 2 is a block diagram showing an example of an in-vehicle sensor included in the in-vehicle system. [Figure 4] FIG. 2 is a functional block diagram of an interior monitoring ECU. [Diagram 5] FIG. 11 is a diagram for explaining an expected recovery time. [Figure 6] FIG. 13 is a diagram showing an example of a passenger correspondence table. [Figure 7] FIG. 2 is a block diagram showing the configuration of a control center. [Figure 8] FIG. 13 is a diagram showing an example of a summary notification image. [Figure 9] 4 is a flowchart for explaining a vehicle-side process. [Figure 10] 13 is a flowchart for explaining emergency response processing. [Figure 11] 13 is a flowchart illustrating a center cooperation process. [Figure 12]13 is a flowchart for explaining a center side process. [Figure 13] 13 is a flowchart illustrating a prescribed message output process. [Figure 14] 13 is a flowchart for setting a caution target based on a fall risk. [Figure 15] 13 is a flowchart of a normal mode return determination process. [Figure 16] FIG. 13 is a block diagram showing a modified example of the configuration of the in-vehicle system. [Figure 17] FIG. 13 is a block diagram showing a modified example of the configuration of the in-vehicle system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] An example of an embodiment of the present disclosure will be described with reference to the drawings. The present disclosure can be implemented with appropriate modifications so as to conform to the laws, regulations and customs of the region to which the following traffic control system Sys is applied.

[0015] <Introduction> Fig. 1 is a diagram showing an example of a schematic configuration of a traffic control system Sys according to the present disclosure. As shown in Fig. 1, the traffic control system Sys according to the present disclosure includes an in-vehicle system IvS mounted on an autonomous driving bus 1, and a control center 2. The in-vehicle system IvS includes an autonomous driving ECU 20 that provides autonomous driving functions, as described below, and an in-vehicle monitoring ECU 30 that monitors the in-vehicle situation. Note that although Fig. 1 shows only one autonomous driving bus 1, there may be multiple autonomous driving buses 1 managed by the control center 2.

[0016] The autonomous bus 1 is a bus equipped with an autonomous driving ECU 20 and configured to be capable of traveling autonomously (automatically) based on the detection results of an external sensor 11. The autonomous bus 1 corresponds to a bus without a driver or other crew member inside the vehicle. The autonomous bus 1 can also be called an unmanned bus, an autonomous bus, a robot bus, or the like. The autonomous bus 1 may be one that periodically travels along a preset route, or may be a so-called on-demand bus that flexibly changes its route based on a boarding request from a user. The autonomous bus 1 may be a shuttle bus or a school bus.

[0017] Of course, the vehicle to which the in-vehicle system IvS is applied is not limited to a passenger vehicle that maintains the form of a bus. The in-vehicle system IvS may be applied to a robot taxi, which is a taxi that runs autonomously without a driver on board. The configuration of the present disclosure is applicable to various types of passenger automobiles. In addition, the present disclosure is not limited to automobiles and can also be applied to trains. Hereinafter, the vehicle equipped with the in-vehicle system IvS will also be referred to as the host vehicle.

[0018] Also, as an example here, the autonomous bus 1 is an electric vehicle, but is not limited to this. The host vehicle may be an engine vehicle or a hybrid vehicle. An electric vehicle refers to a vehicle equipped with only a motor as a drive source. Electric vehicles also include fuel cell vehicles (FCV: Fuel Cell Vehicles). A gasoline vehicle is a vehicle equipped with only an engine as a drive source, and corresponds to a vehicle that runs on fuel such as gasoline. Engine vehicles also include diesel vehicles. A hybrid vehicle refers to a vehicle equipped with an engine and a motor as a power source.

[0019] Control center 2 is a center that manages the operation status, etc. of autonomous bus 1. By wirelessly connecting to wireless base station 3, autonomous bus 1 is configured to be able to communicate with control center 2 via wide area communication network 4, indicated by WAN in Fig. 1. For example, autonomous bus 1 sequentially transmits data indicating its current position, the state of the interior of the vehicle, the surrounding conditions, etc. to control center 2.

[0020] The wide area communication network 4 is, for example, the Internet. The wide area communication network 4 may be, for example, an IP (Internet Protocol) network other than the Internet, or a network provided by a communication carrier, such as a mobile phone network. WAN in FIG. 1 is an abbreviation for Wide Area Network.

[0021] The wireless base station 3 is a communication facility that forms a mobile communication network that complies with the LTE / 4G / 5G / 6G standards (hereinafter referred to as 4G, etc.). The wireless base station 3 is a facility that transmits and receives wireless signals that comply with standards such as LTE to and from the autonomous driving bus 1. The wireless base station 3 is, for example, an evolved NodeB (eNB) that complies with 4G. Of course, the wireless base station 3 may also be a next generation NodeB (gNB) used in 5G. The wireless base station 3 exchanges control signals with a wireless communication device mounted on the autonomous driving bus 1, thereby realizing access to the wide area communication network 4 and, ultimately, data communication between the autonomous driving bus 1 and the control center 2.

[0022] <Overall configuration of the in-vehicle IvS system> The in-vehicle system IvS includes various components as shown in Fig. 2 as an example. That is, the in-vehicle system IvS includes an external sensor 11, a vehicle state sensor 12, a locator 13, a map storage device 14, an in-vehicle sensor 15, a wireless communication device 16, a display 17, and an in-vehicle speaker 18. The in-vehicle system IvS also includes a travel actuator 19, an autonomous driving ECU 20, and an in-vehicle monitoring ECU 30. The ECU in the component names is an abbreviation for Electronic Control Unit, and means an electronic control device.

[0023] Each of the devices such as the external sensor 11 is connected to the in-vehicle monitoring ECU 30 so as to be able to communicate with each other via the in-vehicle network IvN, which is a communication network constructed in the vehicle. Note that the connection between the devices disclosed here is an example and can be changed as appropriate. Some of the components may be connected to each other by dedicated communication lines without going through the in-vehicle network IvN. The topology of the in-vehicle network IvN is not limited to a bus type, and may be a mesh type, a star type, a ring type, or the like. Various standards can be adopted as standards for the in-vehicle network IvN, such as Controller Area Network (hereinafter, CAN: registered trademark), Ethernet (Ethernet is a registered trademark), and FlexRay (registered trademark).

[0024] The external sensor 11 is a sensor that monitors the surroundings of the vehicle. The external sensor 11 is configured to detect the presence and position of a predetermined detection target. For example, a perimeter monitoring camera, a millimeter wave radar, LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), sonar, etc. can be adopted as the external sensor 11. The millimeter wave radar is a device that detects the position of an object present within a detection range by transmitting millimeter waves or quasi-millimeter waves in a predetermined direction and analyzing the received data of the reflected waves that are returned after the transmitted waves are reflected by an object. The LiDAR is a device that generates three-dimensional point cloud data indicating the position of a reflection point for each detection direction by irradiating a laser light. The LiDAR is also called a laser radar. The LiDAR may be a scan type or a flash type.

[0025] The in-vehicle system IvS may include multiple types of external sensors 11 for each detection direction, such as a LiDAR and at least one perimeter monitoring camera. The perimeter monitoring camera is an in-vehicle camera arranged to capture an image of the outside of the vehicle in a predetermined direction. The perimeter monitoring camera includes a front camera arranged on the upper end of the windshield on the interior side of the vehicle or on the front grill to capture an image of the front of the vehicle. The front camera detects various detection targets using a classifier using, for example, a CNN (Convolutional Neural Network) or a DNN (Deep Neural Network).

[0026] The detection target object includes, for example, a moving object such as a pedestrian or another vehicle. The other vehicles include bicycles, motorbikes, and motorcycles. The external sensor 11 is also configured to be able to detect predetermined features. The features that the external sensor 11 detects include road edges, medians, road markings, and three-dimensional structures installed along the road. Examples of road markings include lane markings that indicate the boundaries of lanes, crosswalks, stop lines, guiding strips, safety zones, and regulatory arrows. Examples of three-dimensional structures installed along the road include guardrails, road signs, and traffic lights.

[0027] The object recognition process based on the observation data generated by the external sensor 11 may be executed by an ECU (Electronic Control Unit) outside the sensor, such as the autonomous driving ECU 20. In this case, the various external sensors 11 provide the observation data, such as image data and distance measurement data, as detection result data to the autonomous driving ECU 20. Note that the observation data here refers to, for example, video / image data generated by a camera, 3D point cloud data generated by a LiDAR, and reception result data of exploration waves acquired by a millimeter wave radar / sonar.

[0028] The vehicle state sensor 12 is a group of sensors that detect information related to the state of the host vehicle. The vehicle state sensor 12 includes a vehicle speed sensor, a steering angle sensor, an acceleration sensor, a yaw rate sensor, and the like. The vehicle speed sensor detects the vehicle speed of the host vehicle. The steering angle sensor detects the steering angle, which is the angle of the tire relative to the vehicle longitudinal direction or the axle. The steering angle may also be called a turning angle. The acceleration sensor detects the acceleration of the host vehicle, such as the longitudinal acceleration and lateral acceleration. The yaw rate sensor detects the yaw rate (angular velocity) acting on the host vehicle. The acceleration sensor and the yaw rate sensor may also be called inertial sensors. The vehicle state sensor 12 outputs data indicating the current value (i.e., the detection result) of the physical state quantity to be detected to the in-vehicle network IvN. The detection result output to the in-vehicle network IvN is referred to and used by the autonomous driving ECU 20, the in-vehicle monitoring ECU 30, and the like. Note that the types of sensors that the in-vehicle system IvS has as the vehicle state sensor 12 may be designed appropriately, and it is not necessary to have all of the above-mentioned sensors. Further, sensors other than those described above, such as sensors for detecting the open / closed state of doors, and sensors for detecting the open / closed and locked states of each window, may be provided.

[0029] The locator 13 is a device that generates highly accurate position information of the vehicle by composite positioning that combines multiple pieces of information. The position of the vehicle is expressed by three-dimensional coordinates of latitude, longitude, and altitude, for example. The locator 13 is configured using, for example, a GNSS receiver. The GNSS receiver is a device that sequentially detects the current position of the GNSS receiver by receiving navigation signals (hereinafter, positioning signals) transmitted from positioning satellites that constitute the GNSS (Global Navigation Satellite System). For example, when the GNSS receiver is able to receive positioning signals from four or more positioning satellites, it outputs positioning results every 100 milliseconds. As the GNSS, GPS, GLONASS, Galileo, QZSS, Beidou, etc. can be adopted.

[0030] The locator 13 may sequentially calculate the position of the vehicle by combining the positioning result of the GNSS receiver and the output of the inertial sensor. For example, when the GNSS receiver cannot receive the GNSS signal, such as in a tunnel, the locator 13 performs dead reckoning (i.e., autonomous navigation) using the yaw rate and the vehicle speed. The vehicle position information calculated by the locator 13 is output to the in-vehicle network IvN and used by the autonomous driving ECU 20, the in-vehicle monitoring ECU 30, etc.

[0031] The map storage device 14 is a storage medium that stores map data. The map data stored in the map storage device 14 may be high-precision map data or navigation map data, which is map data for navigation. High-precision map data corresponds to map data that indicates the road structure and the position coordinates of features located along the road with an accuracy that can be used for autonomous driving. The accuracy that can be used for autonomous driving corresponds to a level in which the error between the actual position of each map element and the position registered on the map is suppressed to 10 cm or 20 cm or less, for example. Navigation map data corresponds to map data for navigation, which is relatively less accurate than high-precision map data.

[0032] Map storage device 14 only needs to hold map data for the planned driving area, which is an area in which autonomous bus 1 can drive. Map storage device 14 includes information on road connections in the planned driving area, road shapes, the number of lanes, and the like. The map data held by map storage device 14 preferably also includes information on bus stops and the lanes in which autonomous bus 1 should drive. Lanes in which autonomous bus 1 should drive include bus-only lanes, bus priority lanes, and the like.

[0033] The in-vehicle sensor 15 is a sensor that detects and outputs information indicating the state of passengers in the vehicle. The in-vehicle system IvS includes, as the in-vehicle sensor 15, an in-vehicle camera 151, an in-vehicle microphone 152, a seating sensor 153, an ultrasonic sensor 154, and an infrared camera 155, as shown in FIG.

[0034] The in-vehicle camera 151 is, for example, a visible light camera installed on the ceiling of the vehicle interior so as to be able to capture images of passengers. For example, a CMOS camera or a CCD camera can be used as the in-vehicle camera 151. There may be one in-vehicle camera 151, or multiple in-vehicle cameras may be provided. By providing multiple in-vehicle cameras 151, it is possible to reduce the risk of blind spots occurring in the vehicle, and even if one of the cameras breaks down, it is possible to obtain in-vehicle images using the remaining cameras. Of course, there may be only one in-vehicle camera 151. The in-vehicle camera 151 may be configured using a fisheye lens so as to capture as wide an area as possible. The in-vehicle camera 151 sequentially outputs image data to the in-vehicle monitoring ECU 30. Note that the expression "image data" also includes video signals.

[0035] The in-vehicle microphone 152 is a device (so-called microphone) for acquiring the speech of a passenger. The in-vehicle microphone 152 outputs an electric signal corresponding to the voice in the vehicle to the in-vehicle monitoring ECU 30. The in-vehicle microphones 152 may also be distributed at multiple locations. For example, the in-vehicle microphones 152 may be provided for each seat. The in-vehicle microphones 152 may be arranged so that one microphone includes the vicinity of two adjacent seats in its sound collection range. In addition, the height positions of the heads of a seated person and a standing person are different. For such reasons, the in-vehicle system IvS may be provided with separate in-vehicle microphones 152 for a standing person and for a seated person. The installation position and installation posture of the in-vehicle microphone 152 may be appropriately selected in consideration of the directivity.

[0036] The seating sensor 153 is a sensor that detects whether a passenger is seated, and is provided for each seat, for example. The seating sensor 153 can be, for example, a pressure sensor embedded in the seating surface of each seat.

[0037] The ultrasonic sensor 154 is a device that detects the distance to an object present within a detection range by transmitting ultrasonic waves in a predetermined direction and analyzing the received data of the reflected waves that are returned after the transmitted waves are reflected by the object. The ultrasonic sensor 154 is also called a sonar. The ultrasonic sensor 154 is arranged, for example, in a ceiling portion above each seat, and emits ultrasonic waves toward the seat surface. The ultrasonic sensor 154 outputs a signal indicating the distance to an object present within the detection range to the interior monitoring ECU 30. The detection result of the ultrasonic sensor 154 indicates the occupancy status of the seat, that is, whether the seat is vacant or whether luggage is placed on the seat. The detection result of the ultrasonic sensor 154 also indicates the height of the seat occupant.

[0038] The infrared camera 155 is a camera that generates an infrared image using an image sensor in which elements that detect near-infrared rays are arranged in an array. The value of each pixel in the infrared image indicates the intensity of the infrared rays. The infrared camera 155 may be equipped with a light source module that projects near-infrared rays toward a detection range. The infrared camera 155 may be installed, for example, on a ceiling part in the same manner as the in-vehicle camera 151. The infrared camera 155 sequentially outputs infrared image data to the in-vehicle monitoring ECU 30. The infrared image generated by the infrared camera 155 is used as material for identifying the position, body temperature, posture, etc. of a passenger. The infrared camera 155 may be configured as a thermal camera that measures temperature by detecting the strength of far-infrared rays emitted from a human body. The above-mentioned infrared image may be a thermal image (thermography).

[0039] The types of sensors that the in-vehicle system IvS has as the in-vehicle sensor 15 may be appropriately designed, and it is not necessary to have all of the above-mentioned sensors. In addition, the in-vehicle sensor 15 may include sensors other than those described above, such as a millimeter wave radar, a LiDAR, an infrared camera 155, a photoelectric sensor, a contact sensor, and the like.

[0040] The millimeter wave radar or LiDAR may be placed on the ceiling in place of the ultrasonic sensor 154 or together with the ultrasonic sensor 154. The ultrasonic sensor, millimeter wave radar, and LiDAR correspond to examples of distance measuring sensors that detect the distance to an object using a search wave. Laser light is also included in the concept of a search wave because it is a type of electromagnetic wave.

[0041] In addition, each strap and handrail may be provided with a contact sensor to detect whether or not it is being held by a passenger. The contact sensor may be a pressure-sensitive touch sensor or a capacitance-type touch sensor. Various in-vehicle sensors 15 provide signals to the in-vehicle monitoring ECU 30 to identify the situation inside the vehicle, specifically, the state of the passengers. Note that the state of the passengers may be determined based on the observation data by each in-vehicle sensor 15 or by the in-vehicle monitoring ECU 30. The functional layout may be changed as appropriate. In this example, the in-vehicle monitoring ECU 30 acquires passenger information by analyzing input signals from each in-vehicle sensor 15.

[0042] The wireless communication device 16 is a device for performing wireless communication with an external device such as the control center 2. The wireless communication device 16 includes a communication module for performing wireless communication conforming to a predetermined wide-area wireless communication standard. Various standards such as LTE (Long Term Evolution), 4G, and 5G can be adopted as the wide-area wireless communication standard here. By installing the wireless communication device 16, the vehicle becomes a connected car that can connect to the Internet.

[0043] The wireless communication device 16 outputs information indicating whether data communication with external devices including the control center 2 is possible to the in-vehicle monitoring ECU 30 based on the quality of the received signal, etc. Furthermore, when data communication with the control center 2 is possible, the wireless communication device 16 may measure an index indicating the communication state with the control center 2, such as an average value of the communication speed (so-called throughput) and latency, and output it to the in-vehicle monitoring ECU 30. The latency here refers to the transmission time required for a signal or data transmitted by the wireless communication device 16 to reach the control center 2. When the wireless communication device 16 periodically exchanges control signals for communication confirmation with the control center 2, the wireless communication device 16 may be configured to measure round-trip latency, which is the time required from transmitting a signal addressed to the control center 2 to receiving a response. The round-trip latency may also be called round-trip time (RTT).

[0044] Furthermore, the wireless communication device 16 can calculate indices that directly or indirectly indicate the quality / speed of wireless communication, such as RSRP, RSSI, RSRQ, SINR, and PHR, by using a reference signal (RS) transmitted from the wireless base station 3. The wireless communication device 16 may output these indices of communication quality / communication speed to the in-vehicle monitoring ECU 30.

[0045] Note that RSRP is an abbreviation for Reference Signal Received Power, and RSRP indicates the average received power of the RS per unit resource element. RSSI is an abbreviation for Received Signal Strength Indicator. RSSI indicates a measured value of the power of the entire LTE system band in the OFDM symbol that contains the RS. RSSI and RSRP correspond to index values ​​that indicate the strength of the signal (received radio wave) from the wireless base station 3. Hereinafter, radio wave strength refers to RSSI or RSRP.

[0046] RSRQ is an abbreviation for Reference Signal Received Quality. RSRQ is the ratio of the received power of a cell-specific reference signal to the total power within the reception bandwidth. SINR is an abbreviation for Signal to Noise Interference Ratio, and roughly indicates the ratio of the power of a desired signal to the power of non-desired signals among received signals. The power of non-desired signals refers to, for example, interference waves from other cells / sectors and power originating from thermal noise. Hereinafter, reception quality corresponds to RSRQ or SINR. The larger the value of indexes such as RSRP, RSSI, RSRQ, and SINR, the better the reception quality of the signal from the wireless base station 3.

[0047] PHR stands for Power Headroom and indicates the margin of transmission power. The larger the PHR, the more bandwidth is available for upstream communication. The calculation method for PHR, RSRP, RSSI, and RSRQ can be calculated by the method specified by 3GPP (Third Generation Partnership Project), so the explanation will be omitted here.

[0048] In addition, the wireless communication device 16 may be configured to perform vehicle-to-vehicle communication, which is wireless communication with other vehicles, and road-to-vehicle communication, which is wireless communication with roadside devices. A roadside device refers to communication equipment installed along a road. For example, the wireless communication device 16 may be configured to perform cellular V2X.

[0049] The display 17 is a device that displays images. The display 17 displays an image corresponding to a video signal input from the interior monitoring ECU 30. The display 17 is disposed in a position that is easily visible to passengers, for example, near the ceiling inside the vehicle. More specifically, the display 17 is disposed above the boarding and alighting doors. The display 17 is capable of full-color display, and can be realized by using a liquid crystal display, an OLED (Organic Light Emitting Diode) display, a plasma display, or the like. The in-vehicle system IvS may be equipped with multiple displays 17.

[0050] The in-vehicle speaker 18 is a device that outputs sound. In this disclosure, the expression "sound" includes voice and music. The in-vehicle speaker 18 outputs sound corresponding to an audio signal input from the in-vehicle monitoring ECU 30. The in-vehicle speaker 18 is disposed, for example, on the ceiling or wall inside the vehicle cabin. The in-vehicle system IvS may include multiple in-vehicle speakers 18. For example, the in-vehicle speaker 18 may be provided for each seat.

[0051] The traveling actuator 19 refers to actuators for driving the autonomous bus 1. For example, a braking device, an electronic throttle, a steering actuator, and the like correspond to the traveling actuator 19. An example of a braking device is a brake actuator. The steering actuator includes an EPS (Electric Power Steering) motor. Note that other ECUs, such as a steering ECU that performs steering control, a power unit control ECU that performs acceleration / deceleration control, and a brake ECU, may be interposed between the autonomous driving ECU 20 and the traveling actuator 19.

[0052] The autonomous driving ECU 20 is an ECU that performs some or all of the driving operations on behalf of the user by controlling the driving actuator 19 based on the detection results of the external sensor 11. The autonomous driving ECU 20 is also called an automatic driving device. The autonomous driving ECU 20 is configured as a computer equipped with a CPU (Central Processing Unit), RAM (Random Access Memory), storage, and a bus line connecting these components.

[0053] The autonomous driving ECU 20 has a function equivalent to level 4 defined by the Society of Automotive Engineers (SAE International). Autonomy level 4 is a level at which the system performs all driving tasks except under specific circumstances such as certain unsupported roads and extreme environments. Level 4 corresponds to a level at which the system performs all driving tasks within a pre-set operational design domain (ODD). Level 4 corresponds to so-called highly autonomous driving. The autonomous driving function provided by the autonomous driving ECU 20 may be equivalent to level 3 or level 5, for example.

[0054] The autonomous driving ECU 20 includes a control plan unit G1 and a control execution unit G2 as more detailed functional units. The control plan unit G1 creates a control plan for the vehicle using the detection results of the external sensor 11 and the map data stored in the map storage device 14. The control plan includes a driving route that connects driving positions at each time. For example, the control plan includes long-term route information at the road level, such as which roads to pass and which intersections to turn at, and short-term route information indicating which lanes to drive in in road sections traveled within a certain time period. The control plan may include information such as schedule information for acceleration / deceleration for speed adjustment on the calculated route, schedule information for steering angle control, information on points for lane changes, and timing for notifying passengers of vehicle behavior. The autonomous driving ECU 20 creates a plan for acceleration / deceleration and steering according to the curvature of the road identified based on at least one of the detection results of the external sensor 11 and the map data.

[0055] When it is detected that the vehicle will pass through an intersection within a predetermined time based on at least one of the detection result of the external sensor 11 and the map data, the control planning unit G1 creates a control plan for a driving route in the intersection and stopping according to the lighting state of the traffic lights. When the external sensor 11 detects the presence of an obstacle, such as a parked vehicle on the road, in front of the vehicle, the control planning unit G1 may generate a driving plan for passing the side of the obstacle. In addition, the control planning unit G1 also plans emergency avoidance actions to avoid collisions with other objects, such as avoiding objects that have fallen on the road and control to stop in front of pedestrians. The outline of the control plan created by the control planning unit G1 is also notified to the in-vehicle monitoring ECU 30. Note that part of the process for creating the control plan and part or all of the process for recognizing the vehicle's driving environment may be executed by the control center 2. The arrangement of the functions related to the automatic driving can be changed as appropriate.

[0056] The control execution unit G2 determines the control amount for each traveling actuator 19 based on the control plan created by the control planning unit G1, and outputs a control signal to each traveling actuator 19. The control execution unit G2 also controls the turning on / off of turn signals, headlights, hazard lights, etc. based on the control plan created by the control planning unit G1 and the external environment.

[0057] In this embodiment, the autonomous driving ECU 20 has a fallback mode as an operation mode in addition to a normal mode. The normal mode is an operation mode that is applied when communication with the control center 2 can be normally performed. In the normal mode, the autonomous driving ECU 20 creates a control plan so that the driving speed, acceleration / deceleration acting on the vehicle body, yaw rate, and the like fall within a predetermined standard allowable range.

[0058] On the other hand, the fallback mode is an operation mode that can be applied based on an instruction from the in-vehicle monitoring ECU 30 when a problem occurs in communication with the control center 2. In the fallback mode, the autonomous driving ECU 20 creates a control plan so that the driving speed, acceleration / deceleration acting on the vehicle body, yaw rate, and the like are smaller than those in the normal mode. The fallback mode corresponds to an operation mode in which the allowable values ​​of specifications such as the vehicle speed, acceleration, and angular velocity that can be taken in the control plan are set to values ​​that are smaller by a predetermined amount than those in the normal mode.

[0059] For example, the upper speed limit can be 60km / h in normal mode, but 40km / h in fallback mode. The allowable acceleration / deceleration limit can be 2.0m / s^2 in normal mode, but 1.0m / s^2 in fallback mode. With this configuration, the autonomous driving ECU 20 performs accelerator / brake control in fallback mode so that the acceleration / deceleration limit is 1.0m / s^2 or less. In other words, acceleration and deceleration are performed more gently in fallback mode than in normal mode, reducing the risk of passengers falling over.

[0060] Furthermore, the allowable yaw rate can be set to 1.0 rad / s^2 in normal mode, and 0.5 rad / s^2 in fallback mode. With such a setting, the autonomous driving ECU 20 performs steering control in fallback mode so that the angular velocity is 0.5 rad / s^2 or less. In other words, turning is performed more gently in fallback mode than in normal mode, reducing the risk of passengers falling over.

[0061] Of course, the above values ​​are merely examples, and the allowable control values ​​in the normal mode and the fallback mode can be changed as appropriate. The parameters suppressed in the fallback mode may be only one of the speed, acceleration / deceleration, and yaw rate. The fallback mode may be a mode in which control is more cautious than in the normal mode. Furthermore, the vehicle control in the fallback mode can be called cautious control, and the vehicle control in the normal mode can be called normal control. The cautious control can be understood as control in which passengers are less likely to fall than in the normal control.

[0062] The control specifications in the normal mode are also set in consideration of passenger safety and ride comfort. If the vehicle always operates in the fallback mode, the travel speed will be slowed down, which may cause traffic jams and may reduce passenger convenience due to increased travel time. The control conditions in the normal mode are set in consideration of passenger safety and ride comfort, as well as cooperation with other traffic and convenience corresponding to travel time. The autonomous driving ECU 20 basically operates in the normal mode, but by temporarily running in the fallback mode depending on the communication status with the control center 2, it is possible to satisfy passenger requirements such as safety, ride comfort, and convenience.

[0063] The interior monitoring ECU 30 is an ECU that monitors passengers' conditions based on signals input from the interior sensors 15, and issues announcements such as warnings, directions to disembark, and safety confirmation for people who have fallen, as necessary. The interior monitoring ECU 30 is mainly composed of a computer including a processor 31, a RAM 32, a storage 33, a communication module 34, and a bus connecting these. The processor 31 is hardware for arithmetic processing that is coupled with the RAM 32. The processor 31 is configured to include at least one arithmetic core such as a CPU or a GPU (Graphics Processing Unit). The processor 31 executes various processes by accessing the RAM 32.

[0064] The storage 33 includes a non-volatile storage medium such as a flash memory. The storage 33 stores a passenger monitoring program, which is a program executed by the processor 31. The passenger monitoring program may be configured to be rewritable (updateable) as appropriate via wireless communication. The processor 31 executing the passenger monitoring program corresponds to the execution of a method corresponding to the passenger monitoring program. The communication module 34 is a circuit for communicating with other in-vehicle devices via the in-vehicle network IvN. The communication module 34 is realized using analog circuit elements, ICs, a PHY chip compliant with the communication standard of the in-vehicle network IvN, and the like.

[0065] The interior monitoring ECU 30 executes an interior monitoring program stored in the storage 33 to provide functions corresponding to various functional blocks shown in FIG. 4. That is, the interior monitoring ECU 30 includes, as functional blocks, a specific behavior detection unit F1, a communication state acquisition unit F2, a report processing unit F3, and a response unit F4. The interior monitoring ECU 30 also includes a passenger correspondence table storage unit M1, which is a database in which a passenger correspondence table is stored, and a radio wave intensity map storage unit M2 in which a radio wave intensity map is stored. The passenger correspondence table storage unit M1 corresponds to the notification data storage unit. The passenger correspondence table is data for making announcements / vehicle control corresponding to passenger behavior. The passenger correspondence table will be described in detail later. The radio wave intensity map is data indicating radio wave intensity for each location. The radio wave intensity map storage unit M2 corresponds to a storage medium.

[0066] The passenger correspondence table storage unit M1 and the radio wave intensity map storage unit M2 are each realized using a rewritable non-volatile storage medium. Here, as an example, the passenger correspondence table storage unit M1 and the radio wave intensity map storage unit M2 are realized using a storage area provided in the storage 33. The passenger correspondence table storage unit M1 and the radio wave intensity map storage unit M2 are each configured so that data can be written, read, deleted, etc. by the processor 31. The passenger correspondence table storage unit M1 may be realized using a storage medium that is physically independent from the storage 33. The same applies to the radio wave intensity map storage unit M2. The radio wave intensity map storage unit M2 may also be realized using a volatile memory such as the RAM 32.

[0067] The radio wave strength map may be dynamically updated by distribution from the control center 2. The control center 2 creates and updates the radio wave strength map by collating statistics on the radio wave strength at each point observed by multiple autonomous buses 1. The radio wave strength map may be generated or revised by acquiring the radio wave strength observed by a preceding vehicle through vehicle-to-vehicle communication. The in-vehicle monitoring ECU 30 is provided with a radio wave strength map storage unit M2, which enables it to read ahead (predict) the communication state of the control center 2 at each point through which the vehicle is scheduled to travel.

[0068] The specific behavior detection unit F1 is configured to detect that a passenger has performed a specific specific behavior based on an input signal from the in-vehicle sensor 15. The specific behavior here includes falling and a specific unsafe behavior that increases the risk of falling. Examples of unsafe behavior include standing without holding onto a strap or handrail (hereinafter, strap, etc.), walking while the vehicle is moving, standing up while the vehicle is moving, not wearing a seat belt, and sticking hands / faces out of the window. A state in which unsafe behavior is maintained can also be called an unsafe state. The specific behavior in the present disclosure can also be read as a specific state.

[0069] As a more preferable example, the specific behavior detection unit F1 of this embodiment is configured to be able to detect behaviors that are considered to be bad manners while riding as specific behaviors. Examples of specific behaviors that are considered to be bad manners include loud conversations, talking on a mobile phone, occupying two or more seats by one person, sleeping on a seat, smoking, etc. Other specific behaviors may include behaviors that lead to damage to the vehicle body / equipment, such as hitting / kicking equipment inside the vehicle, and arguments and fights between passengers.

[0070] The specific behavior detection unit F1 detects behaviors such as passengers falling over, standing without holding onto a strap or the like, walking while the vehicle is moving, and standing up while the vehicle is moving by analyzing images from the in-vehicle camera 151. Various specific behaviors are detected using a recognition model using a CNN (Convolutional Neural Network) or a DNN (Deep Neural Network), etc. The same applies to behaviors such as sticking hands / faces out of the window, occupying two or more seats by one person, sleeping on a seat, and smoking.

[0071] The fact that the passenger is not holding onto a strap or the like may be determined by sensor fusion, in other words, by using the detection results of a contact sensor or the like provided on the strap. Actions such as opening a window may be detected based on an input signal from the vehicle state sensor 12 corresponding to the opening and closing of the window. Smoking may be detected from the output of a smoke sensor installed in the vehicle. Actions that may lead to damage, such as hitting / kicking on in-vehicle equipment, and falls may be detected by using the output of an acceleration sensor / vibration sensor in addition. Loud conversations, arguments, and the like may be detected from the volume of the voice collected by the in-vehicle microphone 152 and the contents of the speech. The passenger's actions / states to be detected and the detection algorithms thereof are incorporated in the passenger monitoring program.

[0072] The specific behavior detection unit F1 has sub-functions, a passenger information acquisition unit F11 and a risk assessment unit F12. The passenger information acquisition unit F11 acquires passenger information, which is information about passengers, based on an input signal from the in-car sensor 15. The passenger information includes the number of passengers, the position of each passenger, and the riding posture. The riding posture includes whether the passenger is sitting in a seat or standing. For a standing passenger, a more specific riding posture can include whether the passenger is holding onto a strap or handrail.

[0073] Furthermore, the passenger information may include attribute information such as the passenger's age, physical build, sex, and clothing. The passenger's age may be determined by dividing it into a first age group of 14 years old or younger, a second age group of 15 years old or older and younger than 65 years old, and a third age group of 65 years old or older. The number of age group divisions is an example, and may be divided into four or more age groups. The third age group may be rephrased as an elderly age group. The clothing may include the type of footwear, such as whether the passenger is wearing shoes that are easy to fall in, such as heels, or shoes that are relatively difficult to fall in, such as sneakers. These passenger information may be identified by analyzing the images captured by the in-vehicle camera 151, for example. Of course, the passenger information acquisition unit F11 may acquire information for each passenger by combining the images of the in-vehicle camera 151 and input signals from multiple types of in-vehicle sensors 15, such as the seating sensor 153 and the ultrasonic sensor 154.

[0074] The attribute information for each passenger may include whether or not the passenger uses a walking aid, whether or not the passenger uses a wheelchair, and whether or not the passenger has a disability (handicap). Walking aids include a walking stick and a walker. The above attributes may also be detected by analyzing the image of the in-car camera 151. For example, when the passenger information acquisition unit F11 detects a passenger carrying a white cane by image recognition, the passenger information acquisition unit F11 may pick up the passenger as a person who may have a visual impairment. When the passenger information acquisition unit F11 detects a wheelchair by image recognition, the passenger information acquisition unit F11 may determine that a wheelchair user is on board. In addition, the passenger information acquisition unit F11 may detect a person with a fever by analyzing an infrared image input from the infrared camera 155.

[0075] When the risk assessment unit F12 detects an unsafe behavior of a passenger, it calculates a fall risk Rf, which is the likelihood of a fall, based on attribute information of the subject passenger who performed the unsafe behavior and the content of the unsafe behavior / condition performed. The subject may also be called the behavior performer. The fall risk Rf may be expressed in multiple stages, or may be expressed as a score value from 0 to 100, for example. Here, as an example, the fall risk Rf is evaluated in four stages from 0 to 3.

[0076] The risk assessment unit F12 calculates the fall risk Rf, for example, using the following formula 1. Formula 1: Rf = Ea + Eb + Ec

[0077] The parameter Ea constituting the above formula 1 is a parameter for which a value is set according to the content of the unsafe behavior. The parameter Ea is set to a larger value for behaviors with a higher risk of falling. The parameter Ea is set to the largest when the person is standing without holding onto a strap or the like, the smallest when the person is sitting, and the intermediate value when the person is standing while holding onto a strap or the like. For example, when the person is standing without holding onto a strap or the like, Ea=2 is applied, when the person is sitting, Ea=0 is applied, and when the person is standing while holding onto a strap or the like, Ea=1 is applied. Note that, even if the person is not holding onto a strap or the like, when the person is resting or leaning on a wall or window frame in the vehicle cabin, a value that calculates a higher fall risk Rf than when the person is holding onto a strap or the like, such as 1.5, may be applied. Also, even if the person is seated, when the seat belt is not fastened, the parameter Ea may be set to a predetermined amount larger than when the seat belt is fastened.

[0078] The parameter Eb is a parameter to which a value according to the age of the passenger is applied. If the passenger is determined to belong to the first age group or the third age group, the parameter Eb may be set to a value larger than in other cases. More specifically, if the passenger is determined to belong to the first age group or the third age group, Eb=1 is set, whereas if the passenger is determined to belong to the second age group, Eb=0 is set.

[0079] The parameter Ec is a correction parameter to which a value based on factors other than age is applied. For example, an arbitrary value of 1 or more is set when the passenger is using a walking aid, a wheelchair, or a white cane. Also, 1 or the like is applied when the passenger is estimated to have a physical handicap. A predetermined positive value may also be applied to the parameter Ec when the passenger is pregnant or carrying an infant in a baby carrier or the like. When each parameter includes a decimal point, causing Ea+Eb+Ec to include a decimal point, the final fall risk may be determined by rounding off to the nearest decimal place.

[0080] The above formula 1 is an example. Formula 1 corresponds to a configuration in which the fall risk Rf is calculated using an additive (point-adding) method. Of course, the risk assessment unit F12 may calculate the final fall risk Rf by multiplying risk factors instead of using the additive method. For example, the risk assessment unit F12 may calculate the fall risk Rf using the following formula 1a.

[0081] Formula 1a: Rf=Ea×Eb×Ec In the above formula 1a, it is preferable that Eb and Ec are applied with a value of 1 or more according to various attributes, with a 20-year-old person of a certain body type and wearing sneakers as the reference value (1). For example, in the case of an elderly person, Eb=1.5 or 2 may be set. It is only necessary to design appropriately what kind of person is used as the reference and what values ​​are applied to various attributes. Basically, it is only necessary to set the total fall risk Rf to be evaluated higher for a person who is more likely to fall or who is at higher risk when falling.

[0082] Furthermore, the risk assessment unit F12 may calculate the fall risk Rf taking into consideration the vehicle exterior situation and the control plan acquired from the autonomous driving ECU 20. That is, the fall risk Rf may be calculated according to the following formula 1b using a parameter Ed derived from the vehicle exterior situation and the control plan. The parameter Ed corresponds to a coefficient that corrects the level of the fall risk Rf according to the control plan and the road shape.

[0083] Equation 1b: Rf=Ea×Eb×Ec×Ed For example, the parameter Ed may be set to 1 (reference value) when traveling at a substantially constant speed on a flat straight road, and a value greater than 1 may be applied when stopping, starting, entering a sharp curve, or turning right or left. A straight road here refers to a road with a curvature less than a predetermined value, and a road section with a curvature equal to or greater than a predetermined value is called a curved road. Furthermore, among curved roads, a road with a curvature equal to or greater than a predetermined threshold is called a sharp curve. The risk assessment unit F12 may also apply a predetermined value greater than 1 to the parameter Ed when there is a plan to accelerate or decelerate at a predetermined threshold or greater, or to steer at a steering speed equal to or greater than a predetermined threshold.

[0084] The risk assessment unit F12 may calculate the fall risk Rf according to the riding posture of all passengers, not limited to those who have performed unsafe behavior. The value of the parameter Ea may be determined based on the current riding posture, i.e., whether the passenger is sitting, standing, holding onto a strap, etc.

[0085] The communication status acquisition unit F2 is configured to acquire the current communication status with the control center 2. The current communication status is acquired. That is, the communication status acquisition unit F2 acquires data indicating the current communication status with the control center 2 from the wireless communication device 16. The communication status acquisition unit F2 may also acquire radio wave intensity, reception quality, etc. as information indicating the current wireless communication quality from the wireless communication device 16. The information indicating the current wireless communication quality acquired by the communication status acquisition unit F2 is provided to the response unit F4.

[0086] The report processing unit F3 sequentially executes a process of transmitting data indicating the driving state of the vehicle, such as the current position of the vehicle, the vehicle speed, the traveling direction, and the yaw rate, to the control center 2 as a driving state report. The current position of the vehicle can include an ego lane number indicating the number of lanes from the left or right edge of the road in which the ego lane in which the vehicle is driving is. The report processing unit F3 also transmits an outside image of the vehicle to the control center 2. The outside image of the vehicle refers to images of a front camera, a side camera, or a rear camera, for example. The camera images and the like can be transmitted in accordance with a predetermined protocol such as RTSP (Real Time Streaming Protocol). The images can be transmitted using any codec, such as H.264 or H.265. That is, the report processing unit F3 transmits the outside image of the vehicle to the control center 2 using various real-time image distribution technologies. The transmission of the driving state report by the report processing unit F3 is performed in cooperation with the wireless communication device 16. The report processing unit F3 successively sends driving state reports, enabling an operator Op or the like to manage the operating state of the autonomously driving bus 1. Note that the operator Op here refers to a staff member at the control center 2 who manages the operating status of the autonomously driving bus 1.

[0087] The report processing unit F3 also sequentially transmits in-vehicle video and in-vehicle audio to the control center 2. In-vehicle video refers to video from the in-vehicle camera 151. In-vehicle audio refers to audio picked up by the in-vehicle microphone 152. The transmission of in-vehicle video and the like can be performed in the same manner as the transmission of outside-vehicle video. By transmitting in-vehicle video and in-vehicle audio to the control center 2, the operator Op or the like can grasp the in-vehicle situation of the autonomously driving bus 1. The report processing unit F3 may also periodically transmit data indicating the recognition results of passengers based on video and the like, such as the number of occupants, the body temperature of each occupant, and the riding posture, to the control center 2 as an in-vehicle status report.

[0088] The transmission of the vehicle interior status report is performed periodically when communication with the control center 2 is possible. The transmission of the vehicle interior status report may be triggered by the detection of the occurrence of a specified reporting event, such as the opening or closing of a door at a bus stop or the like, acceleration or deceleration exceeding a specified value (so-called sudden braking), or a yaw rate exceeding a specified value (so-called sudden steering). The detection of the above-mentioned specific behavior can also be included in the reporting event.

[0089] From the viewpoint of avoiding collisions with other vehicles, the report processing unit F3 may be configured to transmit outside-vehicle images rather than inside-vehicle images. With such a configuration, even if the available bandwidth is limited due to deterioration of the wireless communication environment, data showing the driving environment, such as outside-vehicle images, can be transmitted to the control center 2 as much as possible. In another aspect, the report processing unit F3 may be configured to transmit inside-vehicle images rather than outside-vehicle images from the viewpoint of in-vehicle security. With such a configuration, even if the available bandwidth is limited, data showing the state of the passengers can be transmitted to the control center 2 as much as possible, so that the operator Op can quickly respond to passenger falls, unsafe behavior, trouble, etc.

[0090] Additionally, the report processing unit F3 may be configured to sequentially transmit a communication status report, which is a data set indicating the state of wireless communication for each point, to the control center 2. The communication status report may be, for example, a communication packet indicating radio wave strength and reception quality for each point. The communication status report may also include measurement values ​​such as throughput and round-trip latency in communication with the control center 2. The various report data may include vehicle information indicating the source of transmission, transmission time information indicating the transmission time, and the like.

[0091] The response unit F4 performs a process corresponding to a specific passenger behavior (hereinafter, response process) based on the specific passenger behavior detected by the specific passenger behavior detection unit F1. For example, when the specific passenger behavior detection unit F1 detects a passenger falling, it outputs an announcement inquiring about the safety of the passenger who fell. When it detects a passenger falling, the response unit F4 may start decelerating the train to stop. When it detects an unsafe passenger behavior, the response unit F4 performs an announcement encouraging the passenger to take a seat or hold on to a strap or the like.

[0092] The response unit F4 includes a communication state prediction unit F41, a prescribed message output unit F42, and a center cooperation unit F43 as sub-functional units for performing system response processing.

[0093] The communication state prediction unit F41 predicts the transition of wireless communication quality within a predetermined time from the present by comparing the vehicle's planned position at each time indicated in the control plan with the radio wave strength map. In addition, when a problem occurs in data communication with the control center 2 at the present time, the communication state prediction unit F41 calculates the expected recovery time Trc as shown in FIG. 5 by using the radio wave strength map and the control plan.

[0094] The expected recovery time Trc is a predicted value of the time required for the radio wave strength to reach or exceed a predetermined required level Prq. The required level Prq may be a level at which the video signal of the in-vehicle camera 151 can be stably transmitted, or may be a level at which the voice data of the operator Op can be stably received. The required level Prq may be the lowest level at which data communication with the control center 2 is possible. The required level Prq is preset by the designer.

[0095] When a problem occurs in data communication with the control center 2, this includes a state in which data communication is completely impossible. Furthermore, when a problem occurs in data communication with the control center 2, this can include a state in which communication is unstable. A state in which communication is unstable includes, for example, a state in which data communication is possible but the radio wave strength is below a predetermined strength lower limit value Pmn. The strength lower limit value Pmn corresponds to a value at which the packet loss rate is equal to or greater than a predetermined threshold value, or a lower limit value of reception strength at which a received signal can be decoded.

[0096] 5 shows an embodiment in which the intensity lower limit value Pmn is set to a predetermined value less than the required level Prq, but this is not limiting. The intensity lower limit value Pmn and the required level Prq may be the same. In other words, the required level Prq may be used as a threshold for determining that communication is difficult. In addition, when the communication latency is equal to or greater than a predetermined threshold, when the throughput is less than a predetermined threshold, when the packet loss rate is equal to or greater than a predetermined threshold, etc., the communication may be included in the unstable state. Furthermore, when the quality index such as RSRQ or SINR is equal to or less than a predetermined threshold, the communication may be included in the unstable state.

[0097] The prescribed message output unit F42 is configured to refer to a passenger correspondence table and output a prescribed message corresponding to the detected specific behavior by voice. The passenger correspondence table can be data showing the correspondence between the contents of the specific behavior and the output message, as shown in FIG. 6, for example. Such a prescribed message output unit F42 can be called an internal processing unit because the response process is completed inside the vehicle, more specifically, inside the interior monitoring ECU 30. If there is no appropriate message / phrase corresponding to the passenger's behavior, a phrase such as "Contacting the control center" can be used.

[0098] On the other hand, the center cooperation unit F43 is configured to make an in-vehicle announcement corresponding to a specific action by cooperation with a device outside the vehicle. For convenience, a series of processes for obtaining data for announcements regarding a specific action from the control center 2 and outputting the data as audio from the in-vehicle speaker is also referred to as center cooperation processing. Also, audio data transmitted from the control center 2 regarding the detected specific action is also referred to as center audio data.

[0099] For example, the center cooperation unit F43 transmits a response request signal, which is a signal requesting the operator Op to respond to the passenger (e.g., voice input), and waits to receive the voice data input by the operator Op. That is, the center cooperation unit F43 entrusts the operator Op with responding to the passenger in response to the detected specific behavior. Then, as soon as the voice data is received from the control center 2, the voice data is output from the in-vehicle speaker 18.

[0100] The response request signal may include, as summary information, information indicating the content of the detected specific behavior and the location of the target passenger who performed the specific behavior. The information indicating the location of the target may be expressed by a seat number, etc. If the response request signal includes summary information, the server processor 231 can display a summary notification image, and the operator Op can quickly identify the target and issue a warning / suggestion according to the situation. Furthermore, the response request signal may include, in addition to the summary information, an enlarged still image of the target. From its role, the response request signal can be understood as a notification signal that notifies the control center 2 that a specific behavior is being performed.

[0101] Such a center cooperation unit F43 corresponds to a configuration that reports to the control center 2 that a specific action is being performed, and executes voice output and vehicle control based on instructions from the control center 2. Voice output based on instructions from the control center 2 includes outputting the voice of the operator Op to the in-vehicle speaker 18. Note that the center cooperation process may be a process of virtually connecting the microphone 222 for the operator Op and the in-vehicle speaker 18 arranged in the vehicle. Specifically, the center cooperation process may be a process of connecting a telephone line so that a passenger in the vehicle can talk to the operator Op via the in-vehicle microphone 152 and the in-vehicle speaker 18.

[0102] Moreover, the center collaboration unit F43 corresponds to a configuration that suspends output of a prescribed message and waits for instructions from the control center 2 including the operator Op. Among the center collaboration processes, the report that a specific action is being performed can be performed by collaboration between the response unit F4 and the report processing unit F3. The center collaboration unit F43 may have the function of the report processing unit F3. In other words, the report processing unit F3 may be built into the center collaboration unit F43. A specific example of the operation of the response unit F4 will be described separately later.

[0103] <Configuration of Control Center 2> Here, the configuration of the control center 2 will be described. As shown in Fig. 7, the control center 2 includes a communication device 21, an operator HMI 22, and a management server 23. HMI in the names of the components is an abbreviation for Human Machine Interface. Note that the expression "autonomous driving bus 1" as a communication partner of the control center 2 can be read as "in-vehicle system IvS" or "in-vehicle monitoring ECU 30."

[0104] Communication device 21 is communication equipment for communicating data with the autonomously driving bus 1 via the wide area communication network 4 and wireless base station 3. Communication device 21 is connected to the equipment that makes up the wide area communication network 4 using, for example, optical fiber. Communication device 21 receives data transmitted from the autonomously driving bus 1 and outputs the data to management server 23. For example, communication device 21 outputs driving status reports and interior status reports from the autonomously driving bus 1 to management server 23. Communication device 21 also transmits data input from management server 23 to the autonomously driving bus 1. For example, communication device 21 transmits voice data and the like input from management server 23 to the autonomously driving bus 1.

[0105] The operator HMI 22 is equipment that enables the operator Op to check the traveling status and interior status of the autonomous driving bus 1, and to intervene in vehicle control and issue notifications to passengers depending on the situation. The operator HMI 22 includes a display 221, a microphone 222, a speaker 223, and an input device 224.

[0106] The display 221 displays an image based on a video signal input from the management server 23. For example, an outside image and an inside image of the autonomous driving bus 1 are displayed on the display 221. There may be one display 221 or multiple displays 221. The display 221 is placed on a desk for the operator Op. The microphone 222 is a microphone for the operator Op. The microphone 222 collects the voice spoken by the operator Op, converts it into an electrical signal, and outputs it to the management server 23. The speaker 223 converts the audio signal input from the management server 23 into sound and outputs it. The speaker 223 and the microphone 222 may be integrally configured as a headset for the operator Op.

[0107] The input device 224 is a device for accepting instructions and operations from the operator for the autonomous driving bus 1. The input device 224 can be a keyboard, a dedicated device including multiple switches, a touch panel stacked on the display 221, or the like. The operator HMI 22 includes, for example, a talk switch as the input device 224. The talk switch corresponds to a switch for activating the microphone 222 and outputting the speech of the operator Op from the in-vehicle speaker 18 of the autonomous driving bus 1.

[0108] The input device 224 may also include a steering wheel, pedals, a shift lever, a light switch, and the like that are used by the operator Op to remotely operate the autonomous bus 1. The light switch is a switch that changes the lighting state of at least one of the turn signals, headlights, front fog lights (so-called fog lights), and emergency flashers (so-called hazard lights). The concept of a switch is not limited to a button type, but also includes a lever type, a dial type, and a touch sensor type.

[0109] An operation of the operator Op on the input device 224 can be interpreted as an action or instruction input of the operator Op. The input device 224 outputs an electric signal corresponding to an operation performed by the operator Op on the device as an operation signal to the management server 23. The operation signal of the operator Op includes information indicating the operation content of the operator Op.

[0110] The management server 23 is mainly configured as a computer including a server processor 231, a RAM 232, and a storage 233. The server processor 231 is hardware for arithmetic processing coupled with the RAM 232. The server processor 231 includes at least one arithmetic core such as a CPU. The server processor 231 executes various processes by accessing the RAM 232. The storage 233 includes a non-volatile storage medium such as a flash memory. The storage 233 stores a server program as a program executed by the server processor 231. The execution of the above program by the server processor 231 corresponds to the execution of a method for externally adjusting a vehicle interior condition, which is a method corresponding to the server program.

[0111] The management server 23 is communicatively connected to each of the communication device 21 and the operator HMI 22. Appropriate data signals are input to the management server 23 from each of the communication device 21 and the operator HMI 22. In addition, the management server 23 outputs control signals and data signals to each of the communication device 21 and the operator HMI 22 as appropriate.

[0112] For example, the server processor 231 displays on the display 221 the outside video, inside video, and various status information of the autonomously driving bus 1 that are received by the communication device 21. This type of processing corresponds to a vehicle state notification process that notifies the operator Op of the internal or external situation of the autonomously driving bus 1. For example, the server processor 231 may display the outside video, inside video, and various status information of the autonomously driving bus 1 on the display 221 while the autonomously driving bus 1 is in operation. The server processor 231 may also be configured to display the outside video, inside video, and various status information on the display 221 only when a specific operation status confirmation condition is satisfied. Examples of cases where the operation status confirmation condition is satisfied include when a response request signal is received from the autonomously driving bus 1, when an operation to check the operation status is performed by the operator Op, and when the autonomously driving bus 1 has stopped at a bus stop or the like.

[0113] The server processor 231 may be configured to change the display content of the display 221, for example, the screen layout, between when a specific behavior is detected, which is when a specific behavior is detected on the autonomously driving bus 1, and when a specific behavior is not observed normally. When a specific behavior is detected, this corresponds to when a response request signal is received.

[0114] For example, the server processor 231 may be configured to mainly display an image overlooking the entire interior of the vehicle during normal times, while enlarging and displaying an image around the passenger who is performing the specific behavior when the specific behavior is detected. This configuration makes it easier for the operator Op to check the detected specific behavior. In addition, it becomes possible to determine the validity of the detection result, that is, whether or not it is a false detection.

[0115] Furthermore, the server processor 231 may display the vehicle exterior image larger than the vehicle interior image during normal operation, but may display the vehicle interior image larger than the vehicle exterior image when a specific behavior is detected. More specifically, the vehicle exterior image and the vehicle interior image may be displayed in a size ratio of 6:4 or 10:0 during normal operation, but may be displayed in a size ratio of 4:6 or 0:10 when a specific behavior is detected. By primarily displaying the vehicle exterior image during normal operation, it is possible to improve the visibility of the vehicle exterior situation. Furthermore, by primarily displaying the vehicle interior image when a specific behavior is detected, it is possible to improve the visibility of the vehicle interior situation.

[0116] The management server 23 may be configured to allow an operator Op to input whether the detection result of the specific behavior is correct or not. With such a configuration, feedback is given to the detection result of the specific behavior, and the detection accuracy of the specific behavior by the in-vehicle monitoring ECU 30 can be improved.

[0117] Furthermore, when the server processor 231 receives a response request signal from the autonomously driving bus 1, it may display a summary notification image showing an outline of the specific behavior detected by the in-vehicle monitoring ECU 30 on the video from the in-vehicle camera 151, and wait for an input operation from the operator Op. Then, when the server processor 231 acquires speech voice data of the operator Op via the microphone 222, it transmits the voice data to the autonomously driving bus 1 in cooperation with the communication device 21.

[0118] Fig. 8 is an image showing one aspect of the summary notification image. The summary notification image preferably includes information indicating the content of the detected specific behavior and the location of the subject, who is a passenger who performed the specific behavior. For example, as shown in Fig. 8, the summary notification image includes text Im1 indicating the summary, a schematic diagram image Im3 that typically represents the interior of the vehicle, and a location icon image Im4 that indicates the location of the subject. The location icon image Im4 is displayed superimposed on the schematic diagram image Im3 in an area that corresponds to the location of the subject.

[0119] Information indicating the target person's position may be displayed by a seat number or the like. The target person's position information may also be expressed by a number for each section obtained by dividing the vehicle interior into a mesh shape. With the configuration that displays the summary notification image, the operator Op can quickly identify the target person and issue a warning / suggestion according to the situation.

[0120] Also, the summary notification image may include button images for receiving input operations of the operator Op, such as a video confirmation button Im5, a microphone activation button Im6, and a no action required button Im7, as exemplified in FIG. 8. The video confirmation button Im5 is a button image for displaying an image of the in-vehicle camera 151. The server processor 231 may be configured to display an image of the in-vehicle camera 151 based on an input of an operation signal indicating that the video confirmation button Im5 is selected. The microphone activation button Im6 is a button image for activating the microphone 222. The server processor 231 activates the microphone 222 based on an input of an operation signal indicating that the microphone activation button Im6 is selected, and sets the state in which the voice of the operator Op can be output from the in-vehicle speaker 18. The no action required button Im7 is a button image for the operator Op to input that an announcement or the like is not required for the detected action. The server processor 231 may be configured to send a response signal to the autonomously driving bus 1 indicating that no warning or the like is necessary, based on input of an operation signal indicating that the no action required button Im7 has been selected.

[0121] In addition, the summary notification image may include the content of the next event in the operation plan and the remaining distance / remaining time to the event. For example, when a left turn at an intersection is scheduled as the next control event, the server processor 231 may display a summary notification image including the remaining distance / remaining time to the left turn point.

[0122] The voice transmission of the operator Op is configured to be executable at any time based on the operation of the operator Op, not just when a specific behavior is detected. Furthermore, when the management server 23 detects a driving operation such as steering or braking by the operator Op, it transmits a control signal corresponding to the operation content to the autonomous driving bus 1. For example, when the management server 23 detects depression of the brake pedal by the operator Op, it transmits a signal to the autonomous driving bus 1 to temporarily stop control by the autonomous driving ECU 20 and to decelerate the bus in accordance with the amount of pedal depression by the operator Op. The management server 23 can realize override by the operator Op.

[0123] In addition, the management server 23 may be configured to update the radio wave intensity map at a predetermined update interval by collecting information indicating the radio wave intensity at each point from a plurality of vehicles including the autonomous driving bus 1. The update interval may be, for example, one hour, two hours, four hours, or one day. The radio wave intensity at each point indicated by the radio wave intensity map may be a statistical representative value of the radio wave intensity at each point received within a predetermined collection time as a population. The representative value here may be an average value or a median value. It may also be an average value or a median value of a set obtained by removing outliers from the population. An outlier may be a value that is two or three or more times the standard deviation away from the average value or median value of the original population. Various methods such as the Smirnoff-Grubbs test and the Thompson test can be used as a method for determining outliers.

[0124] <Example of operation of the in-vehicle monitoring ECU 30> Here, the operation of the in-vehicle monitoring ECU 30 will be described using the flowchart shown in FIG. 9. For convenience, the series of processes shown in FIG. 9 will be referred to as vehicle-side processing. The flowchart shown in FIG. 9 can be repeatedly executed at a predetermined monitoring period while a predetermined execution condition is satisfied, such as when the power source for traveling is set to ON or when it corresponds to the operation hours / business hours of the autonomously driving bus 1. The monitoring period can be 100 milliseconds, 200 milliseconds, 1 second, or the like. Here, as an example, the vehicle-side processing includes steps S11 to S18. Note that the number of steps constituting the vehicle-side processing and the order of execution can be changed as appropriate. Each step is performed by the processor 31.

[0125] In step S11, the passenger information acquisition unit F11 acquires passenger information based on an input signal from the in-vehicle sensor 15, and the process proceeds to step S12. In this disclosure, acquisition is not limited to being provided from another device / program, but also includes generation / detection by internal calculation. Step S12 is an optional process and may be omitted.

[0126] In step S12, the processor 31 acquires external information, which is information about the outside of the vehicle, based on an input signal from the external sensor 11, and proceeds to step S13. The external information includes, for example, the curvature of the road on which the vehicle is traveling, the ego lane ID, the positional relationship with surrounding vehicles including a preceding vehicle, and the presence or absence of an obstacle ahead. External information related to the road shape, such as the road gradient and the presence or absence of steps / depressions, may be identified by referring to map data corresponding to the current position. The positions and traveling speeds of surrounding vehicles may be acquired by vehicle-to-vehicle communication. Furthermore, obstacle information including lane restrictions due to road construction may be acquired from a roadside unit by wireless communication.

[0127] In step S13, a short-term control plan is obtained from the autonomous driving ECU 20, and the process proceeds to step S14. The short-term control plan refers to a control plan within 30 seconds or 1 minute from the present, for example. In step S14, the communication status acquisition unit F2 acquires the current communication status from the wireless communication device 16, and the process proceeds to step S15.

[0128] In step S15, the specific behavior detection unit F1 judges whether or not there is a passenger who is performing a specific behavior based on the passenger information acquired in step S11. If there is no passenger who is performing a specific behavior, a negative judgment is made in step S15 and this flow ends. On the other hand, if there is a passenger who is performing a specific behavior, the flow proceeds to step S16.

[0129] In step S16, it is determined whether the current communication state is good. For example, if the radio wave strength is equal to or higher than the required level Prq, it is determined that the communication state is good. In addition, it may be determined that the communication state is good under the conditions that the latency of data communication with the control center 2 is less than a predetermined threshold, the throughput is equal to or higher than a predetermined threshold, and the packet loss rate is less than a predetermined threshold. A good communication state corresponds to a case where there is no problem with communication with the control center 2, such as a communication interruption or a delay of a predetermined time or more.

[0130] The conditions for determining that the communication state is good can be changed as appropriate. Determining that the communication state is not good corresponds to determining that a malfunction has occurred in communication with the control center 2. As described above, the malfunction may include a state in which communication is not possible, a state in which a delay of a predetermined value or more has occurred, a packet loss rate of a predetermined value or more, a state in which the throughput is a predetermined value or less, and the like. Step S16 may be replaced with a process for determining whether data communication with the control center 2 is possible.

[0131] If it is determined that the communication state is good, the process proceeds to step S17, where the center cooperation unit F43 transmits a response request signal to the control center 2 and waits for a response from the control center 2. After that, if voice data is received from the control center 2, the in-vehicle speaker 18 is turned on, and the voice sent from the control center 2 is output from the in-vehicle speaker 18. In other words, if the communication state is good, the prescribed message output unit F42 does not operate. This control mode corresponds to a configuration in which the center cooperation unit F43 is operated with priority over the prescribed message output unit F42 when there is no problem in communication with the control center 2.

[0132] If it is determined that the communication state is not good, the process proceeds to step S18, and emergency response processing is executed. The emergency response processing is a sequence that is applied when it is determined that there is a problem in communication with the control center 2. The emergency response processing will be described with reference to FIG. 10.

[0133] 10 is a flowchart showing an example of a series of processes executed as the temporary control process. In this embodiment, the temporary control process includes steps S21 to S26. In step S21, the processor 31 as the communication state prediction unit F41 refers to the radio wave intensity map, calculates an expected recovery time Trc according to the current location and a future control plan, and proceeds to step S22. In step S22, it is determined whether the calculated expected recovery time Trc is less than a predetermined allowable waiting time Tw.

[0134] The allowable waiting time Tw is a threshold value for deciding whether to output a prescribed message or to delegate passenger support to an operator Op. The allowable waiting time Tw is preferably set to a value obtained by adding an expected round-trip latency to an expected value of a delay perception time, which is the time it takes for a human to feel that there is no response / a slow response to an input action. Considering that a typical delay perception time is 2 to 3 seconds, the allowable waiting time Tw is set to 5 seconds as an example in this embodiment. Of course, the specific value of the allowable waiting time Tw may be set to 4 seconds, 8 seconds, 10 seconds, etc.

[0135] The length of the allowable waiting time Tw may be changed depending on the content of the detected specific behavior. For example, if a passenger's fall is detected, the allowable waiting time Tw is set to a relatively short time, such as 3 or 4 seconds. If the detected specific behavior corresponds to an unsafe behavior, the allowable waiting time Tw may be set to 5 or 6 seconds. If the detected specific behavior corresponds to a violation of manners, the allowable waiting time Tw may be set to a relatively long time, such as 8 or 10 seconds.

[0136] If it is determined in step S22 that the predicted recovery time Trc is less than the allowable waiting time Tw, the process proceeds to step S23. On the other hand, if the predicted recovery time Trc is equal to or greater than the allowable waiting time Tw, the process proceeds to step S24.

[0137] In step S23, the center collaboration unit F43 performs center collaboration processing. An example of the center collaboration processing is shown in Fig. 11. The center collaboration processing includes steps S31 to S35. In step S31, it is determined successively whether the communication state has been restored. The communication state has been restored, for example, when the radio wave strength has reached or exceeded the required level Prq. The conditions for determining that the communication state has been restored can be set to be the opposite of the conditions for determining that the communication state is not good / communication is not possible.

[0138] If it is determined that communication with the control center 2 has been restored (is possible), the process proceeds to step S32. If communication with the control center 2 has not been restored even after a certain time has elapsed since the start of the center cooperation process, such as the allowable waiting time Tw, the process proceeds to step S25 as an exception process, and a specified message may be output.

[0139] In step S32, the center cooperation unit F43 transmits a response request signal to the control center 2. In step S33, the process waits for a response from the control center 2. The response from the control center 2 essentially refers to voice data for announcement. The response from the control center 2 can be interpreted as a response from the operator Op. If a response from the operator Op is received, the process proceeds to step S34. In step S34, the in-vehicle speaker 18 is turned on, and the voice messages of the operator Op sent from the control center 2 are output in sequence.

[0140] The above configuration corresponds to a configuration in which a response request signal is sent as soon as the communication state is restored, without waiting for the expected recovery time Trc to elapse. This configuration allows the operator Op to make an announcement more quickly. Of course, in another aspect, the response request signal may be sent after waiting for the expected recovery time Trc to elapse.

[0141] In steps S31 to S32, the radio wave intensity is periodically observed, and the response request signal is transmitted only after it is confirmed that the radio wave intensity has reached the required level Prq. However, the contents of the center cooperation process are not limited to this. For example, in step S31, the response request signal may be periodically transmitted, and it may be determined that the communication state has been restored when a response signal (so-called Ack) indicating that the response request signal has been received is received.

[0142] However, since the emergency response process is executed when the communication state is unstable, there may be a case where the communication state deteriorates again after the response request signal is transmitted, making it impossible to receive a response from the operator Op. In such a case, if no response is obtained from the operator Op even after a predetermined time, for example, a waiting allowable time Tw, has elapsed since the response request signal was transmitted, the process may proceed to step S35 and a prescribed message may be output. In addition, the response unit F4 may execute step S35, i.e., output a voice of a prescribed message corresponding to the action content, on the condition that no voice data has been received from the control center 2 even after the waiting allowable time Tw has elapsed since the detection of the specific action.

[0143] Returning to FIG. 10, the description of step S24 and subsequent steps will be continued. As described above, step S24 and subsequent steps are a processing sequence in the case where the expected recovery time Trc determined based on the radio wave intensity map is equal to or longer than the allowable waiting time Tw. In step S24, it is determined whether the expected recovery time Trc is less than a predetermined fallback threshold Tfb. The fallback threshold Tfb used here is a threshold for switching the automatic driving ECU 20 to a fallback mode, in other words, a threshold for performing automatic driving under control conditions with higher safety. The fallback threshold Tfb is set to, for example, 30 seconds. Of course, the fallback threshold Tfb may be 20 seconds, 45 seconds, or the like, and the specific value can be changed as appropriate.

[0144] If the expected restoration time Trc is less than the fallback threshold Tfb, step S25 is executed, whereas if the expected restoration time Trc is equal to or greater than the fallback threshold Tfb, the process proceeds to step S26.

[0145] In step S25, the prescribed message output unit F42 executes prescribed message output processing. That is, by referring to the passenger correspondence table, it outputs a prescribed message corresponding to the detected specific behavior by voice. For example, if the detected specific behavior is standing without holding onto a strap, it outputs a voice message such as "Please hold onto a strap to prevent falling."

[0146] In step S26, the response unit F4 outputs a fallback instruction signal, which is a control signal requesting the autonomous driving ECU 20 to transition to fallback mode. That is, a signal is output requesting the autonomous driving ECU 20 to reduce the upper speed limit, the acceleration / deceleration tolerance, and the angular velocity tolerance by a predetermined amount when creating an autonomous driving plan. This allows operation to continue in a safer control mode, thereby reducing the risk of passengers falling.

[0147] Note that step S26 may be performed only when the detected specific behavior corresponds to an unsafe behavior, that is, only when an unsafe behavior is detected. Also, in step S26, a prescribed message output process may be performed in parallel with the output of the fallback mode transition instruction signal. For example, the prescribed message may be output by voice as step S26a.

[0148] <Regarding the operation of Management Server 23> Here, the operation of management server 23 will be described using the flowchart shown in FIG. 12. For convenience, the series of processes shown in FIG. 12 will be referred to as center-side processing. The flowchart shown in FIG. 12 can be executed repeatedly at a predetermined response period while a predetermined execution condition is satisfied, such as when the operation hours / business hours of the autonomously driving bus 1 are met. The response period can be 100 milliseconds or 200 milliseconds. Here, as an example, the center-side processing includes steps S41 to S46. Note that the number of steps constituting the center-side processing and the order of execution can be changed as appropriate. Each step is executed by server processor 231.

[0149] First, in step S41, it is determined whether or not a response request signal has been received from the autonomous driving bus 1. If a response request signal has not been received, a negative judgment is made in step S41 and this flow ends. On the other hand, if a response request signal has been received, step S42 is executed. Note that the center-side processing may be started in response to the reception of a response request signal as a trigger. In that case, step S41 may be omitted, and processing may start from step S42.

[0150] In step S42, a predetermined notification sound is output from speaker 223, and a notification image indicating that a passenger performing a specific behavior has been detected inside autonomous driving bus 1 is displayed on display 221. The notification image may be an image including summary information as exemplified in Fig. 8, or may be the interior of the vehicle itself. The notification image may also be an image indicating only that an abnormality has been detected inside the vehicle.

[0151] In step S43, it is determined whether or not voice input by the operator Op has started. For example, when the microphone 222 is set to on, it is determined that voice input by the operator Op has started. If voice input by the operator Op has started, the process proceeds to step S44. In step S44, voice data obtained by converting the voice uttered by the operator Op into a predetermined data format is transmitted to the autonomous driving bus 1. At that time, the management server 23 also transmits a control signal instructing the operator Op to output the voice uttered by the operator Op from the in-vehicle speaker 18.

[0152] On the other hand, if voice input by the operator Op is not started even after waiting for the predetermined time, the process proceeds to step S45. In step S45, it is determined whether or not it has been selected that no response is required for the specific behavior detected this time, based on the operation signal input from the input device 224. If it has been selected that no response is required for the specific behavior detected this time, an instruction signal indicating that no response is required is transmitted in step S44.

[0153] If neither an instruction input indicating that no response is required for the specific action detected this time nor a voice input is made, an exception process may be performed in step S46. The exception process may be an instruction to output a prescribed message, or a process of notifying the operator Op again that a response request has arrived.

[0154] <Effects of the above configuration> The above-mentioned in-vehicle monitoring ECU 30 executes a center cooperation process when it detects the performance of a specific behavior of a passenger in a situation where the communication state is good. That is, the in-vehicle monitoring ECU 30 transmits a response request signal to the control center 2, and the management server 23 provided in the control center 2 notifies the operator Op that a specific behavior has been detected in the vehicle. Then, voice data based on the speech / instruction of the operator Op is output from the in-vehicle speaker 18. In this way, when a specific behavior of a passenger is detected in a situation where the communication state is good, the operator Op is configured to be able to make an announcement such as a warning after checking the image of the in-vehicle camera 151. Therefore, even if a crew member such as a driver is not in the vehicle, it is possible to make an announcement according to the state of the passenger. That is, it is possible to reduce the risk that the system will not respond to a specific behavior of a passenger.

[0155] In addition, when a passenger is detected to be performing a specific action under a situation where a communication failure occurs with the control center 2, the system response is changed according to the length of the expected recovery time Trc, which is the predicted time until the communication situation is restored to the desired required level. In other words, when the expected recovery time Trc is relatively short, the system executes the center cooperation process as in normal times without outputting a prescribed message. On the other hand, when the expected recovery time Trc is relatively long, the system outputs a prescribed message corresponding to the action.

[0156] According to the above configuration, even in a situation where a communication failure occurs with the control center 2, if the communication situation is expected to improve in a relatively short time, the operator Op will respond to the behavior of the passengers. The impression given to the passengers is different between the announcement of the operator Op and a standard / mechanical voice message. It is expected that the passengers are more likely to follow the announcement of the operator Op than the standard / mechanical voice message. In addition, according to the configuration in which the voice of the operator Op is output, it is expected to have the effect of making the passengers aware that the situation inside the vehicle is being watched. In other words, according to the configuration in which the operator Op basically makes announcements, it is possible to give the passengers a sense of security more than a configuration in which only a prescribed message is output. In addition, by making the passengers aware that the operator Op is watching the inside of the vehicle, it is expected to have the effect of suppressing violations of manners and deterring crime.

[0157] In addition, if a configuration does not include the prescribed message output unit F42, if a communication failure occurs, it is not possible to warn passengers about unsafe behavior, etc., until communication is restored. In response to such a problem, the configuration of this embodiment includes the prescribed message output unit F42, and outputs a prescribed message when the predicted recovery time Trc is equal to or greater than a predetermined value. By outputting the prescribed message, it becomes possible to provide a notification in real time according to the behavior or state of passengers, even if a communication failure occurs.

[0158] Furthermore, according to the above configuration, when the in-vehicle monitoring ECU 30 predicts that a communication failure will continue for a certain period of time or more based on the radio wave intensity map, it continues operation in a fallback mode that suppresses the vehicle speed, acceleration, and deceleration. Reducing the vehicle speed, etc. can improve safety and ride comfort. As a result, it is possible to reduce the risk of passengers falling in a situation where the control center 2 cannot monitor the inside of the autonomously driving bus 1 in real time. It is also possible to reduce the risk of collisions with other moving objects in a situation where the control center 2 cannot monitor the outside world of the autonomously driving bus 1 in real time.

[0159] Incidentally, the detection of a specific behavior is performed using a discrimination model constructed by machine learning. Therefore, there is a possibility that the specific behavior detection unit F1 will erroneously detect the performance of a specific behavior by a passenger. In the above configuration, in principle, when the communication state is good, the operator Op will check the video of the in-vehicle camera 151 and then issue a warning. Therefore, it is possible to reduce the possibility of issuing an unnecessary warning based on a false detection.

[0160] In addition, in the above embodiment, when the operator Op judges that an announcement or the like is unnecessary, it is configured to be possible to input that no special measures are necessary. Even if a passenger is actually standing without holding onto a strap or the like, it may be the case that the scene is not serious enough to warrant a warning or that it is unavoidable. In such a case, if the operator Op checks the image of the in-car camera 151 and is configured to judge whether or not a warning is necessary, it is possible to reduce the risk of unnecessary warnings being issued. As a result, it is possible to reduce the risk of causing inconvenience to passengers.

[0161] Although the embodiment of the present disclosure has been described above, the present disclosure is not limited to the above-mentioned embodiment, and various supplements and modifications described below are also included in the technical scope of the present disclosure, and further, various modifications other than those described below can be implemented without departing from the gist of the present disclosure. For example, the various supplements and modifications described below can be implemented in appropriate combinations as long as no technical contradiction occurs. Note that the same reference numerals are used for components having the same functions as the components described above, and their description may be omitted. In addition, when only a part of the configuration is mentioned, the above description can be applied to the other parts.

[0162] <Additional information on standard message output processing> The prescribed message may be repeatedly output at a fixed interval while a communication failure occurs. However, if the same content is continuously notified for a fixed period of time, such as 30 seconds, and the situation inside the vehicle does not change, it is expected that further repeated output of the same message will be ineffective. In addition, continuous output of the same message may cause annoyance or discomfort to passengers.

[0163] For this reason, if the passenger's condition does not change even after the prescribed message has been issued a prescribed number of times / for a prescribed period of time, it is preferable to stop the repeated output of the prescribed message. Fig. 13 is a flowchart showing an example of a specific sequence of processing related to the output of such a prescribed message.

[0164] The flowchart shown in Fig. 13 can be executed as, for example, steps S25, S26a, and S35. When the response unit F4 determines to output a prescribed message in consideration of the expected recovery time Trc, etc., it outputs a voice message corresponding to the contents of the detected specific behavior (step S51). When the response unit F4 outputs a message, it increments the number of times the message has been output by one. In addition, it starts measuring the elapsed time since the start of the prescribed message output process (step S52).

[0165] Then, if the time elapsed since the start of the prescribed message output process is 30 seconds or more (step S53: YES), the response unit F4 stops the repeated output of the prescribed message and ends this flow. The response unit F4 also stops the repeated output of the prescribed message when the specific behavior is resolved (step S54: NO). On the other hand, if the time elapsed since the start of the prescribed message output process is less than 30 seconds and the specific behavior is still continuing (step S54: YES), the process returns to step S51 and outputs the prescribed message again.

[0166] According to the above configuration, it is possible to reduce the possibility that the same message will be repeatedly output and cause discomfort to passengers. Note that the threshold value of 30 seconds for stopping the repeated output of the message is merely an example, and may be 20 seconds, 40 seconds, or the like. The number of times for stopping the repetition of the specified message may be two, three, or four times.

[0167] <Additional information on the selection method for subjects> The in-vehicle monitoring ECU 30 may set passengers to be alerted against unsafe behavior according to the flow shown in Fig. 14. That is, the risk assessment unit F12 calculates a fall risk Rf for each passenger based on the riding posture and attribute information (step S61), and the response unit F4 sets the passenger with the highest calculated fall risk Rf as the person to be alerted (step S62). Then, the response unit F4 issues an alert to the person set in the previous step in a manner according to the current communication state with the control center 2 (step S63). That is, when the communication state is good, the center cooperation process is performed, whereas when a communication failure occurs, a specified message is output.

[0168] According to the above configuration, it is possible to give priority to warning passengers with a high risk of falling Rf. If the situation inside the vehicle changes due to a warning, such as a target person gripping a handrail, the fall risk Rf for each passenger may be recalculated, and a warning may be given to other passengers whose fall risk Rf is still equal to or greater than the predetermined value.

[0169] The response unit F4 may be configured to change the content of the response process according to the risk of falling of the subject. For example, when the risk of falling is equal to or greater than a predetermined value, even if the predicted recovery time Trc is less than the fallback threshold Tfb, a fallback instruction signal may be output and the autonomous driving ECU 20 may be operated in the fallback mode. According to this configuration, when the risk of falling is high, careful control is adopted, and it is possible to reduce the risk of the passenger falling. The response unit F4 may be configured to apply a smaller value as the fallback threshold Tfb as the risk of falling is higher.

[0170] In addition, the number of times the prescribed message is repeatedly output may be changed depending on the level of the risk of falling. For example, when the risk of falling is equal to or greater than a predetermined threshold, the number of times the prescribed message is repeatedly output may be set to three, whereas when the risk of falling is less than the threshold, the number of times the prescribed message is repeatedly output may be set to one or two. When the risk of falling is low, the number of times the prescribed message is repeated may be reduced, thereby reducing the possibility of causing discomfort to passengers.

[0171] <How to return to normal mode from fallback mode> Here, an example of control for returning the autonomous driving ECU 20 from the fallback mode to the normal mode will be described with reference to Fig. 15. For convenience, the flowchart shown in Fig. 15 is also referred to as the normal mode return determination process. The normal mode return determination process includes steps S71 to S74 as an example. This flow can be repeatedly executed at a predetermined determination period while the autonomous driving ECU 20 is operating in the fallback mode. The determination period can be 500 milliseconds, 1 second, 2 seconds, etc.

[0172] First, in step S71, the specific behavior detection unit F1 judges whether or not the unsafe behavior of the passenger continues based on the input signal from the in-car sensor 15. If the unsafe behavior continues, the process proceeds to step S72. On the other hand, if the unsafe behavior has ended, the process proceeds to step S74. The unsafe behavior has ended when the subject holds onto a strap or takes a seat, for example. In step S72, it is judged whether or not a problem still exists in communication with the control center 2. If the communication state has not yet been restored, the process proceeds to step S73. On the other hand, if the communication state has already been restored, step S74 is executed.

[0173] In step S73, a signal to instruct the automatic driving ECU 20 to continue the fallback mode is output to the automatic driving ECU 20, and this flow is terminated. In addition, in step S74, a return instruction signal to instruct the automatic driving ECU 20 to return to the normal mode is output to the automatic driving ECU 20, and this flow is terminated.

[0174] By the above process, when the unsafe behavior of the passenger has ended or the communication state has recovered to the required level, the automatic driving mode returns to the normal mode. In other words, the upper limit of the driving speed and the restrictions on the allowable acceleration / deceleration and yaw rate are relaxed.

[0175] <Additional information on normal mode and fallback mode> The normal mode of the autonomous driving ECU 20 may be an operating mode in which a driving control plan is created in which a value equivalent to the legal speed is set as the upper limit of the vehicle speed, while the fallback mode may be an operating mode in which a control plan is created in which a value that is a predetermined amount smaller than the legal speed is set as the upper limit of the vehicle speed.

[0176] The normal mode may be an operation mode in which a control plan is created with the same acceleration and deceleration as when a human driver drives a route bus as a target value, while the fallback mode may be an operation mode in which driving control is performed at a lower limit speed that is a predetermined amount lower than the above. The acceleration and deceleration when a human driver drives a route bus may be determined based on history data of a human driver driving the same route as the autonomous driving bus 1. In addition, the average acceleration and deceleration may be calculated by statistically analyzing the driving history data of multiple bus drivers, and this may be set as the upper limit of the acceleration and deceleration in the normal mode. The same applies to the angular velocity. For example, a control tolerance in the normal mode may be set based on the yaw rate generated when a human driver turns right or left at an intersection or changes lanes.

[0177] <Modifications of the configuration of the control center 2> Not only the in-vehicle monitoring ECU 30, but also the management server 23 may be configured to determine whether or not a passenger is performing a specific behavior based on the in-vehicle video provided by the in-vehicle monitoring ECU 30. With the above configuration, when the communication state is good, the operator Op can issue a warning about unsafe behaviors overlooked by the in-vehicle monitoring ECU 30. In addition, it is expected that the management server 23 has a processor with higher computing power than the in-vehicle monitoring ECU 30, which requires environmental resistance. With the above configuration, the state of the passenger is determined by a processor with higher computing power, which can reduce the possibility of erroneous detection / missed detection of a specific behavior.

[0178] In addition, in the above, a configuration in which a human being as an operator Op responds to a passenger response request from the interior monitoring ECU 30, i.e., a response request signal, has been exemplified, but the present invention is not limited to this. For example, a response device may be configured to automatically respond to a response request signal. That is, the entity that determines / instructs the content of passenger response in the control center 2 may be a response device, not a human being. The response device may be a device that identifies the state of a passenger from the image of the interior camera 151 and outputs an appropriate response, using a trained artificial intelligence model mainly constructed using a neural network, for example. Such a response device may also be considered as a kind of operator.

[0179] <Additional information regarding calculation of estimated recovery time> The in-vehicle monitoring ECU 30 may calculate the expected recovery time Trc using a reception quality map instead of or in addition to the radio wave intensity map. The reception quality map is data indicating the reception quality, for example, RSRQ, for each location. For example, the response unit F4 may calculate the expected recovery time Trc as the time required for both the radio wave intensity and the reception quality to reach or exceed a predetermined required level. The radio wave intensity map and the reception quality map correspond to communication quality map data. In addition, the communication quality map data may be data indicating the uplink / downlink throughput and round-trip latency for each location. The communication quality map data may also be called a radio wave map.

[0180] <Variations of events to be detected and their response processes> The in-vehicle monitoring ECU 30 may take the passenger's temperature, and if the temperature is detected to be equal to or higher than a predetermined value, may suggest that the passenger transfer to a bus / taxi / ambulance that the system will arrange separately. In addition, in consideration of the passenger's privacy, if the system detects that the passenger has a temperature equal to or higher than a predetermined value, the system may automatically take measures such as increasing the air exchange rate with the air conditioner or opening the windows from the viewpoint of preventing the spread of infection or from other viewpoints.

[0181] The in-vehicle monitoring ECU 30 may be configured to detect excess of passenger capacity or excess weight by identifying the number of passengers and report the result to the control center 2. It is preferable to determine excess of passenger capacity or excess weight before closing the doors or before starting off. When excess of passenger capacity is detected, the control center 2 or the prescribed message output unit F42 may output a voice message encouraging the last person to get off the vehicle. When excess of passenger capacity is detected, the control center 2 may arrange for another vehicle.

[0182] Furthermore, the in-vehicle monitoring ECU 30 may detect that luggage has been left behind and warn passengers who are about to get off the vehicle. The in-vehicle monitoring ECU 30 may also be configured to identify the user as a passenger by performing image recognition of the passenger's face. In a configuration capable of identifying the user, the in-vehicle monitoring ECU 30 may be configured to use user information such as a name linked to a face image to greet the user. In addition, if the user has a seizure or the like while in the vehicle, the user information may be used to contact the user's family or a designated hospital.

[0183] <System Configuration Variations> In the above, the automatic driving ECU 20 and the interior monitoring ECU 30 are configured as separate devices as shown in Fig. 2, but the present invention is not limited to this. For example, as shown in Fig. 16, some or all of the functions of the automatic driving ECU 20 may be provided in the interior monitoring ECU 30. More specifically, the interior monitoring ECU 30 may include a control planning unit G1 and a control execution unit G2.

[0184] 17, the autonomous driving ECU 20 may be equipped with an in-vehicle monitoring unit G3, which is a functional unit equivalent to the in-vehicle monitoring ECU 30. The in-vehicle monitoring unit G3 is configured to execute the same processing as the in-vehicle monitoring ECU 30. The in-vehicle monitoring unit G3 may be equipped in an ECU other than the autonomous driving ECU 20, for example, an HMI control ECU that controls an in-vehicle HMI. The configurations shown in FIGS. 16 and 17 correspond to a configuration in which the autonomous driving ECU 20 and the in-vehicle monitoring ECU 30 are integrated. The in-vehicle monitoring ECU 30 or a device equipped with the in-vehicle monitoring unit G3 corresponds to a passenger monitoring device.

[0185] <Additional remarks> The device, system, and method thereof described in the present disclosure may be realized by a dedicated computer that includes a processor programmed to execute one or more functions embodied by a computer program. The device and method described in the present disclosure may be realized by a dedicated hardware logic circuit. The device and method described in the present disclosure may be realized by one or more dedicated computers configured by a combination of a processor that executes a computer program and one or more hardware logic circuits. For example, some or all of the functions of the interior monitoring ECU 30 may be realized as hardware. The embodiment in which a certain function is realized as hardware includes an embodiment in which the function is realized using one or more ICs. As the processor (computation core), a CPU, an MPU, a GPU, a DFP (Data Flow Processor), or the like can be adopted. Some or all of the functions of the interior monitoring ECU 30 may be realized by combining multiple types of computation processing devices. Some or all of the functions of the interior monitoring ECU 30 may be realized by using a system-on-chip (SoC), an FPGA, an ASIC, or the like. FPGA is an abbreviation for Field-Programmable Gate Array. ASIC stands for Application Specific Integrated Circuit.

[0186] The computer program may be stored in a computer-readable non-transitive tangible storage medium as instructions executed by a computer. A hard-disk drive (HDD), a solid-state drive (SSD), a flash memory, a secure digital (SD) card, or the like may be used as a storage medium for the program. The present disclosure also includes a program for causing a computer to function as the in-vehicle monitoring ECU 30, and a non-transitive tangible storage medium such as a semiconductor memory in which the program is recorded. Although the above describes a modified example of a specific implementation mode of the in-vehicle monitoring ECU 30, the contents of this remark may also be applied to the management server 23. [Explanation of symbols]

[0187] 1 Autonomous driving bus (passenger vehicle), 2 Control center, 15 In-vehicle sensor, 18 In-vehicle speaker, 20 Autonomous driving ECU (computer), 30 In-vehicle monitoring ECU (passenger monitoring device), 22 Operator HMI, 23 Management server, 151 In-vehicle camera, 152 In-vehicle microphone, 221 Display, 222 Microphone, 223 Speaker, 224 Input device, 231 Server processor, F1 Specific behavior detection unit, F11 Passenger information acquisition unit, F12 Risk assessment unit, F2 Communication status acquisition unit, F3 Report processing unit, F4 Response unit, F41 Communication status prediction unit, F42 Specified message output unit, F43 Center cooperation unit, M1 Passenger correspondence table memory unit (notification data memory unit), M2 Radio wave intensity map memory unit (storage medium)

Claims

1. A passenger monitoring device configured to be capable of wireless communication with a control center, the passenger monitoring device being used in a passenger vehicle configured to be able to operate without a crew member under the management of a control center, A report processing unit (F3) that transmits an interior image of the passenger vehicle, which is an image captured by a camera that captures the interior of the vehicle, to the control center; a specific behavior detection unit (F1) that detects whether a passenger is performing a specific behavior based on an input signal from a sensor mounted on the passenger vehicle; a notification data storage unit (M1) storing data for outputting a voice message corresponding to the content of the specific action; A response unit (F4) that performs a response process corresponding to the detected specific behavior based on the detection of the performance of the specific behavior; A communication status acquisition unit (F2) that acquires a current communication status with the control center from a wireless communication device mounted on the passenger vehicle, The response unit, as the response processing, outputting a prescribed message corresponding to the specific action by voice from an in-car speaker (18) that is a speaker installed in the passenger vehicle using the data stored in the notification data storage unit; The vehicle control system is configured to selectively, in parallel, or in sequence perform at least two types of processing: acquiring voice data transmitted from the control center based on the in-vehicle image, and outputting a voice corresponding to the voice data from the in-vehicle speaker; The response unit further comprises: changing the content to be executed as the response process based on a communication state with the control center when the performance of the specific action is detected; When the performance of the specific action is detected, if a problem occurs in communication with the control center, a predicted recovery time is calculated based on communication quality map data indicating wireless communication quality for each point stored in a predetermined storage medium (M2), which is a time until the communication quality with the control center becomes equal to or higher than a predetermined required level; A passenger monitoring device configured to suspend output of the specified message and wait for receipt of the voice data from the control center if the predicted recovery time is less than a predetermined allowable waiting time.

2. The passenger monitoring device according to claim 1, The passenger monitoring device is configured so that the response unit outputs the specified message on condition that the predicted recovery time is equal to or longer than the allowable waiting time.

3. The passenger monitoring device according to claim 1 or 2, The passenger monitoring device, wherein the response unit is configured to change the allowable waiting time depending on the content of the detected specific behavior.

4. The passenger monitoring device according to claim 1, The response unit: the predetermined message is repeatedly output at a predetermined interval while communication with the control center is not possible and the specific behavior of the passenger is continuing, A passenger monitoring device configured to stop outputting the specified message when a predetermined time has elapsed since the start of output of the specified message, or when the specified message has been output a predetermined number of times.

5. A passenger monitoring device according to any one of claims 1 to 4, The response unit: When the performance of the specific action is detected, if communication with the control center is impossible, a recovery expected time is calculated based on communication quality map data indicating wireless communication quality for each point stored in a predetermined storage medium (M2), which is a time until the communication quality with the control center becomes equal to or higher than a predetermined required level, and The passenger monitoring device is configured to, as the response processing, request a computer (20) that creates the control plan to create a control plan in which the driving speed is suppressed by a predetermined amount compared to when communication with the control center is possible when the predicted recovery time is equal to or greater than a predetermined restriction application threshold.

6. A passenger monitoring device according to any one of claims 1 to 5, The response unit: When the performance of the specific action is detected, if communication with the control center is impossible, a recovery expected time is calculated based on communication quality map data indicating wireless communication quality for each point stored in a predetermined storage medium (M2), which is a time until the communication quality with the control center becomes equal to or higher than a predetermined required level, and The passenger monitoring device is configured to, as the response processing, request a computer (20) that creates a control plan to reduce the driving control tolerance value for acceleration or angular velocity by a predetermined amount from a predetermined standard value if the predicted recovery time is equal to or greater than a predetermined restriction application threshold.

7. A passenger monitoring device according to any one of claims 1 to 4, The response unit: When the performance of the specific action is detected, if communication with the control center is impossible, a predicted recovery time is calculated based on communication quality map data indicating wireless communication quality for each point stored in a predetermined storage medium (M2), which is a time until communication quality with the control center becomes equal to or higher than a predetermined required level; The passenger monitoring device is configured to, as the response processing, control the running of the passenger vehicle so that the acceleration or angular velocity acting on the passenger vehicle is smaller than when communication with the control center is possible when the predicted recovery time is greater than or equal to a predetermined restriction application threshold.

8. A passenger monitoring device according to any one of claims 1 to 7, As the specific behavior, a fall and a predetermined unsafe behavior that increases the risk of a fall are registered, The passenger monitoring device includes at least one of the unsafe behaviors: standing without holding onto a strap or handrail, walking while moving, standing up while moving, not wearing a seat belt, and sticking hands / faces out of the window.

9. A passenger monitoring device according to any one of claims 1 to 8, The specific behavior is set to include at least one of loud conversation, talking on a mobile phone, occupying two or more seats by one person, sleeping on a seat, and smoking.

10. A passenger monitoring device according to any one of claims 1 to 9, As the specific behavior, a specific unsafe behavior that increases the risk of falling is registered. the specific behavior detection unit calculates a fall risk, which is a parameter indicating a likelihood that the passenger will fall, using an age of the passenger engaging in the unsafe behavior and a content of the specific behavior as factors; The passenger monitoring device is configured such that the response unit determines the content of the response processing in accordance with the risk of falling in addition to the communication status with the control center.

11. A passenger monitoring device according to any one of claims 1 to 10, the specific behavior detection unit calculates, for each passenger, a fall risk, which is a parameter indicating a likelihood that the passenger will fall, based on at least one of the passenger's age and the presence or absence of a handicap and the passenger's riding posture as factors; The passenger monitoring device is configured such that, when there are multiple passengers whose risk of falling is equal to or greater than a predetermined value, the passenger with the highest risk of falling is set as the target of a warning.

12. A control center (2) using at least one processor (231) for managing the operation of passenger vehicles configured to be able to operate without a crew member on board; A passenger monitoring device (30) used in the passenger vehicle and configured to be able to wirelessly communicate with the control center, The passenger monitoring device includes: A report processing unit (F3) that transmits an interior image of the passenger vehicle, which is an image captured by a camera that captures the interior of the vehicle, to the control center; a specific behavior detection unit (F1) that detects whether a passenger is performing a specific behavior based on an input signal from a sensor mounted on the passenger vehicle; a notification data storage unit (M1) storing data for outputting a voice message corresponding to the content of the specific action; A response unit (F4) that performs a response process corresponding to the detected specific behavior based on the detection of the performance of the specific behavior; a communication status acquisition unit (F2) that acquires a current communication status with the control center from a wireless communication device mounted on the passenger vehicle; a prescribed message output unit (F42) that outputs a prescribed message corresponding to the specific behavior from an in-car speaker (18) that is a speaker installed in the passenger car by voice using the data stored in the notification data storage unit based on the detection of the performance of the specific behavior; a center cooperation unit (F43) that acquires voice data for an in-vehicle announcement for the specific behavior from the control center based on the detection of the performance of the specific behavior, and outputs a voice corresponding to the voice data from the in-vehicle speaker; When there is no problem in communication with the control center, the center cooperation unit operates with priority over the prescribed message output unit, The processor (231) of the control center displaying an image on an operator's display (221) to allow the operator to confirm a situation based on the detection of the performance of the specific behavior; acquiring an operator's speech in response to the specific action via a microphone (222); and transmitting voice data corresponding to the spoken voice of the operator to the passenger monitoring device, The response unit: When the performance of the specific action is detected, if a problem occurs in communication with the control center, a predicted recovery time is calculated based on communication quality map data indicating wireless communication quality for each point stored in a predetermined storage medium (M2), which is a time until the communication quality with the control center becomes equal to or higher than a predetermined required level; A traffic management system configured to suspend output of the specified message and wait for receipt of the voice data from the control center if the predicted recovery time is less than a predetermined allowable waiting time.

13. The traffic management system according to claim 12, The passenger monitoring device transmits a notification signal including information indicating an outline of the detected specific behavior to the control center based on the detection of the specific behavior, A traffic management system, wherein the processor is configured, when receiving the notification signal, to display on the display a summary notification image indicating the summary contained in the notification signal.

Images