Operation system for autonomous traveling vehicles, control method for operation system for autonomous traveling vehicles, and program

Abstract

In this operation system for autonomous traveling vehicles, in which N autonomous traveling vehicles 200 that travel along a set route, and a monitoring server 300 for monitoring the state of the autonomous traveling vehicles are connected to a network 600, the monitoring server includes a monitoring execution unit 301 which has a remote monitoring function and a remote control function for each autonomous traveling vehicle, and a monitoring person terminal 350 which is operated by monitoring persons (m persons, where m<N) and has a display unit, the monitoring server being configured such that, when an alarm is issued from any of the autonomous traveling vehicles, the monitoring server allocates a first monitoring person to respond to the alarm, causes the first monitoring person to respond to the autonomous traveling vehicle that has issued the alarm by intervening using the remote monitoring function and the remote control function of the monitoring execution unit, and causes any one or more of the second to m-th monitoring persons to monitor the autonomous traveling vehicles other than the autonomous traveling vehicle that has issued the alarm.

Description

Autonomous vehicle operation system, and control method and program for the autonomous vehicle operation system

[0001] The present invention relates to an autonomous vehicle operation system, and a control method and program for an autonomous vehicle operation system.

[0002] As an example of an article delivery system using an autonomous vehicle that travels along a set route, there is known an unmanned delivery system disclosed in Patent Document 1. The delivery system of Patent Document 1 is a system that delivers a user-specified package 3 specified by a user terminal 2 using an autonomous mobile device 8.

[0003] Patent document 2 discloses an autonomous vehicle that changes its eye-shaped image depending on the direction in which the autonomous vehicle is moving and the situation of surrounding people and obstacles, in order to be friendly and to reduce the risk of misunderstandings or friction when it comes into contact with moving objects including pedestrians and obstacles.

[0004] JP 2018-58656 A International Publication (WO 2020 / 145189 A1)

[0005] Delivery systems using autonomous vehicles can deliver goods and other items unmanned, offering significant benefits such as reduced labor costs and more efficient work, making them a useful technology.

[0006] When an autonomous vehicle travels on a public road, if there is an obstacle in the direction of travel of the autonomous vehicle, the autonomous vehicle will detect it using surrounding sensors, etc., and if autonomous travel becomes difficult, a management unit for the autonomous vehicle will be required to remotely operate the vehicle to take measures such as repairing, modifying, repairing, or correcting the obstacle, or to continue traveling after these processes. Conventionally, when multiple autonomous vehicles are operated according to their respective purposes, a method in which one monitor monitors one autonomous vehicle in the management unit is inefficient, so a method in which one monitor monitors multiple autonomous vehicles, for example, four autonomous vehicles (monitor:autonomous vehicles=1:4), has been considered.

[0007] When one supervisor monitors four autonomous vehicles, the supervisor must concentrate heavily on monitoring the display, and because each autonomous vehicle also outputs surrounding audio, it can be difficult to distinguish which autonomous vehicle the audio is coming from, for example, on a busy public road. As such, the frequency of remote intervention by the supervisor increases, particularly in busy areas, so it is estimated that the maximum supervisor:autonomous vehicle ratio is 1:2.

[0008] When commercializing delivery using multiple autonomous vehicles, a challenge is how many autonomous vehicles a single monitor can monitor, and an efficient monitoring method that can reduce the cost of monitors per autonomous vehicle is desired.

[0009] In view of the above problems, the present invention aims to provide an efficient autonomous vehicle operation system and program that can reduce the costs and effort required of monitors per autonomous vehicle.

[0010] In order to solve the above problems, the autonomous vehicle operation system of the present invention is an autonomous vehicle operation system in which N autonomous vehicles that travel according to a set route and a monitoring server that monitors the status of the autonomous vehicles are connected to a network, wherein the monitoring server comprises: a monitoring execution unit that has a remote monitoring function and a remote operation function for each autonomous vehicle; and a monitor terminal that is operated by a monitor (m people, and m<N) and has a display unit, and the monitoring server is configured so that when an alarm is issued from any of the autonomous vehicles, it assigns a first monitor to respond to the alarm, and has the first monitor intervene to respond to the autonomous vehicle that issued the alarm using the remote monitoring function and remote operation function of the monitoring execution unit, and has autonomous vehicles other than the autonomous vehicle that issued the alarm monitored by one or more of the second to mth monitors.

[0011] In the above configuration, the monitoring server preferably starts monitoring the autonomous vehicle when an alarm is issued while the autonomous vehicle is traveling along a route or when an alarm is issued due to an obstacle detected by the autonomous vehicle's monitoring sensor. When the monitoring server cancels the alarm for the autonomous vehicle, monitoring of the autonomous vehicle is canceled, and the autonomous vehicle continues its predetermined operations. The monitoring server preferably distributes surrounding video signals transmitted from the N autonomous vehicles to a monitoring pool on the display of the monitor terminal (m units) as video streaming of the N autonomous vehicles. The monitoring server preferably has an automatic alarm notification function that automatically notifies the monitor terminal of an assigned first monitor when an alarm is issued from an autonomous vehicle. The automatic alarm notification function of the monitoring server preferably has an icon display function that indicates the urgency of the alarm so that the monitor can determine it, and the monitoring server has an interrupt intervention function that changes the order in which the monitor terminal responds to the automatic alarm according to the urgency of the response to the alarm, regardless of the order in which the automatic alarms were issued. The monitoring server preferably has a function of automatically assigning a monitor who is not operating the monitor terminal. The monitoring server preferably has a function of automatically assigning observation of autonomous vehicles other than the autonomous vehicle that issued the alert to a monitor other than the first monitor. The monitoring server preferably includes a monitoring pool capable of video streaming with and remote control of each autonomous vehicle, and an automatic assignment function for assigning an available monitor. The alerts displayed on the monitor terminal preferably include any of an alert issued by a sensor of each autonomous vehicle, an alert based on the results of processing information from the sensor by a computing unit of each autonomous vehicle, and an alert detected by the monitoring server based on map information stored in a memory unit. Preferably, in response to a pop-up notification displayed on the monitor terminal indicating the content of the alert issued by the autonomous vehicle, when the monitor terminal determines that the autonomous vehicle should continue autonomous driving, the monitoring server closes the pop-up notification and closes the intervention confirmation screen for the autonomous vehicle.Preferably, when any of the autonomous vehicles issues an alarm, the monitoring server determines that monitor intervention is not required in response to the alarm when a predetermined safety check performed by the autonomous vehicle in response to the alarm matches a safety check performed by the monitor terminal. Preferably, when the autonomous vehicle detects an obstacle or the like and stops, the monitor terminal switches the autonomous vehicle into remote control mode to avoid the obstacle. The monitoring server preferably further includes a linking unit that connects the multiple different autonomous vehicles with facilities such as elevators along the vehicle's travel route. The monitoring server preferably stores map data for getting on and off elevators in a memory unit, and the linking unit is connected to the building management server and the elevator control unit via a network. The map data is preferably two-dimensional map data and / or three-dimensional map data. Preferably, the autonomous vehicle autonomously drives at the elevator boarding floor by receiving two-dimensional map data and / or three-dimensional map data of the boarding floor from a monitoring server, and autonomously drives at the elevator disembarking floor by receiving two-dimensional map data and / or three-dimensional map data of the disembarking floor from the monitoring server.

[0012] The control method for an autonomous vehicle operation system of the present invention includes step 1 in which an emergency alert is issued from the autonomous vehicle to a monitoring server, step 2 in which a response to the emergency alert is assigned to a monitor 1 using an automatic assignment function, step 3 in which a confirmation screen from the autonomous vehicle is sent as video streaming to a monitor terminal of the monitor 1, step 4 in which remote control of the autonomous vehicle is started when the monitor terminal determines that remote control of the autonomous vehicle is necessary, step 5 in which the autonomous vehicle is remotely controlled by the monitor 1, and step 6 in which the remote control is terminated when the specified remote control is no longer necessary.The program may be a program that causes a computer to function as the monitoring server described above.The program may be a program that causes a computer to function as the monitor terminal described above.

[0013] According to the present invention, based on an emergency alert issued from an autonomous vehicle to a monitoring server, the monitoring server automatically assigns a monitor to monitor the autonomous vehicle that issued the emergency alert, and has this monitor respond to the emergency alert for the autonomous vehicle and intervene remotely if necessary.The monitoring server also has other monitors monitor autonomous vehicles other than the autonomous vehicle in question, so that an autonomous vehicle operation system, a control method for an autonomous vehicle operation system, and a program can be provided that can efficiently monitor N autonomous vehicles (N > m) using m monitors.

[0014] 1 is a diagram showing the configuration of an autonomous vehicle operation system according to the first embodiment. FIG. 2 is a schematic diagram showing the functions of a monitoring execution unit. FIG. 3 is a schematic diagram showing a normal monitoring screen of a monitoring pool function. FIG. 4 is a schematic diagram showing a normal monitoring screen of 16 autonomous vehicles by two monitors. FIG. 5 is a schematic diagram showing an example in which monitor 1 remotely operates an autonomous vehicle in response to an emergency alert from one of the 16 autonomous vehicles, and monitor 2 normally monitors the other 15 autonomous vehicles. FIG. 6 is a table showing an example of an emergency alert issued from an autonomous vehicle. FIG. 7 is a flow diagram showing the processing in the monitoring server and the monitor terminal when an emergency alert is issued from an autonomous vehicle in the first embodiment. FIG. 8 is a schematic diagram showing a display screen of the monitor terminal during remote intervention. FIG. 9 is a flow diagram showing a modified example of the processing in the monitoring server when an emergency alert is issued from an autonomous vehicle in the first embodiment. 17 is a diagram showing an example of an operation of an autonomous vehicle according to a first embodiment of the present invention;

[0033] FIG. 18 is a diagram showing an example of an operation of an autonomous vehicle according to a second embodiment of the present invention;

[0034] FIG. 19 is a diagram showing an example of an operation of an autonomous vehicle according to a second embodiment of the present invention;

[0035] FIG. 20 is a diagram showing an example of an operation of an autonomous vehicle according to a first embodiment of the present invention;

[0036] FIG. 21 is a diagram showing an example of an operation of an autonomous vehicle according to a second embodiment of the present invention;

[0037] FIG. 22 is a diagram showing an example of an operation of an autonomous vehicle according to a second embodiment of the present invention;

[0038] FIG. 23 is a diagram showing an example of an operation of an autonomous vehicle according to a second embodiment of the present invention;

[0039] FIG. 24 is a diagram showing an example of an operation of an autonomous vehicle according to a second embodiment of the present invention;

[0039] FIG. 25 is a diagram showing an example of an operation of an autonomous vehicle according to a second embodiment of the present invention;

[0039] FIG. 26 is a diagram showing an example of an operation of an autonomous vehicle according to a second embodiment of the present invention;

[0015] Hereinafter, an autonomous vehicle operation system according to an embodiment of the present invention will be described with reference to the drawings.

[0016] [First Embodiment] FIG. 1 is a diagram showing the configuration of an autonomous vehicle operation system 100 according to the first embodiment. The autonomous vehicle operation system 100 includes N autonomous vehicles 200 that travel along a set route, a monitoring server 300 that monitors the status of the autonomous vehicles 200, a monitor terminal 350, a user terminal 400, and a store server 500, all connected via a network 600. The autonomous vehicles 200 are not limited to those used for delivering goods (luggage), but may also be autonomous vehicles that travel autonomously without being limited in their intended use or function, such as autonomous vehicles for transporting people, autonomous cleaning vehicles, autonomous security vehicles that guard buildings, and autonomous guidance vehicles for various facilities, and are also referred to as autonomous mobile bodies. Here, the autonomous vehicle 200 will be described as an autonomous vehicle for delivering goods. Note that the autonomous vehicles applied to the present invention are not limited to those used for delivering people or luggage, but may also be used for a wide range of purposes, such as cleaning, security, medical care, food distribution, and guidance in factories and various facilities, and can be used both indoors and outdoors.

[0017] The monitoring server 300 includes a monitoring unit having a remote monitoring function and a remote driving function for each autonomous vehicle, and a monitor terminal 350 having a display unit used by monitors (m monitors, where m<N). When an autonomous vehicle 200 issues a report or warning (also referred to as an alert), the monitoring server 300 designates a first monitor to deal with the warning, i.e., respond to the situation, such as that shown in FIG. 6 (described later). The first monitor intervenes and controls the autonomous vehicle 200 that issued the warning using the remote monitoring function and remote operation function, while the second to m monitors monitor the autonomous vehicles 200 other than the autonomous vehicle 200 that issued the alert. Here, the terms "deal with" or "respond" include repair, modification, repair, or correction, as well as measures to allow the autonomous vehicle 200 to continue driving after these processes. The autonomous vehicle 200 monitored by monitor m is represented by "N" in capital letters. Hereinafter, the number of autonomous vehicles 200 monitored by m monitors is defined as N.

[0018] The mechanical configuration of the autonomous vehicle 200 will be described later. In the first embodiment, the autonomous vehicle 200 includes a display unit 103, a monitoring camera 104, an inertial measurement unit (IMU) 112, a calculation unit 130, a delivery route setting unit 131 that sets a route for delivering goods to a user, an authentication unit 132 that performs user authentication using a user terminal 400 (described later), and a status transmission / reception unit 133 that transmits the status of the autonomous vehicle 200 to a monitoring server 300 and receives control signals from the monitoring server 300. The calculation unit 130 processes data related to emergency alerts, such as images captured by the monitoring camera 104 and audio data from a microphone. A deep learning-based algorithm may be used to process the image data and audio data. As a result, information from multiple sensors mounted on the autonomous vehicle 200 (described later) is integrated and processed by the calculation unit 130, and the result is an emergency alert that is transmitted from the status transmission unit 144 to the monitoring server 300 via the network 600.

[0019] The delivery route setting unit 131 sets a route for the autonomous vehicle 200 to travel in the autonomous vehicle 200. As will be described later, the delivery route setting unit 131 may set the delivery route from the store server 500 via the network 600, or may set the route via a CD-ROM, a USB memory, or other recording medium.

[0020] The authentication unit 132 authenticates the user who ordered the product. The user has a user terminal 400. The user terminal 400 is equipped with an authentication presentation unit 402, and an authentication code is displayed on the display device of the user terminal 400. The authentication unit 132 captures the displayed authentication code using the surveillance camera 104 and authenticates the user based on the captured authentication code. If authentication is successful, the autonomous vehicle 200 unlocks the door of the storage unit (not shown) so that the user can remove the product from the storage unit 106. The storage unit of the autonomous vehicle 200 may include a space for storing products and one or more doors for loading and removing products. Each order may be associated with one of the spaces and doors. In this case, the autonomous vehicle 200 can simultaneously deliver two or more products. Furthermore, if one user is authenticated, the other user cannot remove products ordered by the other user, resulting in a delivery system with enhanced safety.

[0021] The status transmitting / receiving unit 133 transmits image information, audio information, and sensor information from the monitoring camera 104 and microphone relating to the status of the autonomous vehicle 200 to the monitoring server 300 (described later) as necessary, and also transmits the status of the autonomous vehicle 200 to the monitoring server 300 upon request from the monitoring execution unit 301 of the monitoring server 300. The monitoring server 300 and the autonomous vehicle 200 communicate with each other via the network 600 via the status transmitting / receiving unit 133.

[0022] The monitoring server 300 is a device that monitors the status of the autonomous vehicle 200. In the first embodiment, the monitoring server 300 includes a monitoring execution unit 301, a monitoring target registration unit 302, a monitor registration unit 303, and a storage unit 304. The monitoring server 300 may be configured using a well-known computer for server use or other purposes. The monitoring server 300 may also be configured with one or more computers. The monitoring server 300 may be a so-called cloud server. The monitoring execution unit 301, the monitoring target registration unit 302, and the monitor registration unit 303, which are functions of the monitoring server 300, are recorded in advance as a program, and the computer functions as the monitoring server 300 by executing the program.

[0023] The monitoring target registration unit 302 registers the autonomous vehicle 200 to be monitored in the monitoring server 300. The status of the registered autonomous vehicle 200 is periodically monitored by the monitoring execution unit 301. If, as a result of the monitoring, an abnormality or other problem is discovered in the autonomous vehicle 200, or if it is determined that regular maintenance is necessary, the maintenance person for the autonomous vehicle 200 or other designated person in charge is contacted.

[0024] The monitor registration unit 303 registers monitors in the monitoring server 300. Password authentication can be used for this registration. Furthermore, monitor registration may be performed using biometric authentication such as facial recognition or fingerprint authentication, for example, Windows Hello (registered trademark). Such biometric authentication can be performed using a web camera installed on the monitor of the monitoring server 300 and / or the monitor terminal 350, or a camera such as a web camera or USB camera connected to the monitoring server 300 via a USB cable. Biometric authentication can provide higher security than password authentication. Registered monitors are assigned autonomous vehicles 200 to be monitored by the monitoring execution unit 301, which will be described later. Therefore, monitors register with the monitor registration unit 303 before the start of their shifts and operate the monitor terminal 350 so that they are deleted from the monitor registration unit 303 when their shifts end. This allows only monitors registered in the monitor registration unit 303 to engage in monitoring work.

[0025] Network 600 is a computer network that connects autonomous vehicle 200, user terminal 400, store server 500, and monitoring server 300. Network 600 may be a well-known computer network, such as a wide area network such as the Internet, a third-generation (referred to as 3G) or fourth-generation (referred to as 4G) mobile phone network, or a locally constructed local area network (LAN).

[0026] (Monitoring of Autonomous Vehicle 200) A signal S1 from the monitoring camera 104 or the monitoring sensor 113 of the autonomous vehicle 200 is transmitted from the status transmitting / receiving unit 133 of the autonomous vehicle 200 to the monitoring server 300 via the network 600. Furthermore, in order to detect contact with a person or an obstacle around the autonomous vehicle 200, a signal S2 detected by a sensor such as the IMU 112 can be transmitted to the monitoring server 300. As a result, the monitoring execution unit 301 displays the signal S1 from the autonomous vehicle 200 on the monitor terminal by video streaming, which will be described later. Furthermore, the monitoring execution unit 301 can remotely control the autonomous vehicle 200 by using a signal S3 in response to an emergency alert from the autonomous vehicle 200, which will be described later. Here, the emergency alert is an alert issued by a sensor of the autonomous vehicle 200 or an alert based on the result of processing information from the sensor by the calculation unit 130 of the autonomous vehicle 200. When remotely operating the autonomous vehicle 200, the monitoring execution unit 301 may output a voice message from the speaker of the autonomous vehicle 200 in advance to alert people in the vicinity. In particular, when remotely operating the autonomous vehicle 200, if the autonomous vehicle is about to turn left on a public road, the monitoring execution unit 301 may output a voice message such as "We are about to turn left. Please be careful," or "We apologize for the inconvenience, but please be careful," or "Excuse me, but please give way," from the speaker of the autonomous vehicle 200 if the autonomous vehicle 200 detects a person or an obstacle and stops traveling.

[0027] FIG. 2 is a schematic diagram showing the functions of the monitoring execution unit 301. The monitoring execution unit 301 has a function of automatically assigning autonomous vehicles 200 to monitors, i.e., an automatic allocation function A of autonomous vehicles 200, and a monitoring pool function B of displaying the autonomous vehicles 200 managed by multiple monitors on the display unit of the monitor terminal 350. FIG. 2 shows that monitor 1 is assigned to autonomous vehicle 200-1 and monitor 2 is assigned to autonomous vehicle 200-2 by the automatic allocation function of the autonomous vehicles 200. Furthermore, video signals from the monitoring cameras of autonomous vehicles 200-1 and 200-2 are sent to the monitoring server 300 via the network 600, and are then sent to the display (display unit) of the monitor terminal by video streaming in the monitoring pool B. Here, video streaming refers to video information including audio information sent from one autonomous vehicle 200.

[0028] The monitor terminal 350 is composed of a so-called PC, a display connected to the PC, and input devices such as a mouse and keyboard as necessary, and is connected to the monitoring server 300 via a network 600 .

[0029] FIG. 3 schematically illustrates a normal monitoring screen of the monitoring pool function. The normal monitoring screen is a screen showing video streaming from N autonomous vehicles 200 displayed on the display of a monitor terminal 350 used by a single monitor. FIG. 3 illustrates N=16 video streaming screens, with one monitor performing normal monitoring of 16 autonomous vehicles 200. In this case, there is one monitor (m=1) and the number of autonomous vehicles 200 (N) is 16 (N=16), so m=1, N=16, and m<N. Here, normal monitoring refers to a state in which the monitor is not remotely operating the autonomous vehicles 200, and is also referred to as observation or passive monitoring. Furthermore, a state in which the monitor is remotely operating the autonomous vehicles 200 is referred to as intervention or check-in. Intervention is also referred to as active monitoring, as opposed to passive monitoring.

[0030] 4 is a schematic diagram showing a normal monitoring screen of 16 autonomous vehicles 200 by two observers. As shown in FIG. 4, observer 1 performs normal monitoring of 16 autonomous vehicles 200, and observer 2 performs normal monitoring (observation) of 16 autonomous vehicles 200.

[0031] FIG. 5 schematically illustrates a case in which monitor-1 remotely controls autonomous vehicle 200-1, one of the 16 autonomous vehicles 200, in response to an emergency alert from autonomous vehicle 200-1 or an alert based on the results of processing by the calculation unit 130 of autonomous vehicle 200-1 of information from sensors mounted on the autonomous vehicle, while monitor-2 normally monitors the other 15 autonomous vehicles 200. FIG. 6 is a table showing an example of an emergency alert issued by autonomous vehicle 200-1. An emergency alert is also referred to as an event. As shown in FIG. 5, when monitor-1 determines that remote control is necessary due to the emergency alert from autonomous vehicle 200-1, he or she begins remote control of autonomous vehicle 200-1. Then, on the monitor terminal 350 used by monitor-1, only the video streaming from autonomous vehicle 200-1 is displayed on the display. Meanwhile, on the monitor terminal 350 used by monitor-2, video streaming of the other 15 autonomous vehicles 200 other than the autonomous vehicle 200-1 remotely operated by monitor-1 is displayed on the display, that is, the monitor is in an observation state (normal monitoring state). Note that, although the above description has been given assuming that there are 16 autonomous vehicles 200, the number of autonomous vehicles 200 that the monitor observes at one time can be adjusted depending on the frequency of emergency alerts. If the frequency of emergency alerts is high, the number of autonomous vehicles 200 that the monitor observes at one time may be eight.

[0032] As shown in FIG. 6 , the monitoring server 300 displays an alarm "DETECTED_CONSTRUCTION_AREA (approaching a construction site)" on the display of the monitor terminal 350, and the monitor is notified of the message "Caution: Construction site." The following alarms are displayed on the display of the monitor terminal 350 via the monitoring server 300 based on signals from the sensors of the autonomous vehicle 200. The alarm "DETECTED_SIREN (emergency vehicle approaching)" is a message stating "Caution: Emergency vehicle." The alarm "DETECTED_STEP_DOWN (step detected)" is a message stating "Caution: Step." The alarm "EMERGENCY_BUTTON_ACTIVATED (emergency stop button activated)" is a message stating "Emergency stop." The alarm "BAD_MAP_QUALITY_AREA (area prone to localization loss)" is a message stating "Self-location unstable." The alarm "STOPPING_TIMEOUT (prolonged stop state)" is a message stating "long-term stop." The "PATH_END_BLOCKED" alert is a "Road Obstruction" message. The "DRIVABLE_AREA_EXIT" alert is a "Driveable Area Exit" message. The "WRONG_ROAD_ENTRY" alert is a "Wrong Road Entry" message. The "UNDEFINED_ROAD_ENTRY" alert is a "Wrong Road Entry" message. The "CROWDED_ENVIRONMENT" alert is a "Crowd Alert" message.

[0033] The following warnings are not based on sensors of the autonomous vehicle 200 or on the results of processing information from the sensors by the calculation unit 130, but are based on warnings from the monitoring server 300 and are displayed on the display of the monitor terminal 350. These warnings are issued based on advance information, such as a two-dimensional or three-dimensional map of the delivery area of ​​the autonomous vehicle 200, pre-stored in the storage unit 304 of the monitoring server 300. The warning "APPROACHING_CROSSING (approaching a crosswalk)" is a message saying "Watch out for the crosswalk." The warning "APPROACHING_TRAFFIC_LIGHT (approaching a traffic light)" is a message saying "Watch out for the traffic light." The warning "ENTERING_SHARED_ROAD (entering a shared road)" is a message saying "Watch out for pedestrians and vehicles not separated." The warning "ENTERING_ELEVATOR (entering the elevator)" is a message saying "Get on the elevator." The warning "ENTERING_NARROW_AREA (entering a narrow area)" is a message saying "Watch out for the narrow passage."

[0034] 7 is a flow diagram showing the processing in the monitoring server 300 and the monitor terminal when an emergency alert is issued from the autonomous vehicle 200 in the first embodiment, and FIG. 8 is a diagram schematically showing the display screen of the monitor terminal during remote intervention. For example, when the autonomous vehicle 200 is automatically traveling along a route for delivery, a situation requiring intervention is detected based on the results of processing by the monitoring sensor or the calculation unit 130, and an emergency alert is issued from the autonomous vehicle 200 to the monitoring server 300 (step ST1). In response to this, the monitoring server 300 assigns a response to the emergency alert to the monitor 1 using an automatic assignment function (step ST2), and sends a confirmation screen from the autonomous vehicle 200 to the monitor terminal 350 of the monitor 1 as video streaming (step ST3). Here, the autonomous vehicle's driving route may include public roads, commercial buildings where the autonomous vehicle enters and exits for deliveries, apartment buildings, and other buildings, outdoor private roads, parks, arenas, factories, and other facilities, and is not limited to any location that the autonomous vehicle 200 can enter. When the monitor determines from the monitor terminal display that remote control of the autonomous vehicle 200 is necessary, the monitor begins remote control of the autonomous vehicle 200 (step ST4). Starting remote control is also referred to as check-in. The autonomous vehicle 200 is remotely controlled by the monitor 1 (step ST5), and when the specified remote control is no longer required, the remote control ends (step ST6). When the remote control ends, the monitor 1 stops intervention (also referred to as check-out, step 7). If there is a next monitoring task, the monitor 1 is automatically assigned by the monitoring server 300, and proceeds to processing of the next assignment screen.

[0035] As shown in Figure 8, during remote intervention, the monitor 1's display 350 displays people and obstacles C around the autonomous vehicle 200 detected by the monitoring sensor at the same time as streaming video of the autonomous vehicle 200.

[0036] The remote operation of autonomous vehicle 200 by monitoring server 300 is performed in response to an alarm that occurs while autonomous vehicle 200 is traveling on a travel route such as a public road, and when the remote operation stops, that is, when intervention is stopped, autonomous vehicle 200 performs the task specific to autonomous vehicle 200, that is, continues the predetermined task. The predetermined task is, for example, a task other than traveling on a travel route such as traveling on a public road using autonomous vehicle 200, such as delivery work, security, or guiding visitors.

[0037] (Variation of Processing in Monitoring Server) FIG. 9 is a flow diagram showing a variation of processing in the monitoring server 300 when an emergency alert is issued to the autonomous vehicle 200 in the first embodiment. First, when the autonomous vehicle 200 is autonomously traveling on a public road for delivery, a monitoring sensor detects a situation requiring intervention, and the autonomous vehicle 200 issues an emergency alert to the monitoring server 300. In step 11, the autonomous vehicle 200 determines whether remote intervention by an observer is required (YES) or not (NO). If remote intervention is required (YES), the autonomous vehicle 200 is temporarily stopped (step ST12), and the monitoring server 300 determines whether an observer is available (YES) or not (NO) (step ST14). If remote intervention is not required (NO) in step 11, the autonomous vehicle 200 continues autonomous traveling (step ST13) and returns to the start. If monitoring server 300 determines in step ST14 that there is an available monitor (YES), the automatic allocation function displays a confirmation screen on monitor terminal 350 of that monitor, notifying the monitor that management of autonomous vehicle 200 has been newly assigned (step ST15). On the other hand, if there is no available monitor (NO) in step ST14, monitoring server 300 continues to stop autonomous vehicle 200 from traveling, and returns to step ST12. In step ST15, the monitor assigned the new autonomous vehicle 200 determines whether remote intervention is required (YES) in response to the emergency alert issued by autonomous vehicle 200 (NO) (step ST17).

[0038] In step ST17, if there is an available observer, the monitoring server 300 instructs the available observer to observe the autonomous vehicles 200 other than the autonomous vehicle 200 to be intervened (step ST18). If the monitoring server 300 determines in step ST17 that there is no available observer, it stops all autonomous vehicles 200 for safety reasons. In step ST18, the observer assigned to the new autonomous vehicle 200 for which an emergency alert has been issued performs remote intervention on the autonomous vehicle 200 (step ST19) and stops the remote intervention in step ST20. If the other autonomous vehicles 200 are stopped in step ST20, the traveling of these autonomous vehicles 200 may be resumed. On the other hand, if remote intervention is not required (NO) in step ST17, the autonomous vehicle 200 may continue to travel (step ST21), or if there is an interrupt process that prioritizes an emergency alert from another autonomous vehicle 200, the autonomous vehicle 200 may continue to stop (step ST22) and the process may return to step ST17.

[0039] FIG. 10 is a schematic diagram of a display screen on which the monitor's terminal is notified of an automatic emergency alert by the automatic alert notification function to the monitor. A list D of autonomous vehicles currently being monitored by the monitor is displayed on the left side, and a list E of autonomous vehicles 200 requiring attention and types of events is displayed on the right side. Other alerts include, for example, the detection of construction sites, traffic signals, intersections, and immobility. Furthermore, video streaming F of autonomous vehicles 200 requiring attention is highlighted with a red or yellow frame. The autonomous vehicles 200 shown in FIG. 10 may be displayed as icons representing the autonomous vehicles 200, rather than as video streaming. Furthermore, when an emergency alert is issued, these icons may be highlighted with a red or yellow frame. Furthermore, the icons may be displayed with the name of the city or project where the autonomous vehicles 200 are operating.

[0040] FIG. 11 is a schematic diagram of a display screen on which multiple emergency alerts are notified to the monitor terminal. Multiple emergency alerts requiring attention from autonomous vehicle 200 are displayed on the left side of the display screen. For example, if autonomous vehicle 200 is stopped in a way that disrupts the flow of people in a crowd, video streaming should be addressed before an autonomous vehicle 200 waiting at a traffic light at a sparsely traveled intersection. In this case, the monitor can perform interrupt processing at the monitor's discretion, rather than following the order of the automatic alert function of monitoring server 300, to intervene in the autonomous vehicle 200 that is stopped in a way that disrupts the flow of people in a crowd.

[0041] 12 is a diagram showing the difference in priority of emergency alerts displayed on the monitor terminal, and shows an example in which the display screen changes color depending on the level of urgency. For example, if a construction site or the like is detected, it is a situation that requires the autonomous vehicle 200 to stop, and therefore it is displayed in red (G) as a high priority that requires immediate stopping, and if it is stopped at an intersection, it is displayed in yellow (H).

[0042] FIG. 13 schematically illustrates a function by which a monitor selects an automatic emergency alert. The automatic emergency alert is displayed in a pop-up window on the monitor's terminal display. If the monitor confirms safety, the monitor presses a button to continue autonomous driving. When the monitor instructs automatic continuation (OK), the pop-up window closes, monitoring of the autonomous vehicle 200 ends, and the monitor waits for the pop-up window for the next autonomous vehicle 200 to appear. The three screens on the right side of FIG. 13 show pop-up windows. For example, at a pedestrian crossing, the autonomous vehicle 200 recognizes a green light, and the monitor simultaneously confirms the green light on the monitor's terminal display. After double-checking, the monitor instructs automatic continuation, and the pop-up window closes. On the other hand, if safety cannot be confirmed, the monitor intervenes (remotely controls) using the monitor's terminal. Intervention is also called check-in. For example, if the autonomous vehicle 200 stops due to an obstacle or a malfunction at an intersection and is unable to move, the monitor switches to remote control mode on the monitor terminal and intervenes in the autonomous vehicle 200, for example, by remotely controlling the vehicle to avoid the obstacle.

[0043] (Monitoring System by a Group of Monitors) As shown in FIG. 14 , which illustrates the operation method of the autonomous vehicle 200 of the present invention, m monitors (m is an integer equal to or greater than 2) can monitor a large number of N (m<N) autonomous vehicles 200. Conventionally, when a single monitor remotely controls one autonomous vehicle, the monitor must remotely control the autonomous vehicle while viewing video of the vehicle on the monitor's terminal. During this remote control, the monitoring server must stop other autonomous vehicles. In contrast, according to the operation method of the autonomous vehicle of the present invention, unlike the conventional method described above, while one monitor remotely controls the autonomous vehicle 200, the other monitors monitor the autonomous vehicles 200 other than the one being remotely controlled by the other monitors. This allows efficient monitoring of N (m<N) autonomous vehicles even when the monitor is taking a toilet break or feeling unwell. Furthermore, if the monitor leaves their seat for a toilet break or when feeling unwell, they may be monitored by the camera of the monitoring server 300. This allows the camera to easily detect when the monitor leaves their seat. In this case, the monitor can be excluded from the monitor list.

[0044] In the autonomous vehicle operation method of the present invention, by setting the autonomous driving rate of autonomous vehicle 200, that is, the total driving time of autonomous vehicle 200 minus the time remotely operated by the monitor, to about 95 to 98%, it is possible to reduce emergency alerts from the autonomous vehicle, thereby reducing the burden on the monitor and reducing the operating costs of the autonomous vehicle. To this end, it is desirable to always keep the maps stored in storage unit 304 of monitoring server 300 up to date.

[0045] (Configuration of Autonomous Vehicle) Next, the configuration of the autonomous vehicle 200 when it is a delivery vehicle will be described. FIG. 15 shows the appearance of the autonomous vehicle according to the first embodiment of the present invention, where (a) is a front perspective view and (b) is a rear perspective view, and FIG. 16 shows the configuration of the control circuit of the autonomous vehicle of FIG. 15. As shown in FIG. 15, the autonomous vehicle 20 is composed of a main body 21 formed in a substantially cubic shape in appearance and wheels 22 serving as a driving unit suspended from the main body 21. The main body 21 of the autonomous vehicle 20 is equipped with a display unit 23, a monitoring camera 24, a speaker 25a, a microphone 25b, a storage unit 26 for items to be transported, a light 27, a direction indicator 28, and a three-dimensional lidar 32. The three-dimensional lidar 32 is used for self-position estimation and detection of people and obstacles. The lights 27 include front lights 27a and 27b disposed at the front of the autonomous vehicle 20 and back lights 27c and 27d disposed at the rear. The storage unit 26 is equipped with a locking and unlocking mechanism (not shown), and is configured so that information regarding the locking and unlocking of the storage unit 26 can be detected by a sensor. Information regarding the locking and unlocking of the storage unit 26 may be transmitted to the monitoring server 300 via the status transmitting / receiving unit 86f.

[0046] The turn indicators 28 are arranged in the shape of ears on the upper left and right sides of the front of the autonomous vehicle 20 so that people around the autonomous vehicle 20 can see the direction to turn from the front, sides, and back, and are made up of, for example, light-emitting diode (LED) lamps.

[0047] 2, the display unit 23 is provided on the front side of the main body 21 and displays an eye-shaped image. The display unit 23 is composed of a display 83 (described later) and a pair of left and right openings 23a, 23b provided on the front side of the main body 21. The display 83 is preferably a display device using liquid crystal, organic electroluminescence (EL), or light-emitting diodes, and more preferably a full-color light-emitting diode display with high-brightness light-emitting diodes arranged at a high density. A speaker 25 and a microphone 25b are provided below the display unit 23.

[0048] The surveillance cameras 24 acquire images of the surroundings of the autonomous vehicle 20 and photograph and monitor the conditions of the passages in front of, to the sides of, and behind the autonomous vehicle 20, as well as pedestrians, etc. The surveillance cameras 24 are composed of a pair of front surveillance cameras 24a, 24b provided on the left and right sides of the front side of the main body 21, and a pair of side surveillance cameras 24c, 24d provided on the left and right sides of the main body 21. The surveillance cameras 24 may be monocular cameras, wide-area cameras, stereo cameras, etc.

[0049] 2, the surveillance camera 24 may include a pair of left and right rear surveillance cameras 24e, 24f at the rear of the main body 21. By using monocular cameras as the front surveillance cameras 24a, 24b and the rear surveillance cameras 24e, 24f, respectively, it is possible to capture images of objects, people, other autonomous vehicles, etc. in a 360-degree area around the autonomous vehicle 20, thereby monitoring the surroundings.

[0050] 16 , autonomous vehicle 20 includes a CPU 81 as control unit 29, which is connected to speaker 25 a, microphone 25 b, battery 82, display 83, memory unit 84, drive unit 85, and detection unit 86. Detection unit 86 is made up of monitoring camera 24 as an imaging unit, an attitude detection sensor 86 a such as an IMU that is a sensor for controlling the traveling state of autonomous vehicle 20, a monitoring sensor 86 b, a distance sensor 86 c, a position sensor 86 d, a bumper sensor 86 e that detects contact with surrounding pedestrians, bicycles, motorbikes, etc., a status transmission / reception unit 86 f that transmits and receives to network 600 (see FIG. 1 ), and an opening / closing sensor 86 g related to locking and unlocking of storage unit 26.

[0051] The CPU 81 and the storage unit 84 control each unit of the device mounted on the autonomous vehicle 20. The CPU 81 is configured by a microprocessor, a microcontroller, or the like.

[0052] The storage unit 84 is configured with nonvolatile memory such as DRAM, a hard disk drive (HDD), flash memory, or a disk using flash memory (SSD). The CPU 81 may be connected to each device using a known method, such as a controller area network (CAN). Furthermore, the program that causes the autonomous vehicle 20 to function is recorded and stored on a storage medium that can be read by a computer that includes the CPU 81 and the storage unit 84, which executes the program. Examples of storage media that can be used include a CD-ROM, a DVD-ROM, and a USB memory. The program may be downloaded from the monitoring server 300 to the computer's storage unit 84 via a wireless network, for example.

[0053] The battery 82 is a power source for the autonomous vehicle 20. The battery 82 is connected to the CPU 81, and the remaining charge of the battery 82 and other information are monitored by the CPU 81.

[0054] The display 83 is a device that constitutes the display unit 23. In this embodiment, a predetermined eye-shaped image is displayed on the display unit 23 based on the state of the autonomous vehicle 20 and / or the image captured by the monitoring camera 24. Here, the state of the autonomous vehicle 20 refers to each driving operation state on the driving route, such as going straight, turning left, turning right, or stopping, as well as image information from the monitoring camera 24 and detection information from the detection unit 86 regarding the autonomous vehicle 20. The detection information from the detection unit 86 is information from the attitude detection sensor 86a, the monitoring sensor 86b, the distance sensor 86c, the position sensor 86d, the bumper sensor 86e, etc. The image information from the monitoring camera 24, the audio information from the microphone 25b, and the detection information from the detection unit 86 regarding the state of the autonomous vehicle 20 are transmitted to the monitoring server 300 by the status transmission / reception unit 86f as necessary.

[0055] The device that displays the eye-shaped image is display 83, and an image according to a predetermined eye-shaped pattern is displayed on display 83 in response to a light emission command from CPU 81 based on a detection signal from surveillance camera 24. When the image is displayed on display 83, a predetermined output linked to the display of the image from speaker 25a may be generated. Furthermore, lights 27 and direction indicators 28 are displayed according to the traveling or stopped state of autonomous vehicle 20. Lights 27 and direction indicators 28 are displayed by a light emission driver (not shown) in response to a light emission command from CPU 81.

[0056] The drive unit 85 is composed of a motor 85a and a driver 85b that drives and controls the motor 85a, and the motor 85a drives the wheels 22. Specifically, a driver L 85b1 and a driver R 85b2 control the motors 85a1 and 85a2, respectively, in response to a control signal from the CPU 81 to drive the left rear wheel 22c and the right rear wheel 22d. The speaker 25a is installed on the front surface of the main body 21 of the autonomous vehicle 20, as described above, and is a device that outputs a predetermined sound.

[0057] The attitude detection sensor 86a is, for example, an inertial measurement unit (IMU), which in this embodiment is an inertial measurement device that measures the angular velocity and angular acceleration of the movement of the autonomous vehicle 20 around each of the roll axis, pitch axis, and yaw axis.

[0058] The monitoring sensor 86b is a sensor used to detect people, obstacles, etc., around the autonomous vehicle 20, particularly those ahead, and measure the distance to the people or obstacles, i.e., distance measurement, two-dimensional and three-dimensional image recognition of people and obstacles, shape and color recognition, and tracking the autonomous vehicle 20's travel path. When the monitoring sensor 86b detects a person or obstacle, it acquires position information such as the position coordinates and the person's eye height. The display 83 may be controlled based on this position information. The monitoring sensor 86b may also use a module that acquires position information using a monocular camera or stereo camera, a lidar, etc. The module may be composed of a CPU or GPU (image processing unit) that processes image data acquired from the monocular camera or stereo camera to generate position information, a storage device, etc. The module using the monocular camera can recognize the shape, color, and pattern of people and obstacles and can also measure approximate distances. The module using the stereo camera is used for distance measurement, three-dimensional recognition of people, vehicles, obstacles, etc., and identifying shapes and colors.

[0059] LIDAR (Laser Imaging Detection and Ranging), also known as laser radar, is a sensor that performs laser image detection and ranging. Two-dimensional LIDAR or three-dimensional LIDAR may be used. Three-dimensional LIDAR can detect laser images ahead of the autonomous vehicle 20, measure the distance to the detected object, and measure the shape of the detected object. Furthermore, it is used for self-position estimation and the detection of people and obstacles for driving control of the autonomous vehicle. When LIDAR is provided, it detects objects ahead of the autonomous vehicle 20 and the distance thereto, and sends data on the distance between the laser image ahead and the detected object to the CPU 81 as a detection signal.

[0060] Furthermore, the monitoring sensor 86b may be disposed on the upper front portion of the autonomous vehicle 20. The monitoring sensor 86b can detect images and measure distances to people, other autonomous vehicles, obstacles, etc. in front of the autonomous vehicle 20 over long distances.

[0061] The distance sensor 86c measures the distance between the autonomous vehicle 20 and obstacles, etc., and is positioned facing forward of the autonomous vehicle 20. It measures the distance to an obstacle ahead on the road by emitting ultrasonic waves or infrared rays toward the obstacle and detecting the reflected waves.

[0062] The position sensor 86d acquires the current position of the autonomous vehicle 20. In this embodiment, the position sensor 86d uses a GNSS receiving unit. It is possible to arbitrarily select whether the position sensor 86d and the attitude detection sensor 86a use separate devices or whether a device in which the GNSS receiving function, gyro sensor function, and acceleration sensor function are integrated into a single package is used.

[0063] The bumper sensor 86e detects contact with nearby people, obstacles, other autonomous vehicles, etc. By detecting contact with nearby people, obstacles, or other autonomous vehicles, the bumper sensor 86e can stop the automatic travel of the autonomous vehicle 20 or make an emergency stop.

[0064] The status transmitting / receiving unit 86f is composed of a communication module capable of public communication, such as third generation (called 3G), fourth generation (called 4G), fifth generation (called 5G), or wireless LAN.

[0065] (User Terminal) The user terminal 400 shown in FIG. 1 is a terminal used by a user when the operation system 100 is used in a delivery system using autonomous vehicles. The user terminal 400 may be a well-known computer, such as a smartphone or tablet computer owned by the user. The functions of the order placing unit 401 and the authentication presenting unit 402, which will be described later, may be recorded as a program in the user terminal 400, and the computer may function as the user terminal 400 by executing the program. This program is recorded and stored in a storage medium that can be read by the computer that stores the program. The storage medium may be a CD-ROM, a USB memory, or other recording medium, or may be read into the computer via the network 600.

[0066] The user terminal 400 includes a product ordering unit 301 that places an order for products to be delivered to the store server 500 (described later), and an authentication presentation unit 402 that authenticates the user via the authentication unit 132 of the autonomous vehicle 200. The order placing unit 401 places an order for products desired for delivery by the autonomous vehicle 200 in response to a user's operation. Specifically, the order placing unit 401 transmits order information to the store server 500, including product information indicating the product, delivery destination information indicating the delivery destination, and orderer information indicating the user who placed the order. When the user receives the product delivered by the autonomous vehicle 200, the authentication presentation unit 402 presents information necessary for user authentication by the authentication unit 132 of the autonomous vehicle 200. The authentication presentation unit 402 also presents the authentication information to the autonomous vehicle 200 by displaying a predetermined authentication code on a display device such as a display of the user terminal 400. The authentication code may be generated using a well-known encoding technique, and may be displayed using a QR code (registered trademark), for example.

[0067] (Store Server) The store server 500 is a device used by a store that delivers merchandise to receive orders from the user terminal 400 and to issue delivery instructions to the autonomous vehicle 200. The store server 500 can be configured using a well-known computer for server use or other purposes. The store server 500 may be configured using one computer or two or more computers.

[0068] When the store server 500 is configured using a single computer, it is assumed that the computer will be installed in the store or a warehouse where products are stored, since it will need to be operated by the store manager or an operator. When the store server 500 is configured using two or more computers, some of the computers may be installed in a remote location, such as a data center. Like the user terminal 400, the store server 500 also has the functions of the order receiving unit 501 (described below) pre-recorded as a program, and the computer can function as the store server 500 by executing the program. The program that causes the store server 500 to function is recorded and stored in a storage medium that can be read by the computer that stores the program. The storage medium may be a CD-ROM, USB memory, or the like, or it may be read into a computer via the network 600.

[0069] The store server 500 includes an order receiving unit 5401 that receives orders from the user terminal 400. The order receiving unit 501 receives the order information sent by the product ordering unit 401, and replies to the user terminal 400 to confirm that the order has been received.

[0070] Second Embodiment FIG. 17 shows the configuration of an autonomous vehicle operation system according to the second embodiment. The autonomous vehicle operation system 100A according to the second embodiment differs from the autonomous vehicle operation system 100 according to the first embodiment in that the monitoring server 300A further includes a linking unit 305 and a linking database 304a in a storage unit 304A. The linking unit 305 has a function of communicatively connecting multiple autonomous vehicles 200 with multiple elevators installed in buildings such as commercial buildings to which the autonomous vehicles 200 are delivered, and with security gates and the like as needed. The linking unit 37 supports various communication protocols to accommodate communication methods specific to the manufacturers of the autonomous vehicles 200. Examples of communication methods include MQTT (MQ Telemetry Transport, a protocol used for device-to-device communication), PLC (Programmable Logic Controller), and OPC UA (an open-source industrial interface developed by the OPC Foundation). Data corresponding to these communication methods used in the linking unit 37 is stored as a linking database 304a in the storage unit 304A.

[0071] The linking unit 305 also connects a plurality of elevators installed in a building or facility with a security gate or the like using an API (Application Program Interface) stored in the linking database 304a.

[0072] The multiple autonomous vehicles 200 may be autonomous vehicles with different uses, such as those for transporting transportation facilities, delivering vehicles, cleaning vehicles, or security vehicles. Furthermore, the autonomous vehicles may be for the same use but manufactured by different companies. Various communication protocols and screens (user interfaces) displayed on the monitor terminal are registered in advance in the coordination database 304a. This allows multiple autonomous vehicles 200A with different manufacturers and connection communication protocols to be registered or learned once in the coordination unit of the monitor server 300A. In the autonomous vehicle operation system 100A of the present invention, autonomous vehicles with different uses can be monitored using a single user interface and various services can be supported.

[0073] (Linkage with Elevator) Figure 18 shows a configuration when linkage with an elevator 41 installed in a building 40 is performed in an autonomous vehicle operation system 100B according to the second embodiment. Figure 18 differs from Figure 17 in that it shows an elevator 41 installed in a building 40, and in that map data 304b required for generating a travel route for the autonomous vehicle 200 is stored in a memory unit 304 of a monitoring server 300B. As shown in Figure 18, the building 40 is made up of multiple floors, and is provided with an elevator 41 for easy movement between the floors. The elevator 41 is movable up and down within a travel path 42 that passes vertically through the building 40 and is composed of an elevator car 43 that moves up and down by a drive unit (not shown), and an elevator control unit 44 that drives and controls the elevator car 43 up and down.

[0074] Elevator control unit 44 is connected via gateway 45a to building management server 45, which manages the entire building 40, and building management server 45 is further connected to monitoring server 300B and monitoring terminal 350 from gateway 45b via network 600. As a result, in autonomous vehicle operation system 100B, autonomous vehicle 200, monitoring server 300B, and building management server 45 are interconnected with building 40 and elevator 41, and electrical equipment such as elevator 41 and security gates (not shown) within building 40 can be controlled via network 600 by monitoring execution unit 301 of monitoring server 300B.

[0075] (Elevator Use Using Three-Dimensional Map Data) Next, elevator use in autonomous vehicle operation system 100B according to this embodiment will be described with reference to Figure 19. First, in step A1, destination information 61b indicating the destination is input to calculation unit 130 of autonomous vehicle 200 by monitoring server 300B or an operation unit of autonomous vehicle 200 (not shown). Next, in step A2, calculation unit 130 of autonomous vehicle 200 detects the current position of autonomous vehicle 200 using a detection signal from position sensor 66d, and acquires current position information 61a. In step A3, calculation unit 130 transmits current position information 61a and destination information 61b to monitoring server 300B via network 600. In response to this, in step A4, the monitoring execution unit 301 of the monitoring server 300B reads out map data 304b of the area corresponding to the driving route from the current position according to the current position information 61a to the destination according to the destination information 61b from the memory unit 304 based on the current position information 61a and the destination information 61b, and in step A5, transmits this map data to the autonomous vehicle 200 via the network 600.

[0076] Map data 304b is registered in storage unit 304 as a database, and map data 304b including the vicinity of the location information can be read based on the location information. Map data 304b is two-dimensional and / or three-dimensional data that includes data on the roads and surrounding environment necessary for autonomous vehicle 200 to travel in the area in which autonomous vehicle 200 can travel, such as information on road boundaries and the positions of traffic lights, and also holds data on the surroundings of buildings and routes on each floor within the building. Furthermore, in the case of a building 40 equipped with an elevator 41, map data 304b1 includes information such as the position information of elevators 41 that can be used to travel to other floors within the building 40.

[0077] In step A6, the calculation unit 130 creates a travel route 61c from the current position to the destination from the current position information 61a, the destination information 61b, and the map data 304b, and transmits the travel route 61c to the monitoring server 300B via the network 600. After creating the travel route 61c in the above-mentioned step A6, the calculation unit 130 creates elevator usage information 61d and elevator control information 61e in step A7, and transmits the elevator usage information 61d and the elevator control information 61e to the monitoring server 300B via the network 600 in step A8.

[0078] When autonomous vehicle 200 approaches the vicinity of the elevator hall in step A9, monitoring server 300B detects autonomous vehicle 200's approach to the elevator based on current position information 61a from autonomous vehicle 200 in step A10, and in step A11 transmits elevator usage information 61d and elevator control information 61e from building management server 45 to elevator control unit 44 via network 600. In response to this, elevator control unit 44 calls elevator 41 at the relevant floor in step A12 based on elevator control information 61e out of elevator usage information 61d and elevator control information 61e.

[0079] In step A13, when the elevator car 43 arrives at the boarding floor and the door opens, in step A14 the elevator control unit 44 outputs a voice message through the speaker 43a inside the elevator car 43 saying, "You are at floor ○. The autonomous vehicle will board now. Please cooperate with boarding," to guide the people inside the elevator car 43 into boarding the autonomous vehicle 200.

[0080] When autonomous vehicle 200 enters elevator car 43 and the door of elevator car 43 closes in step A15, elevator control unit 44 specifies the disembarking floor for elevator car 43 based on elevator control information 61e in step A16, and outputs a voice message from speaker 43a in elevator car 43 based on elevator usage information 61d in step A17 saying, "The autonomous vehicle has boarded on floor ○. This autonomous vehicle will disembark on floor □. Please cooperate with disembarking." to inform people in elevator car 43 that autonomous vehicle 200 will disembark.

[0081] In step A18, when the elevator car 43 starts ascending or descending from the boarding floor, in step A19, the elevator control unit 44, based on the elevator usage information 61d, outputs a voice message from the speaker 41a installed in the elevator hall at the disembarking floor of the autonomous vehicle 200 saying, "The autonomous vehicle is about to disembark. Please cooperate with disembarking." to provide advance notice that the autonomous vehicle 200 will be disembarking.

[0082] In step A20, when elevator car 43 arrives at the disembarking floor and the door opens, in step A21 elevator control unit 44 outputs a voice message from speaker 43a inside elevator car 43 saying, "This is floor X. The autonomous vehicle is disembarking. Please cooperate with disembarking," to inform people inside elevator car 43 that autonomous vehicle 200 is disembarking. This causes autonomous vehicle 200 to disembark from elevator car 43 in step A22. After disembarking, autonomous vehicle 200 autonomously drives to, for example, a conference room on the disembarking floor, and then ends. In this way, autonomous vehicle 200's use of elevator 41 provided in building 40 is completed.

[0083] In the case of autonomous driving using the elevator 41 of the autonomous vehicle 200, in step A10, the monitoring server 300B detects the approach of the autonomous vehicle 200 to the elevator based on current position information 61a from the autonomous vehicle 200, and in step A11 transmits elevator usage information 61d and elevator control information 61e from the building management server 45 to the elevator control unit 44 via the network 600, but the elevator usage information 61d and elevator control information 61e may also be transmitted in advance to the elevator control unit 44 in step A8.

[0084] (Using an elevator using two-dimensional map data) This section describes how to use an elevator using two-dimensional map data in a case where autonomous cleaning vehicle 200 only has the function of recognizing two-dimensional position information. Figure 19 is a flowchart showing the operation of a modified example of elevator use by the autonomous cleaning vehicle of Figure 18. In this modified example of elevator use by autonomous cleaning vehicle 200, steps other than step A4', step A5', and step A15' are the same as when three-dimensional map data is used in Figure 18, and therefore a description of the same steps will be omitted.

[0085] In step A4', monitoring server 300B reads map data 304b from storage unit 304 based on the destination information, etc., and in step A5' transmits this map data to autonomous vehicle 200 via network 600. The map data received by autonomous vehicle 200 in step A5' is two-dimensional map data that includes the first floor of building 40. After autonomous vehicle 200 boards elevator 41 in step A15, autonomous vehicle 200 receives two-dimensional map data of, for example, the third floor, which is the disembarking floor of elevator 41, transmitted from monitoring server 300B in step A15'. As a result, cleaning autonomous vehicle 200 that boards elevator 41 on the first floor of building 40 can get off elevator 41 on the third floor, which is the disembarking floor of elevator 41, in step A22, and then arrive at its destination, for example, a conference room on the third floor, to perform cleaning.

[0086] In autonomous vehicle operation system 100B according to the second embodiment, monitoring server 300B stores map data 304b for getting on and off elevators 41 in storage unit 304, and linking unit 305 is connected to building management server 45 and elevator control unit 44 via network 600. Map data 304b is two-dimensional and / or three-dimensional data that includes data on roads, buildings 40, elevators 41 within buildings 40, and the surrounding environment necessary for autonomous vehicle 200 to travel in the area in which autonomous vehicle 200 can travel, such as information on road boundaries, traffic light positions, etc., and is a map that can be used by autonomous vehicle 200.

[0087] The map that can be used by the autonomous vehicle 200 is data that can centrally manage the map coordinates of maps used by different autonomous vehicles 200, and has the function of converting different map coordinates into absolute coordinates, and is executed by the monitoring execution unit 301 and the program and map data 304b stored in the memory unit.

[0088] According to the above configuration, autonomous vehicle 200 autonomously travels at the boarding floor of elevator 41 by receiving two-dimensional map data and / or three-dimensional map data of the boarding floor from the monitoring server, and autonomously travels at the disembarking floor of elevator 41 by receiving two-dimensional map data and / or three-dimensional map data of the boarding floor from monitoring server 300B. Furthermore, if autonomous vehicle 200 uses three-dimensional map data, when using the elevator, autonomous vehicle 200 receives three-dimensional map data from monitoring server 300B at the boarding floor, for example, disembarks from elevator 41 at the disembarking floor, and autonomously travels to the destination, for example, a conference room.

[0089] The present invention is not limited to the above-described first and second embodiments, and various modifications are possible without departing from the spirit of the present invention. For example, in the above-described embodiments, the autonomous vehicle 200 has been described as an autonomous vehicle for delivery. However, if the autonomous vehicle is used for various tasks such as security, patrol monitoring, disinfection, and cleaning, it may be equipped with various functions depending on the application, such as an alarm, disinfectant sprayer, polisher, or cleaner. Furthermore, the autonomous vehicle may be used as a guide to guide visitors to commercial buildings, public transportation facilities, airports, etc., or as a transport vehicle for delivering goods within hospitals or transporting stretchers carrying patients. If the autonomous vehicle is a cart type with a single-seat seat, it may be used as an autonomous vehicle for transporting patients who are unable or have difficulty walking.

[0090] In FIG. 2 , the monitoring pool is described as being monitored by Monitor 1 and Monitor 2. However, Monitor 1 may be multiple monitors connected to network 600 via the Internet from Japan, and Monitor 2 may be multiple monitors connected to network 600 via the Internet from a country other than Japan. When Monitor 1 connects from Japan, the video streaming delay is 268.68 mS (milliseconds) or less, and the remote control delay is 29.76 mS or less. Furthermore, experiments conducted by the inventors showed that when monitoring from Vietnam, the video streaming delay is 350 mS (milliseconds) or less, and the remote control delay is 60 mS or less. From this, it was confirmed that monitors working in the monitoring pool can stream video and remotely control vehicles from multiple countries, even taking delay times into account. As described below, it was found that two monitors can monitor 20 autonomous vehicles 200.

[0091] FIG. 5 shows an example in which monitor-1 remotely operates an autonomous vehicle in response to an emergency alert from one of the 16 autonomous vehicles, while monitor-2 normally monitors the other 15 autonomous vehicles. However, if the frequency of emergency alerts is low, the number of autonomous vehicles 200 monitored by the monitor at one time may be 16 or more, for example, 20. FIG. 21 shows a case in which monitor-1 remotely operates autonomous vehicle 200-1 in response to an emergency alert from autonomous vehicle 200-1 of the 20 autonomous vehicles 200, or an alert based on the results of processing by a calculation unit 130 of autonomous vehicle 200-1 of information from a sensor mounted on the autonomous vehicle, and monitor-2 normally monitors the other 19 autonomous vehicles 200. When monitor-1 determines that remote operation is necessary due to the emergency alert from autonomous vehicle 200-1, he or she begins remote operation of autonomous vehicle 200-1. Then, only the video streaming from autonomous vehicle 200-1 is displayed on the display of monitor terminal 350 used by monitor-1. On the other hand, the video streaming of the other 19 autonomous vehicles 200 other than autonomous vehicle 200-1 remotely operated by monitor-1 is displayed on the display of monitor terminal 350 used by monitor-2, and the monitor enters an observation state (normal monitoring state).

[0092] 6, an emergency alert is described as being issued while autonomous vehicle 200 is traveling on a public road or the like. However, an emergency alert may also be sent from autonomous vehicle 200 to monitoring server 300 regarding an obstacle captured by monitoring camera 104 while autonomous vehicle 200 is traveling. Obstacles include suspicious objects and suspicious individuals. In this case, when the autonomous vehicle is on a delivery mission and a suspicious individual who is not the intended recipient of a product delivered by the autonomous vehicle is discovered, a monitor may remotely intervene and issue an audio warning from the autonomous vehicle.

[0093] 18 to 20 , it has been described that speaker 43a inside elevator car 43 provides guidance for getting on and off autonomous vehicle 200. However, guidance regarding getting on and off autonomous vehicle 200 may also be displayed on display 43b inside elevator car 43. Guidance regarding getting on and off autonomous vehicle 200 may also be provided by speaker 41a and / or display 46e provided in the elevator hall. Furthermore, images inside elevator 41 and information regarding the space inside elevator 41 may be transmitted from a camera or elevator analysis sensor inside elevator car 43 to monitoring server 300B and / or a monitor terminal via network 600. This allows the monitor to monitor the interior of elevator 41 inside building 40 and also enables remote operation of autonomous vehicle 200.

[0094] 18 , the monitoring server 300B is connected to the elevator control unit 44 via the building management server 45 over the network 600. However, the monitoring server 300B may also be connected to an elevator cloud over the network 600. The elevator cloud is a server managed and operated by, for example, an elevator manufacturer. As a result, according to the autonomous vehicle operation system of the present invention, the monitoring server 300B can smoothly link with the elevators 41, automatic doors, security gates, and other facilities within the building 40 via the linking unit 305, resulting in various advantages such as enabling the operation of autonomous vehicles 200 manufactured by different manufacturers, and significantly improving convenience.

[0095] 20...Autonomous vehicle; 21...Main body; 22...Wheels; 23...Display unit; 24...Surveillance camera; 25a...Speaker; 25b...Microphone; 26...Storage unit; 26a...Mechanism capable of locking and unlocking; 27...Light; 28...Turn indicator; 29...Control unit; 32...3D lidar; 40...Building; 41...Elevator; 41a...Speaker in elevator hall; 42...Travel path; 43...Elevator car; 43a...Speaker in elevator car; 43b...Display in elevator car; 43c...Camera; 43e...Occupant information; 44... Elevator control unit; 45...building management server; 45a, 45b...gateway; 46e...elevator hall display; 61a...current location information; 61b...destination information; 61d...elevator usage information; 61e...elevator control information; 81...CPU; 82...battery; 83...display; 84...memory unit; 85...drive unit; 85a...motor; 85b...driver; 86...detection unit; 86a...posture detection sensor; 86b...monitoring sensor; 86c...distance sensor; 86d...position sensor; 86e...bumper sensor; 86f... Status transmission / reception unit; 86g...opening / closing sensor for locking and unlocking the storage unit; 100...operation system for autonomous vehicle; 103...display unit; 104...surveillance camera; 112...IMU (inertial measurement unit); 130...calculation unit; 131...delivery route setting unit; 132...authentication unit; 133...status transmission / reception unit; 200...autonomous vehicle; 300...monitoring server; 301...monitoring execution unit; 302...monitoring target registration unit; 303...monitoring person registration unit; 304...storage unit; 304a...coordination database; 304b...map data; 305...coordination unit; 350...monitoring User terminal; 400...user terminal; 401...order placing unit; 402...authentication presentation unit; 500...store server; 501...order receiving unit; 600...network; A...automatic allocation function; B...monitoring pool function; C...people and obstacles around the autonomous vehicle; D...list of autonomous vehicles being monitored; E...list of autonomous vehicles and events requiring attention; F...video streaming of autonomous vehicle 200 requiring attention; G, H...display in different colors depending on urgency; S1...surveillance camera signal; S2...IMU detection signal; S3...remote control signal

Claims

1. An autonomous vehicle operation system in which N autonomous vehicles that travel according to a set route and a monitoring server that monitors the status of the autonomous vehicles are connected to a network, wherein the monitoring server comprises: a monitoring execution unit that has a remote monitoring function and a remote operation function for each of the autonomous vehicles; and a monitor terminal that is operated by a monitor (m people, and m<N) and has a display unit, wherein the monitoring server is configured to: when an alarm is issued from any of the autonomous vehicles, assign a first monitor to respond to the alarm, and have the first monitor intervene to respond to the autonomous vehicle that issued the alarm using the remote monitoring function and the remote operation function of the monitoring execution unit, and have autonomous vehicles other than the autonomous vehicle that issued the alarm monitored by one or more of the second to mth monitors.

2. An autonomous vehicle operation system as described in claim 1, wherein when the autonomous vehicle issues the alarm while traveling on a route, or when the alarm is issued due to an obstacle detected by a monitoring sensor of the autonomous vehicle, the monitoring server begins monitoring the autonomous vehicle, and when the alarm for the autonomous vehicle by the monitoring server is lifted, monitoring of the autonomous vehicle is lifted and the autonomous vehicle continues its specified operations.

3. The autonomous vehicle operation system of claim 1, wherein the monitoring server distributes surrounding video signals transmitted from the N autonomous vehicles to a monitoring pool on the display of the monitor terminal (m units) as video streaming of the N autonomous vehicles.

4. An autonomous vehicle operation system as described in claim 1, wherein the monitoring server has an automatic alarm notification function that automatically notifies the assigned first monitor of the alarm when the alarm is issued from the autonomous vehicle.

5. An autonomous vehicle operation system as described in claim 4, wherein the automatic alarm notification function of the monitoring server has an icon display function that indicates the urgency of the alarm so that the monitor can make a judgment, and the monitoring server has an interrupt intervention function that changes the order in which the monitor terminal responds to the automatic alarms depending on the urgency of the response to the alarm, regardless of the order in which the automatic alarms were generated.

6. An autonomous vehicle operation system according to claim 4, wherein the monitoring server has a function of automatically assigning a monitor who is not operating a monitor terminal.

7. An autonomous vehicle operation system as described in claim 6, wherein the monitoring server has a function of automatically assigning the observation of autonomous vehicles other than the autonomous vehicle that issued the alarm to a monitor other than the first monitor.

8. The autonomous vehicle operation system of claim 1, wherein the monitoring server is equipped with a monitoring pool capable of video streaming and remote control of each autonomous vehicle, and an automatic allocation function for allocating available monitors.

9. An autonomous vehicle operation system as described in claim 1, wherein the alerts displayed on the monitor terminal include any of alerts issued by sensors in each autonomous vehicle, alerts based on the results of processing information from the sensors by a computing unit in each autonomous vehicle, and alerts detected by the monitoring server based on map information stored in a memory unit.

10. The autonomous vehicle operation system of claim 1, wherein, in response to a pop-up notification displayed on the monitor terminal indicating the content of a warning issued by the autonomous vehicle, when the monitor terminal determines that the autonomous vehicle should continue autonomous driving, the monitoring server closes the pop-up notification and closes the intervention confirmation screen for the autonomous vehicle.

11. An autonomous vehicle operation system as described in claim 1, wherein when any of the autonomous vehicles issues an alarm, the monitoring server determines that no monitor intervention is required in response to the alarm when a predetermined safety check made by the autonomous vehicle in response to the alarm matches the safety check made by the monitor terminal.

12. An autonomous vehicle operation system as described in claim 1, wherein, when the autonomous vehicle detects an obstacle or the like and stops, the monitor terminal switches the autonomous vehicle into remote control mode to avoid the obstacle.

13. The autonomous vehicle operation system according to claim 1, wherein the monitoring server further includes a linking unit that connects a plurality of different autonomous vehicles with a facility having an elevator that is part of the vehicle's route.

14. An autonomous vehicle operation system as described in claim 13, wherein the monitoring server stores map data for getting on and off the elevator in a memory unit, and the linking unit is connected to a building management server and an elevator control unit via the network.

15. The autonomous vehicle operation system according to claim 14, wherein the map data is two-dimensional map data and / or three-dimensional map data.

16. An autonomous vehicle operation system as described in claim 15, wherein the autonomous vehicle autonomously drives at the elevator boarding floor by receiving two-dimensional map data and / or three-dimensional map data of the boarding floor from the monitoring server, and autonomously drives at the elevator disembarking floor by receiving two-dimensional map data and / or three-dimensional map data of the boarding floor from the monitoring server.

17. A method for controlling an operation system of an autonomous vehicle as described in claim 1, comprising: step 1 in which an emergency alert is issued from an autonomous vehicle to a monitoring server; step 2 in which a response to the emergency alert is assigned to a monitor 1 using an automatic assignment function; step 3 in which a confirmation screen from the autonomous vehicle is sent as video streaming to the monitor terminal of monitor 1; step 4 in which remote control of the autonomous vehicle is initiated when the monitor terminal determines that remote control of the autonomous vehicle is necessary; step 5 in which the autonomous vehicle is remotely controlled by monitor 1; and step 6 in which remote control is terminated when specified remote control is no longer required.

18. A program that causes a computer to function as the monitoring server described in claim 1.

19. A program that causes a computer to function as the monitor terminal described in claim 1.

Images