Remote monitoring device

Abstract

To enable easy and accurate driving resumption determination after an automatic driving vehicle is shifted into a minimum risk state, in a situation where one remote monitoring person monitors a large number of automatic driving vehicles.SOLUTION: A remote monitoring device (2) comprises: an output unit (23) that outputs event information received by a communication unit between a first time when an automatic driving vehicle is shifted into a stopped state, and a second time before a predetermined time with respect to the first time, in a case that the automatic driving vehicle is shifted into the stopped state; and a determination unit (256) that determines, for each automatic driving vehicle, priority information indicating a priority order to be addressed when the communication unit receives, from the plurality of automatic driving vehicles, specific event information indicating an event that may cause shifting of the automatic driving vehicles into the stopped state, wherein the output unit displays the priority information together with information or an image corresponding to each automatic driving vehicle.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a remote monitoring device. [Background technology]

[0002] In autonomous driving at level 4 or higher, when an autonomous vehicle enters a minimum risk state (automatically stopped state), a remote monitor will determine whether autonomous driving can be resumed. Therefore, various technologies have been proposed in the past to assist the remote monitor in making a decision when an autonomous vehicle enters a minimum risk state. For example, Patent Document 1 describes a remote monitoring system that includes an autonomous vehicle and a remote monitoring center, and when the autonomous vehicle automatically stops, the remote monitoring center determines whether the autonomous vehicle can be resumed based on the image received from the camera image transmission unit, and transmits a start signal to the autonomous vehicle if it is determined that the autonomous vehicle can be resumed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-87015 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional technology described in Patent Document 1, the only material available to the supervisor to determine whether or not driving can be resumed is the image captured by the camera. For this reason, the supervisor may not be able to make a decision unless he or she carefully checks the image. As a result, when a large number of autonomous vehicles are monitored by a small number of supervisors, it takes time to check the images, and it may be difficult to quickly determine whether or not driving can be resumed. [Means for solving the problem]

[0005] A remote monitoring device according to one embodiment of the present invention comprises a communication unit that communicates with at least one autonomously driven vehicle and continuously acquires event information indicating at least one of an event detected by a sensor equipped in the autonomously driven vehicle and an event that occurs as a result of a judgment made by the autonomously driven vehicle; an output unit that, when the autonomously driven vehicle transitions to a stopped state, outputs the event information received by the communication unit between a first time when the autonomously driven vehicle transitions to the stopped state and a second time a predetermined time before the first time; and a determination unit that, when the communication unit receives specific event information from each of a plurality of the autonomously driven vehicles indicating an event that may cause the autonomously driven vehicle to transition to the stopped state, determines priority information indicating a high priority for action for each of the autonomously driven vehicles, and the output unit displays the priority information together with information or an image corresponding to each autonomously driven vehicle.

[0006] A remote monitoring device according to another aspect of the present invention comprises a communication unit that communicates with at least one of the autonomous vehicles and continuously acquires event information indicating at least one of an event detected by a sensor equipped in the autonomous vehicle and an event that occurs as a result of a judgment made by the autonomous vehicle; an output unit that, when the autonomous vehicle transitions to a stopped state, outputs a predetermined number of the most recent pieces of event information received by the communication unit, including the event information received when the autonomous vehicle transitioned to the stopped state; and a determination unit that, when the communication unit receives specific event information indicating an event that may cause the autonomous vehicle to transition to the stopped state from each of the autonomous vehicles, determines priority information indicating a high priority for handling for each of the autonomous vehicles, and the output unit displays the priority information together with information or an image corresponding to each autonomous vehicle.

[0007] Each aspect of the present invention may be realized by a computer. In this case, the present invention also includes a program for realizing the remote monitoring device on a computer by causing the computer to operate as each part (software element) of the remote monitoring device, and a computer-readable recording medium on which the program is recorded. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a functional configuration of an autonomously driving vehicle to be monitored by a remote monitoring device according to an embodiment of one aspect of the present invention. FIG. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of the remote monitoring device according to the embodiment. [Diagram 3] 11 is a diagram showing an example of output contents by an output unit included in the remote monitoring device according to the embodiment. FIG. [Figure 4] FIG. 2 is a diagram showing an example of output contents by the output unit. [Diagram 5] FIG. 2 is a diagram showing an example of output contents by the output unit. [Figure 6] FIG. 2 is a diagram showing an example of output contents by the output unit. [Figure 7] FIG. 2 is a diagram showing an example of output contents by the output unit. [Figure 8] FIG. 11 is a diagram showing an example of a decision made by a remote monitor using the remote monitoring device. [Figure 9] 11 is a diagram showing another example of a determination made by a remote monitor using the remote monitoring device. FIG. [Figure 10] 11 is a diagram showing an example of output contents by an output unit included in the remote monitoring device according to the embodiment. FIG. [Figure 11] FIG. 2 is a diagram showing an example of output contents by the output unit. [Figure 12] FIG. 2 is a diagram showing an example of output contents by the output unit. [Figure 13] FIG. 2 is a diagram showing an example of output contents by the output unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] <1. Self-driving vehicles> Before describing the remote monitoring device 2 according to the present invention, a schematic configuration of an autonomously driven vehicle 1 that is the target of monitoring by the remote monitoring device 2 will first be described.

[0010] The autonomous vehicle 1 performs autonomous driving at level 4 or higher. The autonomous vehicle 1 may have a structure that allows a person to ride in a standing position. Specifically, the autonomous vehicle 1 may have a structure similar to that of a bus (particularly a route bus). The autonomous vehicle 1 may have a structure other than that described above. As shown in FIG. 1, the autonomous vehicle 1 includes a sensor 11, a vehicle-side communication unit 12, a vehicle-side memory unit 13, and a vehicle-side control unit 14.

[0011] [Sensor] The sensor 11 is at least one of an image sensor, a radio wave sensor, an optical sensor, and a microphone. The image sensor captures at least one of the inside (hereinafter, inside the vehicle) and the outside (hereinafter, outside the vehicle) of the autonomous vehicle 1 to generate image data. The captured object inside the vehicle includes, for example, a person riding in the autonomous vehicle 1. The captured object outside the autonomous vehicle 1 includes, for example, at least one of a roadway, a lane, a sign, a sidewalk, a pedestrian, a preceding vehicle, a following vehicle, an oncoming vehicle, and an obstacle. The obstacle includes at least one of a building, a curb, a roadside tree, a broken-down vehicle, and a fallen object. In addition, a road closure in a section of the roadway is also considered as an obstacle by the vehicle, and is therefore included in the obstacle. The radio wave (optical) sensor constitutes a Ladar (Lidar) and detects radio waves (light) reflected from surrounding objects. The microphone captures at least one of the voices inside the vehicle and outside the vehicle to generate voice data. The voices inside the vehicle include, for example, the voices of people riding in the autonomous vehicle 1. The sounds outside the vehicle include at least one of the voices of people around the autonomous vehicle 1 and the sound of a collision with the autonomous vehicle 1.

[0012] [Vehicle side communication unit] The vehicle-side communication unit 12 communicates with the remote monitoring device 2. The vehicle-side communication unit 12 according to the present embodiment is configured with, for example, a wireless communication module. That is, the vehicle-side communication unit 12 according to the present embodiment communicates wirelessly with the remote monitoring device 2.

[0013] [Vehicle side storage unit] The vehicle-side storage unit 13 stores a criterion 131 used when the recognition and judgment unit 142 described later performs at least one of recognition and judgment. The vehicle-side storage unit 13 also stores a trained model 132 used when the recognition and judgment unit 142 performs prediction. The trained model 132 according to this embodiment is constructed by, for example, machine learning. Note that the trained model 132 may be constructed by using various conventionally known techniques other than machine learning. The vehicle-side storage unit 13 also stores a three-dimensional map 133 of the driving range of the autonomous vehicle 1. The vehicle-side storage unit 13 according to this embodiment is configured by at least one of a semiconductor memory and a hard disk.

[0014] [Vehicle side control unit] The vehicle-side control unit 14 includes an acquisition unit 141, a recognition determination unit 142, a vehicle-side transmission control unit 143, a vehicle-side reception control unit 144, and a driving control unit 145. Each of the units 141 to 145 of the vehicle-side control unit 14 generates a log every time it operates.

[0015] (Acquisition Department) The acquisition unit 141 acquires data generated by the sensor 11. That is, the autonomously driving vehicle 1 acquires event information indicating an event detected by the sensor 11. The acquisition unit 141 according to this embodiment repeatedly acquires the event information every time a predetermined time elapses.

[0016] (Recognition Judgment Department) The recognition judgment unit 142 recognizes at least one of events inside and outside the vehicle based on the event information (from the sensor 11) acquired by the acquisition unit 141 using a learned model constructed by machine learning, and makes a judgment based on the recognition result.

[0017] The recognition determination unit 142 determines whether or not an abnormality has occurred based on the logs generated by the units 141 to 145. If it is determined that no abnormality has occurred, the recognition determination unit 142 controls the vehicle-side communication unit 12 to transmit the log to the remote monitoring device 2. On the other hand, if it is determined that an abnormality has occurred, the recognition determination unit 142 attaches a problem flag to the log. Then, the recognition determination unit 142 controls the vehicle-side communication unit 12 to transmit the log with the problem flag attached to it to the remote monitoring device 2.

[0018] The recognition determination unit 142 also performs a determination based on the metrics. The metrics include, for example, at least one of the traveling speed, acceleration, accelerator pedal displacement speed, brake pedal displacement speed, and steering wheel rotation speed of the autonomous driving vehicle 1. The determination includes at least one of a determination as to whether or not a predetermined threshold is exceeded, and a determination as to whether or not the operation corresponding to the metrics shows an abnormal tendency. Here, if it is determined that the metrics do not exceed the threshold or do not show an abnormal tendency, the recognition determination unit 142 controls the vehicle-side communication unit 12 to transmit the metrics to the remote monitoring device 2. On the other hand, if it is determined that the metrics exceed the threshold or show an abnormal tendency, the recognition determination unit 142 attaches a problem flag to the metrics. Then, the recognition determination unit 142 controls the vehicle-side communication unit 12 to transmit the metrics with the problem flag attached to the remote monitoring device 2.

[0019] Furthermore, the recognition determination unit 142 recognizes the contents of the user's utterance based on the voice data generated by the microphone, and determines the importance of the contents of the utterance. Here, if it is determined that the importance of the contents of the utterance is relatively low, the recognition determination unit 142 controls the vehicle-side communication unit 12 to transmit the contents of the utterance to the remote monitoring device 2. On the other hand, if it is determined that the importance of the contents of the utterance is relatively high, the recognition determination unit 142 attaches a problem flag to the contents of the utterance. Then, the recognition determination unit 142 controls the vehicle-side communication unit 12 to transmit the contents of the utterance with the problem flag attached to the remote monitoring device 2.

[0020] Further, the recognition determination unit 142 recognizes an object present inside or outside the vehicle based on the image data or the detection result of the Radar, or based on the detection result of the Lidar and the three-dimensional map. Then, the recognition determination unit 142 predicts how the recognized object will move based on the recognition result. Then, the recognition determination unit 142 judges whether or not a danger will occur between the object and the autonomous driving vehicle 1 based on the prediction result. Here, if it is determined that no danger will occur, the recognition determination unit 142 controls the vehicle-side communication unit 12 to transmit the utterance content to the remote monitoring device 2. On the other hand, if it is determined that a danger will occur, the recognition determination unit 142 attaches a problem flag to the judgment result. Then, the recognition determination unit 142 controls the vehicle-side communication unit 12 to transmit the judgment result with the problem flag attached to the remote monitoring device 2.

[0021] The recognition and determination unit 142 performs the above various determinations multiple times. Therefore, when the recognition and determination unit 142 performs a certain determination and the driving control unit 145 controls the autonomous vehicle 1 based on the determination, an event may occur as a result of the determination. For example, when the driving control unit 145 applies the brakes suddenly based on a determination that a pedestrian has jumped out in front, an event may occur in which a person riding in the autonomous vehicle 1 falls over. That is, the autonomous vehicle 1 may also acquire event information indicating an event that has occurred as a result of a determination made by the autonomous vehicle 1. Note that the autonomous vehicle 1 may acquire at least one of event information indicating an event detected by the sensor 11 and event information indicating an event that has occurred as a result of a determination made by the autonomous vehicle 1.

[0022] (Vehicle side transmission control unit) The vehicle-side transmission control unit 143 controls the vehicle-side communication unit 12 to transmit the event information acquired by the sensor 11 to the remote monitoring device 2. In addition, the vehicle-side transmission control unit 143 controls the vehicle-side communication unit 12 to transmit the recognition result, prediction result, and judgment result by the recognition and judgment unit 142 to the remote monitoring device 2. In addition, the vehicle-side transmission control unit 143 controls the vehicle-side communication unit 12 to transmit an alert to the remote monitoring device 2 when the recognition and judgment unit 142 becomes unable to make a judgment.

[0023] (Vehicle side reception control unit) The vehicle-side reception control unit 144 controls the vehicle-side communication unit 12 to receive a stop instruction signal from the remote monitoring device 2. The stop instruction signal is a control signal that instructs a transition to a minimum risk state. The vehicle-side reception control unit 144 controls the vehicle-side communication unit 12 to receive a resume instruction signal from the remote monitoring device 2. The resume instruction signal is a control signal that instructs the autonomously driving vehicle 1 to resume driving. The transmission of a control signal by the remote monitoring device 2 can occur suddenly. For this reason, the vehicle-side reception control unit 144 in this embodiment controls the vehicle-side communication unit 12 so that it can receive a stop instruction signal or a resume instruction signal at any time.

[0024] (Operation control unit) The driving control unit 145 controls at least one of acceleration / deceleration and steering based on the judgment result by the recognition judgment unit 142. In addition, when the recognition judgment unit 142 is unable to make a judgment based on the recognition result of the event, the driving control unit 145 transitions the autonomous vehicle 1 to a minimum risk state. The minimum risk state refers to a state in which the autonomous vehicle 1 is safely stopped while taking into consideration the maximum possible safety of the occupants of the autonomous vehicle 1 and road users other than the autonomous vehicle 1. For example, this corresponds to turning on the hazard lights, warning the following vehicles, and temporarily stopping on the road shoulder at a low speed. Cases in which the recognition judgment unit 142 is unable to make a judgment include cases in which unexpected behavior has occurred at least either inside or outside the vehicle. The driving control unit 145 according to this embodiment also transitions the autonomous vehicle 1 to the minimum risk state when the device-side communication unit 21 receives a stop instruction signal from the remote monitoring device 2. Furthermore, when the vehicle-side communication unit 12 receives a restart instruction signal while the autonomous vehicle 1 is in the minimum risk state, the driving control unit 145 determines whether the recognition determination unit 142 can make a decision based on the recognition result of at least one of the events inside and outside the vehicle. If it is determined that the recognition determination unit 142 can make a decision, the driving control unit 145 cancels the minimum risk state. Then, the driving control unit 145 resumes control of at least one of acceleration / deceleration and steering.

[0025] 2. First embodiment of remote monitoring device Next, a first embodiment of the remote monitoring device 2, which is an aspect of the present invention, will be described in detail.

[0026] [composition] As shown in FIG. 2, the remote monitoring device 2 includes a device-side communication unit 21 (communication unit), a device-side storage unit 22, an output unit 23, an operation unit 24, and a device-side control unit 25.

[0027] [Device side communication section] The device-side communication unit 21 communicates with at least one autonomously driving vehicle 1. The device-side communication unit 21 according to this embodiment is configured with, for example, a wireless communication module. That is, the device-side communication unit 21 according to this embodiment communicates wirelessly with the autonomously driving vehicle 1.

[0028] [Device side storage unit] A database 221 is constructed in the device-side storage unit 22. The database 221 is capable of storing event information, recognition results, prediction results, and judgment results received from the autonomous driving vehicle 1. The device-side storage unit 22 according to this embodiment is configured with at least one of a semiconductor memory and a hard disk. The database 221 may be divided into at least two depending on the information to be stored.

[0029] [Output section] The output unit 23 outputs content according to the control of the device-side control unit 25. The output unit 23 according to this embodiment is at least one display device that displays an image according to the control of the device-side control unit 25. The output unit 23 may be at least one of a communication module that transmits various data and various signals to another device (such as a display device), a terminal for connecting to another device, and a speaker that outputs sound.

[0030] [Operation unit] The operation unit 24 is configured to be operable by a user. The operation unit 24 according to this embodiment is configured with at least one of a keyboard, a pointing device (for example, a mouse), and a touch panel.

[0031] [Device-side control section] The device-side control unit 25 includes a device-side reception control unit 251 (reception control unit), a device-side transmission control unit 252 (transmission control unit), a resumption determination unit 253, an output control unit 254, an extraction unit 255, and a determination unit 256.

[0032] (Device side reception control unit) The device-side reception control unit 251 controls the device-side communication unit 21 to periodically receive, from the autonomous vehicle 1, event information acquired by the autonomous vehicle 1. The device-side reception control unit 251 also controls the device-side communication unit 21 to receive, from the autonomous vehicle 1, the recognition result, prediction result, and judgment result by the recognition judgment unit 142 of the autonomous vehicle 1. The device-side reception control unit 251 also controls the device-side communication unit 21 to receive an alert from the autonomous vehicle 1. The transmission of an alert by the autonomous vehicle 1 can occur suddenly. For this reason, the reception control unit 251 according to this embodiment controls the device-side communication unit 21 to receive an alert at any time.

[0033] (Extraction part) When the device-side communication unit 21 receives a plurality of types of images from one autonomous vehicle 1 and the device-side communication unit 21 receives specific event information from the autonomous vehicle 1, the extraction unit 255 automatically extracts related images from the plurality of types of images. The specific event information is event information indicating an event (hereinafter, an incident) that may cause a transition to a minimum risk state. The incidents include an incident of the autonomous vehicle 1 itself, an incident inside the autonomous vehicle 1, and an incident outside the autonomous vehicle 1. An incident of the autonomous vehicle 1 itself mainly refers to a deviation of the autonomous vehicle 1 from a traffic rule. An incident inside the autonomous vehicle 1 mainly refers to a fall of a person inside the vehicle. An incident outside the autonomous vehicle 1 includes an abnormal approach to an object, an abnormal traffic condition, and the like. The object includes other vehicles, people, obstacles, and the like. The traffic condition includes lane regulation, flooding, snowfall, localized heavy rain, congestion, transmission delay, and the like. The related image is an image that is most related to the incident that has occurred or the minimum risk state associated with the occurrence of the incident. Specifically, the extraction unit 255 uses a trained model (extraction model) that has undergone machine learning to learn the relationship between images inside and outside the vehicle and people or obstacles, for example, to extract images of a person who has fallen, images of people or obstacles that are present in front of the autonomously driven vehicle 1 within a predetermined distance, etc. When the extraction unit 255 receives multiple types of images from multiple autonomously driven vehicles 1, the extraction unit 255 similarly extracts related images when an incident occurs in at least one of the multiple autonomously driven vehicles 1 or the autonomously driven vehicle 1 transitions to a minimum risk state.

[0034] (Decision Section) When the device-side communication unit 21 receives specific event information from multiple autonomously driving vehicles 1, the determination unit 256 determines priority information for each autonomously driving vehicle 1. Priority information is information that indicates the level of priority of an event to be addressed. When the determination unit 256 receives specific event information such as the one listed below, it determines the priority information for the specific event information as follows. - The system analyzes the vehicle's position, speed and lane using images from outside the vehicle and the gyro sensor, and if it receives information indicating that the vehicle is deviating from its lane, it will be given a "high" priority. If the gyro sensor detects sudden braking, but receives information from images outside the vehicle indicating that the cause of the sudden braking was a bicycle suddenly appearing in front of the vehicle, the system will determine the priority as "low." - When images inside the vehicle are analyzed and information is received indicating that a person has fallen, the system gives the vehicle a high priority. If an image outside the vehicle indicates that a pedestrian is on the roadway while the vehicle is traveling, but information is received indicating that no action to avoid the pedestrian (such as reducing speed or changing steering angle) has been detected by autonomous vehicle 1, the priority is determined to be "high." If images outside the vehicle detect that lane restrictions have been implemented due to a traffic accident and information is received that autonomous vehicle 1 is transitioning to a minimum risk state, the priority is determined to be "medium."

[0035] (Output control section) The output control unit 254 controls the output unit 23 to display various monitoring screens S1 to S3 when a predetermined display start condition is met. The display start conditions include turning on the power of the remote monitoring device 2, launching an application, performing a predetermined display start operation (for example, clicking or touching an icon corresponding to a screen), etc. As a result, the output unit 23 displays the first monitoring screen S1, the second monitoring screen S2, the third monitoring screen S3, the operation record screen S4, the alert record screen, or the setting screen.

[0036] ·First monitoring screen The first monitoring screen S1 is a screen for mainly checking various driving conditions of the monitored autonomous vehicle 1. The first monitoring screen S1 according to this embodiment has, for example, as shown in Fig. 3, a vehicle information display area R11, a weather forecast information display area R12, a road traffic information display area R13, and an icon display area R14.

[0037] The vehicle information display area R11 is an area in which individual information of each autonomous vehicle 1 to be monitored is displayed in a list. The individual information includes the vehicle name (for example, vehicle number n, etc.), the name of the on-board emergency response occupant, the number of passengers, the driving mode, the incident occurrence status, etc. The driving mode is information indicating whether the current autonomous vehicle 1 is driving automatically, driving manually, or is out of operation. The output control unit 254 according to this embodiment displays the individual information of the autonomous vehicle 1 in operation together at the top of the vehicle information display area R11, and displays the individual information of the autonomous vehicle 1 that is out of operation together at the bottom of the vehicle information display area R11. In addition, the output control unit 254 according to this embodiment displays the individual information of the autonomous vehicle 1 in operation in a predetermined order. The predetermined order includes the order of the names of the autonomous vehicles 1, the order of registration of the autonomous vehicles 1 in a database, etc. In addition to the above, the individual information may also include the route name (if the autonomous vehicle 1 is a route bus), the number of people who need help getting on and off, and the remaining battery charge (if the autonomous vehicle 1 is an electric vehicle).

[0038] The weather forecast information display area R12 is an area where meteorological information for the area in which the autonomous vehicle 1 being monitored is traveling is displayed. The weather forecast information display area R12 according to this embodiment has a date and area display area R121, a current information display area R122, and one or more future information display areas R123. The output control unit 254 causes today's date and area to be displayed in the date and area display area R121. The output control unit 254 also causes meteorological information for a time period including the current time (9:00 in FIG. 3) to be displayed in the current information display area R122. The meteorological information includes temperature, weather, precipitation, wind direction, wind speed, and the like. The output control unit 254 also causes meteorological information for a time period after the time period including the current time (two time periods of 10:00 and 11:00 in FIG. 3) to be displayed in one or more future information display areas R123. It should be noted that the first monitoring screen S1 does not have to have the weather forecast information display area R12 (the weather forecast information display area R12 may be displayed on another screen).

[0039] The road traffic information display area R13 is an area in which individual road traffic information in the area in which the monitored autonomous vehicle 1 travels is displayed in a list. The road traffic information includes the time and place of occurrence of an impact event, the contents of the impact event, the cause of the impact event, and the like. The impact event is an event that may affect the travel of the autonomous vehicle 1. The impact event includes, for example, traffic regulations, congestion, and the like. The traffic regulations include road closures, lane regulations, speed limits, and the like. The cause of the impact event includes, for example, vehicle breakdowns on the road, traffic accidents, road construction, and the like. When there are multiple pieces of road traffic information, the output control unit 254 according to this embodiment displays the multiple pieces of road traffic information in a list in chronological order in the road traffic information display area R13. Specifically, the impact events are displayed in order of the most recent occurrence times. Note that the first monitoring screen S1 does not need to have the road traffic information display area R13 (the road traffic information display area R13 may be displayed on another screen).

[0040] The icon display area R14 is an area where various icons are displayed. The various icons include a plurality of types of screen switching icons. Some of the screen switching icons (the top four icons in FIG. 3) correspond to various monitoring screens S1 to S3 and an operation record screen S4. When any of these screen switching icons is operated (clicked, touched, etc.), the output control unit 254 displays the monitoring screen S1 to S3 or the operation record screen S4 corresponding to the operated screen switching icon on the output unit 23. When another screen switching icon among the plurality of types of screen switching icons is operated (clicked, touched, etc.), the output control unit 254 displays the alert record screen or the setting screen corresponding to the operated screen switching icon on the output unit 23. The alert record screen is a screen that displays alerts received in the past. The setting screen is a screen for performing various settings of the remote monitoring device 2.

[0041] When the equipment-side communication unit 21 receives an alert (the autonomously driven vehicle 1 has transitioned to a minimum-risk state), the output control unit 254 causes the output unit 23 to display transition information together with information or an image corresponding to the autonomously driven vehicle 1 in which the incident occurred. The transition information is information indicating that the autonomously driven vehicle has transitioned to a minimum-risk state. The output control unit 254 according to this embodiment causes a mark M to be superimposed as transition information on the individual information in the vehicle information display area R11 of the first monitoring screen S1.

[0042] Furthermore, the output control unit 254 according to this embodiment controls the output unit 23 when the determination unit 256 determines the priority information (when an incident occurs in multiple autonomously driven vehicles 1). As a result, the output unit 23 displays the priority information together with information or images corresponding to each autonomously driven vehicle 1. The output control unit 254 according to this embodiment indicates the priority information with the color of a mark indicating transition information. The relationship between the priority order and the color is as follows: - High priority means migration information is displayed in red. - For "medium" priority, migration information is displayed in yellow. - For low priority, migration information is displayed in green.

[0043] This allows the monitor to quickly determine the priority of the actual response. The display manner of the priority information is not limited to color. That is, the priority information may be displayed, for example, by letters, numbers, or the size of transition information, or may be displayed by highlighting individual information, information about incidents, tabs, etc.

[0044] ·Second monitoring screen The second monitoring screen S2 is a screen for mainly providing an overview of the status of the autonomously driven vehicle 1 in operation. As shown in Fig. 4, for example, the second monitoring screen S2 according to this embodiment has a map display area R21 and a timeline display area R22 in addition to a current information display area R122 and an icon display area R14 similar to those of the first monitoring screen S1.

[0045] The map display area R21 is an area where the first image I1 is displayed. The first image I1 is an image in which the second image I2 is superimposed on a map. The map may show the entire driving range of the autonomous vehicle 1, or may show only a part of it. The map display area R21 may be configured to adjust the display range of the map in response to an operation by a user. The second image I2 is an image showing the autonomous vehicle 1 in operation. The display position of the second image I2 on the map shows the current location of the autonomous vehicle 1. The second image I2 moves on the map as the autonomous vehicle 1 travels. This allows the monitor to grasp the direction of movement of the autonomous vehicle 1. The second image I2 may be in the shape of an arrow indicating the traveling direction of the autonomous vehicle 1. In this way, even if the second image I2 moves slowly or stops, the traveling direction of the autonomous vehicle 1 can be easily grasped. The output control unit 254 also causes the third image I3 to be superimposed on the first image I1. The third image I3 is an image showing at least a part of a travel route preset in the autonomous vehicle 1. The travel route may be displayed only as a travel trajectory from a departure point to a current location, or only as a planned route from a current location to a destination. The output control unit 254 further superimposes and displays a fourth image I4 on the first image I1. The fourth image I4 is an image showing at least one of a mark indicating a departure point of the autonomous vehicle 1, a mark indicating a destination, and a mark indicating a waypoint. The fourth image I4 may be displayed for all autonomous vehicles 1 for which the second image I2 is displayed on the map, or may be displayed only for a selected part of the autonomous vehicles 1. The output control unit 254 further superimposes and displays a fifth image I5 on the first image I1. The fifth image I5 is an image showing a speech bubble explaining the status of the autonomous vehicle 1, etc. The status includes the vehicle name, whether there is a delay, whether an incident has occurred, etc.

[0046] The timeline display area R22 is an area where information about an incident that occurred in an autonomous vehicle 1 being monitored is displayed. If no incident has occurred in any of the autonomous vehicles 1 being monitored, the output control unit 254 does not display incident information in the timeline display area R22. On the other hand, when the device-side communication unit 21 receives specific event information from an autonomous vehicle 1 being monitored (an incident occurs in an autonomous vehicle 1), the output control unit 254 displays information about the incident in the timeline display area R22. The information about the incident includes the time of occurrence of the incident, the vehicle name, specific event information (contents of the incident), and the circumstances that led to the incident. When the device-side communication unit 21 receives specific event information from multiple autonomous vehicles 1, the output control unit 254 displays information about the multiple incidents (specific event information) in a list in the order in which the incidents occurred in the timeline display area R22. In addition, even when the output control unit 254 receives specific event information multiple times from one autonomous vehicle 1, the output control unit 254 displays information about the multiple incidents (specific event information) in a list in the order in which the incidents occurred in the timeline display area R22. Having such a timeline display region R22 allows the remote monitor to follow the progress of the autonomous vehicle 1 from before the autonomous vehicle 1 transitions to the minimum-risk state. As a result, it is also possible to stop the autonomous vehicle 1 before the autonomous vehicle 1 transitions to the minimum-risk state on its own.

[0047] Furthermore, when the device-side communication unit 21 receives an alert (the autonomous vehicle 1 transitions to the minimum risk state), the output control unit 254 causes the output unit 23 to display transition information together with information or an image corresponding to the autonomous vehicle 1 in which an incident has occurred. The output control unit 254 according to this embodiment causes a mark M to be superimposed as transition information on the information on the incident in the timeline display area R22 of the second monitoring screen S2. Note that the output control unit 254 may be configured to superimpose the transition information on at least one of the second image I2 and the fifth image I5 in addition to or instead of the information on the incident. Furthermore, when the determination unit 256 determines the priority information, the output control unit 254 according to this embodiment displays the priority information together with information or an image corresponding to each autonomous vehicle 1 in the same manner as in the case of the first monitoring screen S1.

[0048] ·Third monitoring screen The third monitoring screen S3 is a screen for checking at least one of the conditions inside and outside the vehicle in real time. The third monitoring screen S3 according to this embodiment has a display switchable area R31 in addition to an icon display area R14 similar to that of the first monitoring screen S1, as shown in Fig. 5 for example. The display switchable area R31 can be switched to an entire display area R31a or an individual display area R31b (see Fig. 6).

[0049] The whole display area R31a is an area in which at least one of the images of the inside and outside of the vehicle captured by the image sensor of all the autonomous vehicles 1 in operation is displayed in a list. The whole display area R31a has an image display area R311 and a response required alert display area R312. As shown in FIG. 5, for example, the output control unit 254 according to the present embodiment displays at least one of an image of the inside of the vehicle of each autonomous vehicle 1 in operation, an image of the outside of the vehicle (front, rear, side), a reduced first image I1, and a bird's-eye view showing the autonomous vehicle 1 and its surroundings viewed from above in the image display area R311. The output control unit 254 may be configured to display an image generated by an image generation AI based on event information in the image display area R311.

[0050] When an incident occurs in an autonomous vehicle 1 in operation or when the autonomous vehicle 1 transitions to a minimum risk state, the output control unit 254 displays information about the incident in the response required alert display area R312. The information about the incident displayed here is the same as that displayed in the timeline display area R22 of the second monitoring screen S2. In addition, at this time, the output control unit 254 highlights the image of the autonomous vehicle 1 in which an incident has occurred or which has transitioned to a minimum risk state, among the images of each of the multiple autonomous vehicles 1. When multiple images of the autonomous vehicle 1 in which an incident has occurred or which has transitioned to a minimum risk state are displayed, the output control unit 254 highlights the related image extracted by the extraction unit 255. The highlighting mode includes blinking the related image, displaying a frame around the related image, enlarging the related image, and the like. While the incident or the minimum risk state is occurring, the output control unit 254 continues to highlight the related image. This allows the monitor to easily identify the image necessary to confirm the process that led to the occurrence of the incident or the minimum risk state. In addition, the output control unit 254 may be configured to highlight a person, obstacle, etc. that caused an incident or a minimum risk state in the related image, rather than highlighting the entire related image.

[0051] Furthermore, when information about an incident is displayed in the response required alert display area R312, the output control unit 254 displays a response start button B3a in the vicinity (for example, directly below) of the displayed information about the incident. When the response start button B3a is operated (touched, pointed with a cursor, etc.), the output control unit 254 according to this embodiment displays a confirmation screen (not shown) on the third monitoring screen S3. The confirmation screen is provided with a check box for top priority response and a response start button. When the response start button on this confirmation screen is operated, the remote monitoring device 2 is put into a state where it is ready to respond to the incident. When the remote monitoring device 2 is ready to respond, the corresponding individual display area R31b (described later) is displayed, etc., so that the monitor can confirm the state. At that time, when the top priority response check box is checked, the remote monitoring device 2 is ready to respond to the incident with top priority. At this time, the display of the response start button B3a is changed from "response start" to "response in progress". Note that the output control unit 254 may be configured to display a priority order designation button at the display location when the response start button B3a is operated. In this case, when the priority order designation button is operated, the output control unit 254 requests the designation of the actual priority order. Methods of designation include displaying selection buttons or a pull-down menu for the priority order (high / medium / low), displaying an input field for the priority order, and the like. The monitor can designate the priority order to be actually dealt with by operating the selection buttons or pull-down menu, or by inputting information into the input field. Furthermore, when the response start button B3a is operated, the output control unit 254 may put the remote monitoring device 2 into a state in which it can deal with the incident without displaying a confirmation screen or the priority order designation button.

[0052] When an image of the autonomous vehicle 1 in which the incident has occurred (for example, a first screen transition button B3b provided in the heading at the top of the image) is operated after an operation (click, touch, etc.) is performed on a display field (start response button B3a) for information on the incident displayed in the response required alert display area R312, the output control unit 254 switches the display switchable area R31 from the entire display area R31a to the individual display area R31b. As shown in FIG. 6, the individual display area R31b is an area in which at least one of images of the inside and outside of the vehicle captured by an image sensor of one autonomous vehicle 1 in operation is displayed in a list. The individual display areas R31b are provided in the same number as the number of autonomous vehicles 1 in operation, and each individual display area R31b corresponds to each of the multiple autonomous vehicles 1 in operation.

[0053] The individual display area R31b has an image display area R313, a status display area R314, a log display area R315, and a first operation part display area R316.

[0054] The image display area R313 is an area in which an image relating to the corresponding autonomously driven vehicle 1 is displayed among the multiple images displayed in the overall display area R31a. In addition to the image relating to the corresponding autonomously driven vehicle 1, the image display area R313 according to this embodiment displays a map of the area in which the autonomously driven vehicle 1 is traveling. The map may be the same as that displayed in the map display area R21 of the second monitoring screen S2, or may be limited to the current location of the autonomously driven vehicle 1 and its surroundings.

[0055] The status display area R314 is an area that displays the current status of the autonomous vehicle 1. The status of the autonomous vehicle 1 includes the driving type (autonomous driving / manual driving), remaining battery power, vehicle name, number of passengers, driving speed, steering rotation angle, accelerator strength, brake strength, etc.

[0056] The log display area R315 is an area in which events based on logs flagged as problems received from the corresponding autonomously driven vehicle 1 are listed.

[0057] The first operation unit display area R316 is an area where various instruction buttons indicating the contents of instructions to the autonomous vehicle 1 are displayed. The first operation unit display area R316 according to this embodiment has an emergency stop button B3d, a restart button B3e, a return button B3f, and a route change button B3g. When the emergency stop button B3d is operated, the device-side communication unit 21 transmits a stop instruction signal to the autonomous vehicle 1. When the restart button B3e is operated, the device-side communication unit 21 transmits a restart instruction signal to the autonomous vehicle 1. When the return button B3f is operated, the device-side communication unit 21 transmits a return instruction signal to the autonomous vehicle 1 instructing the autonomous vehicle 1 to return to the garage. When the route change button B3g is operated, the device-side communication unit 21 transmits a change instruction signal to the autonomous vehicle 1 instructing the autonomous vehicle 1 to change the travel route.

[0058] - Operation record screen The operation record screen S4 is a screen that displays the past operation records of the autonomous vehicle 1 and allows the monitor to understand the history and background of the incident. The operation record screen S4 according to this embodiment has a past image display area R41, a past state display area R42, a response-required alert display area R43, a past log display area R44, a transition display area R45, and a second operation unit display area R46, as shown in FIG. 7, for example. The output control unit 254 switches the display screen to the operation record screen S4 when an icon in the icon display area R14 is operated, or when a display field (a response start button B3a) for information on the incident displayed in the response-required alert display area R312 of the third monitoring screen S3 is operated (clicked, touched, etc.) and then the display field (for example, a second screen transition button B3c in the display field) is further operated.

[0059] The past image display area R41 is an area in which past images relating to the corresponding autonomous vehicle 1 (at least any of past images of the interior of the autonomous vehicle 1, past images of the outside of the vehicle (front, rear, side), a reduced past first image I1, past bird's-eye views, and past AI images) are displayed.

[0060] The past state display area R42 is an area in which the past states of the autonomous vehicle 1 (driving type (autonomous driving / manual driving), remaining battery level, vehicle name, number of passengers, driving speed, steering rotation angle, accelerator strength, braking strength, etc.) are displayed.

[0061] The response required alert display area R43 is an area that displays information regarding an incident that is displayed in the response required alert display area R43 of the overall display area R31a, and displays information regarding an incident of the autonomously driving vehicle 1 that corresponds to the individual display area R31b currently being displayed.

[0062] The past log display area R44 is an area that displays a list of events based on logs that have been previously received from the corresponding autonomously driven vehicle 1 and that have been flagged as problems.

[0063] The transition display area R45 is an area that displays the transition of the state of the autonomously driving vehicle 1 from the past to the present. States whose transitions are displayed in the transition display area R45 include speed, acceleration, remaining battery power, etc. The transition display area R45 according to this embodiment displays the transition of the state in the form of a graph with time on the horizontal axis and state on the vertical axis.

[0064] The second operation unit display area R46 is an area where various operation units for accepting various operations by the observer are displayed. The second operation unit display area R46 has a current display button B4a, one or more past display buttons B4b, a seek bar B4c, a play / pause button B4d, a fast-rewind button B4e, a fast-forward button B4f, a skip button B4g, and state selection check boxes B4h to B4j. When the current display button B4a is operated, the output control unit 254 displays a current (real-time) image obtained from the autonomously driven vehicle 1 in the image display area R313. By checking this current image, the observer can determine the priority of the response.

[0065] When the past display button B4b is operated, the output control unit 254 displays past images stored in the device-side storage unit 22 in the image display area R313. At this time, if any of the state selection check boxes B4h to B4j is checked, the output control unit 254 displays in the transition display area R45 the transition of the state of the autonomous vehicle 1 at the time of capturing the image, which corresponds to the checked state. At this time, if an incident has occurred in the corresponding autonomous vehicle 1, a mark M indicating that an incident has occurred and a vertical line are displayed in the transition display area R45 at a location corresponding to the time of the incident. At this time, when the play / pause button B4d is operated, the output control unit 254 plays the image. When the play / pause button B4d is operated again, the output control unit 254 pauses the image. When the fast-rewind button B4e or the fast-forward button B4f is operated during image playback, the output control unit 254 fast-forwards or fast-rewinds the image being played. Furthermore, when the skip button B4g is operated during image playback, the output control unit 254 plays the image from a frame a predetermined time before or after the currently displayed frame in the image. This allows the monitor to check past images (the process that led to the occurrence of the incident or minimum risk state) by operating the second operation unit display area R46, for example, when it is difficult to determine the priority of the response just by checking the current image, or when there is a discrepancy between the content displayed in the log display area R315 and the current image.

[0066] By referring to the various information displayed on the third monitoring screen S3, the remote monitor can easily know how the autonomous vehicle 1 has transitioned to the minimum risk state, for example, as shown in FIG. 8 (FIG. 9), in which the autonomous vehicle 1 suddenly brakes due to a pedestrian suddenly jumping out in front of the autonomous vehicle 1 (the pedestrian falls), and the person riding in the autonomous vehicle 1 falls (the person's fallen state continues for a while) due to the sudden braking. In addition, by referring to the various information displayed on the event confirmation screen S7, the remote monitor can easily confirm whether or not the person riding in the autonomous vehicle 1 (the person who had fallen outside the vehicle) has been rescued. As a result, the remote monitor can easily determine whether or not to resume autonomous driving.

[0067] Furthermore, when the device-side communication unit 21 receives an alert (the autonomous vehicle 1 transitions to the minimum risk state), the output control unit 254 causes the output unit 23 to display transition information together with information or an image corresponding to the autonomous vehicle 1 in which the incident has occurred. The output control unit 254 according to this embodiment causes a mark M to be superimposed as transition information on a tab in the display switchable area R31 of the third monitoring screen S3. Note that, in addition to or instead of the tab, the output control unit 254 may be configured to superimpose the transition information on at least one of the images of the third monitoring screen S3. Furthermore, when the determination unit 256 determines the priority information, the output control unit 254 according to this embodiment displays the priority information together with information or an image corresponding to each autonomous vehicle 1 in the same manner as in the case of the first monitoring screen S1.

[0068] (Device side transmission control section) The device-side transmission control unit 252 controls the device-side communication unit 21 to transmit a stop instruction signal to the autonomous vehicle 1 in motion. The device-side transmission control unit 252 according to the present embodiment controls the device-side communication unit 21 to transmit a stop instruction signal based on the user operating the emergency stop button B3d via the operation unit 24. The device-side transmission control unit 252 according to the present embodiment controls the device-side communication unit 21 to transmit a stop instruction signal based on the user operating the emergency stop button B3d via the operation unit 24. The device-side transmission control unit 252 also controls the device-side communication unit 21 to transmit a resume instruction signal to the autonomous vehicle 1 in the minimum risk state. The device-side transmission control unit 252 according to the present embodiment controls the device-side communication unit 21 to transmit a resume instruction signal based on the user operating the operation resume button B3e via the operation unit 24. This allows the vehicle to immediately resume operation if there is no problem after checking the event information.

[0069] (Resumption Judgment Department) The resumption determination unit 253 determines whether the autonomous vehicle 1 has resumed driving after the device-side communication unit 21 transmits a resumption instruction signal. The resumption determination unit 253 according to the present embodiment determines whether the autonomous vehicle 1 has resumed driving based on the event information received by the device-side communication unit 21. Specifically, the resumption determination unit 253 refers to image data and determines whether the autonomous vehicle 1 is traveling. The resumption determination unit 253 may be configured to determine whether the autonomous vehicle 1 has resumed driving based on the processing result of the autonomous vehicle 1. The processing result of the autonomous vehicle 1 includes, for example, that the recognition determination unit 142 has resumed judgment based on the recognition result, that at least one of acceleration / deceleration or steering is being controlled, and the like. As a result, if the sensor 11 of the autonomous vehicle 1 detects the reason why driving cannot be resumed but the supervisor overlooks it, the resumption of driving can be reliably prevented and the autonomous vehicle 1 can be brought into a safe state.

[0070] When the resumption determination unit 253 determines that the autonomous vehicle 1 has not resumed driving, the output control unit 254 according to this embodiment controls the output unit 23. This causes the output unit 23 to output a message to that effect (that the autonomous vehicle 1 has not resumed driving). This makes it possible to deal with a case where, for example, there remains a problem that the sensor 11 of the autonomous vehicle 1 was not able to pick up (that occurred in the blind spot of the camera), and the autonomous vehicle 1 is unable to resume driving, but the supervisor is unaware of this.

[0071] [Effects] In addition, in autonomous driving at level 4 or higher, when an autonomous vehicle enters a minimum risk state, a remote monitor will decide whether or not to resume autonomous driving. In addition, from the viewpoint of business feasibility in autonomous driving services, one remote monitor is required to monitor multiple autonomous vehicles. However, in this case, the remote monitor cannot keep monitoring one autonomous vehicle all the time. For this reason, when an autonomous vehicle enters a minimum risk state, the remote monitor cannot immediately grasp the cause or background of the minimum risk state, and must refer to a large amount of information accumulated up to that point in order to release the minimum risk state. As a result, it may take a long time for the autonomous vehicle to resume driving, or the remote monitor may make an incorrect decision and resume driving even when it is not possible to resume driving.

[0072] However, according to the remote monitoring device 2 according to the present embodiment described above, the monitor can understand in a short time why the autonomously driven vehicle 1 is in the minimum risk state by checking the event information output by the output unit 23 within a limited time period, compared to the case of endlessly checking all the event information (the effort of checking is reduced). As a result, it is possible to determine in a short time whether or not the autonomously driven vehicle 1 in the minimum risk state can resume driving. Therefore, the monitor only needs to deal with the corresponding autonomously driven vehicle 1 when the output unit 23 outputs event information. As a result, one remote monitor can monitor a large number of autonomously driven vehicles 1.

[0073] Furthermore, if the autonomous vehicle 1 monitored by the remote monitoring device 2 according to this embodiment has a structure that allows people to ride in a standing position, there is a risk that people standing inside the autonomous vehicle 1 being toppled over, and the autonomous vehicle 1 being monitored is more likely to transition to a minimum risk state than an autonomous vehicle 1 that rides with all people seated. However, with the above configuration, the monitor can remotely monitor multiple autonomous vehicles 1 relatively easily even in such a situation.

[0074] <3. Second embodiment of remote monitoring device> Next, a second embodiment of the remote monitoring device according to the present invention will be described. For the sake of convenience, the same reference numerals will be used to designate components having the same functions as those described in the first embodiment, and the description thereof will not be repeated.

[0075] [composition] As shown in FIG. 2, the remote monitoring device 2A of this embodiment includes an apparatus side communication unit 21 (communication unit), an apparatus side memory unit 22, an output unit 23, and an operation unit 24 similar to those of the remote monitoring device 2 of the first embodiment described above, as well as an apparatus side control unit 25A.

[0076] [Device-side control section] The device side control unit 25A includes an output control unit 254A in addition to a device side reception control unit 251 (reception control unit), a device side transmission control unit 252 (transmission control unit), and a resume judgment unit 253 similar to those of the device side control unit 25 of the first embodiment described above.

[0077] (Output control section) The output control unit 254A controls the output unit 23 to display the monitoring screen S5 when a predetermined display start condition is satisfied. The display start condition includes the power of the remote monitoring device 2 being turned on, the launching of an application, and the performance of a predetermined operation. As a result, the output unit 23 displays the monitoring screen S5. The monitoring screen S5 according to this embodiment has an overall map display area R51, a vehicle display area R52, and an alert display area R53, as shown in FIG. 10, for example. The overall map display area R51 is an area showing a map of the entire driving range of the autonomous vehicle 1 to be monitored. The vehicle display area R52 is an area showing a list of information related to the autonomous vehicle 1 being monitored. The output unit 23 displays the vehicle name, route name, operation status, and battery level (remaining amount) in the vehicle display area R52 as information related to the autonomous vehicle 1. The alert display area R53 is an area showing a list of information related to alerts previously received from the autonomous vehicle 1 being monitored. The output unit 23 displays, in the alert display area R53, information about the alert, such as the occurrence time, the vehicle name, the type of the alert, and the response status. The response status is information indicating the response status of the remote monitor to the alert. Note that the monitoring screen S5 does not necessarily have to have the alert display area R53.

[0078] In addition, the output control unit 254A controls the output unit 23 to display the alert detection screen S6 when the device-side communication unit 21 receives an alert (the autonomous vehicle 1 transitions to the minimum risk state). As a result, the output unit 23 displays the alert detection screen S6. The alert detection screen S6 according to this embodiment has a vehicle name display area R61, an alert summary display area R62, a status display area R63, a partial map display area R64, and an image display area R65, as shown in FIG. 11, for example. The vehicle name display area R61 is an area that shows information that identifies the autonomous vehicle 1 that has sent the alert. The alert summary display area R62 is an area that shows the reason why the autonomous vehicle 1 has sent the alert. The output unit 23 displays the reason based on the judgment result acquired by the output control unit 254A from the database 221 in this alert summary display area R62. The status display area R63 is an area that shows the current status of the autonomous vehicle 1. In the status display area R63, the output unit 23 displays at least one of the remaining battery level, driving speed, accelerator strength, brake strength, and steering as the status of the autonomously driven vehicle 1. The partial map display area R64 is an area that shows the current position of the autonomously driven vehicle 1 that has sent the alert and a map of its vicinity. The image display area R65 is an area that displays at least one of an image of the interior and an image of the exterior of the vehicle captured by the autonomously driven vehicle 1. The image display area R65 according to this embodiment is capable of displaying both an image of the interior of the vehicle and an image of the exterior of the vehicle.

[0079] In addition, the output control unit 254A controls the output unit 23 to display the event confirmation screen S7 when the first switching button B6a of the alert detection screen S6 is operated. As a result, the output unit 23 displays the event confirmation screen S7. The event confirmation screen S7 according to this embodiment has, as shown in FIG. 12, a vehicle name display area R61, a status display area R63, a partial map display area R64, and an image display area R65 similar to those of the alert detection screen S6, as well as an event information display area R71, a message information display area R72, and an operation panel display area R73. The event information display area R71 is a display area for event information of the target autonomous vehicle. The message information display area R72 is an area that displays a message (e.g., auxiliary information related to instructions) to the monitor regarding the autonomous vehicle 1 to be monitored. The operation panel display area R73 is an area that has operation buttons (e.g., a play button, a pause button, a fast-rewind button, a seek bar, etc.) related to the playback of the video displayed in the image display area R65. When the user operates the operation button, the output control unit 254A displays a moving image based on the event information in the image display region R65 in a mode according to the operation content.

[0080] The output unit 23, under the control of the output control unit 254A, displays (outputs) in the event information display area R71 the event information received by the device-side communication unit 21 between a first time when the autonomous vehicle 1 transitions to the minimum risk state and a second time a predetermined time before the first time. The first time may be the time when the driving control unit 145 of the autonomous vehicle 1 decides to transition to the minimum risk state, or the time when the autonomous vehicle 1 actually transitions to the minimum risk state and the stop of the autonomous vehicle 1 is completed, or any time between them. The interval between the first time and the second time is, for example, about 5 to 10 minutes. The output unit 23 also outputs the time when the event occurred and the content of the event under the control of the output control unit 254A. The output unit 23 according to this embodiment outputs the time when the event occurred and the content of the event in a table format. This allows the monitor to understand in chronological order what happened to the autonomous vehicle 1 that transitioned to the minimum risk state. The output unit 23 displays the number of pieces of event information (e.g., three) within the range that can be displayed at one time in the event information display area R71. This allows the observer to understand at a glance why the autonomously driven vehicle 1 is in the minimum risk state, without having to scroll through the columns of event information or move to the next screen.

[0081] Moreover, the output unit 23 displays the event information and the first image I1 under the control of the output control unit 254A. The first image I1 is an image (e.g., an arrow) in which a second image I2 indicating the current location or traveling direction of the autonomous vehicle 1 is superimposed on a map, as shown in FIG. 12, for example. This allows the monitor to know in more detail the event that occurred to the autonomous vehicle 1 that has transitioned to the minimum risk state. In particular, the monitor can more quickly grasp the current location and traveling direction of the autonomous vehicle 1, and can more quickly determine whether or not to start traveling. In addition, the first image I1 according to this embodiment may be an image in which a third image I3 indicating the traveling trajectory or planned course of the autonomous vehicle 1 is further superimposed on the map. This allows the monitor to more quickly grasp the current location, traveling direction, traveling trajectory, or planned course of the autonomous vehicle 1, and can more quickly determine whether or not to start traveling. The output unit 23 also displays the first image I1, the second image I2 and the third image I3 in the overall map display area R51 of the monitoring screen S5, the partial map display area R64 of the alert detection screen S6, and the partial map display area R64 of the vehicle instruction screen S8 described later.

[0082] By referring to the various information displayed on this event confirmation screen S7, the remote monitor can easily know the circumstances that led to the transition to the minimum risk state, for example, as shown in FIG. 8 (FIG. 9), in which the autonomous vehicle 1 suddenly brakes due to a pedestrian suddenly jumping out in front of the autonomous vehicle 1 (the pedestrian falls), and the person riding in the autonomous vehicle 1 falls (the person's fallen state continues for a while) due to the sudden braking. In addition, by referring to the various information displayed on the event confirmation screen S7, the remote monitor can easily confirm whether or not the person riding in the autonomous vehicle 1 (the person who fell outside the vehicle) has been rescued. As a result, the remote monitor can easily determine whether or not to resume autonomous driving.

[0083] In addition, the output control unit 254A controls the output unit 23 to display the vehicle instruction screen S8 when the second switching button B6b on the alert detection screen S6 or the event confirmation screen S7 is operated. As a result, the output unit 23 displays the vehicle instruction screen S8 as shown in FIG. 13, for example. The vehicle instruction screen S8 according to this embodiment has a vehicle name display area R61, a status display area R63, a partial map display area R64, and an image display area R65 similar to those of the alert detection screen S6, as well as an instruction button display area R81 and a message display area R82. The instruction button display area R81 is an area having various instruction buttons (an emergency stop button B3d, a resume operation button B3e, a return to warehouse button B3f, and a route change button B3g similar to those of the third monitoring screen S3) that indicate the contents of instructions to the autonomously driven vehicle 1. When the user operates any of the instruction buttons B3d to B3g, the device-side transmission control unit 252 controls the device-side communication unit 21 to transmit a control signal instructing an operation corresponding to the operated instruction button B3d to B3g to the autonomously driven vehicle 1. When the emergency stop button B3d is operated, the device-side communication unit 21 transmits a stop instruction signal to the autonomously driven vehicle 1.

[0084] In addition, the output unit 23 may be configured, under the control of the output control unit 254, to output a predetermined number of the most recent event information out of the multiple event information received by the device side communication unit 21, including the event information received when the autonomous vehicle 1 transitioned to a minimum risk state.

[0085] <4.Other> The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims.

[0086] For example, if the output unit 23 includes a speaker, the output unit 23 may be configured to output sound as well as the event information under the control of the output control unit 254. This allows the monitor to know in more detail the event that has occurred in the autonomously driven vehicle 1 that has transitioned to the minimum risk state.

[0087] Moreover, the functions of the remote monitoring device 2 can be realized by a program for making a computer function as the device 2 and for making a computer function as each control block of the device 2 (particularly each unit included in the device-side control unit 25). In this case, the device 2 includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The functions described in the above embodiment are realized by executing the program using the control device and the storage device. The program may be recorded on one or more computer-readable recording media, not on a temporary basis. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0088] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0089] Furthermore, the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0090] According to each aspect of the present invention described above, by providing effects that are suitable for safe and sustainable public transportation such as buses, it can contribute to the achievement of Goal 11 of the Sustainable Development Goals (SDGs), "Sustainable cities and communities".

[0091] 〔summary〕 The remote monitoring device of aspect 1 of the present invention is configured to include a communication unit that communicates with at least one autonomously driving vehicle to acquire event information indicating at least one of an event detected by a sensor equipped in the autonomously driving vehicle and an event that occurs as a result of a judgment made by the autonomously driving vehicle, a reception control unit that controls the communication unit to periodically receive the event information acquired by the autonomously driving vehicle from the autonomously driving vehicle, and an output unit that, when the autonomously driving vehicle transitions to a minimum risk state, outputs the event information received by the communication unit between a first time when the autonomously driving vehicle transitioned to the minimum risk state and a second time a predetermined time before the first time.

[0092] A remote monitoring device according to a second aspect of the present invention may be configured in the above-mentioned first aspect so that the output unit outputs a time when the event occurred and a content of the event.

[0093] A remote monitoring device according to aspect 3 of the present invention may be configured in the above-mentioned aspect 1 or 2 such that the output unit displays a screen having a display area for the event information, and displays a number of pieces of event information within the range that can be displayed at one time in the display area.

[0094] A remote monitoring device according to aspect 4 of the present invention is any one of aspects 1 to 3 above, wherein the sensor is an image sensor that captures images of at least one of the inside and outside of the autonomously driving vehicle, and the output unit is configured to display the event information as well as a first image.

[0095] A remote monitoring device according to aspect 5 of the present invention is any one of aspects 1 to 4 above, wherein the sensor is a microphone that captures audio from at least one of the inside and outside of the autonomously driving vehicle, and the output unit is configured to output the event information as well as output audio.

[0096] The remote monitoring device of aspect 6 of the present invention, in any one of aspects 1 to 5 above, may further include a transmission control unit that controls the communication unit to send a resume instruction signal to the autonomous vehicle in the minimum risk state, instructing the autonomous vehicle to resume driving, based on a specified operation performed by a user.

[0097] A remote monitoring device according to aspect 7 of the present invention may be configured in the above-mentioned aspect 6 such that after the communication unit transmits the resume instruction signal, a resumption judgment unit judges whether the autonomous vehicle has resumed driving, and the output unit outputs a signal to that effect if the resumption judgment unit judges that the autonomous vehicle has not resumed driving.

[0098] A remote monitoring device according to aspect 8 of the present invention, in any one of aspects 1 to 7 above, may be configured such that the autonomously driven vehicle to be monitored has a structure capable of carrying a person in a standing position.

[0099] The remote monitoring device of aspect 9 of the present invention is configured to include a communication unit that communicates with at least one autonomous vehicle to acquire event information indicating at least one of an event detected by a sensor equipped in the autonomous vehicle and an event that occurs as a result of a judgment made by the autonomous vehicle, a reception control unit that controls the communication unit to periodically receive the event information acquired by the autonomous vehicle from the autonomous vehicle, and an output unit that, when the autonomous vehicle transitions to a minimum risk state, outputs a predetermined number of the most recent event information from the multiple event information received by the communication unit, including the event information received when the autonomous vehicle transitioned to the minimum risk state.

[0100] A remote monitoring device according to aspect 10 of the present invention, in aspect 1 or 9 above, may be configured so that when the communication unit receives specific event information from multiple autonomous vehicles, each of the specific event information indicating an event that may cause a transition to the minimum risk state, the output unit displays a list of the multiple specific event information in the order in which the events that may cause the transition occurred.

[0101] The remote monitoring device of aspect 11 of the present invention, in aspect 1 or 9 above, may further include a determination unit that, when the communication unit receives specific event information from each of the multiple autonomous vehicles indicating an event that may cause a transition to the minimum risk state, determines priority information indicating the level of priority for action for each autonomous vehicle, and the output unit may be configured to display the priority information together with information or an image corresponding to each autonomous vehicle.

[0102] A remote monitoring device according to aspect 12 of the present invention may be configured in the above-mentioned aspect 7 such that the resumption determination unit determines whether or not the autonomous vehicle has resumed driving based on the event information received by the communication unit. [Explanation of symbols]

[0103] 1. Self-driving vehicles 11 Sensors 12 Vehicle side communication unit 13 Vehicle side memory unit 131 Standards 132 trained models 133 3D Map 14 Vehicle side control unit 141 Acquisition Department 142 Recognition judgment unit 143 Vehicle side transmission control unit 144 Vehicle side receiving control unit 145 Operation control unit 2. 2A remote monitoring device 21 Device side communication unit (communication unit) 22 Device side storage unit 221 Database 23 Output section 24 Control section 25, 25A Equipment side control unit 251 Device side reception control unit (reception control unit) 252 Equipment side transmission control section 253 Resumption Judgment Department 254, 254A Output control section 255 Extraction part 256 Decision Section

Claims

1. a communication unit that communicates with at least one of the autonomously driven vehicles and continuously acquires event information indicating at least one of an event detected by a sensor equipped in the autonomously driven vehicle and an event that occurs as a result of a determination made by the autonomously driven vehicle; and an output unit that outputs, when the autonomously driven vehicle transitions to a stopped state, the event information received by the communication unit between a first time, which is a predetermined time between a time when the transition is determined and a time when the transition is completed, and a second time before the time when the transition is determined; a determination unit that, when the communication unit receives specific event information indicating an incident that is an event that may cause the autonomously driven vehicles to transition to the stopped state, determines priority information indicating a level of priority for handling for each of the autonomously driven vehicles; and an extraction unit that extracts, when the communication unit receives multiple types of camera images from one autonomously driven vehicle and the communication unit receives specific event information from the autonomously driven vehicle, a relevant image that is most relevant to the incident that has occurred or the stopped state that accompanies the incident from the multiple types of camera images; Equipped with The output unit displaying the priority information together with information corresponding to each autonomous vehicle and camera images corresponding to each autonomous vehicle; When multiple camera images of an autonomous vehicle that has experienced an incident or transitioned to a minimum risk state are displayed, the relevant image is highlighted. A remote monitoring device characterized by:

2. A communication unit that communicates with at least one of the autonomous vehicles and continuously acquires event information indicating at least one of events detected by a sensor equipped in the autonomous vehicle and events that occur as a result of a decision made by the autonomous vehicle; and an output unit that, when the autonomously driven vehicle transitions to a stopped state, outputs the event information received by the communication unit between a first time, which is a time when the transition is determined, and a second time before the first time; a determination unit that, when the communication unit receives specific event information indicating an incident that is an event that may cause the autonomously driven vehicles to transition to the stopped state, determines priority information indicating a level of priority for handling for each of the autonomously driven vehicles; and an extraction unit that extracts, when the communication unit receives multiple types of camera images from one autonomously driven vehicle and the communication unit receives specific event information from the autonomously driven vehicle, a relevant image that is most relevant to the incident that has occurred or the stopped state that accompanies the incident from the multiple types of camera images; Equipped with The output unit displaying the priority information together with information corresponding to each autonomous vehicle and camera images corresponding to each autonomous vehicle; When multiple camera images of an autonomous vehicle that has experienced an incident or transitioned to a minimum risk state are displayed, the relevant image is highlighted. A remote monitoring device characterized by:

3. A communication unit that communicates with at least one of the autonomous vehicles and continuously acquires event information indicating at least one of an event detected by a sensor equipped in the autonomous vehicle and an event that occurs as a result of a decision made by the autonomous vehicle; and an output unit that outputs the event information received by the communication unit between a first time when the transition of the autonomously driven vehicle to a stopped state is completed and a second time before the time when the transition is determined; a determination unit that, when the communication unit receives specific event information indicating an incident that is an event that may cause the autonomously driven vehicles to transition to the stopped state, determines priority information indicating a level of priority for handling for each of the autonomously driven vehicles; and an extraction unit that extracts, when the communication unit receives multiple types of camera images from one autonomously driven vehicle and the communication unit receives specific event information from the autonomously driven vehicle, a relevant image that is most relevant to the incident that has occurred or the stopped state that accompanies the incident from the multiple types of camera images; Equipped with The output unit displaying the priority information together with information corresponding to each autonomous vehicle and camera images corresponding to each autonomous vehicle; When multiple camera images of an autonomous vehicle that has experienced an incident or transitioned to a minimum risk state are displayed, the relevant image is highlighted. A remote monitoring device characterized by:

4. a communication unit that communicates with at least one of the autonomously driven vehicles and continuously acquires event information indicating at least one of an event detected by a sensor equipped in the autonomously driven vehicle and an event that occurs as a result of a determination made by the autonomously driven vehicle; and an output unit that outputs, when the autonomously driven vehicle transitions to a stopped state, the event information received by the communication unit between a first time, which is a predetermined time between a time when the transition is determined and a time when the transition is completed, and a second time before the time when the transition is determined; a determination unit that, when the communication unit receives specific event information indicating an incident that is an event that may cause the autonomously driven vehicles to transition to the stopped state, determines priority information indicating a level of priority for handling for each of the autonomously driven vehicles; and Equipped with The output unit an overall display area in which camera images from a plurality of vehicles are displayed; an individual display area in which the camera image of the selected vehicle is displayed; It is possible to switch between the following: In the entire display area, the priority information is displayed together with information corresponding to each autonomously driven vehicle and a camera image; When a response start button is operated while the entire display area is displayed, the display area transitions to an individual display area of ​​the vehicle where the incident occurred. A remote monitoring device characterized by:

5. A communication unit that communicates with at least one of the autonomous vehicles and continuously acquires event information indicating at least one of an event detected by a sensor equipped in the autonomous vehicle and an event that occurs as a result of a decision made by the autonomous vehicle; and an output unit that, when the autonomously driven vehicle transitions to a stopped state, outputs the event information received by the communication unit between a first time, which is a time when the transition is determined, and a second time before the first time; a determination unit that, when the communication unit receives specific event information indicating an incident that is an event that may cause the autonomously driven vehicles to transition to the stopped state, determines priority information indicating a level of priority for handling for each of the autonomously driven vehicles; and Equipped with The output unit an overall display area in which camera images from a plurality of vehicles are displayed; an individual display area in which the camera image of the selected vehicle is displayed; It is possible to switch between the following: In the entire display area, the priority information is displayed together with information corresponding to each autonomously driven vehicle and a camera image; When a response start button is operated while the entire display area is displayed, the display area transitions to an individual display area of ​​the vehicle where the incident occurred. A remote monitoring device characterized by:

6. A communication unit that communicates with at least one of the autonomous vehicles and continuously acquires event information indicating at least one of an event detected by a sensor equipped in the autonomous vehicle and an event that occurs as a result of a decision made by the autonomous vehicle; and an output unit that outputs the event information received by the communication unit between a first time when the transition of the autonomously driven vehicle to a stopped state is completed and a second time before the time when the transition is determined; a determination unit that, when the communication unit receives specific event information indicating an incident that is an event that may cause the autonomously driven vehicles to transition to the stopped state, determines priority information indicating a level of priority for handling for each of the autonomously driven vehicles; and Equipped with The output unit an overall display area in which camera images from a plurality of vehicles are displayed; an individual display area in which the camera image of the selected vehicle is displayed; It is possible to switch between the following: In the entire display area, the priority information is displayed together with information corresponding to each autonomously driven vehicle and a camera image; When a response start button is operated while the entire display area is displayed, the display area transitions to an individual display area of ​​the vehicle where the incident occurred. A remote monitoring device characterized by:

7. a communication unit that communicates with at least one of the autonomously driven vehicles and continuously acquires event information indicating at least one of an event detected by a sensor equipped in the autonomously driven vehicle and an event that occurs as a result of a determination made by the autonomously driven vehicle; and an output unit that, when the autonomously driven vehicle transitions to a stopped state, outputs a predetermined number of most recent pieces of event information from the plurality of pieces of event information received by the communication unit, the most recent pieces of event information including the event information received when the autonomously driven vehicle transitions to the stopped state; a determination unit that, when the communication unit receives specific event information indicating an incident that is an event that may cause the autonomously driven vehicles to transition to the stopped state, determines priority information indicating a level of priority for handling for each of the autonomously driven vehicles; and an extraction unit that extracts, when the communication unit receives multiple types of camera images from one autonomously driven vehicle and the communication unit receives specific event information from the autonomously driven vehicle, a relevant image that is most relevant to the incident that has occurred or the stopped state that accompanies the incident from the multiple types of camera images; Equipped with The output unit displaying the priority information together with information corresponding to each autonomous vehicle and camera images corresponding to each autonomous vehicle; When multiple camera images of an autonomous vehicle that has experienced an incident or transitioned to a stopped state are displayed, the relevant image is highlighted. A remote monitoring device characterized by:

8. a communication unit that communicates with at least one of the autonomously driven vehicles and continuously acquires event information indicating at least one of an event detected by a sensor equipped in the autonomously driven vehicle and an event that occurs as a result of a determination made by the autonomously driven vehicle; and an output unit that, when the autonomously driven vehicle transitions to a stopped state, outputs a predetermined number of most recent pieces of event information from the plurality of pieces of event information received by the communication unit, the most recent pieces of event information including the event information received when the autonomously driven vehicle transitions to the stopped state; a determination unit that, when the communication unit receives specific event information indicating an incident that is an event that may cause the autonomously driven vehicles to transition to the stopped state, determines priority information indicating a level of priority for handling for each of the autonomously driven vehicles; and Equipped with The output unit an overall display area in which camera images from a plurality of vehicles are displayed; an individual display area in which the camera image of the selected vehicle is displayed; It is possible to switch between the following: In the entire display area, the priority information is displayed together with information corresponding to each autonomously driven vehicle and a camera image; When a response start button is operated while the entire display area is displayed, the display area transitions to an individual display area of ​​the vehicle where the incident occurred. A remote monitoring device characterized by:

Images