Information processing method, information processing program, and information processing device

Abstract

An information processing device 10 according to an embodiment comprises a determination unit 15B and an output control unit 15C. The output control unit 15C determines whether or not an event requiring remote control has occurred in a mobile body 20. If it is determined that the event has occurred, the output control unit 15C outputs remote control request information relating to the remote control to an operator terminal 30 at a timing based on situation information relating to the mobile body 20.

Description

Information processing method, information processing program, and information processing device

[0001] An embodiment of the present invention relates to an information processing method, an information processing program, and an information processing device.

[0002] In recent years, autonomously moving objects such as self-driving vehicles have become known. In such autonomous driving technology, when an event requiring remote control occurs in the moving object, a remote control request is immediately notified to an operator.

[0003] However, there are cases where the situation of the moving object at the time of the occurrence of the event makes it difficult for the operator to immediately start remote control, or where it is difficult to return to autonomous movement even if remote control is started. If a remote control request is notified to the operator despite such a situation, the operator may be tied up for an unnecessarily long time, which may make it difficult to efficiently manage the operator's resources.

[0004] Patent No. 7335350

[0005] The present invention has been made in view of the above, and an object of the present invention is to provide an information processing method, an information processing program, and an information processing device that can efficiently manage operator resources.

[0006] The information processing method of the embodiment is an information processing method executed by an information processing device, and includes a judgment step of judging whether an event requiring remote control has occurred in a mobile object, and an output control step of outputting remote control request information regarding remote control to an operator terminal at a timing based on situation information regarding the mobile object if it is determined that the event has occurred.

[0007] FIG. 1 is an explanatory diagram of an example of an information processing system according to an embodiment. FIG. 2 is a functional block diagram of an example of an information processing system according to an embodiment. FIG. 3A is a schematic diagram of an example of a display screen. FIG. 3B is a schematic diagram of an example of a display screen. FIG. 3C is a schematic diagram of an example of a display screen. FIG. 4 is a sequence diagram showing an example of the flow of information processing executed in the information processing system. FIG. 5 is a flowchart showing an example of the flow of information processing executed by an information processing device. FIG. 6 is a hardware configuration diagram.

[0008] Hereinafter, embodiments of an information processing method, an information processing program, and an information processing device will be described in detail with reference to the accompanying drawings.

[0009] FIG. 1 is an explanatory diagram of an example of an information processing system 1 according to this embodiment.

[0010] The information processing system 1 includes an information processing device 10, one or more mobile objects 20, and one or more operator terminals 30. The mobile objects 20 and the information processing device 10 are communicatively connected via a network NW or the like. The information processing device 10 and the operator terminal 30 are communicatively connected via a network NW or the like.

[0011] The network NW that communicatively connects the information processing device 10 and the mobile object 20 and the network NW that communicatively connects the information processing device 10 and the operator terminal 30 may be the same network or different networks.

[0012] The information processing device 10 is an information processing device that executes management of the mobile object 20 and the operator terminal 30. The information processing device 10 is a dedicated or general-purpose computer.

[0013] The mobile object 20 is an autonomously movable object that is subject to remote control by an operator P. The information processing system 1 includes one or more mobile objects 20.

[0014] The remote control refers to a task related to remote control of the mobile object 20 by the operator P. The remote control includes at least one of remote operation of the mobile object 20 and remote monitoring of the mobile object 20.

[0015] The moving body 20 is, for example, an autonomously moving vehicle, a robot having a movement function and moving autonomously, etc. An autonomously moving vehicle may be called a self-driving vehicle. In this embodiment, a form in which the moving body 20 is a vehicle capable of autonomous movement will be described as an example.

[0016] The operator terminal 30 is an information processing device for remotely controlling the mobile body 20. The operator terminal 30 is operated by an operator P who is in charge of remotely controlling the mobile body 20. The information processing system 1 includes one or more operator terminals 30.

[0017] FIG. 2 is a functional block diagram of an example of the information processing system 1 according to the present embodiment.

[0018] The mobile object 20 includes a communication unit 21, an input unit 22, an output unit 23, a sensor 24, an imaging unit 25, a drive unit 26, a storage unit 27, and a control unit 28. The communication unit 21, the input unit 22, the output unit 23, the sensor 24, the imaging unit 25, the drive unit 26, the storage unit 27, and the control unit 28 are communicatively connected via a bus or the like.

[0019] The communication unit 21 communicates with the information processing device 10 via a network NW or the like. The input unit 22 accepts various operations by a user. The output unit 23 outputs various information. The output unit 23 may be a display that displays various information, a speaker that outputs various sounds, a light that lights up or flashes, etc.

[0020] The sensor 24 is a sensor that detects the state of the moving body 20. The sensor 24 is, for example, an internal sensor and an external sensor.

[0021] The internal sensors are sensors that obtain observation information about the internal environment of the mobile object 20. The observation information includes information such as the position, speed, acceleration, steering angle, steering direction, accelerator depression angle, and vibration of the mobile object 20. Specifically, the internal sensors include position sensors (Global Navigation Satellite System (GNSS), Global Positioning System (GPS)). The internal sensors also include, for example, an inertial measurement unit (IMU), an acceleration sensor, a speed sensor, a speed sensor such as a gyro sensor, a rotary encoder, a steering angle sensor, a steering direction sensor, an accelerator depression angle detection sensor, and a vibration sensor.

[0022] The external sensor is a sensor that obtains distance information, which indicates the distance from the mobile object 20 to an object in the vicinity of the mobile object 20.

[0023] The external sensor is, for example, a distance sensor. The distance sensor is, for example, a millimeter-wave radar, a laser sensor, a distance image sensor, etc. The laser sensor is, for example, a two-dimensional LiDAR (Laser Imaging Detection and Ranging) sensor or a three-dimensional LiDAR sensor installed parallel to a horizontal plane.

[0024] The sensor 24 sequentially detects the state of the moving body 20 in chronological order, and sequentially outputs the detection results to the control unit 28 .

[0025] The image capturing unit 25 is a camera that captures images of the surroundings of the moving body 20 and obtains captured image data. The captured image data will be referred to as captured image in the following description. The moving body 20 may be provided with a plurality of image capturing units 25 each with a different image capturing direction.

[0026] The driving unit 26 is a dedicated control device for devices mounted on the mobile body 20 and driving the devices related to the movement of the mobile body 20. The driving unit 26 is, for example, an ECU (Electronic Control Unit) that controls the engine, a motor driver that controls the motor, etc. The driving unit 26 controls the driving of the devices under the control of the control unit 28.

[0027] The storage unit 27 stores various types of data. The storage unit 27 is, for example, a random access memory (RAM), a semiconductor memory element such as a flash memory, a hard disk, an optical disk, etc. The storage unit 27 may be a storage device provided outside the mobile object 20. The storage unit 27 may also be a storage medium that stores or temporarily stores programs and various types of information downloaded via a local area network (LAN), the Internet, etc.

[0028] The control unit 28 is a computer that executes information processing in the mobile object 20 .

[0029] The control unit 28 includes a communication control unit 28A, a drive control unit 28B, and a mobile object information transmission unit 28C. The communication control unit 28A, the drive control unit 28B, and the mobile object information transmission unit 28C are realized by one or more processors. For example, each of the above units may be realized by causing a processor such as a CPU (Central Processing Unit) to execute a program, i.e., by software.

[0030] Each of the above units may be realized by a processor such as a dedicated IC, i.e., hardware. Each of the above units may also be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0031] The communication control unit 28A controls communication with the information processing device 10 .

[0032] The drive control unit 28B controls the drive unit 26 to autonomously move the mobile body 20. The drive control unit 28B determines the situation around the mobile body 20 based on the detection results of the sensor 24, etc., and controls the accelerator amount, brake amount, steering angle, etc., to control the drive unit 26 so that the mobile body 20 moves autonomously.

[0033] Furthermore, the drive control unit 28B executes remotely controlled driving based on remote control information received from the operator terminal 30 via the information processing device 10. Remotely controlled driving means driving in accordance with remote control by the operator terminal 30. During remotely controlled driving, the drive control unit 28B controls the accelerator amount, brake amount, steering angle, etc. based on the remote control information received from the operator terminal 30, and controls the drive unit 26 so that the mobile body 20 drives in accordance with operation instructions from the operator terminal 30.

[0034] The mobile object information transmitting unit 28C transmits the mobile object information to the information processing device 10 via the communication control unit 28A and the communication unit 21.

[0035] The mobile object information is information related to the traveling environment of the mobile object 20. The mobile object information includes, for example, at least one of the detection results of the sensor 24, the video captured by the image capturing unit 25, and the analysis results of the traveling environment.

[0036] The mobile object information transmission unit 28C derives an analysis result by analyzing, using a known image processing method, the captured image obtained by the image capturing unit 25. The mobile object information transmission unit 28C also derives an analysis result of the traveling environment of the mobile object 20 by analyzing, using a known analysis method, at least one of the captured image, the detection result of the sensor 24, and the map data.

[0037] The analysis results include information about the environment around the mobile body 20. For example, the analysis results include information representing obstacles present around the mobile body 20, the travel area of ​​the mobile body 20, the status of objects around the mobile body 20, etc. Information representing obstacles present around the mobile body 20 includes information representing the type of obstacle, the size of the obstacle, the relative position and relative distance of the obstacle to the mobile body 20, etc. Information representing the travel area of ​​the mobile body 20 includes information representing the road conditions on which the mobile body 20 is traveling, etc. Road conditions include the type of travel route, information about traffic signs such as crosswalks and traffic lights present around the travel route, the congestion status of the road, etc. The status of objects around the mobile body 20 includes information representing the number of objects present around the mobile body 20, the type of object, information about whether the object is in contact with the mobile body 20, the relative position and relative distance of the object to the mobile body 20, etc. An object is, for example, a person. The type of object includes, for example, the gender and age of the person. Furthermore, the type of object may further include behavioral information that indicates the behavior of a person, such as a parent holding a child, taking a photo, viewing an advertisement, etc. The age of a person may be estimated, for example, from the captured video using a known method.

[0038] The mobile object information transmitting unit 28C transmits the mobile object information to the information processing device 10 via the communication control unit 28A.

[0039] Next, the operator terminal 30 will be described.

[0040] The operator terminal 30 includes a communication unit 31, an input unit 32, an output unit 33, a storage unit 34, and a control unit 35. The communication unit 31, the input unit 32, the output unit 33, the storage unit 34, and the control unit 35 are communicatively connected via a bus or the like.

[0041] The communication unit 31 communicates with the information processing device 10 via a network NW or the like. The input unit 32 accepts various operations by an operator P. The output unit 33 outputs various information. The output unit 33 is a display that displays various information, a speaker that outputs various sounds, etc.

[0042] The storage unit 34 stores various types of data. The storage unit 34 is, for example, a semiconductor memory element such as a RAM or a flash memory, a hard disk, an optical disk, etc. The storage unit 34 may be a storage device provided outside the operator terminal 30. The storage unit 34 may also be a storage medium that stores or temporarily stores programs and various types of information downloaded via a LAN, the Internet, etc.

[0043] The control unit 35 is a computer that executes information processing in the operator terminal 30 .

[0044] The control unit 35 includes a communication control unit 35A, an output control unit 35B, and a remote control control unit 35C. The communication control unit 35A, the output control unit 35B, and the remote control control unit 35C are implemented by one or more processors. For example, each of the above units may be implemented by causing a processor such as a CPU to execute a program, i.e., by software.

[0045] Each of the above units may be realized by a processor such as a dedicated IC, i.e., hardware. Each of the above units may also be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0046] Furthermore, at least one of the above-described units included in the control unit 35 may be mounted on an external information processing device communicably connected to the operator terminal 30 via a network NW or the like.

[0047] The communication control unit 35A controls communication with the information processing device 10 .

[0048] The output control unit 35B outputs remote control request information received from the information processing device 10 via the communication control unit 35A and the communication unit 31 to the output unit 33. The remote control request information is information requesting at least one of remote operation and remote monitoring from the operator P. The information requesting remote operation from the operator P is information representing a request for remote operation of the mobile object 20. The remote control request information may include at least one of mobile object information received from the mobile object 20 and status information, which will be described later.

[0049] For example, the output control unit 35B displays the remote control request information received from the information processing device 10 on the display of the output unit 33. If the remote control request information includes video captured by the information processing device 10, the output control unit 35B displays the video captured by the mobile object 20 on the output unit 33. By visually checking the remote control request information displayed on the output unit 33, the operator P can remotely monitor and remotely control the mobile object 20, such as remotely operating it.

[0050] The remote operation control unit 35C remotely controls the moving body 20 in response to the operation of the input unit 32 by the operator P. For example, the remote operation control unit 35C transmits remote operation information corresponding to an operation instruction for the moving body 20 input in response to the operation of the input unit 32 by the operator P, together with the moving body ID (identification) of the moving body 20, to the information processing device 10. The moving body ID is identification information for the moving body 20. The information processing device 10 transmits the remote operation information to the moving body 20 identified by the moving body ID. The moving body 20 controls the drive unit 26 based on the remote operation information, thereby moving the moving body 20 in response to the remote operation by the operator P. Therefore, the moving body 20 is remotely controlled by the operator P who operates the operator terminal 30.

[0051] Next, the information processing device 10 will be described.

[0052] The information processing device 10 includes a communication unit 11, an input unit 12, an output unit 13, a storage unit 14, and a control unit 15. The communication unit 11, the input unit 12, the output unit 13, the storage unit 14, and the control unit 15 are communicatively connected via a bus or the like.

[0053] The communication unit 11 communicates with the mobile object 20 and the operator terminal 30 via a network NW or the like. The input unit 12 accepts various operations by a user. The output unit 13 outputs various information. The output unit 13 is a display that displays various information, a speaker that outputs various sounds, etc.

[0054] The storage unit 14 stores various types of data. The storage unit 14 is, for example, a semiconductor memory element such as a RAM or a flash memory, a hard disk, an optical disk, or the like. The storage unit 14 may be a storage device provided outside the information processing device 10. The storage unit 14 may also be a storage medium that stores or temporarily stores programs and various types of information downloaded via a LAN, the Internet, or the like. At least a portion of the information stored in the storage unit 14 may also be stored in a storage device such as an external server device communicably connected to the information processing device 10.

[0055] The control unit 15 is a computer that executes information processing in the information processing device 10 .

[0056] The control unit 15 includes an information receiving unit 15A, a determination unit 15B, and an output control unit 15C. The information receiving unit 15A, the determination unit 15B, and the output control unit 15C are realized by one or more processors. For example, each of the above units may be realized by causing a processor such as a CPU to execute a program, i.e., by software.

[0057] Each of the above units may be realized by a processor such as a dedicated IC, i.e., hardware. Each of the above units may also be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units. Furthermore, at least one of the above units included in the information processing device 10 may be mounted on an external information processing device communicatively connected to the information processing device 10 via a network NW or the like.

[0058] The information receiving unit 15A receives information from the mobile object 20 and the operator terminal 30 via the communication unit 11 .

[0059] For example, when the information receiving unit 15A receives remote control information from the operator terminal 30, it transmits the remote control information to the mobile object 20 identified by the mobile object ID included in the remote control information.

[0060] The information receiving unit 15A also receives mobile object information from the mobile object 20. The information receiving unit 15A also receives communication status information indicating the communication status with the mobile object 20. For example, the information receiving unit 15A receives communication status information indicating the communication status with the mobile object 20 via the network NW from the communication unit 11, an external communication status monitoring device, or the like. The communication status information is expressed, for example, by a communication speed expressed in bits per second (bps). Furthermore, when the mobile object 20 and the information processing device 10 communicate using multiple communication paths, the communication status information may be information indicating the communication speed for each communication path. A communication path may be referred to as a channel. Furthermore, a deterioration in the communication status indicates, for example, that the communication speed indicated by the communication status information is below a predetermined threshold.

[0061] The determination unit 15B determines whether an event requiring remote control has occurred in the moving object 20.

[0062] An event is an event that requires remote control. Specifically, an event represents a situation in which the mobile object 20 requires at least one of remote operation, remote assistance, and remote monitoring.

[0063] A situation requiring remote control refers to a situation requiring remote control of the mobile body 20 by the operator P. A situation requiring remote assistance refers to a situation requiring auxiliary remote control of the mobile body 20 by the operator P. Auxiliary remote operation means that the mobile body 20 autonomously executes some of the processes related to its travel and executes other processes related to its travel in response to remote operation information from the operator P. A situation requiring remote monitoring refers to a situation requiring monitoring of the mobile body 20 by the operator P.

[0064] Specifically, an event requiring remote control represents, for example, a situation in which the mobile object 20 cannot move autonomously.

[0065] The judgment unit 15B analyzes the mobile body information and communication status information received from the information receiving unit 15A and determines whether the mobile body 20 is in a situation where it cannot move autonomously, thereby determining whether an event requiring remote control has occurred.

[0066] For example, assume that the mobile body 20 is a mobile body 20 that autonomously moves along a predetermined planned travel route. In this case, for example, the determination unit 15B analyzes the mobile body information and determines that the mobile body 20 has stopped at a point other than a planned stop point indicated on the planned travel route, and determines that an event requiring remote control has occurred.

[0067] Also, assume that the moving body 20 is pre-programmed to stop autonomous movement when it determines that a situation matching a predetermined emergency stop condition has occurred. For example, stop requirement information indicating a condition for stopping the autonomous movement is pre-stored in the memory unit 27 of the moving body 20. Then, when the detection result by the sensor 24 and the captured image acquired by the image capture unit 25 match the stop requirement information, the drive control unit 28B of the moving body 20 determines that a condition matching the emergency stop condition has occurred and controls the drive unit 26 to stop the autonomous movement. In this case, when the determination unit 15B receives an emergency stop signal from the moving body 20 indicating that the moving body 20 has stopped due to the determination that a situation matching the emergency stop condition has occurred, it may determine that an event requiring remote control of the moving body 20 has occurred. The emergency stop condition is, for example, a requirement matching a standby condition, which will be described later.

[0068] The determination unit 15B may be configured such that an operator P operating the operator terminal 30 determines, based on the mobile body information of the mobile body 20, that an event that makes the mobile body 20 unable to move autonomously has occurred. In this case, the operator terminal 30 receives an operation instruction from the operator P via the input unit 32, and transmits a stop instruction signal to the mobile body 20. The drive control unit 28B of the mobile body 20 that receives the stop instruction signal controls the drive unit 26 to stop the movement, and transmits an emergency stop signal to the information processing device 10. Then, when the emergency stop signal is received from the mobile body 20, the determination unit 15B may determine that an event that requires remote control of the mobile body 20 has occurred.

[0069] The timing for transmitting the emergency stop signal from the mobile object information transmitter 28C of the mobile object 20 to the information processing device 10 may be after the mobile object 20 has moved to an area where it can safely stop and stopped. For example, the mobile object 20 is pre-programmed so that, when it detects a deterioration in the communication conditions with the information processing device 10, it continues autonomous movement until it reaches an area where it can safely stop, and stops when it reaches that area. For example, when it detects a deterioration in the communication conditions while traveling through an intersection area, it is pre-programmed so that it continues autonomous movement until it leaves the intersection area, and stops when it reaches that area.

[0070] In this case, the drive control unit 28B of the moving body 20 may transmit the emergency stop signal to the information processing device 10 when it detects a deterioration in the communication conditions of the information processing device 10, or may control the drive unit 26 to stop when it reaches an area beyond the intersection area, and when it reaches and stops in that area, transmit the emergency stop signal to the information processing device 10. Furthermore, the drive control unit 28B of the moving body 20 may detect recovery of the communication conditions after reaching the area, and, if it determines from the captured video and the detection results of the sensor 24, etc. that the conditions around the moving body 20 pose little traffic risk if autonomous movement were to resume, may control the drive unit 26 to return to autonomous movement without transmitting an emergency stop signal to the information processing device 10.

[0071] When it is determined that an event has occurred, the output control unit 15C outputs remote control request information regarding remote control to the operator terminal 30 at a timing based on the status information regarding the moving body 20.

[0072] In detail, when the determination unit 15B determines that an event requiring remote control has occurred in the moving object 20 , the output control unit 15C acquires status information regarding the moving object 20 .

[0073] The status information related to the mobile object 20 is information that indicates the status of the mobile object 20 in which an event has occurred. The status information includes mobile object information and communication status information of the mobile object 20. The output control unit 15C acquires the status information related to the mobile object 20 by acquiring the mobile object information and communication status information with the mobile object 20 received from the mobile object 20 by the information receiving unit 15A.

[0074] The output control unit 15C then determines whether the status information of the moving body 20 satisfies the standby condition. If the status information of the moving body 20 satisfies the standby condition, the output control unit 15C waits for the output of remote control request information for the moving body 20 to the operator terminal 30. If the status information of the moving body 20 does not satisfy the standby condition, the output control unit 15C outputs the remote control request information for the moving body 20 to the operator terminal 30. Through these processes, the output control unit 15C outputs the remote control request information to the operator terminal 30 at a timing based on the status information regarding the moving body 20.

[0075] The standby condition is information that represents a condition for waiting for the output of remote control request information to the operator terminal 30. In other words, the standby condition is a condition for determining to wait for immediate notification of remote control request information for a mobile body 20 in which the occurrence of an event has been determined to be, to the operator terminal 30. The standby condition is also a condition for determining whether the status information of the mobile body 20 is information that represents at least one of the following: it would be inefficient for the operator P to immediately start remote control of the mobile body 20 in which an event has occurred; and it would be difficult to immediately start remote control of the mobile body 20 in which an event has occurred.

[0076] The standby condition may be information indicating a situation in which remote control of the mobile body 20 would be difficult or inefficient if an attempt were made to immediately control the mobile body 20 by remote control. Specifically, the standby condition may represent situation information such as a user being within a predetermined range from the mobile body 20, a user making contact with the mobile body 20, the mobile body 20 being unable to cross a target crossing area ahead in the direction of travel, a situation in which the travel route of the mobile body 20 is difficult to travel, a situation in which an obstacle exists on the travel route of the mobile body 20 and the obstacle cannot be avoided even by remote control, a situation in which a bus or taxi boarding or disembarking point or a facility entrance / exit exists near the travel route of the mobile body 20 and a certain travel risk is estimated even by remote control, a situation in which communication conditions with the mobile body 20 deteriorate but communication is possible, etc.

[0077] The output control unit 15C analyzes the mobile object information included in the situation information to determine whether a user is present within a predetermined range from the mobile object 20. The predetermined range may be set in advance. A user whose presence within the predetermined range is determined may be, for example, a child, but is not limited to a child. A child may be, for example, a person whose estimated age is below a predetermined age. Furthermore, a user whose presence within the predetermined range is determined may be a user whose behavior is represented by specific behavioral information, such as viewing an advertisement, taking a photo, or purchasing a product offered by the mobile object 20. The type of object, such as the behavioral information representing the behavior of these children or people, is included in the analysis results described above. Therefore, the output control unit 15C may read the analysis results included in the mobile object information to identify the type of object and behavioral information and use them in the determination process.

[0078] In this embodiment, a case will be described in which the user is a child as an example. The output control unit 15C identifies the age of a person present around the moving body 20 and the relative distance to the person, for example, using the captured video included in the moving body information, the detection results of the sensor 24, and the analysis results of the driving environment. Through this identification process, the output control unit 15C determines whether a child is present within a predetermined range from the moving body 20. For example, the output control unit 15C identifies a child present around the moving body 20 by analyzing the captured video, and determines whether a child is present within a predetermined range from the moving body 20 by estimating the distance from the information processing device 10 to the child using ranging information of the height of the feet of people around the moving body 20 obtained by a laser sensor such as a two-dimensional LiDAR included in the sensor 24.

[0079] When a user is present within a predetermined range from the moving body 20, even if the moving body 20 is immediately resumed by remote control, there is a possibility that the moving body 20 will immediately stop its autonomous movement due to a user present near the moving body 20. For example, the drive control unit 28B of the moving body 20 may control the moving body 20 to stop its autonomous movement when it detects contact with an object or the presence of an object within a predetermined range by analyzing at least one of the detection results by the sensor 24 and the captured video captured by the capture unit 25. Furthermore, when a user such as a child is present within a predetermined range from the moving body 20, the safety of the moving body 20's remotely controlled movement may not be ensured. Therefore, by including situation information indicating that a user is present within a predetermined range from the moving body 20 in the standby condition, it is possible to prevent remote control request information from being output to the operator P in situations where it is difficult or inefficient for the operator P to immediately start remote control.

[0080] The output control unit 15C also analyzes the moving object information included in the situation information to determine whether the user has come into contact with the moving object 20. The user to be determined to have come into contact with the moving object 20 may be, for example, a child, but is not limited to a child. In this embodiment, an example in which the user is a child will be described. The output control unit 15C identifies the age of people present around the moving object 20 and the relative distance to the people, for example, using the captured video included in the moving object information, the detection results of the sensor 24, and the analysis results of the traveling environment. Through this identification process, the output control unit 15C determines whether a child has come into contact with the moving object 20.

[0081] The output control unit 15C also analyzes the mobile object information included in the situation information to determine whether an obstacle exists on the traveling path of the mobile object 20. An obstacle is an object that may make it difficult for the mobile object 20 to continue traveling if it comes into contact with the object, or that may cause some kind of damage or impede traveling to at least one of the mobile object 20 and the object if the mobile object 20 tries to come into contact with or pass over the object without avoiding the object. Specifically, an obstacle may be, for example, a parked vehicle on the traveling path, a vehicle such as a bus or taxi that is temporarily stopped on the traveling path, a vehicle that is parked in a garage or parking area on the traveling path, or some other object or person on the traveling path. If the analysis result included in the mobile object information indicates that an obstacle exists within a predetermined range from the mobile object 20, the output control unit 15C determines that an obstacle exists on the traveling path of the mobile object 20.

[0082] In a situation where an obstacle such as a parked vehicle or a stopped vehicle is present on the travel path, people may get on or off the vehicle, or the vehicle may remain stopped for a certain period of time. In such a case, even if the mobile unit 20 is immediately caused to resume travel by remote control, it may be difficult to move the mobile unit 20 by remote control. Therefore, by including situation information indicating the presence of an obstacle on the travel path of the mobile unit 20 in the standby condition, it is possible to prevent remote control request information from being output to the operator P in a situation where it is difficult or inefficient for the operator P to immediately start remote control.

[0083] The output control unit 15C also analyzes the mobile object information included in the situation information to determine whether the mobile object 20 is unable to cross a target crossing area ahead in the direction of travel. For example, the autonomously moving mobile object 20 may not have the function to determine whether to cross an intersection area without traffic lights, or the autonomously moving mobile object 20 may not be permitted by law to cross an intersection area. Note that the mobile object 20's crossing determination and crossing operation can technically be performed by remote control. The mobile object 20 may also be pre-programmed to stop its autonomous movement just before an intersection area without traffic lights. In this case, the output control unit 15C identifies information indicating the mobile object 20's travel area and the mobile object 20's position included in the mobile object information to determine that the mobile object 20 is stopped just before crossing a target crossing area, such as an intersection. If the output control unit 15C determines that the mobile object 20 is in this state, the output control unit 15C determines that the mobile object 20 is unable to cross the target area ahead in the direction of travel.

[0084] Furthermore, when the output control unit 15C determines by analyzing the mobile object information included in the situation information that the traffic volume of the target crossing area that the mobile object 20 plans to cross is equal to or greater than the traffic volume that the mobile object 20 can cross by autonomous movement, it determines that the situation is such that it is impossible for the mobile object 20 to cross the target crossing area ahead in the traveling direction. Furthermore, when the output control unit 15C determines by analyzing the mobile object information included in the situation information that a traffic light installed in the target crossing area that the mobile object 20 plans to cross is lit in red, indicating a stop sign, it determines that the situation is such that it is impossible for the mobile object 20 to cross the target crossing area ahead in the traveling direction.

[0085] A convoy of other mobile bodies 20 traveling through the target crossing area ahead in the traveling direction of the mobile body 20 may continue for more than a predetermined time, and in such a situation, it would be difficult to immediately start remote control even if remote control request information were immediately output to the operator P. Therefore, by including in the standby condition situation information indicating that it is impossible for the mobile body 20 to cross the target crossing area ahead in the traveling direction, it is possible to prevent remote control request information from being output to the operator P in a situation where it is difficult or inefficient for the operator P to immediately start remote control.

[0086] The output control unit 15C also analyzes the mobile object information included in the situation information to determine whether the driving route of the mobile object 20 is in a difficult driving situation. For example, when the output control unit 15C determines that the driving route of the mobile object 20 is in a difficult driving situation when the output control unit 15C determines that the driving route of the mobile object 20 is in a difficult driving situation when the output control unit 15C determines that the mobile object 20 is traveling on a narrow road and is passing another mobile object 20 or is in a situation where passing another mobile object 20 is difficult, by analyzing the mobile object information included in the situation information. The output control unit 15C may also analyze the mobile object information included in the situation information to determine that the traffic congestion or congestion situation on the driving route of the mobile object 20 is equal to or greater than a situation representing a predetermined degree of traffic congestion or congestion, and determine that the driving route of the mobile object 20 is in a difficult driving situation.

[0087] When operator P, who is a remote operator, performs control to return to autonomous driving from a situation in which autonomous driving is not possible, if the mobile body 20 passes another traffic participant while traveling on a "narrow road" or "near an entrance or exit of a bus or building," there is a risk that the mobile body 20 will not be able to pass through by remote control, or that passing through will disrupt surrounding traffic. In such cases, it is preferable not to output remote control request information to operator P until the situation is resolved. Therefore, by including situation information indicating that the traveling route of the mobile body 20 is in a difficult situation, it is possible to prevent remote control request information from being output to operator P in a situation in which it is difficult or inefficient for operator P to immediately start remote control.

[0088] Furthermore, the output control unit 15C analyzes the communication status information included in the status information to determine whether the communication status with the mobile object 20 has deteriorated. For example, the output control unit 15C determines whether the current communication speed with the mobile object 20 represented by the communication status information included in the status information is less than a threshold for receiving a predetermined amount of information per unit time from the mobile object 20. If the output control unit 15C determines that the communication speed is less than the threshold, it determines that the communication status with the mobile object 20 has deteriorated.

[0089] When the communication conditions between the mobile body 20 and the information processing device 10 deteriorate, the communication becomes unstable, making safe and stable remote control difficult. Therefore, by including in the standby condition status information indicating a deterioration in the communication conditions with the mobile body 20, it is possible to prevent the operator P from remotely controlling the mobile body 20 in an environment where safety or stability has deteriorated.

[0090] As described above, when the moving body 20 determines that a condition matching the emergency stop requirement has occurred, the moving body 20 stops its autonomous movement even at a location other than the planned stopping location. That is, when the drive control unit 28B of the moving body 20 determines that a condition matching the emergency stop requirement has occurred, the drive control unit 28B controls the drive unit 26 to stop the autonomous movement and transmits an emergency stop signal to the information processing device 10. Also, as described above, the emergency stop requirement may be a requirement matching the standby condition.

[0091] In this embodiment, an example will be described in which at least some of the emergency stop requirements and the standby conditions are met. Therefore, when a situation occurs in which at least one of the standby conditions is met, the moving body 20 is unable to move autonomously and stops moving autonomously.

[0092] When the output control unit 15C determines that the status information of the mobile object 20 in which the above-mentioned event occurred satisfies the standby condition, the output control unit 15C waits for the output of remote control request information for the mobile object 20 to the operator terminal 30 .

[0093] As described above, the remote control request information is information requesting at least one of remote operation and remote monitoring from the operator P. The information requesting remote operation from the operator P is information representing a request for remote operation of the mobile body 20.

[0094] The information requesting the operator P to perform remote operation may include information representing at least one of the remote operation and remote monitoring. The output control unit 15C may generate information representing at least one of the remote operation and remote monitoring by acquiring the cause of the event that occurred in the mobile body 20 by analyzing the mobile body information, etc. The cause of the event that occurred in the mobile body 20 is, for example, at least one of the above-mentioned standby conditions used to determine the occurrence of the event.

[0095] The information representing the content of at least one of the remote operation and remote monitoring is, for example, remote operation of the mobile body 20 to avoid obstacles, safety confirmation around the mobile body 20 while stopped, instruction to the mobile body 20 to start moving, safety confirmation around the mobile body 20 after starting moving, audio output instruction, etc. The safety confirmation around the mobile body 20 while stopped, instruction to the mobile body 20 to start moving, and safety confirmation around the mobile body 20 after starting moving are, for example, safety confirmation around the mobile body 20 before crossing an intersection area, instruction to the mobile body 20 to start moving, safety confirmation around the mobile body 20 until the mobile body 20 has completed crossing the intersection area, etc. The audio output instruction is, for example, an instruction to output a predetermined sound from the output unit 23 of the mobile body 20, etc.

[0096] When the status information of the mobile body 20 in which the event has occurred satisfies the standby condition, the output control unit 15C can output the auxiliary information to the operator terminal 30. In other words, when the status information of the mobile body 20 in which the event has occurred satisfies the standby condition, the output control unit 15C can also output the auxiliary information to the operator terminal 30.

[0097] The assistance information is information relating to assistance in remote control of the moving body 20 .

[0098] The auxiliary information includes, for example, at least one of the following: information indicating that the moving body is in a situation where it cannot move autonomously; a predicted waiting time until remote control begins; information indicating the reason why it is difficult to begin remote control immediately; measure information indicating measures to improve the situation where it is difficult to begin remote control; video footage of the area around the moving body 20; and information indicating that a remote response is required but remote response cannot be performed immediately.

[0099] When the status information of the mobile body 20 in which the event has occurred satisfies the standby condition, the output control unit 15C generates information indicating that the mobile body 20 is in a situation in which it is unable to move autonomously. The information indicating that it is in a situation in which it is unable to move autonomously may be information from the status information of the mobile body 20 that satisfies the standby condition.

[0100] The output control unit 15C calculates the predicted waiting time as the predicted time required for the situation information of the mobile object 20 to change from a state where the situation information satisfies the waiting condition to a state where the situation information does not satisfy the waiting condition. For example, assume that the output control unit 15C analyzes the mobile object information included in the situation information and determines that a traffic light installed in a target crossing area where the mobile object 20 is scheduled to cross is lit red, indicating a travel stop, and determines that the mobile object 20 is unable to cross the target crossing area ahead in its traveling direction. In this case, the output control unit 15C estimates the time required for the traffic light to change from red to green, indicating permission to travel, and calculates this time as the predicted waiting time. Alternatively, the output control unit 15C may input the situation information of the mobile object 20 and calculate the predicted time as the predicted waiting time using a learning model that outputs a predicted time required for the situation represented by the situation information to change to a state where the waiting condition is not satisfied. Alternatively, the output control unit 15C may calculate the predicted waiting time from statistics of past waiting times.

[0101] In addition, the output control unit 15C generates information indicating standby conditions that match the status information of the moving body 20 as information indicating the reason why it is difficult to start remote control immediately.

[0102] Furthermore, the output control unit 15C pre-stores, in association with the standby conditions, policy information indicating policies for improving the situation represented by the standby conditions that match the status information of the mobile body 20. Then, the output control unit 15C reads the policy information corresponding to the standby conditions that match the status information of the mobile body 20, and generates policy information for improving the situation in which it is difficult to start remote control.

[0103] In addition, the output control unit 15C acquires the captured video of the mobile body 20 included in the mobile body information received from the mobile body 20, thereby acquiring the captured video of the periphery of the mobile body 20.

[0104] Then, the output control unit 15C outputs to the operator terminal 30 auxiliary information including at least one of the following that has been generated, acquired, or calculated: information indicating that the mobile body 20 is in a situation where it cannot move autonomously; a predicted waiting time until remote control begins; information indicating the reason why it is difficult to begin remote control immediately; measure information indicating measures to improve the situation where it is difficult to begin remote control; captured video of the area around the mobile body 20; and information indicating that a remote response is required but remote response cannot be performed immediately.

[0105] The output control unit 35B of the operator terminal 30 that receives the auxiliary information from the information processing device 10 outputs the received auxiliary information to the output unit 33. Therefore, the information processing device 10 can output the auxiliary information of the mobile body 20, in which the above-mentioned event has occurred and which is waiting to output remote control request information, to the operator terminal 30 before outputting the remote control request information. Therefore, the information processing device 10 can provide the operator P with at least one of the following before providing the remote control request information: information indicating that the mobile body 20 is in a situation where it cannot move autonomously, a predicted waiting time until the start of remote control, information indicating the reason why it is difficult to start remote control immediately, measure information indicating measures to improve the situation where it is difficult to start remote control, captured video of the area around the mobile body 20, and information indicating that a remote response is required but that immediate remote response is not possible.

[0106] Therefore, by checking the auxiliary information output to the output unit 33, the operator P can confirm in advance, before receiving remote control request information from the information processing device 10, at least one of the following: that the mobile body 20 is in a situation where it cannot move autonomously, the predicted waiting time until the start of remote control, the reason why it is difficult to start remote control immediately, measures to improve the situation where it is difficult to start remote control, captured video of the area around the mobile body 20, and the situation where a remote response is necessary but a remote response cannot be made immediately. Furthermore, the operator P can make advance preparations for remote control of the mobile body 20.

[0107] Furthermore, the operator P can consider implementing measures to create a situation in which remote control can be initiated. For example, assume that the standby condition matching the situation information of the moving object 20 is contact with the moving object 20, and the measure information is a display on the output unit 33 of an instruction button for instructing audio output prompting the child to release contact with the moving object 20. In this case, when the instruction button is operated by the operator P through the input unit 32, an instruction signal for the audio output is transmitted to the moving object 20 via the information processing device 10. The drive control unit 28B of the moving object 20 that has received the instruction signal controls the speaker of the output unit 23 to output the audio represented by the instruction signal. Through this control, for example, audio prompting the child to release contact with the moving object 20 is output from the output unit 23 of the moving object 20, thereby executing the measure, thereby making it possible to suitably improve a situation in which remote control is difficult to initiate. In this case, the voice of the operator P may be output from the output unit 23 of the moving object 20. Furthermore, such a measure may be autonomously executed by the drive control unit 28B of the moving object 20 in which the above-mentioned event occurred. For example, the drive control unit 28B of the moving body 20 may control the output unit 23 to output a sound urging the child to release contact with the moving body 20 when an emergency stop is made due to the detection of contact between the child and the moving body 20.

[0108] In addition, when the status information of the mobile body 20 where the event occurred satisfies the standby condition and an operator P is performing monitoring of the mobile body 20 without captured video, the output control unit 15C may output auxiliary information including the captured video captured by the mobile body 20 to an operator terminal 30 operated by the operator P.

[0109] In addition, if the content of remote control represented by the mobile body information of the mobile body 20 changes while waiting for the output of remote control request information for the mobile body 20 to be output to the operator terminal 30, the output control unit 15C may output auxiliary information to the operator terminal 30 including information representing the content of remote control before the change and the content of remote control after the change.

[0110] For example, assume that the mobile body 20 detects a parked vehicle as an obstacle, determines that the emergency stop requirements are met, stops autonomous movement, and transmits an emergency stop signal to the information processing device 10. Then, assume that the output control unit 15C of the information processing device 10 determines that the status information of the mobile body 20 for which the event has been determined indicates the presence of an obstacle, a parked vehicle, on the travel path of the mobile body 20, which is one of the waiting conditions. In this scenario, the parked vehicle may move while the output control unit 15C is waiting to output remote control request information for the mobile body 20 to the operator terminal 30. In this case, for example, the output control unit 15C may output auxiliary information to the operator terminal 30 when it determines that an event has occurred, the auxiliary information including information representing the remote control content indicating avoidance of the parked vehicle, and when it detects the movement of the parked vehicle, the auxiliary information including information representing the remote control content indicating an auxiliary operation to resume departure of the mobile body 20.

[0111] When the output control unit 15C determines that the status information of the mobile object 20 in which the above-mentioned event has occurred does not satisfy the standby condition, it outputs remote control request information for the mobile object 20 to the operator terminal 30.

[0112] In other words, when the output control unit 15C determines that the status information of the mobile body 20 in which the above-mentioned event occurred satisfies the standby condition, it waits for the output of remote control request information for the mobile body 20 to the operator terminal 30, and when the situation represented by the status information improves and no longer satisfies the standby condition, it outputs the remote control request information for the mobile body 20 to the operator terminal 30.

[0113] For example, assume that the situation information indicates that a child has contacted the moving object 20, and the output control unit 15C determines that the situation information satisfies the standby condition. In this case, the output control unit 15C transmits an output instruction signal to the moving object 20 to prompt the moving object 20 to release the contact. The drive control unit 28B of the moving object 20, upon receiving the output instruction signal, outputs the information prompting the release of the contact from at least one of the display and speaker included in the output unit 23. This allows the child in contact with the moving object 20 to be prompted to release the contact. The drive control unit 28B may control the output to stop when it determines that the child has released the contact with the moving object 20 based on the detection results of the sensor 24 or the captured video. Then, when the situation information including the moving object information received from the moving object 20 indicates that the child has released the contact with the moving object 20, the output control unit 15C of the information processing device 10 determines that the situation represented by the situation information has improved and no longer satisfies the standby condition. The output control unit 15C may then output remote control request information for the moving object 20 to the operator terminal 30.

[0114] Also, assume a case where the output control unit 15C determines that the status information satisfies the standby condition because the status information indicates that a user who is purchasing a product provided by the mobile unit 20 is within a predetermined range from the mobile unit 20. In this case, if the status information including the mobile unit information received from the mobile unit 20 indicates that the user has moved outside the predetermined range of the mobile unit 20, the output control unit 15C determines that the status indicated by the status information has improved and the standby condition no longer is satisfied. For example, assume that the mobile unit 20 sells products by loading them into lockers mounted on the mobile unit 20. Also assume that the sensor 24 of the mobile unit 20 includes an opening / closing detection sensor that detects the opening and closing of the locker. In this case, if the analysis result included in the mobile unit information received from the mobile unit 20 indicates that the user has moved from within the predetermined range to outside the predetermined range and the detection result of the opening / closing detection sensor indicates that the locker has been opened or closed, the output control unit 15C may determine that the user has moved outside the predetermined range of the mobile unit 20. Then, the output control unit 15C may output the remote control request information for the moving body 20 to the operator terminal 30.

[0115] Further, for example, assume that the situation information indicates that a certain number of children or more are present within a predetermined range from the moving body 20, and the output control unit 15C determines that the situation information satisfies the standby condition. In this case, the output control unit 15C transmits to the moving body 20 an output instruction signal for information urging children to move away from the moving body 20. The drive control unit 28B of the moving body 20, upon receiving the output instruction signal, outputs the information urging children to move away from the moving body 20 from at least one of the display and speaker included in the output unit 23. This makes it possible to urge children present within the predetermined range from the moving body 20 or people such as adults around the children to move away from the moving body 20. The drive control unit 28B may control the output to stop when it determines, based on the detection results of the sensor 24 or the captured video, that the number of children present within the predetermined range from the moving body 20 has fallen below the certain number, or that a person such as an adult has picked up a child, or the like. When the status information including the mobile object information received from the mobile object 20 indicates that a child has left the predetermined range of the mobile object 20, the output control unit 15C of the information processing device 10 determines that the status indicated by the status information has improved and the standby condition is no longer satisfied. Then, the output control unit 15C outputs remote control request information for the mobile object 20 to the operator terminal 30.

[0116] Also, for example, assume that the status information indicates a deterioration in the communication conditions, and the output control unit 15C determines that the status information satisfies the standby condition. In this case, the output control unit 15C may transmit instruction information to the mobile body 20 indicating an instruction to reduce the amount of data of the mobile body information transmitted to the information processing device 10 per unit time, so that the mobile body information is transmitted from the mobile body 20 to the information processing device 10 at the current communication speed with the mobile body 20 indicated by the communication status information included in the status information. Upon receiving the instruction information, the communication control unit 28A of the mobile body 20 adjusts the amount of data of the mobile body information transmitted to the information processing device 10 per unit time so that it can be transmitted at the current communication speed by lowering the image quality of the captured video included in the mobile body information, compressing the mobile body information, adjusting the vehicle speed, etc. Then, when the amount of data of the mobile body information received per unit time from the mobile body 20 becomes less than the current communication speed, the output control unit 15C determines that the status indicated by the status information has improved and the standby condition is no longer satisfied. Then, the output control unit 15C may output the remote control request information for the moving body 20 to the operator terminal 30.

[0117] Furthermore, a mode indicating whether the process for improving the communication conditions is to be performed autonomously by the mobile body 20 or through remote intervention by an operator P may be set in advance in the information processing device 10 and the mobile body 20. For example, assume a situation in which a mode indicating that the process for improving the communication conditions is to be performed autonomously by the mobile body 20 is set in the information processing device 10 and the mobile body 20. In this case, when the communication control unit 28A of the mobile body 20 detects a deterioration in the communication conditions, it may adjust the amount of data of the mobile body information transmitted to the information processing device 10 per unit time so that it can be transmitted at the current communication speed by lowering the image quality of the captured video included in the mobile body information, compressing the mobile body information, adjusting the vehicle speed, etc. Also, assume a situation in which a mode indicating that the process for improving the communication conditions is to be performed through remote intervention by an operator P is set in the information processing device 10 and the mobile body 20. In this case, when the output control unit 15C of the information processing device 10 determines that the situation information indicates a deterioration in the communication conditions, it may output remote control request information for the mobile body 20 to the operator terminal 30.

[0118] Furthermore, when the output control unit 15C determines that the situation represented by the situation information has improved due to an improvement in the communication speed with the mobile body 20 and that the standby condition is no longer satisfied, the output control unit 15C may output remote control request information, the remote control content of which includes a request for an instruction to return to autonomous movement, to the operator terminal 30. Furthermore, when the output control unit 15C determines that the situation represented by the situation information has improved due to a reduction in the data amount of mobile body information received from the mobile body 20 and that the standby condition is no longer satisfied, the output control unit 15C may output remote control request information, the remote control content of which includes a request for remote operation to move to a location with a better communication environment, to the operator terminal 30.

[0119] Also, for example, assume that the output control unit 15C determines that the wait condition is satisfied because the situation information indicates the presence of an obstacle, such as a parked vehicle on the driving route or a vehicle such as a bus or taxi that is temporarily stopped on the driving route. In this case, if the situation around the mobile body 20 indicated by the situation information indicates that the number of vehicles and the number of people getting on and off the vehicles within, for example, a rectangular area ahead of the mobile body 20 are below a certain number, the output control unit 15C determines that the situation indicated by the situation information has improved and the wait condition is no longer satisfied. Then, the output control unit 15C may output remote control request information for the mobile body 20 to the operator terminal 30.

[0120] Also, for example, assume a case where the status information indicates that the mobile body 20 traveling on a narrow road is passing another mobile body 20 or that passing another mobile body 20 is difficult, and the output control unit 15C determines that the status information satisfies the wait condition. In this case, if the output control unit 15C determines that the status of the mobile body 20 indicated by the status information indicates that the mobile body 20 has completed passing the other mobile body 20 or has exited from within a predetermined peripheral area of ​​the other mobile body 20, the output control unit 15C determines that the status indicated by the status information has improved and no longer satisfies the wait condition. Then, the output control unit 15C may output remote control request information for the mobile body 20 to the operator terminal 30.

[0121] Also, for example, assume that the output control unit 15C determines that the wait condition is satisfied because the status information indicates that the mobile body 20 is stopped just before crossing a target crossing area such as an intersection. In this case, if the situation around the mobile body 20 indicated by the status information indicates, for example, that there are no other mobile bodies 20 traveling in a direction intersecting the target crossing area with the planned traveling direction of the mobile body 20, or that the other mobile bodies 20 have stopped and taken an action to give way to the mobile body 20 and encourage it to move, the output control unit 15C determines that the situation indicated by the status information has improved and the wait condition is no longer satisfied. Then, the output control unit 15C may output remote control request information for the mobile body 20 to the operator terminal 30.

[0122] In addition, if the output control unit 15C determines that the status information of the mobile body 20 in which the above-mentioned event occurred satisfies the waiting conditions, it may output remote control request information to the operator terminal 30 a predetermined period before the time when the situation represented by the status information becomes one in which the waiting conditions are not satisfied.

[0123] In this case, the output control unit 15C calculates a predicted waiting time required from the time when the status information of the mobile body 20 in which the event occurred satisfies the waiting condition until the situation represented by the status information improves and the waiting condition is no longer satisfied. The predicted waiting time may be calculated in the same manner as described above. Then, the output control unit 15C can output the remote control request information to the operator terminal 30 before the calculated predicted waiting time has elapsed after determining that the status information of the mobile body 20 in which the event occurred satisfies the waiting condition. In other words, the output control unit 15C can also output the remote control request information to the operator terminal 30 before the calculated predicted waiting time has elapsed after determining that the status information of the mobile body 20 in which the event occurred satisfies the waiting condition. For example, the output control unit 15C transmits the remote control request information to the mobile body 20 a predetermined time before the predicted waiting time. This predetermined time may be shorter than the predicted waiting time and may be set in advance.

[0124] For example, there are cases where the operator P needs to understand the status of the mobile body 20 in advance before receiving the remote control request information at the operator terminal 30. In this case, if the remote control request information for the mobile body 20 is transmitted to the operator terminal 30 when the status information for the mobile body 20 indicates a situation that does not satisfy the standby condition, it may be difficult for the operator P to immediately start remote control of the mobile body 20. Therefore, the output control unit 15C outputs the remote control request information to the operator terminal 30 a predetermined period before the situation indicated by the status information indicates a situation that does not satisfy the standby condition, thereby enabling the output control unit 15C to assist the operator P in immediately starting remote control of the mobile body 20.

[0125] In addition, the output control unit 15C may identify the content of remote control for the mobile body 20 based on the status information of the mobile body 20 in which the event occurred, and further determine whether to wait for the output of remote control request information to the operator terminal 30 depending on the content of the remote control.

[0126] For example, if the remote control content represents remote control with a low load that is less than a predetermined load on the operator P, the output control unit 15C, upon determining the occurrence of an event, outputs remote control request information for the mobile body 20 to the operator terminal 30. On the other hand, if the remote control content represents remote control with a high load that is equal to or greater than a predetermined load on the operator P, the output control unit 15C outputs the remote control request information to the operator terminal 30 at a timing based on the situation information on the mobile body 20, as described above. The output control unit 15C associates each remote control content with load information that represents the degree of load on the operator P of the remote control represented by the content, and stores the information in the storage unit 14. Then, the output control unit 15C reads the load information corresponding to the identified remote control content from the storage unit 14, thereby identifying the load on the operator P for the remote control content and using the information for the above-mentioned determination.

[0127] Furthermore, for example, if the remote control content indicates a task that requires an immediate response, the output control unit 15C may output remote control request information for the mobile body 20 to the operator terminal 30 when determining that an event has occurred. On the other hand, if the remote control content does not indicate a task that requires an immediate response, the output control unit 15C outputs the remote control request information to the operator terminal 30 at a timing based on the situation information related to the mobile body 20, as described above. The output control unit 15C associates information indicating the urgency of the task for the mobile body 20 indicated by the remote control content with each remote control content in advance and stores the information in the storage unit 14. Then, the output control unit 15C reads information indicating the urgency corresponding to the identified remote control content from the storage unit 14, thereby identifying the urgency of the remote control content and using the information in the above determination.

[0128] Furthermore, when the state in which the situation represented by the situation information satisfies the standby condition continues for a predetermined time or more, the output control unit 15C of the information processing device 10 may output remote control request information to the operator terminal 30, even if the situation satisfies the standby condition when the predetermined time has elapsed since it was determined that the situation information satisfies the standby condition. The predetermined time may be determined based on the load status of the operator P, the urgency of the task, the task type, surrounding environment information obtainable from sensors such as a camera or a microphone, etc.

[0129] In addition, if the state in which the standby condition is satisfied continues for more than the specified time, the output control unit 15C may output a message to the operator terminal 30 indicating that the situation satisfies the standby condition after the specified time has elapsed, but that remote control request information has been output.

[0130] Furthermore, the output control unit 15C may output to the operator terminal 30, in different formats, remote control request information that the operator P can respond to immediately and remote control request information that the operator P cannot respond to immediately.

[0131] The output control unit 35B of the operator terminal 30 that has received the remote control request information outputs the received remote control request information to the output unit 33. Furthermore, if auxiliary information is received before the remote control request information is received, the output control unit 35B of the operator terminal 30 outputs the auxiliary information to the output unit 33. Then, when the remote control request information is received, the output control unit 35B outputs the remote control request information to the output unit 33.

[0132] 3A and 3B are schematic diagrams of an example of a display screen 40 output to the output unit 33 of the operator terminal 30. FIG.

[0133] 3A is a schematic diagram of an example of a display screen 40A including auxiliary information 44. The display screen 40A is an example of the display screen 40. When the output control unit 35B of the operator terminal 30 receives the auxiliary information 44 from the information processing device 10, the output control unit 35B displays the display screen 40A including the auxiliary information 44 on the output unit 33.

[0134] The display screen 40A includes, for example, a captured image 42 captured by the moving object 20 and auxiliary information 44. Fig. 3A shows, as an example, an example in which the auxiliary information 44 includes information indicating "approaching pedestrians," which is an example of information indicating a reason why immediate remote control intervention is difficult.

[0135] By visually checking the displayed auxiliary information 44, the operator P can carry out necessary preparations before starting remote control of the moving body 20.

[0136] 3B is a schematic diagram of an example of a display screen 40B including remote control request information 46. Display screen 40B is an example of the display screen 40. When the output control unit 35B of the operator terminal 30 receives the remote control request information 46 from the information processing device 10, it displays the display screen 40B including the remote control request information 46 on the output unit 33.

[0137] The display screen 40B includes, for example, a captured image 42 captured by the mobile object 20 and remote control request information 46. Fig. 3B shows an example of a form in which the remote control request information 46 includes information indicating "safety confirmation," which is an example of information requesting remote monitoring from the operator P. As shown in Fig. 3B, the output control unit 35B preferably outputs the remote control request information 46 in a display form that prompts the operator P to pay close attention. The display form that prompts attention means, for example, displaying the remote control request information 46 in a form that visually prompts greater attention, such as in bold or yellow, compared to other information.

[0138] By visually checking the remote control request information 46 , the operator P can execute the remote control indicated by the remote control request information 46 on the mobile body 20 .

[0139] There are cases where the operator P monitors a plurality of moving bodies 20. In such cases, information about each of the plurality of moving bodies 20 is displayed on the display screen 40. In such cases, it is preferable that the output control unit 35B of the operator terminal 30 arranges and displays information about the moving body 20, such as the captured video 42 captured by the moving body 20 and represented by the remote control request information 46 received from the information processing device 10, in an area, for example, in the center of the display screen 40 where the operator P is watching, and displays the remote control request information 46 on the information.

[0140] When the operator P remotely controls the moving body 20, the output control unit 35B preferably outputs to the output unit 33 a display screen 40 including information about the moving body 20 to be remotely controlled.

[0141] 3C is a schematic diagram of an example of a display screen 40C. The display screen 40C is an example of the display screen 40. In detail, the display screen 40C is an example of the display screen 40 displayed on the output unit 23 during remote control of the moving body 20 by the operator P.

[0142] For example, the display screen 40C includes a captured image 42, remote control request information 46, monitoring support information 48, sensor information 50, and a button image 52.

[0143] Specifically, the display screen 40C includes a screen in which, superimposed on the captured image 42, are information 48A and information 48B representing analysis results of an obstacle identified from the detection results of the sensor 24 included in the mobile object information, information 48C representing the travel path of the mobile object 20, and information 48D representing the end point of the remote operation of the mobile object 20. The display screen 40C also includes a screen in which one or more images representing the detection results of the sensor 24 of the mobile object 20 are arranged around an icon representing the mobile object 20. The display screen 40C may also include information 46B representing the content of the remote operation. The display screen 40C may further include information 46A representing the situation around the mobile object 20 and information 48E representing the state of the mobile object 20. The information 48A to information 48E are examples of monitoring support information 48. The display screen 40C may also include a button image 52 that the operator P uses to instruct the start and end of the remote operation.

[0144] Furthermore, the output control unit 35B may adjust the amount of information displayed on the display screen 40C depending on the content of remote control by the operator P. For example, the output control unit 35B may display a larger amount of information on the display screen 40C as the status information of the mobile object 20 indicates a situation requiring more detailed remote control.

[0145] Next, an example of the flow of information processing executed by the information processing system 1 of this embodiment will be described.

[0146] FIG. 4 is a sequence diagram showing an example of the flow of information processing executed by the information processing system 1 of this embodiment.

[0147] The mobile object information transmitting unit 28C of the mobile object 20 acquires the mobile object information (step S100) and transmits it to the information processing device 10 (step S102).

[0148] The determination unit 15B of the information processing device 10 determines whether an event requiring remote control has occurred in the moving object 20 (step S104).

[0149] When the output control unit 15C of the information processing device 10 determines that an event has occurred, it outputs remote control request information to the operator terminal 30 at a timing based on status information including mobile body information received from the mobile body 20 and communication status information representing the communication status between the mobile body 20 and the information processing device 10 (step S106).

[0150] The output control unit 35B of the operator terminal 30 outputs the remote control request information received from the information processing device 10 to the output unit 33 (step S108).

[0151] The remote operation control unit 35C transmits remote operation information corresponding to an operation instruction for the moving object 20 input in response to the operation of the input unit 32 by the operator P, together with the moving object ID of the moving object 20, to the information processing device 10 (step S110). The information processing device 10 transmits the remote operation information to the moving object 20 identified by the moving object ID (step S112). The moving object 20 controls the drive unit 26 based on the remote operation information, thereby moving the moving object 20 in response to the remote operation by the operator P. Thus, the moving object 20 is remotely controlled by the operator P operating the operator terminal 30.

[0152] Next, an example of the flow of information processing executed by the information processing device 10 will be described.

[0153] 5 is a flowchart showing an example of the flow of information processing executed by the information processing device 10. The information processing device 10 repeatedly executes the information processing shown in FIG. 5 at predetermined time intervals.

[0154] The determination unit 15B determines whether an event requiring remote control has occurred in the mobile object 20 (step S200). For example, the determination unit 15B analyzes the mobile object information and communication status information received from the information receiving unit 15A and determines whether the mobile object 20 is in a situation where it cannot move autonomously, thereby determining whether an event requiring remote control has occurred. If the determination in step S200 is negative (step S200: No), the routine ends. If the determination in step S200 is positive (step S200: Yes), the routine proceeds to step S202.

[0155] In step S202, the output control unit 15C acquires situation information related to the moving object 20 in which the occurrence of the event was determined in step S200 (step S202).

[0156] The output control unit 15C determines whether the situation information acquired in step S202 satisfies the standby condition (step S204). As described above, the standby condition indicates that the situation information indicates at least one of the following: a user is present within a predetermined range from the mobile body 20; the user has come into contact with the mobile body 20; an obstacle is present on the travel path of the mobile body 20; the mobile body 20 is unable to cross the target crossing area ahead in the travel direction; the mobile body 20 is in a difficult-to-travel state along the travel path; and communication conditions with the mobile body 20 have deteriorated.

[0157] If the output control unit 15C determines that the status information satisfies the standby condition (step S204: Yes), the process proceeds to step S206. In step S206, the output control unit 15C outputs the auxiliary information to the operator terminal 30 (step S206). Then, the process returns to step S202.

[0158] On the other hand, if the output control unit 15C determines that the status information does not satisfy the standby condition (step S204: No), the output control unit 15C proceeds to step S208. In step S208, the output control unit 15C outputs remote control request information for the moving object 20 to the operator terminal 30 (step S208). Then, this routine ends.

[0159] The output control unit 35B of the operator terminal 30 that has received the remote control request information outputs the received remote control request information to the output unit 33. By checking the remote control request information, the operator P can start remote control of the moving body 20.

[0160] As described above, the information processing device 10 of this embodiment includes the determination unit 15B and the output control unit 15C. The output control unit 15C determines whether an event requiring remote control has occurred in the mobile object 20. If the output control unit 15C determines that an event has occurred, it outputs remote control request information regarding remote control to the operator terminal 30 at a timing based on situation information regarding the mobile object 20.

[0161] In the prior art, when an event requiring remote control occurs in the mobile body 20, a remote control request or the like is immediately notified to the operator P. However, the situation of the mobile body 20 at the time of the occurrence of the event may be such that it is difficult for the operator P to immediately start remote control, or such that it is difficult to return to autonomous movement even after remote control is started. If the remote control request is notified to the operator P despite such a situation, the operator P may be tied up for an unnecessarily long time, which may make it difficult to efficiently manage the resources of the operator P.

[0162] On the other hand, in the information processing device 10 of this embodiment, when it is determined that an event has occurred, the output control unit 15C outputs remote control request information regarding remote control to the operator terminal 30 at a timing based on the status information regarding the mobile object 20. In this way, in the information processing device 10 of this embodiment, the remote control request information is output to the operator terminal 30 operated by the operator P at a timing based on the status information of the mobile object 20.

[0163] Therefore, the information processing device 10 of this embodiment can suppress the output of remote control request information to the operator terminal 30 in situations where it is difficult for the operator P to immediately start remote control or where it is difficult to return to autonomous movement even after starting remote control. Therefore, the information processing device 10 of this embodiment can suppress unnecessary restraint of the operator P. Furthermore, the information processing device 10 of this embodiment can suppress the time required for the operator P to control the moving body 20.

[0164] Therefore, the information processing device 10 of this embodiment can efficiently manage the resources of the operator P.

[0165] Next, an example of the hardware configuration of the information processing device 10, the mobile object 20, and the operator terminal 30 according to the above embodiment and modified example will be described.

[0166] FIG. 6 is a diagram illustrating an example of a hardware configuration of the information processing device 10, the mobile object 20, and the operator terminal 30 according to the above embodiment and the modified example.

[0167] The information processing device 10, mobile body 20, and operator terminal 30 in the above embodiment and modified example have a hardware configuration that utilizes a conventional computer, in which a CPU (Central Processing Unit) 80, a ROM (Read Only Memory) 82, a RAM (Random Access Memory) 84, an I / F 86, etc. are interconnected by a bus 88.

[0168] The CPU 80 is a computing device that controls the information processing device 10, the mobile object 20, and the operator terminal 30 of the above-described embodiment and modified examples. The ROM 82 stores programs and the like that realize information processing by the CPU 80. The RAM 84 stores data necessary for various processes by the CPU 80. The I / F 86 is an interface that is connected to a storage unit, an input unit, an output unit, a sensor, a communication unit, and the like, and is used to send and receive data.

[0169] In the information processing device 10, the mobile object 20, and the operator terminal 30 of the above-described embodiment and modified example, the CPU 80 reads a program from the ROM 82 onto the RAM 84 and executes it, thereby realizing each of the above-described functional units on the computer.

[0170] The programs for executing the above processes executed by the information processing device 10, the mobile object 20, and the operator terminal 30 in the above-described embodiment and modified examples may be stored in a hard disk drive (HDD). Also, the programs for executing the above processes executed by the information processing device 10, the mobile object 20, and the operator terminal 30 in the above-described embodiment and modified examples may be provided by being pre-installed in the ROM 82.

[0171] Furthermore, the programs for executing the above processes executed by the information processing device 10, mobile object 20, and operator terminal 30 of the above embodiments and modified examples may be stored in an installable or executable file format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disk), or flexible disk (FD) and provided as a computer program product. Furthermore, the programs for executing the above information processes executed by the information processing device 10, mobile object 20, and operator terminal 30 of the above embodiments and modified examples may be stored on a computer connected to a network such as the Internet and provided by downloading via the network. Furthermore, the programs for executing the above information processes executed by the information processing device 10, mobile object 20, and operator terminal 30 of the above embodiments and modified examples may be provided or distributed via a network such as the Internet.

[0172] Although the above describes an embodiment, the embodiment is presented as an example and is not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0173] The present technology can also be configured as follows. (1) An information processing method executed by an information processing device, comprising: a determination step of determining whether an event requiring remote control has occurred in a moving object; and an output control step of, when it is determined that the event has occurred, outputting remote control request information regarding remote control to an operator terminal at a timing based on situation information regarding the moving object. (2) The information processing method according to (1), in which the event represents a situation in which the moving object is unable to move autonomously. (3) The information processing method according to (1) or (2), in which the event represents a situation in which the moving object requires at least one of remote operation, remote assistance, and remote monitoring. (4) The information processing method according to any one of (1) to (3), in which, when the situation information satisfies a standby condition, the output control step waits for output of the remote control request information to the operator terminal, and when the standby condition is not satisfied, outputs the remote control request information to the operator terminal. (5) The information processing method according to (4), wherein the output control step, when the status information satisfies the standby condition, outputs the auxiliary information to the operator terminal before outputting the remote control request information to the operator terminal. (6) The information processing method according to (4) or (5), wherein the output control step, when the status information changes to a state where the standby condition is not satisfied while waiting for output of the remote control request information to the operator terminal, outputs the remote control request information to the operator terminal. (7) The information processing method according to any one of (4) to (6), wherein, when the status information satisfies the standby condition, the output control step outputs the remote control request information to the operator terminal a predetermined period before the situation represented by the status information becomes a state where the standby condition is not satisfied. (8) The information processing method described in any one of (4) to (7), wherein the output control step identifies the content of remote control for the moving body based on the status information, and waits for the output of the remote control request information to the operator terminal according to the identified content of remote control.(9) The information processing method according to (8), wherein the content of the remote control includes at least one of a load on an operator remotely controlling the mobile object and an urgency of a task. (10) The information processing method according to any one of (4) to (9), wherein the output control step outputs the remote control request information to the operator terminal when a state in which the situation represented by the situation information satisfies the standby condition continues for a predetermined time or more after the predetermined time has elapsed since it was determined that the situation information satisfies the standby condition. (11) The information processing method according to any one of (4) to (10), wherein the standby condition indicates a situation in which movement by remote control would be difficult or inefficient if an attempt were made to immediately control the mobile object by remote control. (12) The information processing method according to any one of (4) to (11), wherein the output control step is capable of outputting auxiliary information related to assistance in remote control of the mobile object to the operator terminal when the situation information satisfies the standby condition. (13) The information processing method according to any one of (4) to (13), wherein the auxiliary information includes at least one of information indicating that the moving body is in a situation where it cannot move autonomously, a predicted waiting time until remote control is started, information indicating a reason why it is difficult to start remote control immediately, policy information indicating a policy for improving a situation where it is difficult to start remote control, captured video of the area around the moving body, and information indicating that a remote response is required but not possible immediately. (14) The information processing method according to any one of (4) to (13), wherein the output control step is capable of, when the situation information satisfies the standby condition, calculating a predicted waiting time required until the situation represented by the situation information becomes a situation where the standby condition is not satisfied, and outputting the remote control request information to the operator terminal a predetermined time before the predicted waiting time. (15) An information processing program for causing a computer to execute the following steps: a determination step for determining whether an event requiring remote control has occurred in a mobile object; and an output control step for outputting remote control request information regarding remote control to an operator terminal at a timing based on status information regarding the mobile object when it is determined that the event has occurred.(16) An information processing device comprising: a judgment unit that judges whether an event requiring remote control has occurred in a moving object; and an output control unit that, when it is determined that the event has occurred, outputs remote control request information regarding remote control to an operator terminal at a timing based on status information regarding the moving object.

[0174] 10 Information processing device 15B Determination unit 15C Output control unit 30 Operator terminal

Claims

1. An information processing method executed by an information processing device, comprising: a determination step of determining whether an event requiring remote control has occurred in a mobile object; and an output control step of outputting remote control request information regarding remote control to an operator terminal at a timing based on status information regarding the mobile object when it is determined that the event has occurred.

2. The information processing method according to claim 1, wherein the event represents a situation in which the mobile object is unable to move autonomously.

3. The information processing method according to claim 1, wherein the event represents a situation in which the mobile object requires at least one of remote operation, remote assistance, and remote monitoring.

4. The information processing method according to claim 1, wherein the output control step waits for the output of the remote control request information to the operator terminal if the status information satisfies a standby condition, and outputs the remote control request information to the operator terminal if the standby condition is not satisfied.

5. The information processing method according to claim 4, wherein the output control step outputs auxiliary information regarding assistance in remote control of the mobile body to the operator terminal before outputting the remote control request information to the operator terminal when the status information satisfies the standby condition.

6. The information processing method according to claim 4, wherein the output control step outputs the remote control request information to the operator terminal if the status information changes to a state that does not satisfy the waiting condition while waiting for the output of the remote control request information to the operator terminal.

7. The information processing method according to claim 4, wherein the output control step, when the status information satisfies the standby condition, outputs the remote control request information to the operator terminal a predetermined period before the situation represented by the status information becomes one in which the standby condition is not satisfied.

8. An information processing method as described in claim 4, wherein the output control step identifies the content of remote control for the mobile body based on the status information, and waits for the output of the remote control request information to the operator terminal depending on the identified content of remote control.

9. The information processing method according to claim 8, wherein the remote control content includes at least one of the load on an operator remotely controlling the mobile object and the urgency of a task.

10. The information processing method of claim 4, wherein the output control step outputs the remote control request information to the operator terminal when the situation represented by the situation information continues to satisfy the standby condition for a predetermined time or more, after the predetermined time has elapsed since it was determined that the situation information satisfies the standby condition.

11. The information processing method according to claim 4, wherein the standby condition indicates a situation in which remote control of the moving object would be difficult or inefficient if an attempt were made to immediately control the moving object by remote control.

12. An information processing method according to claim 4, wherein the output control step is capable of outputting auxiliary information relating to assistance in remote control of the mobile object to the operator terminal when the status information satisfies the standby condition.

13. The information processing method of claim 12, wherein the auxiliary information includes at least one of information indicating that the mobile body is in a situation where it cannot move autonomously, a predicted waiting time until remote control begins, information indicating the reason why it is difficult to begin remote control immediately, measure information indicating measures to improve the situation where it is difficult to begin remote control, captured video of the area around the mobile body, and information indicating that a remote response is required but cannot be performed immediately.

14. The information processing method of claim 4, wherein the output control step is capable of calculating, when the status information satisfies the standby condition, a predicted standby time required for the status represented by the status information to become a status that does not satisfy the standby condition, and outputting the remote control request information to the operator terminal a predetermined time before the predicted standby time.

15. An information processing program for causing a computer to execute the following steps: a determination step for determining whether an event requiring remote control has occurred in a mobile object; and an output control step for outputting remote control request information regarding remote control to an operator terminal at a timing based on status information regarding the mobile object if it is determined that the event has occurred.

16. An information processing device comprising: a judgment unit that judges whether an event requiring remote control has occurred in a moving object; and an output control unit that, when it is determined that the event has occurred, outputs remote control request information regarding remote control to an operator terminal at a timing based on status information regarding the moving object.

Images