Information processing device, program, and information processing method

Abstract

To provide an information processing apparatus, a program, and an information processing method, which are new and improved, capable of improving an operability of a remote operation system that remotely operates a movement body.SOLUTION: An information processing apparatus includes a calculation unit that calculates an index indicating an execution risk by autonomous control of a movement body relating to an event occurred in the movement body operated by the autonomous control or operator control, an incident management unit that manages information on an incident where the event occurred in the movement body is considered as an incident, and the index calculated by the calculation unit, a notification control unit that controls notification to an operator who manages the movement body based on the index calculated by the calculation unit, and an operation control unit that switches control of the movement body from the autonomous control to the operator control according to an operation by the operate who receives the notification.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a program, and an information processing method.

Background Art

[0002] In recent years, with the progress of robot-related technologies, the development of systems for operating robots that autonomously perform work has been underway. Such systems are expected as a means to solve the labor shortage associated with the declining birthrate and aging population.

[0003] For example, by using technologies such as artificial intelligence (AI) and map data within and between buildings, systems and technologies have been developed that can autonomously operate robots and enable robots to autonomously move and travel within and between buildings.

[0004] On the other hand, currently, events may occur during the operation of a robot that are difficult for the robot to autonomously handle or judge. In such cases, the robot transmits a notification requesting an operation to an operator at a remote location. Then, the operator who has confirmed the notification switches the robot that has transmitted the notification from the state of operating autonomously to a state where it can be manually operated. This may make it possible to solve events that are difficult for the robot to handle or judge.

[0005] For example, in Patent Document 1 below, when it is determined that it is difficult for a communication robot to respond only with autonomous control, based on a condition list created from the voice of the dialogue partner, it connects to the terminal of an operator who satisfies the conditions required by the dialogue partner, and a technique is disclosed in which the operator remotely conducts a dialogue.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] However, in the technology described in Patent Document 1, if it is determined that it is difficult to respond using only autonomous control by multiple robots (hereinafter also referred to as mobile units), the responses to the events (hereinafter also referred to as events) are connected to the operator's terminal in the order in which they occurred, regardless of the difficulty level of the events. Therefore, the operator may not be able to respond immediately to events with a high degree of difficulty.

[0008] Therefore, the present invention has been made in view of the above problems, and the object of the present invention is to provide a novel and improved information processing device, program, and information processing method that can further improve the operability of a remote operation system for remotely operating a mobile object. [Means for solving the problem]

[0009] To solve the above problems, according to one aspect of the present invention, a calculation unit calculates an index indicating the execution risk by autonomous control for each of a plurality of mobile bodies that operate under autonomous control or operator control, with respect to each event that occurs in the plurality of mobile bodies; an incident management unit that manages the information of the incident and the index calculated by the calculation unit, treating each of the events that occur in the plurality of mobile bodies as an incident; a notification control unit that, based on the incident information and the index managed by the incident management unit, prioritizes notifications to the operator managing the mobile bodies regarding events in which the index meets a predetermined standard, in order of increasing index level; and an event is selected by the operator who receives the notification. In that case, a communication connection is established between the operator's remote control terminal and the mobile object in which the event described above is occurring. Further operation by the aforementioned operator The first moving body An information processing device is provided, which includes an operation control unit that switches the control from autonomous control to operator control.

[0013] The calculation unit may calculate the index based on the information relating to the event and the operating status of the moving body.

[0014] Information relating to the operating status of the mobile body includes information about the surrounding environment in which the mobile body operates. or The moving body Overall width Information of It may include at least one of these.

[0015] The calculation unit may calculate the index based on a calculation model obtained by machine learning, which uses a set of information relating to the event and the operating status of the moving body and the index as training data.

[0016] The aforementioned index may include at least one of the following with respect to the aforementioned event: an index indicating the difficulty of execution by autonomous control of the mobile body, or an index indicating the degree of loss in the event of failure to execute by autonomous control.

[0017] Furthermore, in order to solve the above problems, according to another aspect of the present invention, the computer comprises: a calculation function that calculates an index indicating the execution risk by autonomous control for each of the multiple mobile bodies operating under autonomous control or operator control, for each event occurring in the multiple mobile bodies; an incident management function that manages the information of the incident and the index calculated by the calculation function, treating each event occurring in the multiple mobile bodies as an incident; a notification control function that, based on the incident information and the index managed by the incident management function, prioritizes notifications to the operator managing the mobile bodies regarding events in which the index meets a predetermined standard, in order of increasing index level; and an event being selected by the operator's operation upon receiving the notification. In that case, a communication connection is established between the operator's remote control terminal and the mobile object in which the event described above is occurring. Further operation by the aforementioned operator The first moving body A program is provided that functions as an operation control function that switches the control from autonomous control to operator control.

[0018] Also, in order to solve the above problems, according to another aspect of the present invention, regarding each event that occurs in a plurality of mobile bodies operating under autonomous control or operator control, calculating an index indicating the execution risk of each of the plurality of mobile bodies by the autonomous control, regarding each of the events that occur in the plurality of mobile bodies as an incident, managing the information of the incident and the index indicating the calculated execution risk respectively, based on the managed information of the incident and the index, preferentially controlling in descending order of the index the notification to the operator who manages the mobile body regarding the event that the index satisfies a predetermined criterion, and selecting one event by the operation of the operator who received the notification In that case, a communication connection is established between the operator's remote control terminal and the mobile object in which the event described above is occurring. By further operation of the operator The first moving body switching the control from the autonomous control to the operator control, and an information processing method executed by a computer is provided.

Advantages of the Invention

[0019] As described above, according to the present invention, by allowing the operator to select the mobile body to be handled under operator control after checking the index indicating the execution risk, it is possible to handle events in descending order of the index, and it is possible to further improve the operability of the remote operation system for remotely operating the mobile body.

Brief Description of the Drawings

[0020] [Figure 1] It is a schematic diagram of the information processing system 1 according to the present embodiment. [Figure 2] It is a block diagram showing an example of the functional configuration of the information processing system 1 according to the present embodiment. [Figure 3] It is a flowchart showing an example of the operation process of the robot 10 of the information processing system 1 according to the present embodiment. [Figure 4] It is a flowchart showing an example of the operation process of the remote operation terminal 20 of the information processing system 1 according to the present embodiment.

Embodiments for Carrying Out the Invention

[0021] Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in detail. In this specification and the drawings, for components having substantially the same functional configuration, the same reference numerals are given and duplicate description is omitted.

[0022] Also, in this specification and the drawings, when N robots are distinguished and expressed as robot 10-1, robot 10-2, ···, robot 10-N, respectively, if there is no particular need to distinguish each robot, it is simply referred to as robot 10. Similarly, for M remote operation terminals 20-1, 20-2, ··· 20-M and M operators OP-1, OP-2, ··· OP-M, if there is no particular need to distinguish them, they are simply referred to as remote operation terminal 20 and operator OP.

[0023] <Embodiment> <<1. Overview of Information Processing System>> First, referring to FIG. 1, the overview of the information processing system according to this embodiment will be described.

[0024] FIG. 1 is a schematic diagram of an information processing system 1 according to this embodiment. As shown in FIG. 1, the information processing system 1 according to this embodiment includes N robots 10-1, 10-2, ··· 10-N, M remote operation terminals 20-1, 20-2, ··· 20-M, and an information processing server 30.

[0025] N and M are arbitrary integers independently selected from each other. However, in order to remotely control the robot 10 more efficiently, M may be an integer smaller than N.

[0026] The N robots 10-1, 10-2, ··· 10-N, the M remote operation terminals 20-1, 20-2, ··· 20-M, and the information processing server 30 are interconnected via a network 5, respectively.

[0027] Robot 10 is an example of a mobile device. Robot 10 is capable of autonomous movement and can also be controlled (referred to as remote operation) from the remote control terminal 20. Furthermore, robot 10 can send notifications to the remote control terminal 20 and transmit data in response to requests from the remote control terminal 20.

[0028] Here, a robot is defined as a mechanical device that is a so-called "artificial human," a "device that performs actions or tasks similar to those of a human," or a "device that performs actions or tasks through computer control based on instructions." For example, a robot may be defined as a "machine used in industry that has an automatic control manipulation function or movement function and can perform various tasks by program" (JIS B 0134-1998), a "robot that serves humans" (JIS B 0187:2005), or an "intelligent mechanical system having three elemental technologies: sensors, intelligence / control systems, and drive systems" (Ministry of Economy, Trade and Industry Robot Policy Study Group).

[0029] In the information processing system 1 according to this embodiment, the robot 10 can be used for various purposes or operations. For example, the robot 10 can be used for security purposes such as patrolling and monitoring at facilities under security, for inspection of public facilities or public equipment, at manufacturing sites (such as equipment inspection within factories or transport of manufactured parts or materials), for transporting building components at construction sites or for checking the construction status, for transporting passengers or luggage in the passenger transport sector, for guiding tourists, or for inspecting luggage, for inspecting cargo or cargo vehicles in the logistics sector, or for transporting cargo.

[0030] The remote control terminal 20 is a device used by the operator OP to monitor and remotely control the robot 10. For example, the remote control terminal 20 is a device that includes a terminal such as a personal computer (PC) or a smartphone, and peripheral devices connected to these terminals. The remote control terminal 20 is capable of outputting data regarding the status of the robot 10, issuing movement instructions to the robot 10 based on the operator OP's input operations, outputting notifications from the robot 10, and monitoring the status of the operator OP.

[0031] In the information processing system 1 according to this embodiment, the remote control terminal 20 can be used for various purposes or operations. For example, the remote control terminal 20 can present an image to the operator OP that includes the robot 10 and the environment surrounding the robot 10, thereby allowing the operator OP to understand the environment surrounding the robot 10. As a result, the remote control terminal 20 can remotely move the robot 10 based on the input of the operator OP.

[0032] Furthermore, if there are multiple remote control terminals 20, one operator OP may be assigned to each remote control terminal 20. For example, as shown in Figure 1, remote control terminal 20-1 is used by operator OP-1, and remote control terminal 20-2 is used by operator OP-2.

[0033] The information processing server 30 is the device on which the main system of the information processing system 1 operates. For example, the information processing server 30 manages various information related to the operation and status of all N robots 10-1, 10-2, ..., 10-N, and transmits this information to the remote control terminal 20. The information processing server 30 also manages various information for all M remote control terminals 20-1, 20-2, ..., 20-M, and all operators OP-1, OP-2, ..., OP-M who use the remote control devices.

[0034] Furthermore, the information processing server 30 may be located in any location. For example, the information processing server 30 may be located in the same location as the remote control terminal 20, or it may be located in a different location.

[0035] Network 5 is a communication network connecting the robot 10, the remote control terminal 20, and the information processing server 30. Network 5 enables the mutual transmission and reception of data between the robot 10, the remote control terminal 20, and the information processing server 30. Network 5 may be the Internet, a satellite communication network, a mobile communication network, a LAN (Local Area Network), or a WAN (Wide Area Network), etc.

[0036] Furthermore, the communication methods between the robot 10, the remote control terminal 20, and the information processing server 30 and the network 5 may be the same or different. For example, the robot 10 may connect to the network 5 using wireless LAN communication so that it can connect to the network 5 from any location. On the other hand, the remote control terminal 20 and the information processing server 30 may connect to the network 5 using wired LAN communication so that they can be installed in predetermined locations and connected to the network 5. However, the communication methods between the robot 10 and the network 5, and between the remote control terminal 20 and the information processing server 30 and the network 5 are not limited to those described above and may use other communication methods.

[0037] In the information processing system 1, the robot 10 performs tasks assigned by the operator OP (hereinafter also referred to as "tasks") through autonomous control. However, during the performance of these tasks, the robot 10 may encounter situations that it can resolve on its own, or situations that it finds difficult to resolve on its own.

[0038] When robot 10 encounters an event, it calculates an index indicating the execution risk through autonomous control for the event that occurred, identifies the event as an incident, and transmits the details of the incident and the calculated result of the execution risk index to the information processing server 30. The index indicating the execution risk through autonomous control of robot 10 is just one example of an index indicating the execution risk of a mobile object; for example, it could be an index indicating the difficulty of execution through autonomous control of robot 10, or an index indicating the degree of loss in the event of execution failure through autonomous control.

[0039] The information processing server 30 receives the incident details and indicators showing the execution risk transmitted from the robot 10, and stores and manages the incident details and indicators showing the execution risk.

[0040] The remote control terminal 20 notifies the operator OP of the details of the incident stored in the information processing server 30 and indicators showing the risk of execution. Upon receiving the notification regarding these indicators, the operator OP operates the remote control terminal 20 to select an incident to handle remotely.

[0041] Furthermore, when an incident is selected by the operator OP, the remote control terminal 20 requests the information processing server 30 to connect to the robot 10 where the selected incident is occurring.

[0042] The information processing server 30 connects the robot 10 where the incident selected by the operator OP has occurred with the remote control terminal 20 that has requested a connection.

[0043] The remote control terminal 20 displays the camera feed from the robot 10. The operator OP decides whether to switch the control of the robot 10 from autonomous control to operator control. If operator control is selected, the operator switches to remote control via the remote control terminal 20 and performs remote control via the remote control terminal 20 to resolve any incidents that occur with the robot 10.

[0044] The details of the functional configuration example of the information processing system 1 according to this embodiment will be described below with reference to Figure 2.

[0045] <<2. Example of Functional Configuration of Information Processing System 1>> Figure 2 is an explanatory diagram illustrating an example of the functional configuration of the information processing system 1 according to this embodiment. As shown in Figure 2, the information processing system 1 according to this embodiment includes N robots 10-1, 10-2, ...10-N, M remote control terminals 20-1, 20-2, ...20-M, and an information processing server 30.

[0046] (Robot 10) Robot 10 is an example of an information processing device. Robot 10 comprises a robot control unit 110, a LAN 112, a camera unit 113, a sensor unit 114, a display unit 115, a speaker 116, a microphone 117, a robot communication unit 118, an AI unit 119, a movement unit 120, an information storage unit 121, and a battery unit 122.

[0047] The robot control unit 110 processes various data acquired by the robot 10 and controls the overall operation of the robot 10. The robot control unit 110 may consist of, for example, software including a program that controls the overall operation of the robot 10, and hardware including a processor and memory on which the software is installed. Examples of hardware on which the software is installed include terminals such as a PC (Personal Computer), tablet, microcomputer, or smartphone.

[0048] Furthermore, as shown in Figure 2, the robot control unit 110 includes a task management unit 123, an event detection unit 124, and a risk calculation unit 125.

[0049] LAN112 is the network within the robot 10. The robot control unit 110, the robot communication unit 118, and the AI ​​unit 119 are connected to each other via LAN112, and communicate and send / receive data.

[0050] The camera unit 113 includes at least one imaging device capable of acquiring environmental images of the robot 10. The camera unit 113 may include, for example, a 360-degree camera capable of acquiring environmental images of the entire surroundings of the robot 10. Alternatively, the camera unit 113 may include a plurality of fisheye cameras provided in each direction of the robot 10.

[0051] The sensor unit 114 includes at least one sensor capable of sensing various parameters of the environment surrounding the robot 10. The sensor unit 114 may acquire sensing information by corresponding sensors, for example, by sensing temperature, humidity, illuminance, atmospheric pressure, vibration, distance from the robot 10 to an object, or changes in the tilt of the robot 10. The sensor unit 114 may also acquire sensing information by corresponding sensors by sensing the generation of smoke, chemical substances, and static electricity. In the above, "sensing" includes not only detecting various parameters of the environment surrounding the robot 10, but also further measuring, identifying, and analyzing the detected parameters.

[0052] The type of sensor included in the sensor unit 114 is not particularly limited. The sensor unit 114 may include, for example, one or more of the following: a thermometer, a hygrometer, an illuminometer, a barometer, a LiDAR (Light Detection And Ranging) sensor, or a thermal imaging device (thermographic camera).

[0053] Furthermore, the sensor unit 114 may acquire sensing information about the environment surrounding the robot 10 from a sensor device installed outside the robot 10 via the robot communication unit 118. In such a case, the sensor unit 114 may acquire sensing information from the external sensor device, for example, by 920MHz band multi-hop wireless communication.

[0054] The images acquired by the camera unit 113 and the sensing information acquired by the sensor unit 114 are examples of environmental information surrounding the moving object.

[0055] The display unit 115 outputs various information in the form of images or the like, in accordance with the control of the robot control unit 110. The display unit 115 may include, for example, a display that outputs various information in the form of images, or a lamp that outputs various information by emitting light.

[0056] The speaker 116 outputs sound around the robot 10 in accordance with the control of the robot control unit 110. For example, the speaker 116 may output sound effects corresponding to the robot 10's movements, synthesized speech corresponding to the robot 10's movements, or speech input by the operator OP around the robot 10.

[0057] Microphone 117 picks up sounds from around the robot 10. For example, microphone 117 may pick up the voices of people present around the robot 10, or ambient sounds around the robot 10.

[0058] The robot communication unit 118 is a communication interface for connecting to network 5. For example, the robot communication unit 118 may be a communication interface that can connect wirelessly or via wired connection to network 5, or to a base station that can connect to network 5. However, the robot communication unit 118 may also be a communication interface that can connect to network 5 via a mobile phone network or a wireless LAN.

[0059] The robot communication unit 118 transmits, for example, information related to incidents managed by the task management unit 123 to the information processing server 30. The robot communication unit 118 also transmits environmental images of the robot 10's surroundings, acquired by the camera unit 113, to the remote control terminal 20. The robot communication unit 118 also receives control information related to remote operation from the remote control terminal 20.

[0060] The AI ​​unit 119 recognizes the environment around the robot 10 based on the environmental image of the robot 10 captured by the camera unit 113 and the sensing information of various parameters of the environment around the robot 10 sensed by the sensor unit 114, and makes judgments and decisions according to the recognized environment. The AI ​​unit 119 may, for example, use a machine learning algorithm to recognize the environment around the robot 10 from the environmental image and sensing information, and make judgments and decisions according to the recognized environment.

[0061] The event detection unit 124 detects events that have occurred to the robot 10 based on the results of the AI ​​unit 119's recognition of the environment around the robot 10. For example, the event detection unit 124 may determine that an event has occurred if the AI ​​unit 119 performs person detection and a person is detected, or if the AI ​​unit 119 performs object detection and a specific object is detected.

[0062] The risk calculation unit 125 is an example of a calculation unit and calculates an index indicating the execution risk of the robot 10 through autonomous control in relation to an event that occurred to the robot 10. The robot control unit 110 then controls the robot 10 to respond through autonomous control if the calculated index indicating the execution risk is low, and transmits the details of the event that occurred and the calculated execution risk to the information processing server 30 if the calculated index indicating the execution risk is high. For example, the risk calculation unit 125 calculates an index indicating the execution risk of the robot 10 through autonomous control based on the event that occurred to the robot 10, information related to the operating status of the robot 10, and the judgment result from the AI ​​unit 119.

[0063] Furthermore, a list may be prepared in advance that includes the relationship between events that occur with the robot 10, information related to the operating status of the robot 10, and an index indicating the risk of execution by autonomous control of the robot 10. In this case, the risk calculation unit 125 may calculate an index indicating the risk of execution by autonomous control of the robot 10 based on the pre-prepared list and the information related to the operating status of the robot 10. The information related to the operating status of the robot 10 is an example of information related to the operating status of a mobile body, and includes, for example, at least one of the surrounding environment information in which the robot 10 operates or the characteristic information of the robot 10. Furthermore, the AI ​​unit 119 may generate a risk calculation model using machine learning technology that uses the events that occur with the robot 10 and the information related to the operating status of the robot 10 as training data.

[0064] The mobile unit 120 is a moving mechanism capable of moving the robot 10 to any position based on control by the robot control unit 110. The mobile unit 120 may be a moving mechanism of various types, such as a wheeled, legged, tracked, or air cushion type. Furthermore, the mobile unit 120 may be a moving mechanism capable of three-dimensional movement. For example, the mobile unit 120 may be a moving mechanism capable of moving through the air using rotor blades, or a moving mechanism capable of moving on or underwater using a screw.

[0065] The task management unit 123 manages information related to the tasks performed by the robot 10. Alternatively, the task management unit 123 may also be an example of an incident management unit, managing incident information and indicators showing the execution risk through autonomous control related to the incident.

[0066] The information storage unit 121 stores characteristic information of the robot 10 as an example of characteristic information of a moving object. The characteristic information of the robot 10 includes, for example, information about the robot's appearance such as its size, shape, overall width, or height. The characteristic information of the robot 10 may also include, for example, information about its movement characteristics such as its movement speed, movement range, or movable area.

[0067] The battery unit 122 supplies power to each part of the robot 10. The battery unit 122 may include a secondary battery, such as a lithium-ion secondary battery.

[0068] (Remote control terminal 20) As shown in Figure 2, the remote control terminal 20 includes a control unit 210, a LAN 211, a communication unit 212, a camera unit 213, a sensor unit 214, a display unit 215, a speaker 216, a microphone 217, an operation unit 218, and a power supply unit 219.

[0069] The control unit 210 processes various data acquired by the remote control terminal 20 and controls the overall operation of the remote control terminal 20. The control unit 210 may consist of, for example, software including a program that controls the overall operation of the remote control terminal 20, and hardware including a processor and memory on which the software is installed. Examples of hardware on which the software is installed include terminals such as personal computers, tablets, microcomputers, or smartphones.

[0070] As shown in Figure 2, the control unit 210 includes a notification control unit 220, an image generation unit 221, and an operation control unit 222.

[0071] The notification control unit 220 controls notifications to the operator OP who manages the robot 10, based on the incident indicators accumulated in the all-robot incident management unit 313.

[0072] Various methods can be applied to notify the operator OP of the notification control unit 220. For example, when an incident is accumulated in the all-robot incident management unit 313, the notification control unit 220 may display an incident information notification screen on the display unit 215. Alternatively, when an indicator calculated by the risk calculation unit 125 meets a predetermined condition, the notification control unit 220 may output a notification voice to the speaker 216 informing the operator that the risk of autonomous control execution is high.

[0073] Furthermore, the notification control unit 220 may prioritize notifications for each indentation that occurs in the N robots 10-1, 10-2, ..., 10-N, in order of the highest index calculated by the risk calculation unit 125. For example, the notification control unit 220 may display a screen on the display unit 215 that highlights incidents and robots in order of the highest index calculated by the risk calculation unit 125.

[0074] The image generation unit 221 generates a display image by superimposing a virtual image corresponding to the robot 10 onto an environmental image of the robot 10 acquired by the camera unit 113 of the robot 10.

[0075] Various methods can be applied to generate the display image of the image generation unit 221. For example, if the camera unit 113 includes a 360-degree camera capable of acquiring images of the entire surroundings of the robot 10, the image generation unit 221 may correct the images of the entire surroundings of the robot 10 and generate an environmental image from the corrected images. Alternatively, if the camera unit 113 includes multiple fisheye cameras provided in each direction of the robot 10, the image generation unit 221 may synthesize the images captured by the multiple fisheye cameras provided in the front, back, left, and right of the robot 10 into a seamless omnidirectional image to generate an environmental image that provides an aerial view of the environment around the robot 10 (excluding the robot 10).

[0076] In this case, the environmental image described above does not include an image of the robot 10 on which the camera unit 113 is installed. Therefore, the image generation unit 221 can generate a display image in which both the robot 10 and the environment surrounding the robot 10 can be viewed simultaneously by superimposing a virtual image of the robot 10 at the position corresponding to the robot 10 in the generated environmental image.

[0077] The method for generating environmental images showing the environment around the robot 10 is not limited to the above. For example, environmental images showing the environment around the robot 10 may be generated from images captured by a surveillance camera installed in the space where the robot 10 is located, or from images captured by a webcam connected to capture the front, back, left, and right sides of the robot 10.

[0078] The motion control unit 222 switches the control of the robot 10 from autonomous control to operator control in response to an operation by operator OP that has been notified by the notification control unit 220.

[0079] Various methods can be applied to the method of switching the remote control of the motion control unit 222. For example, when an incident is selected by the operator OP, the operator OP requests the information processing server 30 to connect to the robot 10 where the incident is occurring, and when the operator OP switches to remote control while connected to the robot 10, the control of the robot 10 may be switched from autonomous control to operator control. Alternatively, when an incident is selected by the operator OP, the operator OP requests the information processing server 30 to connect to the robot 10 where the incident is occurring, and the control may be switched from autonomous control to operator control when connected to the robot 10.

[0080] The method of selection by the operator OP may be, for example, based on operation by the control unit 218, based on voice recognition by the microphone 217, or based on gesture operation using the sensor unit 214.

[0081] LAN211 is the network within the robot 10. The control unit 210 and the communication unit 212 are connected via LAN211, and they communicate and send / receive data.

[0082] The communication unit 212 is a communication interface for connecting to network 5. For example, the communication unit 212 may be a communication interface that can connect wirelessly or via wired connection to network 5, or to a base station that can connect to network 5. However, the communication unit 212 may also be a communication interface that can connect to network 5 via a mobile phone network or a wireless LAN.

[0083] The camera unit 213 includes an imaging device capable of acquiring images of the area around the remote control terminal 20. The camera unit 213 may also include, for example, an imaging device that images the operator OP or the area around the operator OP.

[0084] The sensor unit 214 includes at least one sensor capable of sensing the environment surrounding the remote control terminal 20 or various parameters of the operator OP. The sensor unit 214 may acquire sensing information, for example, vital signs such as the operator OP's body temperature, heart rate, blood pressure, or respiratory rate, by the corresponding sensor. The sensor unit 214 may also acquire sensing information regarding the environment surrounding the remote control terminal 20 or the environment surrounding the operator OP from a sensor device provided outside the remote control terminal 20 via the communication unit 212.

[0085] The display unit 215 displays an environmental image generated by the image generation unit 221 included in the control unit 210. The display unit 215 may include a display device such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or a touch panel display. The display unit 215 displays a display image generated by the image generation unit 221 using an environmental image captured by the camera unit 113 of the robot 10. The operator OP can understand the environment around the robot 10 by viewing the display image displayed on the display unit 215.

[0086] Furthermore, the display unit 215 may, in accordance with the control of the notification control unit 220, display a notification screen indicating that the difficulty of execution by autonomous control is high. The notification screen indicating that the difficulty of execution by autonomous control is high may be, for example, a pop-up screen containing information about the incident with the highest index calculated by the risk calculation unit 125 and the robot in which the incident occurred. Alternatively, the notification screen indicating that the difficulty of execution by autonomous control is high may be a display screen sorted in descending order of the index calculated by the risk calculation unit 125.

[0087] The speaker 216 outputs sound to the area around the remote control terminal 20 in accordance with the control of the control unit 210. For example, the speaker 216 may output sound associated with the control by the control unit 210, or, when the robot 10 is being remotely controlled, sound from around the robot 10 picked up by the microphone 117 on the robot 10 to the area around the remote control terminal 20.

[0088] Furthermore, the speaker 216 may output a notification sound indicating that the difficulty of performing the autonomous control is high, in accordance with the control of the notification control unit 220.

[0089] The microphone 217 picks up sounds from the vicinity of the remote control device 102. For example, the microphone 217 may pick up sounds spoken by an operator OP in the vicinity of the remote control terminal 20.

[0090] The control unit 218 includes an input device for receiving operations from the operator OP. The control unit 218 may include, for example, an input device such as a keyboard, mouse, touch panel, trackpad, directional keys, or control lever. With this configuration, the control unit 218 can receive an operation from the operator OP to switch from autonomous control to operator control when the operator OP receives a notification indicating that the difficulty of performing the operation by autonomous control is high.

[0091] The power supply unit 219 supplies power to each part of the remote control terminal 20 from an external power source or an internal power source. For example, the power supply unit 219 may transform the input power and then supply the power input from the external power source to each part of the remote control terminal 20.

[0092] (Information processing server 30) As shown in Figure 2, the information processing server 30 includes a mobile robot operation system unit 310, a total robot management unit 311, a total robot task management unit 312, and a total robot incident management unit 313.

[0093] The mobile robot operation system unit 310 is the device on which the main system of the information processing system 1 operates. The mobile robot operation system unit 310 controls the transmission and reception of information regarding each task of the N robots 10-1, 10-2, ..., 10-N to the all-robot management unit 311. The mobile robot operation system unit 310 also manages various information such as the M remote control terminals 20-1, 20-2, ..., 20-M, and the operators OP-1, OP-2, ..., OP-M who are operating each remote control terminal.

[0094] The All Robot Management Unit 311 manages information on all robots connected to the Information Processing Server 30. For example, the All Robot Management Unit 311 manages information on all robots.

[0095] The All Robot Task Management Unit 312 manages task information for all robots connected to the Information Processing Server 30. For example, the All Robot Task Management Unit 312 manages information related to the operating status of all robots and a list of tasks for all robots.

[0096] The All Robot Incident Management Unit 313 manages incident information for all robots connected to the Information Processing Server 30. For example, the All Robot Incident Management Unit 313 manages a list of incidents that have occurred for each of the robots, and a list of indicators that show the execution risk by autonomous control of robot 10 corresponding to each incident.

[0097] The functional configuration example of the information processing system 1 according to this embodiment has been described above. Next, an example of the operation processing related to remote control of the information processing system 1 according to this embodiment will be described with reference to Figures 3 and 4.

[0098] <<3. Example of Operation Process of the Information Processing System According to This Embodiment>> Figure 3 is a flowchart showing an example of the operation process of the robot 10 in the information processing system 1 according to this embodiment. First, the robot 10 executes the task assigned by the all-robot task management unit 312 of the information processing server 30 through autonomous control (S101).

[0099] Next, the robot 10 determines whether the task has been completed (S102). If the task is completed (S102 / Yes), the autonomous control of the robot 10 according to this embodiment is completed (S114), and the robot 10 terminates processing. On the other hand, if the task is not completed (S102 / No), the robot 10 executes the task autonomously (S103).

[0100] While robot 10 is executing a task under autonomous control (S103), it determines whether or not an event has occurred (S104). If no event has occurred (S104 / No), the process returns to S102. On the other hand, if an event has occurred (S104 / Yes), robot 10 calculates an index indicating the execution risk that has occurred based on the information related to the occurrence of the event (S105). Then, robot 10 identifies the occurred event as an incident and transmits information regarding the occurrence of the incident to the information processing server 30 (S106). At this time, robot 10 continues another task under autonomous control until an operator makes a decision (S107).

[0101] While robot 10 is performing another task under autonomous control (S107), it determines whether or not there is an instruction from the information processing server 30 to respond to an incident under autonomous control (S108). If there is an instruction to respond to an incident under autonomous control (S108 / Yes), robot 10 responds to the incident under autonomous control (S115), sends a message to the information server indicating that the incident response is complete (S116), resumes the task under autonomous control (S113), and returns to 102. On the other hand, if there is no instruction to respond to an incident under autonomous control (S108 / No), the process proceeds to S109.

[0102] If there is no instruction to handle the incident autonomously (S108 / No), the robot 10 determines whether or not there is a connection request with the remote control terminal 20 (S109). If there is no connection request with the remote control terminal 20 (S109 / No), the process returns to S107. On the other hand, if there is a connection request with the remote control terminal 20 (S109 / Yes), the process proceeds to S110.

[0103] If the information processing server 30 requests a connection to the remote control terminal 20 (S109 / Yes), the robot 10 is connected to the remote control terminal 20 by the information processing server 30, interrupts its handling of another task by autonomous control, and once connected, receives remote control input from the remote control terminal 20 to the robot 10 (S110).

[0104] Robot 10 responds to the incident that occurs through remote control by operator OP (S111).

[0105] Once the incident response is completed remotely, the robot 10 disconnects from the remote control terminal 20 (S112).

[0106] Then, robot 10 resumes the task under autonomous control (S113), and the processes from S102 to S113 are repeated until it is determined in S102 that the task has been completed (S102 / Yes).

[0107] The above describes an example of the operation processing of the robot 10 in the information processing system 1 according to this embodiment. Next, referring to Figure 4, an example of the operation processing of the remote control terminal 20 when an incident occurs in S106 shown in Figure 3 will be described.

[0108] Figure 4 is a flowchart showing an example of the operation process of the remote control terminal 20 of the information processing system 1 according to this embodiment. First, the remote control terminal 20 determines whether or not autonomous control of the robot 10 has been completed (S201). If autonomous control of the robot 10 is completed (S201 / Yes), the remote control terminal 20 terminates its processing. On the other hand, if autonomous control of the robot 10 is not completed (S201 / No), the remote control terminal 20 obtains incident information stored in the information processing server 30 from the information processing server 30 (S202).

[0109] The remote control terminal 20 then displays the acquired incident information on its display unit 215 and notifies the operator (S203).

[0110] Then, the remote control terminal 20 determines, based on the operator OP's response, whether or not the incident is one that the operator OP can handle remotely (S204). If the incident is one that the operator OP can handle (S204 / Yes), the process proceeds to S205. If the incident is not one that the operator OP can handle (S204 / No), the remote control terminal 20 instructs the information processing server 30 to have the robot 10 handle the event autonomously (S212), and the process proceeds back to S201.

[0111] If the incident is determined to be one that requires remote intervention by an operator (S204 / Yes), the operator selects the relevant incident (S205).

[0112] The remote control terminal 20 then sends a connection request between the robot 10 and the remote control terminal 20 to the information processing server 30 (S206). The information processing server 30 connects the remote control terminal 20 to the robot 10 where the incident occurred.

[0113] Then, the remote control terminal 20 receives an image of the environment surrounding the robot 10 captured by the camera unit 113 of the robot 10 (S207), and the image generation unit 221 generates a display image by superimposing a virtual image of the robot 10 onto the image of the environment surrounding the robot 10, thereby generating the display image generated by the image generation unit 221 (S208).

[0114] The display unit 215 then displays the display screen generated by the image generation unit 221 (S209).

[0115] Then, the remote control terminal 20 transmits a remote control signal in response to the operator OP's operation via the control unit 218, and remotely controls the robot 10 (S210). As a result, the robot 10 switches from autonomous control to operator OP's remote control, and responds to the incident through operator OP's remote control, as shown in S111.

[0116] Then, once the remote control of operator OP by the operation unit 218 is complete, the remote control terminal 20 disconnects the connection between the remote control terminal 20 and the robot 10 in accordance with the operator OP's operation (S211), and the remote control terminal 20 in this embodiment terminates its processing. Meanwhile, the robot 10 resumes its work under autonomous control, as shown in S113.

[0117] The operation examples of the information processing system 1 according to this embodiment have been described above. Next, the effects and advantages of this embodiment will be explained.

[0118] <<4. Effects of this embodiment>> According to the embodiment described above, various effects and advantages can be obtained. For example, in the remote control terminal 20 of this embodiment, when an incident occurs in the robot 10, the operator OP can choose whether or not to switch from autonomous control to operator control, and the operator OP can select which incidents to control, thereby reducing the burden on the operator OP.

[0119] Furthermore, even if the number of incidents occurring in each of the multiple robots 10 increases, by calculating the execution risk through the autonomous control of the robots, the operator OP can more easily determine which of the multiple incidents should be addressed by the operator OP, especially those that should be handled remotely.

[0120] Furthermore, the notification control unit 220 controls notifications for incidents in which an indicator showing the risk of execution by autonomous control of the robot 10 meets a predetermined standard. As a result, in incidents with a low risk of execution by autonomous control, the robot 10 will respond by autonomous control, which can improve the operational efficiency of the operation center using the information processing system 1.

[0121] <<5. Supplementary Information>> Although preferred embodiments of the present invention have been described in detail above with reference to the attached drawings, the present invention is not limited to these examples. It is clear to any person with ordinary skill in the art to which the present invention belongs that various modifications or alterations can be conceived within the scope of the technical idea described in the claims, and these are also understood to naturally fall within the technical scope of the present invention.

[0122] For example, the information processing device according to the present invention may be the robot 10, the remote control terminal 20, or the information processing server 30 according to this embodiment. Furthermore, the configurations of the robot 10, the remote control terminal 20, and the information processing server 30 according to this embodiment may be those of other devices. For example, the robot 10 may be equipped with a notification control unit 220, and the remote control terminal 20 may be equipped with an event detection unit 124 and a risk calculation unit 125.

[0123] Furthermore, the steps in processing the operation of the robot 10 and the remote control terminal 20 in the information processing system 1 according to this embodiment do not necessarily have to be processed chronologically in the order shown in the explanatory diagram. For example, each step in processing the operation of the remote control terminal 20 may be processed in an order different from the order shown in the explanatory diagram, or may be processed in parallel.

[0124] Furthermore, it is possible to create computer programs that enable the CPU, ROM, RAM, and other hardware components built into the robot 10, remote control terminal 20, and information processing server 30 to perform functions equivalent to those of the aforementioned configurations of the robot 10, remote control terminal 20, and information processing server 30. A computer-readable recording medium containing such computer programs can also be provided. [Explanation of symbols]

[0125] 1. Information Processing System 5 Network 10 Robots OP Operator 110 Robot Control Unit 112 LAN 113 Camera Section 114 Sensor section 115 Display section 116 speakers 117 Mike 118 Robot Communications Department 119 AI Department 120 Mobile Unit 121 Information storage section 122 Battery section 123 Task Management Department 124 Event detection unit 125 Risk Calculation Unit 20 Remote Control Terminals 210 Control Unit 220 Notification Control Unit 221 Image Generation Unit 222 Operation Control Unit 211 LAN 212 Communications Department 213 Camera Section 214 Sensor section 215 Display section 216 speakers 217 Mike 218 Operation section 219 Power supply section 30 Information Processing Server 310 Mobile Systems Department 311 All Robot Management Department 312 All Robot Task Management Department 313 All Robot Incident Management Department

Claims

1. A calculation unit calculates an index indicating the execution risk of each of the multiple mobile bodies operating under autonomous control or operator control, with respect to each event that occurs to the multiple mobile bodies. An incident management unit manages the incident information and the index calculated by the calculation unit, with each of the events that occurred in the plurality of moving objects being treated as an incident. A notification control unit, based on the incident information and indicators managed by the incident management unit, controls notifications to the operator managing the mobile object regarding events in which the indicators meet predetermined criteria, prioritizing those with higher indicators. If an event is selected by the operator who has received the notification, an operation control unit establishes a communication connection between the operator's remote control terminal and the mobile body in which the event is occurring, and switches the control of the mobile body from autonomous control to operator control by further operation of the operator, An information processing device equipped with the following features.

2. The calculation unit described above, Based on the information relating to the aforementioned events and the operating status of the moving body, the index is calculated. The information processing apparatus according to claim 1.

3. The information relating to the operating status of the moving body includes at least one of the following: information about the surrounding environment in which the moving body operates, or information about the overall width of the moving body. The information processing apparatus according to claim 2.

4. The calculation unit described above, The index is calculated based on a calculation model obtained by machine learning, using the information relating to the aforementioned events and the operating status of the moving body, and the index, as training data. The information processing apparatus according to claim 3.

5. The aforementioned index includes at least one of the following with respect to the aforementioned event: an index indicating the difficulty of execution by autonomous control of the mobile body, or an index indicating the degree of loss in the event of failure to execute by autonomous control. The information processing apparatus according to claim 1.

6. Computers, A calculation function that calculates an index indicating the execution risk of each of the multiple mobile bodies operating under autonomous control or operator control, with respect to each event that occurs in the multiple mobile bodies operating under autonomous control, An incident management function manages each of the events that occur in the aforementioned multiple moving objects as an incident, and manages the incident information and the indicator calculated by the calculation function, respectively. Based on the incident information and indicators managed by the incident management function, a notification control function prioritizes notifications to the operator managing the mobile object regarding events in which the indicators meet predetermined criteria, in order of highest indicator value. If an event is selected by the operator who has received the notification, a communication connection is established between the operator's remote control terminal and the mobile object in which the event is occurring, and the control of the mobile object is switched from autonomous control to operator control by further operation of the operator, A program characterized by its ability to function in a certain way.

7. With respect to each event that occurs in a plurality of mobile bodies operating under autonomous control or operator control, an index is calculated that indicates the execution risk of each of the plurality of mobile bodies under autonomous control. Each of the events that occurred in the aforementioned multiple moving objects is treated as an incident, and the information of the incident and the indicator showing the calculated execution risk are managed accordingly. Based on the managed incident information and the indicators, notifications to the operator managing the mobile object regarding events in which the indicators meet predetermined criteria are prioritized in order of the highest indicators, If an event is selected by the operator who has received the notification, a communication connection is established between the operator's remote control terminal and the mobile body in which the event is occurring, and the control of the mobile body is switched from autonomous control to operator control by further operation of the operator. A computer-based information processing method, including [a specific example].

Images