Control device, control method, and program

Abstract

To reliably resume teaching playback traveling suspended due to remote control.SOLUTION: A controller includes a teaching playback traveling control unit that allows a moving body to autonomously travel along a predetermined route, a remote control traveling control unit that changes traveling modes of the moving body from autonomous traveling, which is controlled by the teaching playback traveling control unit, to remote control traveling which is controlled with a remote control terminal, and allows the moving body to travel according to an entry manipulation performed at the remote control terminal, and a return traveling control unit that, when the remote control traveling of the moving body is terminated, allows the moving body to travel to return to a change position, at which the traveling modes are changed from autonomous traveling to remote control traveling, on the basis of an environment map created during the remote control traveling and the traveling route along which the remote control traveling is carried out.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a control device, a control method, and a program. [Background technology]

[0002] In recent years, with the advancement of robot-related technology, autonomously performing robots (hereinafter referred to as mobile robots) and systems for operating and monitoring mobile robots have been developed and sold as a means of solving the social issue of labor shortages caused by population decline.

[0003] In order to operate multiple such mobile vehicles simultaneously at multiple locations, it is being considered to operate the mobile vehicles autonomously under normal circumstances, while having an operator remotely monitor and operate the mobile vehicles when a problem occurs.

[0004] Furthermore, a technology called "taught-to-play driving" has been developed to allow a mobile body to travel autonomously. In taught-to-play driving, a user operates a mobile body to travel a travel route in advance (hereinafter referred to as taught driving), and the mobile body stores an environmental map of the travel route and the travel route, thereby allowing the mobile body to travel autonomously along the travel route stored in the taught driving.

[0005] For example, Patent Document 1 listed below discloses a travel control system that remotely controls a mobile object traveling on a patrol route to leave the patrol route and then returns the mobile object to the patrol route after the remote control is completed. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 8-241123 Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the cruise control system disclosed in Patent Document 1, when remote control of a moving object performing taught playback travel is completed and the moving object returns to a position on the patrol route, no determination is made as to whether the moving object has returned to the position where it transitioned to the remote control state. Therefore, in the cruise control system disclosed in Patent Document 1, it may be difficult to return the moving object to taught playback travel again. This is because, in taught playback travel, the position and speed of the moving object are continuously recorded, and resuming an interrupted taught playback travel requires the moving object to accurately return to the interrupted position.

[0008] Therefore, the present invention has been made in consideration of the above problems, and an object of the present invention is to provide a new and improved control device, control method, and program that can reliably resume teaching playback traveling that was interrupted by remote control by interrupting the teaching playback traveling of a moving body by remote control, and then, when the remote control is completed, the moving body autonomously returns to the position where it was switched to remote control, and determines whether it has returned to the position where it transitioned to the remote control state. [Means for solving the problem]

[0009] In order to solve the above problem, according to one aspect of the present invention, there is provided a teaching playback traveling control unit that causes a moving body to travel autonomously along a predetermined route; a remotely controlled traveling control unit that switches the traveling of the moving body from the autonomous traveling by the teaching playback traveling control unit to remotely controlled traveling by a remote control terminal, and causes the moving body to travel based on input operations from the remote control terminal; a return traveling control unit that, when the remotely controlled traveling of the moving body has ended, causes the moving body to return to a switching position where the autonomous traveling was switched to the remotely controlled traveling based on an environmental map created during the remotely controlled traveling and the traveling route during the remotely controlled traveling; and a resumption determination unit that determines whether the moving body has returned to the switching position. a restart failure notification unit that notifies the remote control terminal of a failure in the return of the moving object when it is determined that the moving object has not returned to the switching position; and wherein the resumption determination unit determines that the moving body has returned to the switching position when a difference between an image acquired by an imaging device mounted on the moving body at the switching position and an image acquired by the imaging device at the current position becomes equal to or less than a threshold value.

[0010] When it is determined that the moving object has returned to the switching position, the teaching playback traveling control unit may cause the moving object to resume the autonomous traveling along the predetermined route.

[0013] When it is determined that the moving object has not returned to the switching position, the return travel control unit may cause the moving object to return to the switching position based on position information of the switching position.

[0015] The predetermined route may be a route that is taught to the moving body in advance.

[0016] Furthermore, in order to solve the above problem, according to another aspect of the present invention, there are provided a method for detecting a vehicle's autonomous driving along a predetermined route, a method for switching the driving of the vehicle from the autonomous driving to remotely controlled driving by a remote control terminal and driving the vehicle based on an input operation from the remote control terminal, a method for returning the vehicle to a switching position where the driving was switched from the autonomous driving to the remotely controlled driving based on an environmental map created during the remotely controlled driving and the driving route during the remotely controlled driving, when the remotely controlled driving of the vehicle has ended, and a method for determining whether the vehicle has returned to the switching position. when it is determined that the moving object has not returned to the switching position, notifying the remote control terminal of a failure to return the moving object; and when a difference between an image acquired by an imaging device mounted on the moving body at the switching position and an image acquired by the imaging device at the current position becomes equal to or less than a threshold value, it is determined that the moving body has returned to the switching position.

[0017] Furthermore, in order to solve the above problem, according to another aspect of the present invention, a computer includes a teaching playback traveling control unit that causes a moving body to travel autonomously along a predetermined route; a remotely controlled traveling control unit that switches the traveling of the moving body from the autonomous traveling by the teaching playback traveling control unit to remotely controlled traveling by a remote control terminal, and causes the moving body to travel based on input operations from the remotely controlled terminal; a return traveling control unit that, when the remotely controlled traveling of the moving body has ended, causes the moving body to return to a switching position where the switching from the autonomous traveling to the remotely controlled traveling was made based on an environmental map created during the remotely controlled traveling and the traveling route during the remotely controlled traveling; and a resumption determination unit that determines whether the moving body has returned to the switching position. a restart failure notification unit that notifies the remote control terminal of a failure in the return of the moving object when it is determined that the moving object has not returned to the switching position; and when a difference between an image acquired by an imaging device mounted on the moving body at the switching position and an image acquired by the imaging device at the current position becomes equal to or less than a threshold value, the resumption determination unit determines that the moving body has returned to the switching position. [Effects of the Invention]

[0018] As described above, according to the present invention, it is possible to reliably resume interrupted teaching playback travel by remote control, thereby further improving the operability of the mobile body operation system. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a block diagram showing a functional configuration of a remote operation system according to a first embodiment of the present invention. [Figure 2] 2 is a block diagram showing detailed functional configurations of a mobile object control unit and a mobile object storage unit according to the first embodiment of the present invention. FIG. [Figure 3] FIG. 3 is a flowchart illustrating the flow of operations of a moving object control unit according to the first embodiment of the present invention. [Figure 4] FIG. 10 is a block diagram showing detailed functional configurations of a mobile object control unit and a mobile object storage unit of a mobile object according to a second embodiment of the present invention. [Figure 5]FIG. 11 is a block diagram showing detailed functional configurations of a mobile object control unit and a mobile object storage unit of a mobile object according to a third embodiment of the present invention. [Figure 6] FIG. 11 is a flowchart illustrating the flow of operations of a moving object control unit according to the third embodiment of the present invention. [Figure 7] 1 is a block diagram showing the functional configuration of a remote operation system including a plurality of mobile objects and a plurality of remote control terminals. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant explanations will be omitted.

[0021] <1. First embodiment> (Overall composition) First, the overall configuration of a remote operation system 10 according to a first embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the functional configuration of the remote operation system 10 according to this embodiment.

[0022] 1, the remote operation system 10 includes a mobile object 100, a remote control terminal 200, and a mobile object operation system unit 300. The mobile object 100, the remote control terminal 200, and the mobile object operation system unit 300 are connected via a network 400 so as to be able to transmit and receive data to and from each other.

[0023] The mobile object 100 is a robot that is capable of autonomous movement and also capable of moving and running under control (so-called remote operation) from the remote control terminal 200. The mobile object 100 is capable of, for example, autonomous movement and running, connection to the mobile object operation system unit 300, notification to the remote control terminal 200, movement and running under remote control from the remote control terminal 200, and data transmission in response to a request from the remote control terminal 200.

[0024] Here, a robot is a mechanical device defined as a so-called "artificial human," "a device that moves or performs tasks similar to those of a human," or "a device that moves or performs tasks by being controlled by a computer based on instructions." A robot may be, for example, a mechanical device defined as "a machine for industrial use that has an automatically controlled manipulation or movement function, can perform various tasks by program, and is capable of being used in industry" (JIS B 0134-1998), "a robot that serves humans" (JIS B 0187:2005), or "an intelligent mechanical system that has three elemental technologies: sensors, an intelligence / control system, and a drive system" (Robot Policy Study Group, Ministry of Economy, Trade and Industry).

[0025] The mobile body 100 of the remote operation system 10 according to this embodiment is used for various purposes or various operations. The mobile body 100 may be used, for example, for security purposes such as patrol surveillance of a facility under security, for inspection of public facilities or public equipment, for monitoring locations such as manufacturing sites (such as inspecting equipment within a factory or transporting manufactured parts or materials) or construction sites (such as transporting building components or checking the status of construction), or for service purposes such as the passenger business field (such as transporting passengers or luggage, guiding tourists to tourist spots, or inspecting luggage), or the logistics business field (such as inspecting cargo or cargo vehicles, or transporting cargo).

[0026] The remote control terminal 200 is a device used by the operator OP when remotely controlling the moving object 100. The remote control terminal 200 may include, for example, a terminal such as a personal computer (PC), a tablet, a microcomputer, or a smartphone, and peripheral devices connected to these terminals. The remote control terminal 200 can output data regarding the status of the moving object 100, instruct the moving object 100 to move based on an input operation by the operator OP, output notifications from the moving object 100, or monitor the status of the operator OP. For example, the remote control terminal 200 can present an image including the moving object 100 and the environment surrounding the moving object 100 to the operator OP, thereby allowing the operator OP to understand the environment surrounding the moving object 100. The operator OP can remotely control the moving object 100 while checking the environment surrounding the moving object 100.

[0027] The network 400 is a communication network that interconnects the mobile object 100, the remote control terminal 200, and the mobile object operation system unit 300. The network 400 allows data to be transmitted and received between the mobile object 100, the remote control terminal 200, and the mobile object operation system unit 300. The network 400 may be, for example, the Internet, a satellite communication network, a mobile communication network, a LAN (Local Area Network), or a WAN (Wide Area Network).

[0028] Furthermore, the communication method between the mobile object 100 and the network 400 and the communication method between the remote control terminal 200 and the network 400 may be the same communication method or may be different communication methods. For example, the mobile object 100 may connect to the network 400 using wireless LAN communication to connect to the network 400 at any location. On the other hand, the remote control terminal 200 may connect to the network 400 using wired LAN communication to connect to the network 400 at the installation location. However, the communication methods between the mobile object 100, the remote control terminal 200, and the mobile object operation system unit 300 and the network 400 are not limited to the above and may be other communication methods.

[0029] The mobile object operation system unit 300 is a device on which a main system that manages the entire remote operation system 10 operates. The mobile object operation system unit 300 manages, for example, information on the mobile object 100, the remote control terminal 200, and the operator OP operating the remote control terminal 200, manages the operation status of the mobile object 100, manages the connection between the mobile object 100 and the remote control terminal 200, and manages operation information of the operator OP. The installation location of the mobile object operation system unit 300 is not limited, and may be, for example, the same location as the remote control terminal 200 or a different location from the remote control terminal 200.

[0030] (Detailed configuration) Furthermore, the detailed configuration of the remote operation system 10 will be described with reference to FIG.

[0031] (Configuration of moving body 100) As shown in FIG. 1, the mobile object 100 includes a mobile object control unit 110, a mobile object memory unit 120, a camera unit 101, a sensor unit 102, a display unit 103, a speaker 104, a microphone 105, a mobile object communication unit 106, an AI unit 107, a battery unit 108, and a movement unit 109.

[0032] The mobile object control unit 110 processes various data acquired by the mobile object 100 and controls the overall operation of the mobile object 100. The mobile object control unit 110 may be configured, for example, by software including a program for controlling the overall operation of the mobile object 100, and hardware on which the software is installed. Examples of hardware on which software is installed include a CPU (Central Processing Unit), RAM (Read Only Memory), and ROM (Read Only Memory). Furthermore, the mobile object control unit 110 may be configured with a DSP (Digital Signal Processor), a microprocessor, an IC (Integrated Circuit), or the like instead of a CPU. A more detailed configuration of the mobile object control unit 110 will be described later with reference to FIG. 2.

[0033] It is also possible to create a program that causes the above-mentioned hardware, such as the CPU, ROM, and RAM, to perform functions equivalent to those of the mobile object control unit 110. It is also possible to provide a computer-readable recording medium on which the program is recorded.

[0034] The mobile object storage unit 120 is a recording device that stores various information and images under the control of the mobile object control unit 110. The mobile object storage unit 120 may store, for example, basic information and map information about the location where the mobile object 100 autonomously moves, environmental map data around the mobile object 100 created by the mobile object control unit 110, and the travel route, information indicating the sensing results by the sensor unit 102, log information of the mobile object control unit 110, information about the shape of the mobile object 100 (e.g., the shape, overall length, overall width, or height of the mobile object 100), or information about the movement characteristics of the mobile object 100 (e.g., the movement speed, movement range, or movable area of ​​the mobile object 100). A more detailed configuration of the mobile object storage unit 120 will be described later with reference to FIG. 2.

[0035] The information stored in the mobile object storage unit 120 (for example, log information of the mobile object control unit 110 stored in the mobile object storage unit 120, information about the shape of the mobile object 100, or information about the movement characteristics of the mobile object 100) may be used in the remote control terminal 200. The information stored in the mobile object storage unit 120 may be used, for example, to display the state of the mobile object 100 on the remote control terminal 200 when remotely controlling the mobile object 100 with the remote control terminal 200, or may be used to set the amount of movement of the mobile object 100 in response to a predetermined amount of input operation on the remote control terminal 200.

[0036] The camera unit 101 includes at least one imaging device capable of acquiring images and videos of the environment surrounding the mobile object 100. The camera unit 101 may include, for example, multiple web cameras or fisheye cameras provided in each direction of the mobile object 100. In such a case, the remote control terminal 200 can generate an environmental image that provides a bird's-eye view of the environment surrounding the mobile object 100 from above by seamlessly synthesizing images and videos from the multiple cameras transmitted from the mobile object 100 in all directions. Alternatively, the camera unit 101 may include a 360-degree camera capable of acquiring environmental images capturing the entire periphery of the mobile object 100.

[0037] However, the camera unit 101 is not limited to the above and may have other configurations. The camera unit 101 may include, for example, a so-called network camera that is installed on the moving object 100 so as to capture images of the front, back, left, and right of the moving object 100.

[0038] The sensor unit 102 includes at least one type of sensor capable of sensing parameters of the environment surrounding the mobile object 100. The sensor unit 102 may include a ToF (Time of flight) sensor, a LiDAR (Light Detection and Ranging), a millimeter wave radar, or the like, which are capable of creating a map of the surrounding environment and estimating the self-position of the mobile object 100.

[0039] The sensor unit 102 may further include sensors capable of sensing, for example, temperature, humidity, illuminance, atmospheric pressure, vibration, position information, the distance from the mobile body 100 to an object, or changes in the inclination of the mobile body 100, and may additionally include sensors capable of sensing the generation of smoke, chemical substances, static electricity, etc. Note that the term "sensing" in the above includes not only detecting various parameters of the environment around the mobile body 100, but also further measuring, discriminating, and analyzing the detected parameters.

[0040] There is no particular limitation on the type of sensor included in the sensor unit 102. The sensor unit 102 may include, for example, one or more of a thermometer, a hygrometer, an illuminance meter, a barometer, or a thermal image measuring device (thermography camera).

[0041] Furthermore, the sensor unit 102 may acquire sensing information about the environment surrounding the mobile object 100 from a sensor device provided outside the mobile object 100 via the mobile object communication unit 106. In such a case, the sensor unit 102 may acquire the sensing information from the external sensor device by, for example, 920 MHz band multi-hop wireless communication.

[0042] In this embodiment, the moving object 100 is described as having both the camera unit 101 and the sensor unit 102, but this embodiment is not limited to this example. The moving object 100 may, for example, have only the camera unit 101 or only the sensor unit 102. When the moving object 100 has only the camera unit 101, the camera unit 101 may include at least two or more imaging devices. Furthermore, when the moving object 100 has only the sensor unit 102, the sensor unit 102 may include at least two or more sensors.

[0043] The display unit 103 outputs various information as images or the like in accordance with the control of the mobile object control unit 110. The display unit 103 may include, for example, a liquid crystal display (LCD) device that outputs various information as images, an OLED (organic light emitting diode) display device (organic EL display device), a touch panel display device, or a lamp that outputs various information by emitting light.

[0044] The speaker 104 outputs sound around the moving object 100 in accordance with the control of the moving object control unit 110. The speaker 104 may output, for example, sound effects corresponding to the movement of the moving object 100, synthesized voices corresponding to the movement of the moving object 100, or voices input by an operator OP, around the moving object 100.

[0045] The microphone 105 collects sounds around the moving object 100. The microphone 105 may collect, for example, the voices of people around the moving object 100 or environmental sounds around the moving object 100.

[0046] The mobile communication unit 106 is a communication interface for connecting to the network 400. The mobile communication unit 106 may be, for example, a communication interface that can be connected wirelessly or by wire to the network 400 or a base station that can be connected to the network 400, or may be a communication interface that can be connected to the network 400 via a network such as a mobile phone communication network or a wireless LAN.

[0047] The AI ​​unit 107 recognizes the environment around the moving object 100 based on an image or video of the environment around the moving object 100 captured by the camera unit 101 and sensing information obtained by sensing each parameter of the environment around the moving object 100 by the sensor unit 102, and makes a judgment and decision according to the recognized environment. The AI ​​unit 107 may recognize the environment around the moving object 100 from the environmental image and sensing information by using, for example, a machine learning algorithm, and make a judgment and decision according to the recognized environment. Note that the result of the judgment by the AI ​​unit 107 may be used to set the amount of movement of the moving object 100 in response to a predetermined amount of input operation on the remote control terminal 200.

[0048] The battery unit 108 supplies power to each component of the mobile object 100. The battery unit 108 may include a secondary battery such as a lithium-ion secondary battery.

[0049] The moving unit 109 is a moving mechanism capable of moving the moving body 100 to any position based on control by the moving body control unit 110. The moving unit 109 may be, for example, a moving mechanism of various types such as a wheel type, a leg type, a crawler type, or an air cushion type, a moving mechanism capable of three-dimensional movement, a moving mechanism capable of moving in the air using rotors or the like, or a moving mechanism capable of moving on or underwater using a screw or the like.

[0050] (Configuration of remote control terminal 200) The remote control terminal 200 includes a camera unit 201 , a sensor unit 202 , a display unit 203 , a speaker 204 , a microphone 205 , an operation unit 206 , a remote control terminal communication unit 207 , a power supply unit 208 , and a remote control terminal control unit 210 .

[0051] The camera unit 201 includes an imaging device capable of acquiring images or videos of the surroundings of the remote control terminal 200. The camera unit 201 may include, for example, one or more web cameras, fisheye cameras, or 360-degree cameras that are provided in the remote control terminal 200 and capture images of the operator OP or the surroundings of the operator OP, and that can acquire environmental images capturing the entire surroundings of the remote control terminal 200.

[0052] However, the camera unit 201 is not limited to the above and may have other configurations. The camera unit 201 may include, for example, a network camera arranged to capture an image of the operator OP of the remote control terminal 200 or the area around the operator OP.

[0053] The sensor unit 202 includes at least one type of sensor capable of sensing the environment around the remote control terminal 200 or each parameter of the operator OP. The sensor unit 202 may include a sensor capable of sensing vital signs of the operator OP, such as body temperature, heart rate, blood pressure, or respiratory rate. The sensor unit 202 may also acquire sensing information on the environment around the remote control terminal 200 or the environment around the operator OP from a sensor device provided outside the remote control terminal 200 via the remote control terminal communication unit 207.

[0054] The display unit 203 displays an image of the environment around the moving object 100. The display unit 203 may include a display device such as a liquid crystal display device, an OLED display device (organic EL display device), or a touch panel display device. The display unit 203 displays an image or video generated by the remote control terminal control unit 210 using an environmental image captured by the camera unit 101 of the moving object 100, as well as various information about the moving object 100. The operator OP can grasp the environment around the moving object 100 by visually checking the display image displayed on the display unit 203.

[0055] When the camera unit 101 includes a 360-degree camera, the display unit 203 can display an image of the entire surroundings of the moving object 100. When the camera unit 101 includes multiple fisheye cameras provided in each direction of the moving object 100, the display unit 203 can display an omnidirectional environmental image (an image of the environment around the moving object 100 viewed from above) that is a seamless combination of images captured by the multiple fisheye cameras.

[0056] The speaker 204 outputs sound around the remote control terminal 200 in accordance with the control of the remote control terminal control unit 210. The speaker 204 may output, for example, sound related to the control of the remote control terminal control unit 210 or sound around the mobile object 100 picked up by the microphone 105 of the mobile object 100 while the mobile object 100 is being remotely controlled.

[0057] The microphone 205 picks up sounds around the remote control terminal 200. For example, the microphone 205 may pick up a voice spoken by an operator OP around the remote control terminal 200.

[0058] The operation unit 206 is an input device for receiving operations from an operator OP. The operation unit 206 can receive input operations from the operator OP that instruct the direction and speed of movement of the moving body 100. The operation unit 206 may include input devices such as a keyboard, a mouse, a touch panel, a trackpad, a cross key, a joypad, a controller, or an operation lever.

[0059] The remote control terminal communication unit 207 is a communication interface for connecting to the network 400. The remote control terminal communication unit 207 may be, for example, a communication interface that can be connected to the network 400 or a base station wirelessly or by wire, or may be a communication interface that can be connected to the network 400 via a network such as a mobile phone communication network or a wireless LAN.

[0060] The power supply unit 208 supplies power from an external power supply or a built-in power supply to each unit of the remote control terminal 200. For example, the power supply unit 208 may transform the power input from the external power supply and then supply the power to each unit of the remote control terminal 200.

[0061] The remote control terminal control unit 210 processes various data acquired by the remote control terminal 200 and controls the overall operation of the remote control terminal 200. The remote control terminal control unit 210 may be configured, for example, by software including a program that controls the overall operation of the remote control terminal 200, and hardware on which the software is installed. Examples of hardware on which software is installed include terminals such as a personal computer, a tablet, a microcomputer, or a smartphone.

[0062] (Configuration of the mobile object control unit 110 and the mobile object storage unit 120) Next, the detailed configuration of the mobile object control unit 110 and the mobile object storage unit 120 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the detailed functional configuration of the mobile object control unit 110 and the mobile object storage unit 120.

[0063] As shown in Figure 2, the mobile object control unit 110 includes a SLAM (Simultaneous Localization and Mapping) unit 111, a driving state control unit 112, a teaching driving control unit 113, a teaching playback driving control unit 114, a remote control receiving unit 115, a remote control driving control unit 116, a return driving control unit 117, and a resumption determination unit 118.

[0064] The SLAM unit 111 creates environmental map data around the mobile object 100 and performs SLAM to estimate the mobile object 100's own position. The SLAM unit 111 may simultaneously create environmental map data around the mobile object 100 and estimate the mobile object 100's own position using, for example, an image of the surrounding environment captured by the camera unit 101 or distance measurement information of the surrounding environment sensed by the sensor unit 102. The mobile object 100 can move autonomously based on the environmental map data created by the SLAM unit 111 and the estimated own position. In this embodiment, the SLAM unit 111 is described as using either the camera unit 101 or the sensor unit 102, but the present embodiment is not limited to this example. The SLAM unit 111 may use both the camera unit 101 and the sensor unit 102. In such a case, the camera unit 101 includes at least one imaging device, and the sensor unit 102 includes at least one sensor.

[0065] The traveling state control unit 112 controls the state of the traveling control of the moving body 100 to any one of taught traveling, taught playback traveling, remotely controlled traveling, and return traveling. The traveling state control unit 112 may control the state of the traveling control of the moving body 100 based on an instruction from the remote control terminal 200 or a determination by the resumption determination unit 118.

[0066] When the state of the travel control of the moving body 100 is taught travel, the operation of the moving body 100 is controlled by the taught travel control unit 113. When the state of the travel control of the moving body 100 is taught playback travel, the operation of the moving body 100 is controlled by the taught playback travel control unit 114. When the state of the travel control of the moving body 100 is remotely controlled travel, the operation of the moving body 100 is controlled by the remotely controlled travel control unit 116. When the state of the travel control of the moving body 100 is return travel, the operation of the moving body 100 is controlled by the return travel control unit 117.

[0067] When the mobile object 100 is in a teaching travel state, the teaching travel control unit 113 causes the mobile object 100 to travel under the control of the remote control terminal 200, etc., and also causes the SLAM unit 111 to create environmental map data during teaching travel and store the travel route. The created environmental map data and travel route are stored in the mobile object storage unit 120.

[0068] When the mobile body 100 is in a teaching playback driving state, the teaching playback driving control unit 114 causes the mobile body 100 to drive so as to follow the driving route during teaching driving, using the environmental map data created during teaching driving by the teaching driving control unit 113 and the stored driving route (i.e., causes the mobile body 100 to drive in teaching playback).

[0069] When the moving body 100 is in a remotely controlled traveling state, the remote control receiving unit 115 receives control information for the moving body 100 from the remote control terminal 200, and outputs the control information for controlling the traveling of the moving body 100 to the remote control traveling control unit 116. In addition, the remote control receiving unit 115 outputs information received from the remote control terminal 200 instructing the start or end of remotely controlled traveling to the traveling state control unit 112.

[0070] When the mobile object 100 is in a remotely controlled traveling state, the remotely controlled traveling control unit 116 causes the mobile object 100 to travel based on control information for controlling the traveling of the mobile object 100 received by the remotely controlled traveling receiving unit 115. The remotely controlled traveling control unit 116 also stores the position information of the mobile object 100 when the traveling state of the mobile object 100 switches from taught-replay traveling to remotely controlled traveling in the mobile object storage unit 120, and causes the SLAM unit 111 to create environmental map data during remotely controlled traveling and store the traveling route. The created environmental map data and traveling route are stored in the mobile object storage unit 120 and are used to return the mobile object 100 to the position where it switched from taught-replay traveling to remotely controlled traveling after the remotely controlled traveling ends.

[0071] When the moving body 100 is in the return traveling state, the return traveling control unit 117 causes the moving body 100 to travel to a position where the teaching playback traveling has been switched to remotely controlled traveling, using the environmental map data created during remotely controlled traveling by the remotely controlled traveling control unit 116 and the stored traveling route. In this way, when the remotely controlled traveling by the remote operation terminal 200 is completed, the return traveling control unit 117 can cause the moving body 100 to return to the position where the teaching playback traveling was interrupted.

[0072] When the moving body 100 in the return travel state completes its return travel, the resume determination unit 118 determines whether the position information of the moving body 100 stored in the moving body storage unit 120 at the time of switching from taught playback travel to remotely controlled travel matches the position information of the moving body 100 at the time control by the return travel control unit 117 is completed, and outputs the determination result to the travel state control unit 112. If the two match, the travel state control unit 112 sets the travel control state of the moving body 100 to taught playback travel and causes the moving body 100 to resume the suspended taught playback travel. On the other hand, if the two do not match, the travel state control unit 112 leaves the travel control state of the moving body 100 as return travel and causes the return travel control unit 117 to retry traveling the moving body 100 to the position where the travel was switched from taught playback travel to remotely controlled travel.

[0073] The moving object storage unit 120 includes an environmental map data storage unit 121, a teaching data storage unit 122, a traveling route storage unit 123, and a switching position storage unit .

[0074] The environmental map data storage unit 121 accumulates and stores the environmental map data created by the SLAM unit 111. The teaching data storage unit 122 accumulates and stores the traveling route of the mobile body 100 during teaching traveling. The traveling route storage unit 123 accumulates and stores the traveling route of the mobile body 100 during remotely controlled traveling. The switching position storage unit 124 accumulates and stores position information when the mobile body 100 is switched from teaching playback traveling to remotely controlled traveling.

[0075] (Example of operation) Next, an example of the operation of the mobile object control unit 110 in the remote operation system 10 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart illustrating the flow of the operation of the mobile object control unit 110 according to this embodiment.

[0076] In the remote operation system 10, first the mobile object operation system unit 300 starts up, and then the mobile object 100 and the remote operation terminal 200 start up.

[0077] As shown in FIG. 3, first, the traveling state of the moving body 100 is determined (S101).

[0078] When the traveling state is a teaching traveling state (S101 / teaching traveling state), the mobile body 100 starts teaching traveling under the control of the remote control terminal 200 or the like (S121), and creates environmental map data for the teaching traveling and stores the traveling route until the teaching traveling ends (S122 / Yes). When the teaching traveling ends (S122 / Yes), the mobile body 100 stores the created environmental map data and the stored traveling route in the mobile body storage unit 120 (S123), travels to the initial position (S124), and then ends the operation.

[0079] On the other hand, if the traveling state is the taught playback traveling state (S101 / taught playback traveling), the mobile body 100 checks whether the environmental map data and traveling route during the taught traveling are stored in the mobile body storage unit 120 (S102). If the environmental map data and traveling route during the taught traveling are not stored (S102 / No), the mobile body 100 travels to its initial position (S124) and then terminates its operation. If the environmental map data and traveling route during the taught traveling are stored (S102 / Yes), the mobile body 100 starts taught playback traveling using the stored environmental map data and traveling route (S103). The mobile body 100 travels so as to follow the traveling route during the taught traveling until the taught playback traveling is completed (S104 / Yes).

[0080] If the moving body 100 receives a request to start remotely controlled traveling from the remote control terminal 200 during taught-reproduction traveling (S105 / Yes), the moving body 100 switches the traveling state from taught-reproduction traveling to remotely controlled traveling. The moving body 100 also stores the position information at the time of switching from taught-reproduction traveling to remotely controlled traveling, and starts creating environmental map data and storing the traveling route (S106). Thereafter, the moving body 100 starts remotely controlled traveling based on the control command received from the remotely controlled terminal 200 (S107). The moving body 100 continues creating environmental map data and storing the traveling route during remotely controlled traveling until remotely controlled traveling is completed (S108 / Yes).

[0081] If the remotely controlled traveling is completed (S108 / Yes), the moving body 100 ends the creation of the environmental map data and the storage of the traveling route (S109). After that, the moving body 100 performs the return traveling to the position where the teaching playback traveling was switched to the remotely controlled traveling (hereinafter referred to as the switching position) using the environmental map data created during the remotely controlled traveling and the stored traveling route (S110).

[0082] When the return travel is completed, the moving object 100 determines whether or not it has returned to the switching position (S111). The determination of whether or not the moving object 100 has returned to the switching position can be made based on the captured image or video by the camera unit 101, or sensing information by the sensor unit 102. For example, the moving object 100 may determine that it has returned to the switching position when the difference between the position information of the moving object 100 stored when switching from taught playback travel to remotely controlled travel and the position information of the moving object 100 after the return travel is equal to or less than a threshold. Furthermore, the moving object 100 may determine that it has returned to the switching position when the difference between the image captured by the camera unit 101 when switching from taught playback travel to remotely controlled travel and the image captured by the camera unit 101 after the return travel is equal to or less than a threshold. Furthermore, the moving object 100 may determine whether or not it has returned to the switching position based on sensing information by multiple sensors included in the sensor unit 102.

[0083] If it is determined that the moving body 100 has returned to the switching position (S111 / Yes), the moving body 100 ends the return travel and resumes the taught playback travel (S113). If it is determined that the moving body 100 has not returned to the switching position (S111 / No), the moving body 100 moves to the position indicated by the position information of the moving body 100 that was stored when the moving body 100 switched from taught playback travel to remotely controlled travel (S112). Thereafter, the moving body 100 ends the return travel and resumes the taught playback travel (S113).

[0084] As described above, according to this embodiment, when the mobile body 100 switches from taught playback traveling to remotely controlled traveling, the mobile body 100 can return to the switching position based on the environmental map data and the traveling route during remotely controlled traveling. Furthermore, the mobile body 100 determines whether or not it has reached the switching position by returning, and if it has not returned to the switching position, it can return to the switching position by moving to the position indicated by the position information of the switching position. This allows the mobile body 100 to reliably resume taught playback traveling, further improving the operability of the remote operation system 10.

[0085] 2. Second embodiment (Configuration of mobile object control unit 110A and mobile object storage unit 120A) A remote operation system 10 according to a second embodiment of the present invention will be described with reference to Fig. 4. Fig. 4 is a block diagram showing detailed functional configurations of a mobile object control unit 110A and a mobile object storage unit 120A of a mobile object 100 according to this embodiment.

[0086] As shown in FIG. 4, the second embodiment differs from the first embodiment in that the mobile object control unit 110A further includes a terminal image receiving unit 501, and the mobile object storage unit 120A includes a switching position image storage unit 502 instead of the switching position storage unit 124.

[0087] The terminal image receiving unit 501 receives image data generated by the remote control terminal 200. The image data generated by the remote control terminal 200 is, for example, an omnidirectional image of the mobile object 100 (an image overlooking the environment around the mobile object 100 from above) that is a seamless combination of images captured by multiple fisheye cameras provided in each direction of the mobile object 100. The image data received by the terminal image receiving unit 501 is accumulated and held, for example, in the switching position image holding unit 502 of the mobile object storage unit 120A.

[0088] When switching from taught playback traveling to remotely controlled traveling, the remotely controlled traveling control unit 116 receives image data generated by the remotely controlled terminal 200 at the terminal image receiving unit 501 and stores it in the switching position image holding unit 502 of the moving object storage unit 120A. In this way, the resumption determination unit 118 can determine whether the moving object 100 has returned to the switching position by comparing the image data generated by the remotely controlled terminal 200 when switching from taught playback traveling to remotely controlled traveling with the image data generated by the remotely controlled terminal 200 when returning traveling is completed.

[0089] (Example of operation) Next, an example of the operation of the mobile object control unit 110A in the remote operation system 10 according to the second embodiment will be described with reference to FIG.

[0090] 3, when the mode is switched from taught playback traveling to remotely controlled traveling, the moving body 100 receives image data generated by the remote control terminal 200 and stores the received image data in the switching position image storage unit 502. As a result, the switching position image storage unit 502 stores image data that was captured at the position where the mode was switched from taught playback traveling to remotely controlled traveling and then synthesized by the remote control terminal 200.

[0091] 3, the moving body 100 determines that it has returned to the switching position if the difference between the image data received from the remote control terminal 200 when switching from taught playback traveling to remotely controlled traveling and the image data received from the remote control terminal 200 when returning traveling is completed is equal to or less than a threshold. For example, the moving body 100 may determine whether the difference between the image data received when switching from taught playback traveling to remotely controlled traveling is equal to or less than a threshold by comparing the image data received when switching from taught playback traveling to remotely controlled traveling with the image data received when returning traveling is completed on a pixel-by-pixel basis.

[0092] Also, in step S112 of Figure 3, if it is determined that the moving body 100 has not returned to the switching position (S111 / No), the moving body 100 may move to a position where the image data received from the remote control terminal 200 matches the image data received from the remote control terminal 200 when the moving body 100 was switched from taught playback driving to remote control driving.

[0093] As described above, according to this embodiment, the moving body 100 can determine whether or not it has returned to the switching position where the teaching playback traveling was switched to the remotely controlled traveling, based on image data generated by the remote control terminal 200. This allows the moving body 100 to determine with higher accuracy whether or not it has returned to the switching position, thereby enabling the teaching playback traveling to be reliably resumed, thereby further improving the operability of the remote operation system 10.

[0094] 3. Third Embodiment (Configuration of the mobile object control unit 110B and the mobile object storage unit 120) A remote operation system 10 according to a third embodiment of the present invention will be described with reference to Fig. 5. Fig. 5 is a block diagram showing detailed functional configurations of a mobile object control unit 110B and a mobile object storage unit 120 of a mobile object 100 according to this embodiment.

[0095] As shown in FIG. 5, the third embodiment differs from the first embodiment in that a mobile object control unit 110B further includes a restart failure notification unit 503.

[0096] When the resumption determination unit 118 determines that the moving body 100 has not returned to the switching position where the teaching playback traveling was switched to remotely controlled traveling, the restart failure notification unit 503 notifies the remote control terminal 200 that the teaching playback traveling has not resumed. Furthermore, the traveling state control unit 112 switches the state of the traveling control of the moving body 100 from returning traveling to remotely controlled traveling, thereby enabling the moving body 100 to travel to the switching position by remote control from the remote control terminal 200. Note that in the remotely controlled traveling for making the moving body 100 travel to the switching position, environmental map data for the remotely controlled traveling is not created, and the traveling route is not stored.

[0097] Next, an example of the operation of the mobile object control unit 110B in the remote operation system 10 according to this embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart illustrating the flow of the operation of the mobile object control unit 110B according to this embodiment.

[0098] 3, the moving body 100 stops the teaching playback traveling and starts the remotely controlled traveling (S107), and then completes the remotely controlled traveling (S108 / Yes). After the remotely controlled traveling is completed, the moving body 100 finishes creating the environmental map data and storing the traveling route (S109).

[0099] 6, the moving body 100 performs return travel to the switching position where the taught playback travel was switched to remotely controlled travel, using the environmental map data created during remotely controlled travel and the stored travel route (S110). When the return travel is completed, the moving body 100 determines whether or not it has returned to the switching position (S111). When it is determined that the moving body 100 has returned to the switching position (S111 / Yes), the moving body 100 ends the return travel and resumes taught playback travel (S113). When it is determined that the moving body 100 has not returned to the switching position (S111 / No), the moving body 100 moves to the position indicated by the position information of the moving body 100 stored when the taught playback travel was switched to remotely controlled travel (S112).

[0100] The moving body 100 further determines whether the moving body 100 has returned to the switching position by movement based on the position information of the moving body 100 stored when switching from taught playback traveling to remotely controlled traveling (S114). If it is determined that the moving body 100 has returned to the switching position (S114 / Yes), the moving body 100 ends the returning traveling and resumes taught playback traveling (S113).

[0101] If it is determined that the moving body 100 has not returned to the switching position (S114 / No), the moving body 100 notifies the remote control terminal 200 via the restart failure notifying unit 503 that the taught playback traveling has not been restarted (S115). Thereafter, the moving body 100 switches the traveling control state of the moving body 100 to remotely controlled traveling (S116), thereby transitioning to a state in which it can accept control from the remote control terminal 200.

[0102] When the operator OP, who is monitoring the moving object 100 with the remote control terminal 200, confirms on the remote control terminal 200 a notification from the moving object 100 that the taught playback traveling has not resumed, the operator OP requests the moving object operation system unit 300 to connect to the moving object 100. This allows the operator OP to remotely control the moving object 100 to travel via the remote control terminal 200 connected to the moving object 100. The remotely controlled moving object 100 starts remotely controlled traveling based on the control command received from the remote control terminal 200 (S117). The moving object 100, which has returned to the switching position by remotely controlled traveling using the remote control terminal 200, can resume taught playback traveling after the remotely controlled traveling is completed (S118 / Yes) (S113).

[0103] As described above, according to this embodiment, if the moving body 100 is unable to return to the switching position by the return travel, the moving body 100 transmits a notification to the remote control terminal 200 that the taught-reproduction travel has not been resumed. This allows the moving body 100 to return to the switching position and resume the taught-reproduction travel by remote control of the operator OP operating the remote control terminal 200. Therefore, the moving body 100 can suppress failure to resume taught-reproduction travel that was interrupted by remote-controlled travel, thereby further improving the operability of the remote operation system 10.

[0104] Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person skilled in the art to which the present invention pertains can conceive of various modifications and alterations within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present invention.

[0105] 7, the remote operation system 10 may include, for example, N mobile objects 100-1, 100-2, ..., 100-N, M remote control terminals 200-1, 200-2, ..., 200-M, and a mobile object operation system unit 300. M and N are any integers independent of each other and equal to or greater than 2. However, in order to efficiently operate the mobile objects 100 using the remote control terminals 200, M may be an integer smaller than N.

[0106] N mobile objects 100-1, 100-2, ..., 100-N are connected to M remote control terminals 200-1, 200-2, ..., 200-M via network 400 so as to be able to send and receive data to and from each other. M remote control terminals 200-1, 200-2, ..., 200-M are operated by M operators OP-1, OP-2, ..., OP-M, respectively, and can remotely control each of the mobile objects 100.

[0107] According to this configuration, the remote operation system 10 can, for example, operate and monitor a plurality of moving bodies 100 simultaneously, thereby improving the operating efficiency of the moving bodies 100. [Explanation of symbols]

[0108] 10 Remote Operation System 100 Mobile 101 Camera Club 102 Sensor unit 103 Display section 104 Speaker 105 Microphone 106 Mobile Communications Department 107 AI Department 108 Battery section 109 Mobile Division 110, 110A, 119B Mobile control unit 111 SLAM Department 112 Running condition control unit 113 Teaching travel control unit 114 Teaching playback travel control unit 115 Remote control receiver 116 Remotely controlled driving control unit 117 Return driving control unit 118 Resumption judgment section 120,120A Mobile memory unit 121 Environmental map data storage unit 122 Teaching data storage unit 123 Route Maintenance Unit 124 Switching position holding unit 200 Remote Control Terminal 300 Mobile Operation Systems Department 400 Network 501 Terminal image receiving unit 502 Switching position image storage unit 503 Resume failure notification section

Claims

1. a teaching playback travel control unit that causes the moving body to travel autonomously along a predetermined route; a remote control travel control unit that switches the travel of the moving body from the autonomous travel by the teaching playback travel control unit to remote control travel by a remote control terminal, and causes the moving body to travel based on an input operation from the remote control terminal; a return travel control unit that, when the remotely controlled traveling of the moving body is completed, causes the moving body to return to a switching position where the autonomous traveling has been switched to the remotely controlled traveling, based on the environmental map created during the remotely controlled traveling and the traveling route during the remotely controlled traveling; a restart determination unit that determines whether the moving object has returned to the switching position; a restart failure notification unit that notifies the remote control terminal of a failure in the return of the moving object when it is determined that the moving object has not returned to the switching position; Equipped with The control device, wherein the resumption determination unit determines that the moving body has returned to the switching position when the difference between the image acquired by the imaging device mounted on the moving body at the switching position and the image acquired by the imaging device at the current position becomes less than or equal to a threshold value.

2. The control device according to claim 1 , wherein, when it is determined that the moving object has returned to the switching position, the teaching playback travel control unit causes the moving object to resume the autonomous travel along the predetermined route.

3. The control device according to claim 1 , wherein, when it is determined that the moving object has not returned to the switching position, the return travel control unit causes the moving object to return to the switching position based on position information of the switching position.

4. 4. The control device according to claim 1, wherein the predetermined route is a route that is taught to the moving body in advance.

5. A step of autonomously traveling the mobile object along a predetermined route; switching the traveling of the moving body from the autonomous traveling to remotely controlled traveling by a remote control terminal, and causing the moving body to travel based on an input operation from the remote control terminal; When the remotely controlled traveling of the moving body is completed, returning the moving body to a switching position where the autonomous traveling was switched to the remotely controlled traveling based on the environmental map created during the remotely controlled traveling and the traveling route during the remotely controlled traveling; determining whether the moving object has returned to the switching position; when it is determined that the moving object has not returned to the switching position, notifying the remote control terminal of a failure to return the moving object; Including, A control method using a computing device, in which if the difference between an image acquired by an imaging device mounted on the moving body at the switching position and an image acquired by the imaging device at the current position becomes less than a threshold value, it is determined that the moving body has returned to the switching position.

6. Computer, a teaching playback travel control unit that causes the moving body to travel autonomously along a predetermined route; a remote control travel control unit that switches the travel of the moving body from the autonomous travel by the teaching playback travel control unit to remote control travel by a remote control terminal, and causes the moving body to travel based on an input operation from the remote control terminal; a return travel control unit that, when the remotely controlled traveling of the moving body is completed, causes the moving body to return to a switching position where the autonomous traveling has been switched to the remotely controlled traveling, based on the environmental map created during the remotely controlled traveling and the traveling route during the remotely controlled traveling; a restart determination unit that determines whether the moving object has returned to the switching position; a restart failure notification unit that notifies the remote control terminal of a failure in the return of the moving object when it is determined that the moving object has not returned to the switching position; It functions as A program that causes the resumption determination unit to determine that the moving body has returned to the switching position when a difference between an image acquired by an imaging device mounted on the moving body at the switching position and an image acquired by the imaging device at a current position becomes equal to or less than a threshold value.

Images