Remote control device, remote control method, and program

Abstract

To provide a remote control device, a remote control program, and a remote control method that can further improve the usability of operators who remotely control mobile bodies.SOLUTION: A remote control device includes: an image generation unit that generates a display image in which a virtual image corresponding to a mobile body is superimposed on an environmental image around the mobile body to be controlled; and a communication unit that transmits information about input operation from an operator of the mobile body to the mobile body. The image generation unit controls a display range of the environmental image in the display image based on a movement amount of the mobile body corresponding to the input operation of a predetermined amount.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a remote control device, a remote control method, and a program. [Background technology]

[0002] In recent years, with the advancement of robot-related technologies, the development of systems for operating robots that perform tasks autonomously is underway. Such systems are expected to be a means of resolving labor shortages caused by the declining population.

[0003] For example, technology has been developed that allows robots to operate autonomously using technology such as artificial intelligence (AI). The following Patent Document 1 discloses a system that allows a robot to move autonomously within a building and between buildings by using map data within the building and between buildings.

[0004] On the other hand, currently, events that are difficult for a robot to handle or judge may occur while the robot is working. In such cases, the robot sends a notification to a remote operator requesting operation. The operator confirms the notification and switches the robot that sent the notification from an autonomous operating state to a state where it can be manually operated. In this way, the robot resolves the event that has occurred by being manually operated by the operator. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-3602 A Summary of the Invention [Problem to be solved by the invention]

[0006] When an operator (hereinafter also referred to as a manipulator) operates multiple robots (hereinafter also referred to as mobile objects) with different movement characteristics (e.g., movement speed, movement range, movable environment, etc.), it is desirable for the operator to operate the mobile object after grasping the movement characteristics of the mobile object to be operated. However, it is difficult for the operator to grasp all of the movement characteristics of multiple mobile objects.

[0007] In addition, an operator may switch between moving objects with significantly different movement characteristics. In such a case, the operation feel of the moving object after switching is completely different from that of the moving object before switching, and the moving characteristics are not immediately grasped, so that the operator may be slow to respond to manual operation of the moving object or may operate the moving object incorrectly.

[0008] Therefore, it has been difficult for the operator to remotely control the moving object efficiently.

[0009] Therefore, the present invention has been made in consideration of the above problems, and an object of the present invention is to provide a remote control device that enables an operator to operate a mobile object at a remote location, and that can further improve the usability for the operator remotely operating the mobile object. [Means for solving the problem]

[0010] In order to solve the above problem, according to one aspect of the present invention, Multiple objects with different movement characteristics of mobile Each an image generating unit that generates a display image by superimposing a virtual image corresponding to the moving object on a surrounding environment image; Multiple objects with different movement characteristics to be operated Mobile Each a communication unit that transmits information regarding an input operation from an operator to the mobile body, and the image generating unit generates the image corresponding to a predetermined amount of the input operation. Multiple objects with different movement characteristics to be operated Mobile Each Based on the amount of movement, When the movement amounts of the plurality of moving objects having different movement characteristics to be operated are different, the difference between the moving objects is reduced. A remote control device is provided for controlling a display range of the environmental image on the display image.

[0011] The image generation unit may expand the display range of the environmental image as the amount of movement of the moving body corresponding to the specified amount of the input operation is greater, and may reduce the display range of the environmental image as the amount of movement of the moving body corresponding to the specified amount of the input operation is smaller.

[0012] The image generating unit, in the display image, The above different Has mobility properties The size of the display range of the environmental image may be controlled so that the amount of movement of the moving body in response to the predetermined amount of the input operation is substantially the same among a plurality of moving bodies.

[0013] The image generating section may generate the environmental image overlooking the surroundings of the moving body by synthesizing images captured by a plurality of imaging devices mounted on the moving body.

[0014] The amount of movement of the moving object in response to the predetermined amount of the input operation may be set based on information regarding movement characteristics of the moving object.

[0015] The information regarding the movement characteristics may be obtained from the moving object.

[0016] The information on the movement characteristics may be information indicating the movement speed, movement range, movable area, shape, overall length, overall width, or height of the moving object.

[0017] The image generating section may generate the display image by further superimposing an image indicating the movement characteristic on the environmental image.

[0018] The image indicating the movement characteristic may be an image indicating information regarding an amount of movement of the moving object corresponding to the predetermined amount of the input operation.

[0019] The remote control device is The above different Has mobility properties The multiple moving objects may each be remotely controlled.

[0020] In order to solve the above problem, according to another aspect of the present invention, a processing device is provided for controlling an operation target. Multiple objects with different movement characteristics of mobile Each generating a display image in which a virtual image corresponding to the moving object is superimposed on a surrounding environment image; Multiple objects with different movement characteristics to be operated Mobile Each and transmitting information regarding an input operation from an operator to the mobile unit; When the movement amounts of the plurality of moving objects having different movement characteristics to be operated are different, The display range of the environmental image in the display image corresponds to a predetermined amount of the input operation. Multiple objects with different movement characteristics to be operated Mobile Each Based on the amount of movement , so that the difference between the moving bodies is small. A method of remote control is provided.

[0021] In order to solve the above problem, according to another aspect of the present invention, a computer is an object to be operated. Multiple objects with different movement characteristics of mobile Each an image generating unit that generates a display image by superimposing a virtual image corresponding to the moving object on a surrounding environment image; Multiple objects with different movement characteristics to be operated Mobile Each a communication unit that transmits information regarding an input operation from an operator to the mobile body, and the image generating unit generates a Multiple objects with different movement characteristics to be operated Mobile Each Based on the amount of movement, When the movement amounts of the plurality of moving objects having different movement characteristics to be operated are different, the difference between the moving objects is reduced. A program is provided that controls a display range of the environmental image in the display image. Effect of the Invention

[0022] As described above, according to the present invention, it is possible to further improve usability for an operator who remotely controls a moving object. [Brief description of the drawings]

[0023] [Figure 1] 1 is a block diagram showing an example of a functional configuration of a remote control system according to a first embodiment of the present invention. [Figure 2A] 10A to 10C are schematic diagrams illustrating examples of display images generated by an image generating unit for moving objects having different amounts of movement corresponding to a predetermined amount of input operation. [Figure 2B] 10A to 10C are schematic diagrams illustrating examples of display images generated by an image generating unit for moving objects having different amounts of movement corresponding to a predetermined amount of input operation. [Figure 2C]10A to 10C are schematic diagrams illustrating examples of display images generated by an image generating unit for moving objects having different amounts of movement corresponding to a predetermined amount of input operation. [Diagram 3] 4 is a flowchart showing the flow of operations of the remote control device according to the embodiment. FIG. [Figure 4] FIG. 11 is a block diagram showing an example of an overall configuration of a remote control system according to a first modified example. [Diagram 5] FIG. 11 is a block diagram showing an example of an overall configuration of a remote control system according to a second modified example. [Figure 6] FIG. 13 is a block diagram showing an example of an overall configuration of a remote control system according to a third modified example. [Figure 7A] FIG. 11 is a schematic diagram showing an example of a display image generated by an image generating unit in a remote control system according to a second embodiment of the present invention. [Figure 7B] FIG. 11 is a schematic diagram showing an example of a display image generated by an image generating unit in a remote control system according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and duplicated explanations will be omitted.

[0025] <1. First embodiment> (1.1. Overview) First, an overview of a remote control system according to a first embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of a functional configuration of a remote control system 100 according to this embodiment.

[0026] As shown in FIG. 1, a remote control system 100 includes, for example, two moving objects 101-1 and 101-2 and a remote control device 102.

[0027] Two mobile objects 101-1 and 101-2 are connected to a remote control device 102 so as to be able to transmit and receive data to and from each other via a network 103. The remote control system 100 is a system in which an operator OP remotely controls two mobile objects 101-1 and 101-2 located in remote locations using the remote control device 102.

[0028] Two moving bodies 101-1, 101-2 (collectively referred to as moving body 101 when there is no need to distinguish between them) are robots that can move autonomously and can also move and run under control (also referred to as remote operation) from a remote operation device 102. Specifically, the moving body 101 can move and run autonomously, notify the remote operation device 102, move and run under remote operation from the remote operation device 102, and transmit data in response to a request from the remote operation device 102.

[0029] Here, a robot is a mechanical device defined as a so-called "android," "a device that performs actions or tasks similar to those of a human being," or "a device that performs actions or tasks by being computer-controlled based on instructions." For example, a robot may be a mechanical device defined as "a machine that has an automatically controlled manipulation or movement function, can execute various tasks by programs, and is used in industry" (JIS B 0134-1998), "a robot that serves humans" (JIS B 0187:2005), or "an intelligent mechanical system that has three elemental technologies: sensors, intelligence / control system, and drive system" (Robot Policy Study Group, Ministry of Economy, Trade and Industry).

[0030] In the remote control system 100 according to the present embodiment, the mobile body 101 may be used for various applications or operations. For example, the mobile body 101 may be used for security applications such as patrol monitoring in a facility to be guarded. The mobile body 101 may be used for applications such as inspection of public facilities or public equipment. The mobile body 101 may also be used in places such as manufacturing sites (inspection of equipment in a factory, or transport of manufacturing parts or materials, etc.) or construction sites (transportation of building parts, checking of construction status, etc.). Furthermore, the mobile body 101 may be used in the passenger business field (transportation of passengers or luggage, guidance of tourist spots, or inspection of luggage, etc.) or the logistics business field (inspection of cargo or cargo vehicles, or transport of cargo, etc.), etc.

[0031] The remote control device 102 is a device used when the operator OP remotely controls the moving object 101. For example, the remote control device 102 is a device including a terminal such as a personal computer (PC) or a smartphone, and peripheral devices connected to these terminals. Specifically, the remote control device 102 can output data regarding the state of the moving object 101, instruct the moving object 101 to move based on an input operation of the operator OP, output a notification from the moving object 101, or monitor the state of the operator OP. For example, the remote control device 102 can present an image including the moving object 101 and the surrounding environment of the moving object 101 to the operator OP, thereby allowing the operator OP to understand the surrounding environment of the moving object 101. In this way, the remote control device 102 can move the moving object 101 by remote control based on an input of the operator OP.

[0032] The network 103 is a communication network that connects the mobile object 101 and the remote control device 102. The network 103 enables data to be transmitted and received between the mobile object 101 and the remote control device 102. The network 103 may be the Internet, a satellite communication network, a mobile communication network, a LAN (Local Area Network), a WAN (Wide Area Network), or the like.

[0033] Furthermore, the communication method between the mobile object 101 and the network 103 and the communication method between the remote control device 102 and the network 103 may be the same communication method or different communication methods. For example, the mobile object 101 may be connected to the network 103 using wireless LAN communication so that it can connect to the network 103 at any position. On the other hand, the remote control device 102 may be connected to the network 103 using wired LAN communication so that it can be installed at a predetermined position and connect to the network 103. However, the communication method between the mobile object 101 and the network 103 and the communication method between the remote control device 102 and the network 103 are not limited to the above and may be another communication method.

[0034] In the remote control system 100 according to this embodiment, the moving body 101 basically moves and travels autonomously, while transmitting an image of the environment around the moving body 101 to the remote control device 102. However, the moving body 101 may encounter a situation (also called an incident) that is difficult for the moving body 101 to solve or judge by itself.

[0035] In such a case, the mobile object 101 transmits a notification to the remote control device 102 requesting the operator OP to remotely control the mobile object 101, and the remote control device 102 notifies the operator OP of the received notification. This allows the operator OP to remotely control the mobile object 101 that transmitted the notification. Therefore, the mobile object 101 can be remotely controlled by the operator OP, who has visually recognized an image of the environment around the mobile object 101, via the remote control device 102, thereby resolving the encountered incident.

[0036] However, if the moving characteristics of the moving bodies 101-1 and 101-2 are different, for example, when the operator OP remotely operates the moving body 101-1 after remotely operating the moving body 101-2, it becomes difficult to immediately grasp the moving characteristics of the moving body 101 to be operated. Therefore, the operator OP who remotely operates the moving body 101 without grasping the moving characteristics of the moving body 101 may move the moving body 101 excessively.

[0037] In the remote operation system 100 according to this embodiment, the display range of the environmental image of the surroundings of the moving object 101, which is presented to the operator OP, is controlled based on the movement characteristics of the moving object 101 to be operated. This allows the remote operation device 102 to control the size of the display range of the environmental image so that the difference in the movement amount of the moving object 101 in response to a predetermined amount of input operation is reduced between multiple moving objects 101 with different movement characteristics. Therefore, the remote operation device 102 can prevent the operator OP from moving the moving object 101 excessively.

[0038] The remote operation device 102 may reduce the difference in the amount of movement of the moving object 101 to an extent that the operator OP does not feel the difference in the amount of movement of the moving object 101 for a predetermined amount of input operation. For example, the remote operation device 102 may reduce the difference in the amount of movement of the moving object 101 to an extent that the difference in the amount of movement of the moving object 101 for a predetermined amount of input operation is 10% or less of the amount of movement of the moving object 101.

[0039] (1.2. Detailed configuration) Next, a detailed configuration of each device of the remote control system 100 according to this embodiment will be described with reference to FIG.

[0040] (Mobile 101) The mobile object 101 includes, for example, a mobile object control unit 110, a mobile object communication unit 111, an AI unit 112, a battery unit 113, a moving unit 114, a camera unit 115, a sensor unit 116, a display unit 117, a speaker 118, and a microphone 119.

[0041] The mobile object control unit 110 processes various data acquired by the mobile object 101 and controls the overall operation of the mobile object 101. The mobile object control unit 110 may be configured, for example, by software including a program for controlling the overall operation of the mobile object 101 and hardware including a processor and memory on which the software is installed. Examples of the hardware on which the software is installed include terminals such as a personal computer, a tablet, a microcomputer, or a smartphone.

[0042] The mobile communication unit 111 is a communication interface for connecting to the network 103. For example, the mobile communication unit 111 may be a communication interface that can be connected wirelessly or wired to the network 103 or a base station that can be connected to the network 103. However, the mobile communication unit 111 may also be a communication interface that can be connected to the network 103 via a network such as a mobile phone communication network or a wireless LAN.

[0043] The moving unit 114 is a moving mechanism capable of moving the moving body 101 to an arbitrary position based on the control by the moving body control unit 110. The moving unit 114 may be a moving mechanism of various types, such as a wheel type, a leg type, a caterpillar type, or an air cushion type. The moving unit 114 may also be a moving mechanism capable of three-dimensional movement. For example, the moving unit 114 may be a moving mechanism capable of moving in the air using a rotor or the like, or a moving mechanism capable of moving on or underwater using a screw or the like.

[0044] The camera unit 115 includes at least one imaging device capable of acquiring an environmental image of the surroundings of the moving body 101. The camera unit 115 may include, for example, a 360-degree camera capable of acquiring an environmental image capturing an image of the entire surroundings of the moving body 101. The camera unit 115 may also include multiple fisheye cameras provided in each direction of the moving body 101. In such a case, images captured by the multiple fisheye cameras provided on the front, rear, left and right sides of the moving body 101 are synthesized by the remote control device 102 into a seamless omnidirectional image, thereby generating an environmental image that overlooks the environment around the moving body 101 from the sky.

[0045] The sensor unit 116 includes at least one type of sensor capable of sensing each parameter of the environment surrounding the mobile object 101. The sensor unit 116 may acquire sensing information obtained by sensing, for example, temperature, humidity, illuminance, air pressure, vibration, the distance from the mobile object 101 to an object, or a change in the inclination of the mobile object 101 using a corresponding sensor. The sensor unit 116 may also acquire sensing information obtained by sensing the generation of smoke, chemical substances, static electricity, and the like using a corresponding sensor. Note that the "sensing" in the above includes not only sensing various parameters of the environment surrounding the mobile object 101, but also further measuring, discriminating, and analyzing the sensed parameters.

[0046] There is no particular limitation on the type of sensor included in the sensor unit 116. The sensor unit 116 may include, for example, one or more of a thermometer, a hygrometer, an illuminometer, a barometer, a LiDAR (Light Detection And Ranging) sensor, or a thermal image measuring device (thermography camera).

[0047] Furthermore, the sensor unit 116 may acquire sensing information on the environment around the mobile object 101 from a sensor device provided outside the mobile object 101 via the mobile object communication unit 111. In such a case, the sensor unit 116 may acquire the sensing information from the external sensor device by, for example, 920 MHz band multi-hop wireless communication.

[0048] In the above description, the moving object 101 is described as including the camera unit 115 and the sensor unit 116, but the present embodiment is not limited to the above example. For example, the moving object 101 may include only the camera unit 115 without including the sensor unit 116.

[0049] The display unit 117 outputs various information as images or the like in response to control by the mobile object control unit 110. The display unit 117 may include, for example, a display that outputs various information as images, or a lamp that outputs various information by emitting light, or the like.

[0050] The speaker 118 outputs sound around the moving object 101 in response to the control of the moving object control unit 110. For example, the speaker 118 may output sound effects in response to the movement of the moving object 101, synthesized sound in response to the movement of the moving object 101, or sound input by the operator OP, around the moving object 101.

[0051] The microphone 119 picks up sounds around the moving object 101. For example, the microphone 119 may pick up sounds of people around the moving object 101 or environmental sounds around the moving object 101.

[0052] The AI ​​unit 112 recognizes the environment around the moving object 101 based on an environmental image around the moving object 101 captured by the camera unit 115 and sensing information obtained by sensing each parameter of the environment around the moving object 101 by the sensor unit 116, and makes a judgment and decision according to the recognized environment. The AI ​​unit 112 may recognize the environment around the moving object 101 from the environmental image and sensing information by using a machine learning algorithm, for example, and make a judgment and decision according to the recognized environment.

[0053] The battery unit 113 supplies power to each component of the mobile object 101. The battery unit 113 may include a secondary battery such as a lithium ion secondary battery.

[0054] (Remote control device 102) The remote control device 102 includes, for example, a control unit 120 including an image generation unit 126, a communication unit 121, a camera unit 124, a sensor unit 125, a display unit 127, a speaker 128, a microphone 129, an operation unit 130, a mobile object information storage unit 122, and a power supply unit 123.

[0055] The communication unit 121 is a communication interface for connecting to the network 103. For example, the communication unit 121 may be a communication interface that can be connected wirelessly or wired to the network 103 or a base station that can be connected to the network 103. However, the communication unit 121 may also be a communication interface that can be connected to the network 103 via a network such as a mobile phone communication network or a wireless LAN.

[0056] The control unit 120 processes various data acquired by the remote control device 102 and controls the overall operation of the remote control device 102. The control unit 120 also includes an image generation unit 126 that generates a display image to be displayed on the display unit 127. The control unit 120 may be configured, for example, with software including a program that controls the overall operation of the remote control device 102 and hardware including a processor and memory in which the software is installed. Examples of the hardware in which the software is installed include terminals such as a personal computer, a tablet, a microcomputer, or a smartphone.

[0057] The camera unit 124 includes an imaging device capable of acquiring an image of the surroundings of the remote operation device 102. The camera unit 124 may include, for example, an imaging device that captures an image of the operator OP or the surroundings of the operator OP.

[0058] The sensor unit 125 includes at least one type of sensor capable of sensing the environment around the remote control device 102 or each parameter of the operator OP. The sensor unit 125 may acquire sensing information obtained by sensing vital signs of the operator OP, such as body temperature, heart rate, blood pressure, or respiratory rate, using a corresponding sensor. The sensor unit 125 may acquire sensing information on the environment around the remote control device 102 or the environment around the operator OP from a sensor device provided outside the remote control device 102 via the communication unit 121.

[0059] The image generating unit 126 generates a display image by superimposing a virtual image corresponding to the moving object 101 on an environmental image around the moving object 101 captured by the camera unit 115 of the moving object 101 .

[0060] For example, if the camera unit 115 includes a 360-degree camera capable of acquiring images capturing the entire surroundings of the moving body 101, the image generation unit 126 can generate an environmental image by correcting the images capturing the entire surroundings of the moving body 101 and cutting out the corrected images.

[0061] Furthermore, when the camera unit 115 includes multiple fisheye cameras installed in each direction of the moving body 101, the image generation unit 126 can generate an environmental image that provides a bird's-eye view of the environment around the moving body 101 (but not including the moving body 101) from above by combining images captured by the multiple fisheye cameras installed in the front, back, left, and right of the moving body 101 into a continuous, omnidirectional image.

[0062] Here, the environmental image generated by connecting the images captured by the fisheye camera does not include an image of the moving body 101 on which the fisheye camera itself is installed. Therefore, the image generating unit 126 can generate a display image in which the moving body 101 and the environment around the moving body 101 are overlooked from the sky by superimposing a virtual image of the moving body 101 at a position corresponding to the moving body 101 in the generated environmental image.

[0063] Note that the method of generating the environmental image showing the environment around the moving object 101 is not limited to the above. For example, the environmental image showing the environment around the moving object 101 may be generated from an image captured by a surveillance camera installed in the space where the moving object 101 exists.

[0064] Furthermore, the image generating unit 126 can control the display range of the environmental image around the moving object 101 in the display image, based on information about the movement characteristics of the moving object 101.

[0065] Specifically, the image generating unit 126 may control the display range of the environmental image around the moving object 101 in the display image based on the amount of movement of the moving object 101 corresponding to a predetermined amount of input operation by the operator OP. For example, the image generating unit 126 may enlarge the display range of the environmental image as the amount of movement of the moving object 101 corresponding to the predetermined amount of input operation increases, and may reduce the display range of the environmental image as the amount of movement of the moving object 101 corresponding to the predetermined amount of input operation decreases. In this way, the image generating unit 126 can control the size of the display range of the environmental image so that the difference in the amount of movement of the moving object 101 for a predetermined amount of input operation decreases between moving objects 101 with different movement characteristics.

[0066] The display unit 127 displays the environmental image generated by the image generation unit 126 included in the control unit 120. The display unit 127 may include a display device such as a liquid crystal display (LCD) device, an organic light emitting diode (OLED) display device, or a touch panel display device. As will be described later, a display image generated by the image generation unit 126 using an environmental image captured by the camera unit 115 of the moving object 101 is displayed on the display unit 127. The operator OP can grasp the environment around the moving object 101 by visually checking the display image displayed on the display unit 127.

[0067] The speaker 128 outputs sound around the remote control device 102 in response to control by the control unit 120. For example, the speaker 128 may output sound associated with the control by the control unit 120, or sound around the mobile object 101 picked up by the microphone 119 of the mobile object 101 when the mobile object 101 is being remotely controlled, around the remote control device 102.

[0068] The microphone 129 picks up sounds around the remote operation device 102. For example, the microphone 129 may pick up sounds spoken by the operator OP around the remote operation device 102.

[0069] The operation unit 130 includes an input device for receiving operations from the operator OP. The operation unit 130 may include input devices such as a keyboard, a mouse, a touch panel, a track pad, a cross key, or an operation lever. With this, the operation unit 130 can receive an input operation from the operator OP that instructs the direction and speed of travel of the moving object 101.

[0070] The moving object information storage unit 122 stores information on the movement characteristics of the moving objects 101 (specifically, two moving objects 101-1 and 101-2) connected to the remote control device 102. Specifically, the moving object information storage unit 122 stores information on the movement characteristics such as the movement speed, movement range, or movable area of ​​the two moving objects 101-1 and 101-2 that may be remotely controlled by the remote control device 102. Furthermore, the moving object information storage unit 122 may store information on the shape, overall length, overall width, height, or the like of the two moving objects 101-1 and 101-2.

[0071] The information on the movement characteristics of the moving object 101 stored in the moving object information storage unit 122 is used to set the movement direction and movement amount of the moving object 101 in response to an input operation from the operation unit 130. For example, the information on the movement characteristics of the moving object 101 may be used to set the movement amount of the moving object 101 in response to a predetermined amount of input operation from the operation unit 130.

[0072] The power supply unit 123 supplies power from an external power supply or an internal power supply to each unit of the remote control device 102. For example, the power supply unit 123 may supply the power input from the external power supply to each unit of the remote control device 102 after transforming the input power.

[0073] Here, a display image generated by the image generating unit 126 for the moving objects 101A, 101B, and 101C, which have different amounts of movement corresponding to a predetermined amount of input operation, will be described with reference to Figs. 2A to 2C. Figs. 2A to 2C are schematic diagrams showing an example of a display image generated by the image generating unit 126 for the moving objects 101A, 101B, and 101C, which have different amounts of movement corresponding to a predetermined amount of input operation. In Figs. 2A to 2C, the amount of movement corresponding to the predetermined amount of input operation is different for the moving object 101. B ,101 A , 101C in that order. Get bigger .

[0074] For example, in the case of the moving body 101A, in which the amount of movement corresponding to the predetermined amount of input operation is a reference value, the image generating unit 126 generates a display image 301A by superimposing a virtual image 321A on an environment image 311A ​​in which the display range is not enlarged or reduced, as shown in Fig. 2A. In the case of the moving body 101B, in which the amount of movement corresponding to the predetermined amount of input operation is smaller than that of the moving body 101A, the image generating unit 126 generates a display image 301B by superimposing a virtual image 321B on an environment image 311B in which the display range is reduced from that of the environment image 311A, as shown in Fig. 2B. In the case of the moving body 101C, in which the amount of movement corresponding to the predetermined amount of input operation is larger than that of the moving body 101A, the image generating unit 126 generates a display image 301C by superimposing a virtual image 321C on an environment image 311C in which the display range is enlarged from that of the environment image 311A, as shown in Fig. 2C.

[0075] According to this, the image generation unit 126 can reduce the difference in the amount of movement of the moving bodies 101A, 101B, 101C in the virtual images 321A, 321B, 321C in response to a specified input operation by controlling the display range of the environmental images 311A, 311B, 311C in each of the display images 301A, 301B, 301C.

[0076] Therefore, the operator OP can operate the moving object 101 with a similar operation feeling even if the moving amount of the moving object 101 in response to a predetermined input operation is different in the display image 301. Even when the operator OP operates moving objects 101 with different movement characteristics, the remote operation device 102 can unify the operation feeling of each moving object 101 to such an extent that the difference is not felt, thereby improving the usability of the operator OP.

[0077] (1.3. Example of operation) Next, an example of the operation of the remote control device 102 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a flow chart showing the flow of the operation of the remote control device 102 according to this embodiment.

[0078] First, in the remote control system 100 of this embodiment, the flow until the moving body 101 switches from a state in which it moves and travels autonomously (hereinafter also referred to as an autonomous control state) to a state in which it moves and travels by remote control (hereinafter also referred to as a remote control state) will be described.

[0079] In the remote operation system 100, a moving object 101 is mutually connected to a remote operation device 102 via a network 103. The moving object 101 performs autonomous control of a moving object control unit 110 to set a moving route, acquire captured images by a camera unit 115, acquire sensing information by a sensor unit 116, and move and travel autonomously. At the same time, the moving object 101 transmits the captured images acquired by the camera unit 115 and part of the sensing information acquired by the sensor unit 116 to the remote operation device 102 under the autonomous control of the moving object control unit 110. Meanwhile, the remote operation device 102 generates a display image in an image generation unit 126 using the captured images received from the moving object 101, etc. The generated display image is displayed on an operator OP on a display unit 127.

[0080] Normally, the mobile object 101 operates by autonomous judgment and autonomous control. However, when a predetermined occurrence (event) is confirmed, the mobile object 101 suspends the autonomous control and notifies the remote control device 102 of a request for remote control. Hereinafter, the occurrence in which the mobile object 101 judges that the autonomous control should be suspended is referred to as an "incident."

[0081] Specifically, when the moving body 101 determines that an object detected on the moving route is an obstacle to the moving and cannot be avoided only by the autonomous moving and traveling of the moving body 101, it detects an incident caused by the object. In such a case, the moving body 101 notifies the remote operation device 102 of the occurrence of the incident, and may also notify the remote operation device 102 of a remote operation request from the operator OP.

[0082] There is no particular limitation on the method by which the moving body 101 detects an object that is an obstacle to the moving and traveling. For example, the moving body 101 may detect an object that is an obstacle to the moving and traveling based on at least one of the captured image acquired by the camera unit 115 and the sensing information acquired by the sensor unit 116 (e.g., sensing information of an ultrasonic sensor or a millimeter wave radar).

[0083] Furthermore, the moving body 101 may recognize the type of object detected on the route of movement and travel. For example, the moving body 101 may acquire information for identifying the type of the detected object by recognizing the captured image acquired by the camera unit 115 or the sensing information acquired by the sensor unit 116 using a machine learning algorithm of the AI ​​unit 112. As a machine learning algorithm used for recognizing such objects, a machine learning algorithm that has been trained in advance using images of objects that may be obstacles to movement and travel as teacher data can be adopted.

[0084] For example, there are various types of objects that can be obstacles to travel, such as objects that can be avoided without changing the route the mobile body 101 plans to travel (such as a small tree branch), and objects that can be avoided by changing the route the mobile body 101 plans to travel (such as a large fallen tree blocking the road). Therefore, by classifying objects that can be obstacles to travel in advance in consideration of frequency and seriousness, the mobile body 101 can determine whether or not it is possible to autonomously resolve the incident and whether or not to request remote operation by the operator OP, depending on the type of object detected.

[0085] According to the above flow, after the notification requesting remote operation is transmitted from the mobile object 101 to the remote operation device 102, the remote operation device 102 receives the remote operation request transmitted from the mobile object 101 that detected the occurrence of an incident (S101), as shown in Fig. 3. The remote operation request from the mobile object 101 may be output to the operator OP via the display unit 127, or may be output to the operator OP via the speaker 128, for example.

[0086] Next, the remote control device 102 acquires information on the movement characteristics of the moving object 101 to be remotely controlled from the moving object information storage unit 122 that exists inside the remote control device 102 or outside the remote control device 102 (S102).

[0087] Next, the remote control device 102 receives an image of the environment around the moving object 101 captured by the camera unit 115 of the moving object 101 (S103).

[0088] Next, the image generating unit 126 of the remote control device 102 generates a display image by superimposing the virtual image of the moving object 101 on the environmental image around the moving object 101 (S104).

[0089] The display image generated by the image generation unit 126 is presented to the operator OP via the display unit 127 (S105). This allows the operator OP to visually recognize the display image and appropriately remotely control the moving object 101. At this time, the remote control device 102 may output information on the movement characteristics of the moving object 101 (particularly, information on the movement amount of the moving object 101 corresponding to a predetermined operation amount) to the operator OP as an image via the display unit 127. In addition, the remote control device 102 may output information on the movement characteristics of the moving object 101 as sound to the operator OP via the speaker 128.

[0090] The remote control device 102 accepts an input operation to the mobile object 101 from the operator OP who has viewed the display image (S106), and transmits information related to the input operation from the operator OP to the mobile object 101 (S107).

[0091] This allows the remote control device 102 to present a display image including an environmental image around the moving object 101 to the operator OP, and also to transmit an input operation from the operator OP to the moving object 101 to the moving object 101 .

[0092] As described above, according to the remote operation device 102 of this embodiment, when remotely operating moving objects 101 with different movement characteristics, it is possible to unify the operation feel of each moving object 101 in the display image visually recognized by the operator OP. Therefore, the remote operation device 102 can improve the usability of the operator OP.

[0093] (1.4. Modifications) (First Modification) Next, a remote control system 100A according to a first modified example of this embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the overall configuration of the remote control system 100A according to the first modified example.

[0094] As shown in FIG. 4, a remote control system 100A according to the first modification is different from the remote control system 100 shown in FIG.

[0095] The mobile object information storage unit 122 is a storage unit that stores information on the movement characteristics of each mobile object 101 in which the mobile object information storage unit 122 is provided. When the mobile object 101 transmits a notification requesting remote control to the remote control device 102, the mobile object 101 also transmits information on the movement characteristics stored in the mobile object information storage unit 122 to the remote control device 102.

[0096] According to this, the remote operation system 100A according to this modification can obtain real-time information on the movement characteristics of the remotely operated moving object 101 by the remote operation device 102. Therefore, the remote operation system 100A according to this modification can obtain real-time information on the moving object 101, and therefore can further improve the usability for the operator OP.

[0097] The mobile object information storage unit 122 may be provided in an external server connected to the remote control device 102 via the network 103. In this case, when the remote control device 102 remotely controls the mobile object 101, the remote control device 102 can similarly acquire information on the movement characteristics of the mobile object 101 to be remotely controlled from the mobile object information storage unit 122 provided in the external server.

[0098] (Second Modification) Next, a remote control system 100B according to a second modified example of this embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing an example of the overall configuration of the remote control system 100B according to the second modified example.

[0099] 5, a remote control system 100B according to a first modification includes N (N is an integer equal to or greater than 3) moving objects 101-1, 101-2, ... 101-N and a remote control device 102. The remote control system 100B according to a second modification differs from the remote control system 100 shown in FIG 1 in that the number of moving objects 101 connected to the remote control device 102 is greater.

[0100] N mobile objects 101-1, 101-2, ..., 101-N (collectively referred to as mobile objects 101 when there is no need to distinguish between them) are connected to a remote control device 102 via a network 103 so as to be able to transmit and receive data to and from each other. The remote control system 100B is a system in which an operator OP remotely controls the N mobile objects 101-1, 101-2, ..., 101-N that are located in remote locations using the remote control device 102.

[0101] As shown in the remote control system 100B according to the second modification, the number of moving objects 101 included in the remote control system 100B is not particularly limited. However, it is considered that as the number of moving objects 101 included in the remote control system 100B increases, an operator OP who operates a remote control device 102 will remotely control moving objects 101 having various movement characteristics.

[0102] The remote operation system 100B according to this modification can reduce the difference in the amount of movement of the moving object 101 on the display image for a given amount of operation between moving objects 101 having different movement characteristics. Therefore, the remote operation system 100B according to this modification can unify the operation feel of each moving object 101 for an operator OP who remotely operates a larger number of moving objects 101, thereby further improving the usability of the operator OP.

[0103] (Third Modification) Next, a remote control system 100C according to a third modified example of this embodiment will be described with reference to Fig. 6. Fig. 6 is a block diagram showing an example of the overall configuration of the remote control system 100C according to the third modified example.

[0104] 6, a remote control system 100C according to the third modification includes N (N is an integer equal to or greater than 2 and M≦N) moving objects 101-1, 101-2, ..., 101-N, and M (M is an integer equal to or greater than 2) remote control devices 102-1, ..., 102-M. The remote control system 100C according to the second modification differs from the remote control system 100 shown in FIG. 1 in that a plurality of moving objects 101 and a plurality of remote control devices 102 are connected to a network 103.

[0105] N mobile objects 101-1, 101-2, ..., 101-N (collectively referred to as mobile objects 101 when they are not distinguished from one another) are connected to M remote operation devices 102-1, ..., 102-M (collectively referred to as remote operation devices 102 when they are not distinguished from one another) so as to be able to transmit and receive data to and from one another via a network 103. The remote operation system 100C is a system in which M operators OP-1, ..., OP-M remotely operate the N mobile objects 101-1, 101-2, ..., 101-N located in remote locations by using each of the M remote operation devices 102-1, ..., 102-M.

[0106] As shown in the remote operation system 100C according to the third modification, the number of remote operation devices 102 included in the remote operation system 100C is two or more and is equal to or less than the number of moving objects 101 (M≦N). This is because, when the number of remote operation devices 102 is greater than the number of moving objects 101, the number of moving objects 101 per operator is at least one or less. In the remote operation system 100C according to the third modification, by appropriately allocating the remote operation device 102 and the operator OP to the moving object 101 to be remotely operated in response to a remote operation request from the moving object 101, it is possible to efficiently monitor and remotely operate more moving objects 101.

[0107] The remote operation system 100C according to this modification is capable of controlling the amount of movement of the moving object 101 on the display image relative to a predetermined amount of operation. Therefore, the remote operation system 100C according to this modification can optimize the amount of movement of the moving object 101 on the display image for the operator OP, thereby further improving the usability of the operator OP.

[0108] <2. Second embodiment> A remote control system 100 according to a second embodiment of the present invention will be described with reference to Fig. 7A and Fig. 7B. The remote control system 100 according to the second embodiment is different from the remote control system 100 according to the first embodiment in that the image generating unit 126 generates a display image on which an information image showing information related to the movement characteristics of the moving object 101 is further superimposed. Fig. 7A and Fig. 7B are schematic diagrams showing an example of a display image generated by the image generating unit 126.

[0109] 7A and 7B, the image generating unit 126 may generate a display image 301 in which a virtual image 321 of the moving object 101 and an information image 331 showing information on the movement characteristics of the moving object 101 are superimposed on an environmental image 311 around the moving object 101. For example, the image generating unit 126 may superimpose the information image 331 on any of the four corners of the environmental image 311 so as not to obstruct the view of an operator OP who remotely controls the moving object 101.

[0110] The information image 331 may include, for example, character data indicating the moving speed, moving range, or movable area of ​​the moving object 101. The information image 331 may further include character data indicating the name, moving location, and size of the moving object 101. The image generating unit 126 can generate the above-mentioned information image 331 by using information on the moving characteristics of the moving object 101 stored in the moving object information storage unit 122.

[0111] Furthermore, the image generating unit 126 may superimpose on the display image 301 an information image 331 that indicates the movement characteristics of the moving object 101 using an arrow or hatching or the like.

[0112] According to this, the operator OP can grasp the movement characteristics of the moving object 101 by visually checking the information image 331 included in the display image 301, and can correct the operation feeling of the moving object 101 within his / her own sense. Therefore, the remote operation device 102 can improve the usability of the operator OP when the operator OP operates the moving object 101 having different movement characteristics.

[0113] In the above, the technology according to the second embodiment is described as being used in combination with the technology according to the first embodiment, but the present invention is not limited to the above example. In the remote operation system 100 according to the second embodiment, the image generating unit 126 may superimpose a virtual image 321 of the moving object 101 and an information image 331 of the moving object 101 on an environmental image 311 whose display range is not controlled based on the amount of movement of the moving object 101 corresponding to a predetermined amount of input operation by the operator OP.

[0114] Although the preferred embodiment of the present invention has been described in detail above with reference to the accompanying drawings, the present invention is not limited to such an example. It is clear that a person having ordinary knowledge in the technical field to which the present invention pertains can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present invention.

[0115] It is also possible to create a program for causing hardware such as a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory) built into a computer to perform functions equivalent to those of the configuration of the remote control device 102 according to the first or second embodiment of the present invention. Furthermore, a computer-readable recording medium having the program recorded therein may also be provided. [Explanation of symbols]

[0116] 100, 100A, 100B, 100C Remote Control System 101,101A,101B,101C Mobile object 102 Remote control device 103 Network 110 Mobile control unit 111 Mobile Communications Department 112 AI Department 113 Battery section 114 Mobile Division 115 Camera Department 116 Sensor section 117 Display section 118 Speaker 119 Mike 120 Control section 121 Communications Department 122 Mobile information storage unit 123 Power supply section 124 Camera Division 125 Sensor section 126 Image Generation Unit 127 Display section 128 Speaker 129 Mike 130 Operation section OP Operator

Claims

1. an image generating unit that generates a display image by superimposing a virtual image corresponding to each of a plurality of moving objects having different movement characteristics to be operated on an environmental image surrounding the moving object; a communication unit that transmits, to the moving body, information regarding an input operation from an operator of each of the plurality of moving bodies having different movement characteristics that are the operation targets; Equipped with A remote control device, wherein the image generation unit controls the display range of the environmental image in the display image so as to reduce the difference between the moving bodies when the movement amounts of each of the multiple moving bodies having different movement characteristics that are the target of operation differ based on the movement amounts of each of the multiple moving bodies having different movement characteristics that are the target of operation corresponding to a predetermined amount of the input operation.

2. The remote control device according to claim 1 , wherein the image generation unit expands the display range of the environmental image as the amount of movement of the moving body corresponding to the specified amount of the input operation increases, and reduces the display range of the environmental image as the amount of movement of the moving body corresponding to the specified amount of the input operation decreases.

3. The remote control device according to claim 2, wherein the image generation unit controls the size of the display range of the environmental image so that, in the display image, the amount of movement of the moving body in response to the specified amount of the input operation is approximately the same among multiple moving bodies having the different movement characteristics.

4. The remote control device according to any one of claims 1 to 3, wherein the image generation unit generates the environmental image overlooking the surroundings of the moving body by synthesizing images captured by a plurality of imaging devices mounted on the moving body.

5. 5. The remote control device according to claim 1, wherein the amount of movement of the moving object in response to the predetermined amount of the input operation is set based on information about a movement characteristic of the moving object.

6. The remote control device according to claim 5 , wherein the information regarding the movement characteristics is acquired from the moving object.

7. A remote control device as described in claim 5 or 6, wherein the information regarding the movement characteristics is information indicating the movement speed, movement range, movable area, shape, overall length, overall width, or height of the moving body.

8. The remote control device according to claim 5 , wherein the image generating unit generates the display image by further superimposing an image indicating the movement characteristic on the environmental image.

9. A remote control device as described in Claim 8, wherein the image indicating the movement characteristics is an image indicating information regarding the amount of movement of the moving body corresponding to the specified amount of the input operation.

10. The remote control device according to any one of claims 1 to 9, wherein the remote control device remotely controls each of a plurality of moving objects having different movement characteristics.

11. By the processing unit, generating a display image in which a virtual image corresponding to each of a plurality of moving objects having different movement characteristics to be operated is superimposed on an environmental image surrounding the moving object; transmitting, to the moving body, information regarding an input operation from an operator of each of a plurality of moving bodies having different movement characteristics to be operated; Including, A remote control method in which, when the movement amounts of multiple moving bodies having different movement characteristics that are the target of operation are different, the display range of the environmental image in the display image is controlled so as to reduce the differences between the moving bodies based on the movement amounts of each of the multiple moving bodies having different movement characteristics that are the target of operation corresponding to a predetermined amount of the input operation.

12. Computer an image generating unit that generates a display image by superimposing a virtual image corresponding to each of a plurality of moving objects having different movement characteristics to be operated on an environmental image surrounding the moving object; a communication unit that transmits, to the moving body, information regarding an input operation from an operator of each of the plurality of moving bodies having different movement characteristics that are the operation targets; Function as a A program that causes the image generation unit to control the display range of the environmental image in the display image so as to reduce the difference between the moving bodies when the movement amounts of multiple moving bodies having different movement characteristics that are the target of operation differ based on the movement amounts of each of the multiple moving bodies having different movement characteristics that are the target of operation corresponding to a predetermined amount of the input operation.

Images