Control device, control method, control program, and control system

Abstract

To more efficiently operate a moving object that moves autonomously.SOLUTION: The present invention relates to a control system which comprises a moving object and a remote terminal operated by an operator, in which the moving object has a control device that controls a movement of the moving object mounted thereon. The moving object device of the present invention includes: an autonomous travel control unit that autonomously travels the moving object along a predetermined route; a remote operation travel controlling unit that makes the moving object travel based on an input operation from the remote terminal by switching the travel of the moving object from the autonomous travel controlled by the autonomous travel control unit to a remote operation travel controlled by the remote terminal operated by an operator; and a communication quality measuring unit that measures a communication quality of a communication connected to a network when the moving object performs an autonomous operation travel. A predetermined operation is performed when the communication quality measured by the communication quality measuring unit is equal to or lower than a predetermined communication quality.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a control device, a control method, a control program, and a control system, and for example, to a system for operating a robot as a mobile body that performs work autonomously. [Background technology]

[0002] In recent years, with the advancement of robot-related technologies, the development of systems that operate autonomous mobile robots is underway. Such systems are expected to be a means of resolving labor shortages caused by population decline.

[0003] For example, systems and technologies are being developed that use technologies such as artificial intelligence (AI) and map data within and between buildings to enable robots to operate autonomously and move autonomously within and between buildings.

[0004] Meanwhile, currently, events that are difficult for the robot to deal with or judge autonomously may occur while the robot is working (for example, when an increase in obstacles on the travel path makes it impossible for the robot to travel autonomously). In such cases, the robot sends a notification to a remote operator requesting operation. The operator then confirms the notification and switches the robot that sent the notification from an autonomously operating state to a state where it can be operated manually. The operator then manually operates the robot on-site or remotely to deal with the event that has occurred, and switches it back to an autonomously operating state, returning the robot to an autonomously operating state. This may make it possible to resolve events that are difficult for the robot to deal with or judge autonomously.

[0005] For example, Patent Document 1 discloses a technology in which, when a communication robot determines that it is difficult to respond through autonomous control alone, it connects to the terminal of an operator who meets the conditions required by the conversation partner based on a conditions list created from the conversation partner's voice, and the operator remotely converses with the robot. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2009-90420 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, with the technology described in Patent Document 1, when the communication robot uses a wireless network to autonomously drive in a place with a poor wireless network environment (for example, a place where radio waves do not reach, a place where radio waves are weak, or a place where the network communication speed is slow), it is not possible to transmit information about the communication robot (for example, robot information (sensor information, battery information, location information, etc.), log information, camera footage, etc.). Furthermore, when dealing with a conversation partner in a place with a poor wireless network environment, if it is determined that it is difficult to deal with the conversation through autonomous control alone, it is not possible to connect to the terminal of an operator that meets the conditions required by the conversation partner, and the operator is unable to communicate remotely.

[0008] In view of the above problems, there is a demand for a control device, a control method, a control program, and a control system that can operate an autonomously moving object more efficiently. [Means for solving the problem]

[0009] The first invention is a control device for controlling the operation of a moving body, which comprises an autonomous driving control unit that causes the moving body to drive autonomously along a predetermined route, a remotely controlled driving control unit that switches the driving of the moving body from the autonomous driving by the autonomous driving control unit to remotely controlled driving by a remote terminal operated by an operator, and causes the moving body to drive based on input operations from the remote terminal, and a communication quality measurement unit that measures the communication quality of communications connecting to a network during the autonomous driving of the moving body, and is characterized in that a predetermined operation is performed when the communication quality measured by the communication quality measurement unit is below a predetermined level.

[0010] The second control program of the present invention is characterized in that it causes a computer mounted on a control device that controls the operation of a moving body to function as an autonomous driving control unit that causes the moving body to autonomously drive along a predetermined route, a remotely controlled driving control unit that switches the driving of the moving body from autonomous driving by the autonomous driving control unit to remotely controlled driving by a remote terminal operated by an operator and causes the moving body to drive based on input operations from the remote terminal, a communication quality measurement unit that measures the communication quality of communications connecting to a network during the autonomous driving of the moving body, and a means for performing a predetermined operation when the communication quality measured by the communication quality measurement unit is below a predetermined level.

[0011] The third invention is a control method performed by a control device that controls the operation of a moving object, the control device comprising an autonomous driving control unit, a remote-controlled driving control unit, and a communication quality measurement unit, the autonomous driving control unit causes the moving object to autonomously drive along a predetermined route, the remote-controlled driving control unit switches the driving of the moving object from the autonomous driving by the autonomous driving control unit to remote-controlled driving by a remote terminal operated by an operator, and drives the moving object based on an input operation from the remote terminal, the communication quality measurement unit measures communication quality during the autonomous driving of the moving object, and performs a predetermined operation when the communication quality measured by the communication quality measurement unit is equal to or lower than a predetermined value.

[0012] The fourth invention of the present invention is a control system comprising a moving body and a remote terminal operated by an operator, the moving body being equipped with a control device for controlling the operation of the moving body, the control device having an autonomous driving control unit that causes the moving body to autonomously drive along a predetermined route, a remotely controlled driving control unit that switches the driving of the moving body from the autonomous driving by the autonomous driving control unit to remotely controlled driving by the remote terminal operated by the operator, and causes the moving body to drive based on input operations from the remote terminal, and a communication quality measurement unit that measures the communication quality of communications connecting to a network during the autonomous driving of the moving body, and is characterized in that a predetermined operation is performed when the communication quality measured by the communication quality measurement unit is below a predetermined level. Effect of the Invention

[0013] According to the present invention, an autonomously moving object can be operated more efficiently. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing an overall configuration of a remote operation system according to a first embodiment. [Diagram 2] 2 is a block diagram showing detailed configurations of a mobile object control unit and a mobile object storage unit according to the first embodiment. FIG. [Diagram 3] 5 is a flowchart showing the flow of operations of a moving object control unit (manually operated travel control unit) according to the first embodiment. [Figure 4] 5 is a flowchart showing the flow of operations of a moving object control unit (autonomous driving control unit) according to the first embodiment. [Diagram 5] 5 is a flowchart showing the flow of operations of a moving object control unit (remotely operated travel control unit) according to the first embodiment. [Figure 6] 13 is a block diagram showing detailed configurations of a mobile object control unit and a mobile object storage unit according to a second embodiment. FIG. [Figure 7] 10 is a flowchart showing the flow of operations of a moving object control unit (an autonomous driving control unit and a map communication status reflecting unit) according to a second embodiment. [Figure 8] FIG. 11 is a block diagram showing the overall configuration of a remote operation system according to a third embodiment. [Figure 9] FIG. 13 is a block diagram showing detailed configurations of a mobile object control unit and a mobile object storage unit according to a third embodiment. [Figure 10] 13 is a flowchart (part 1) showing the flow of operations of a mobile object control unit (an autonomous driving control unit and a communication relay mobile object call request unit) according to the third embodiment. [Figure 11] 13 is a flowchart (part 2) showing the flow of operations of a mobile object control unit (an autonomous driving control unit and a communication relay mobile object call request unit) according to the third embodiment. [Figure 12] FIG. 11 is an explanatory diagram (part 1) showing an example of the operation of a mobile object and a communication relay mobile object according to the third embodiment. [Figure 13] FIG. 13 is an explanatory diagram (part 2) showing an example of the operation of the mobile body and the communication relay mobile body according to the third embodiment. [Figure 14] FIG. 13 is a block diagram showing the overall configuration of a remote operation system according to a fourth embodiment. [Figure 15] FIG. 13 is a block diagram showing the overall configuration of a remote operation system according to a fifth embodiment. [Figure 16] 13 is an explanatory diagram showing an example of the operation of a mobile body and a communication relay mobile body according to the fifth embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] (A) First embodiment Hereinafter, a first embodiment of a control device, a control method, a control program, and a control system according to the present invention will be described in detail with reference to the drawings. In this embodiment, an example in which the control device, the control program, and the control method of the present invention are applied to a mobile object control unit that controls a mobile object will be described. Also, in this embodiment, an example in which the control system of the present invention is applied to a remote operation system will be described.

[0016] (A-1) Configuration of the First Embodiment First, with reference to FIG. 1, the overall configuration of a remote operation system 10 according to the first embodiment will be described.

[0017] Fig. 1 is a block diagram showing the functional configuration of a remote operation system 10 according to the first embodiment. Note that the reference characters in parentheses in Fig. 1 are reference characters that are used only in the second embodiment described later.

[0018] 1, the remote operation system 10 includes a mobile object 100, a remote control terminal 200, and a mobile object operation system unit 300. The mobile object 100, the remote control terminal 200, and the mobile object operation system unit 300 are connected via a network 400 so as to be able to transmit and receive data to and from each other.

[0019] The mobile object 100 is a robot capable of moving and running autonomously, and also capable of moving and running under the control of the remote control terminal 200. The mobile object 100 is capable of performing, for example, autonomous moving and running, connection to the mobile object operation system unit 300, sending and receiving data to the mobile object operation system unit 300, notifying the remote control terminal 200, moving and running by remote control from the remote control terminal 200, and transmitting data in response to a request from the remote control terminal 200.

[0020] Here, a robot is a mechanical device defined as a so-called "android," "a device that performs actions or tasks similar to those of a human," or "a device that performs actions or tasks by being computer-controlled based on instructions." A robot may be, for example, a mechanical device defined in "a machine for industrial use that has an automatically controlled manipulation or movement function, can execute various tasks by programs, and is used in industry" (JIS B 0134-1998), "a robot that serves humans" (JIS B 0187:2005), or "an intelligent mechanical system that has three elemental technologies: a sensor, an intelligence / control system, and a drive system" (Robot Policy Study Group, Ministry of Economy, Trade and Industry).

[0021] The mobile body 100 of the remote operation system 10 according to this embodiment is used for various purposes or operations. The mobile body 100 may be used for, for example, patrol, monitoring, security, and other purposes in a facility to be guarded, inspection of public facilities and equipment, monitoring at locations such as manufacturing sites (inspection of equipment in a factory, or transport of manufacturing parts or materials, etc.) and construction sites (transportation of building parts, or confirmation of construction status, etc.), and service purposes such as the passenger business field (transportation of passengers or luggage, guidance of tourist spots, or inspection of luggage, etc.) and the logistics business field (inspection of cargo or cargo vehicles, or transport of cargo, etc.).

[0022] The remote control terminal 200 is a device used when the operator OP remotely monitors or remotely controls the moving object 100 connected to the moving object operation system unit 300. The remote control terminal 200 may include, for example, a terminal such as a personal computer (PC), a tablet PC, a microcomputer, or a smartphone, and peripheral devices connected to these terminals. The remote control terminal 200 can, for example, output data related to the state of the moving object 100, instruct the moving object 100 to move based on an input operation of the operator OP, output a notification from the moving object 100, operate the moving object operation system unit 300, or monitor the state of the operator OP. The remote control terminal 200 can, for example, present an image including information on the moving object 100 and the environment around the moving object 100 to the operator OP, so that the operator OP can grasp the state of the moving object 100 and the environment around the moving object 100. As a result, when the moving body 100 is moving and traveling autonomously, the environment around the moving body 100 can be confirmed and monitored, and when remote control of the moving body 100 becomes necessary, the moving body 100 can be remotely controlled.

[0023] The network 400 is a communication network that connects the mobile body 100, the remote control terminal 200, and the mobile body operation system unit 300 to each other. The network 400 can transmit and receive data between the mobile body 100, the remote control terminal 200, and the mobile body operation system unit 300. The network 400 can use various communication networks. The network 400 may be connected, for example, by the Internet, a satellite communication network, a mobile communication network, a LAN (Local Area Network), or a WAN (Wide Area Network). In addition, the communication method between the mobile body 100 and the network 400 and the communication method between the remote control terminal 200 and the network 400 may be the same communication method or may be different communication methods. For example, the mobile body 100 may connect to the network 400 using wireless LAN communication in order to connect to the network 400 at an arbitrary position. On the other hand, the remote control terminal 200 may connect to the network 400 using wired LAN communication in order to connect to the network 400 at the installation position. However, the communication method between the mobile object 100, the remote control terminal 200, and the mobile object operation system unit 300 and the network 400 is not limited to the above, and may be another communication method.

[0024] The mobile object operation system unit 300 is a device on which a main system that manages and operates the entire remote operation system 10 operates. The mobile object operation system unit 300 may include, for example, terminals such as a PC, a PC for a server, and a notebook PC, and peripheral devices connected to these terminals. The mobile object operation system unit 300 can manage, for example, information on the mobile object 100, the remote control terminal 200, and the operator OP operating the remote control terminal 200, manage the operation status of the mobile object 100, manage the connection between the mobile object 100 and the remote control terminal 200, and manage operation information of the operator OP. The installation location of the mobile object operation system unit 300 is not limited, and may be installed in the same place as the remote control terminal 200, or in a different place from the remote control terminal 200, for example.

[0025] Next, the detailed configuration of the remote operation system 10 will be described with reference to FIG.

[0026] First, the configuration of the moving body 100 will be described.

[0027] As shown in FIG. 1, the mobile body 100 includes a mobile body control unit 101, a camera unit 102, a sensor unit 103, a display unit 104, a speaker 105, a microphone 106, a moving unit 107, a mobile body LAN hub unit 108, a mobile body communication unit 109, an AI unit 110, a mobile body operation unit 111, a mobile body memory unit 112, and a mobile body battery unit 113.

[0028] The mobile object control unit 101 processes various data acquired by the mobile object 100 and controls the overall operation of the mobile object 100. The mobile object control unit 101 may be configured, for example, by software including a program (control program according to the embodiment) for controlling the overall operation of the mobile object 100, and hardware (computer) on which the software is installed. An example of the hardware on which the software is installed is a computer having a central processing unit (CPU), a read only memory (RAM), and a read only memory (ROM). The mobile object control unit 101 may also be configured by a computer having a digital signal processor (DSP), a microprocessor, or an integrated circuit (IC) instead of a CPU. A more detailed configuration of the mobile object control unit 101 will be described later with reference to FIG. 2.

[0029] It is also possible to create a program for causing the above-mentioned hardware such as the CPU, ROM, and RAM to perform functions equivalent to those of the mobile object control unit 101. It is also possible to provide a computer-readable recording medium on which the program is recorded.

[0030] The camera unit 102 includes at least one imaging device capable of acquiring images and videos of the environment around the moving body 100. The camera unit 102 may include, for example, one Web camera provided in front of the moving body 100, or multiple Web cameras provided in each of the front, back, left and right directions. When the camera unit 102 includes multiple imaging devices, the remote control terminal 200 may generate an environmental image that overlooks the environment around the moving body 100 from the sky by seamlessly synthesizing the images and videos of the cameras transmitted from the moving body 100 in all directions. Alternatively, the camera unit 102 may include a fisheye camera capable of acquiring an environmental image that captures a wide range of the moving body 100, or a 360-degree camera that can acquire an environmental image that captures the entire surroundings of the moving body 100.

[0031] However, the camera unit 102 is not limited to the above and may have another configuration. The camera unit 102 may include, for example, a network camera that moves (pans and tilts) in the forward, backward, left, and right directions of the moving object 100 to capture images in all directions.

[0032] The sensor unit 103 includes at least one or more sensors capable of sensing various information of the environment surrounding the mobile body 100. The sensor unit 103 also includes at least one type of sensor. The sensor unit 103 may include a ToF (Time of flight) sensor, a LiDAR (Light Detection and Ranging), or a millimeter wave radar capable of creating a map of the surrounding environment and estimating the self-position of the mobile body 100. The sensor unit 103 may further include a sensor capable of sensing, for example, temperature, humidity, illuminance, air pressure, vibration, position information, the distance from the mobile body 100 to an object, or a change in the inclination of the mobile body 100, and may additionally include a sensor capable of sensing the generation of smoke, chemical substances, static electricity, and the like. Note that the "sensing" in the above includes not only sensing various information of the environment surrounding the mobile body 100, but also further measuring, discriminating, and analyzing the sensed information.

[0033] The type of sensor included in the sensor unit 103 is not particularly limited. The sensor unit 103 may include, for example, one or more of a thermometer, a hygrometer, an illuminometer, a barometer, or a thermal image measuring device (thermography camera). Furthermore, the sensor unit 103 may acquire sensing information on the environment around the mobile body 100 from a sensor device provided outside the mobile body 100 via the mobile communication unit 109. In such a case, the sensor unit 103 may acquire the sensing information from the external sensor device by, for example, 920 MHz band multi-hop wireless communication.

[0034] In this embodiment, the moving object 100 is described as including both the camera unit 102 and the sensor unit 103, but this embodiment is not limited to such an example. The moving object 100 may, for example, include only the camera unit 102 or only the sensor unit 103. In that case, the camera unit 102 or the sensor unit 103 includes at least two or more imaging devices or sensors.

[0035] The display unit 104 outputs various information in the form of characters, graphs, images, videos, etc., according to the control of the mobile object control unit 101. The display unit 104 may include, for example, a liquid crystal display (LCD) device that outputs various information, an organic light emitting diode (OLED) display device, a touch panel display device, or a lamp that outputs various information by emitting light.

[0036] The speaker 105 outputs sounds around the moving body 100 in response to the control of the moving body control unit 101. The speaker 105 may output, for example, sound effects in response to the movement of the moving body 100, synthesized voice in response to the movement of the moving body 100, or voice input by an operator OP, around the moving body 100.

[0037] The microphone 106 picks up sounds around the moving object 100. The microphone 106 may pick up, for example, the voices of people around the moving object 100, or environmental sounds around the moving object 100.

[0038] The mobile LAN hub unit 108 is a network device that transmits and receives data, packets, etc., and can transmit and receive data, packets, etc. between the mobile control unit 101, the AI ​​unit 110, and the mobile communication unit 109. The mobile LAN hub unit 108 may include network devices such as a switching hub, a bridging hub, and a router.

[0039] The mobile communication unit 109 is a communication interface for connecting to the network 400. The mobile communication unit 109 may be, for example, a communication interface that can be connected wirelessly or wired to a base station that can be connected to the network 400, or may be a communication interface that can be connected to the network 400 via a network such as a mobile phone communication network or a wireless LAN.

[0040] The AI ​​unit 110 recognizes the environment around the moving object 100 based on an image or video of the environment around the moving object 100 captured by the camera unit 102 and sensing information of each information of the environment around the moving object 100 sensed by the sensor unit 103, and makes a judgment and decision according to the recognized environment. The AI ​​unit 110 may recognize the environment around the moving object 100 from the environmental image and sensing information by using a machine learning algorithm, for example, and makes a judgment and decision according to the recognized environment. Note that the result of the judgment by the AI ​​unit 110 may be used to set the amount of movement of the moving object 100 in response to a predetermined amount of input operation at the remote control terminal 200.

[0041] The mobile object operation unit 111 is an input device for accepting operations of the mobile object, and can accept input operations for instructing the direction and speed of travel of the mobile object 100. The mobile object operation unit 111 is used when operating the mobile object 100 near the mobile object 100. The mobile object operation unit 111 may include input devices such as a keyboard, a mouse, a touch panel, a track pad, a cross key, a joypad, a controller, or an operation lever.

[0042] The mobile object storage unit 112 is a recording unit that stores various information and images according to the control of the mobile object control unit 101. The mobile object storage unit 112 may store, for example, basic information and map information of the location where the mobile object 100 moves autonomously, environmental map data around the mobile object 100 created by the mobile object control unit 101, and a travel route, information indicating a sensing result by the sensor unit 103, log information of the mobile object control unit 101, information on the shape of the mobile object 100 (for example, the shape, overall length, overall width, or height of the mobile object 100), or information on the movement characteristics of the mobile object 100 (for example, the movement speed, movement range, or movable area of ​​the mobile object 100). A more detailed configuration of the mobile object storage unit 112 will be described later with reference to FIG. 2.

[0043] The information stored in the mobile object storage unit 112 (e.g., log information of the mobile object control unit 101, information on the shape of the mobile object 100, information on the movement characteristics of the mobile object 100, etc.) may be used by the remote control terminal 200 or the mobile object operation system unit. For example, when remotely operating the mobile object 100 with the remote control terminal 200, the mobile object operation system unit 300 may use the information stored in the mobile object storage unit 112 to display the state of the mobile object 100 on the remote control terminal 200, and the mobile object operation system unit 300 may use the information to set the amount of movement of the mobile object 100 in response to a predetermined amount of input operation on the remote control terminal 200.

[0044] The mobile body battery unit 113 supplies power to each unit of the mobile body 100. The mobile body battery unit 113 may be configured to include a secondary battery such as a lithium ion secondary battery. In Fig. 1, the mobile body battery unit 113 is connected to the mobile body control unit 101, the mobile body LAN hub unit 108, the mobile body communication unit, and the AI ​​unit 110, but it can also be connected to other devices of the mobile body 100 that require a power supply as necessary.

[0045] The moving unit 107 is a moving mechanism capable of moving the moving body 100 to any position based on the control by the moving body control unit 101. The moving unit 107 may be, for example, a moving mechanism of various types such as a wheel type, a leg type, a crawler type, or an air cushion type, a moving mechanism capable of three-dimensional movement, a moving mechanism capable of moving in the air using a rotor or the like, or a moving mechanism capable of moving on or underwater using a screw or the like.

[0046] Next, the configuration of the remote control terminal 200 will be described.

[0047] The remote control terminal 200 comprises a remote control terminal control unit 201, a camera unit 202, a sensor unit 203, a display unit 204, a speaker 205, a microphone 206, a remote control unit 207, a remote control terminal LAN hub unit 208, a remote control terminal communication unit 209, a remote control terminal memory unit 210, and a power supply unit 211.

[0048] The camera unit 202 includes an imaging device capable of acquiring an image or video of the surroundings of the remote control terminal 200. The camera unit 202 may include, for example, one or more Web cameras provided in the remote control terminal 200 that capture images of the operator OP or the surroundings of the operator OP, a fisheye camera capable of acquiring an environmental image capturing a wide range of the remote control terminal 200, or a 360-degree camera capable of acquiring an environmental image capturing an entire surrounding of the remote control terminal 200. However, the camera unit 202 is not limited to the above, and may have other configurations. The camera unit 202 may include, for example, the operator OP of the remote control terminal 200, or a network camera that moves (pans and tilts) in the forward, backward, left, right, and right directions to capture images of the front, rear, left, and right of the operator OP.

[0049] The sensor unit 203 includes at least one sensor capable of sensing the environment around the remote control terminal 200 or each piece of information about the operator OP. The sensor unit 203 also includes one or more types of sensors. The sensor unit 203 may include a sensor capable of sensing vital signs such as the body temperature, heart rate, blood pressure, or respiratory rate of the operator OP. The sensor unit 203 may also acquire sensing information about the environment around the remote control terminal 200 or the environment around the operator OP from a sensor device provided outside the remote control terminal 200 via the remote control terminal communication unit 209.

[0050] The display unit 204 displays an environmental image around the moving object 100. The display unit 204 may include a display device such as a liquid crystal display device, an OLED display device (organic electroluminescence display device), or a touch panel display device. When the moving object 100 is moving autonomously, the display unit 204 displays an image or video generated by the remote control terminal control unit 201 using an environmental image captured by the camera unit 102 of the moving object 100 connected to the moving object operation system unit 300, and various information about the moving object 100. When the moving object 100 needs to be remotely controlled, the display unit 204 displays an image or video generated by the remote control terminal control unit 201 using an environmental image captured by the camera unit 102 of only the moving object 100 that needs to be remotely controlled, and various information about the moving object 100. As a result, when the moving body 100 is moving autonomously, the operator OP can check and monitor the environment around the moving body 100 by visually checking the display image displayed on the display unit 204, and when the moving body 100 needs to be remotely controlled, the operator OP can check and operate the moving body 100 that needs to be remotely controlled.

[0051] When the camera unit 102 includes a 360-degree camera, the display unit 204 can display an image capturing the entire surroundings of the moving body 100. When the camera unit 102 includes multiple cameras provided in each direction of the moving body 100, the display unit 204 can display an omnidirectional environmental image (an image overlooking the environment around the moving body 100 from the sky) in which images captured by the multiple cameras are seamlessly synthesized.

[0052] The speaker 205 outputs sound to the surroundings of the remote control terminal 200 in response to the control of the remote control terminal control unit 201. The speaker 205 may output, for example, sound related to the control of the remote control terminal control unit 201, or sound around the moving object 100 picked up by the microphone 106 of the moving object 100 when the moving object 100 is being remotely controlled, to the surroundings of the remote control terminal 200.

[0053] The microphone 206 picks up sounds around the remote control terminal 200. The microphone 206 may pick up, for example, a voice spoken by an operator OP around the remote control terminal 200.

[0054] The remote control unit 207 is an input device for receiving operations from an operator OP. The remote control unit 207 can receive input operations from the operator OP to instruct the direction and speed of travel of the moving body 100. The remote control unit 207 may include input devices such as a keyboard, a mouse, a touch panel, a track pad, a cross key, a joypad, a controller, or an operation lever.

[0055] The remote control terminal LAN hub unit 208 is a network device that transmits and receives data, packets, etc., and can transmit and receive data, packets, etc. between the remote control terminal control unit 201, the AI ​​unit 110, and the remote control terminal communication unit 209. The mobile LAN hub unit 108 may include network devices such as a switching hub, a bridging hub, and a router.

[0056] The remote control terminal communication unit 209 is a communication interface for connecting to the network 400. The remote control terminal communication unit 209 may be, for example, a communication interface connectable to the network 400 or a base station wirelessly or with a wire, or may be a communication interface connectable to the network 400 via a network such as a mobile phone communication network or a wireless LAN.

[0057] The power supply unit 211 supplies power from an external power supply or a built-in power supply to each unit of the remote control terminal 200. For example, the power supply unit 211 may transform the power input from the external power supply and then supply the power to each unit of the remote control terminal 200.

[0058] Next, the detailed configuration of the mobile object control unit 101 and the mobile object storage unit 112 will be described with reference to FIG.

[0059] FIG. 2 is a block diagram showing the detailed functional configuration of the mobile object control unit 101 and the mobile object storage unit 112. As shown in FIG.

[0060] As shown in FIG. 2, the mobile object control unit 101 includes a SLAM (Simultaneous Localization and Mapping) unit 501, a driving route designation unit 502, a driving state control unit 503, a manually operated driving control unit 504, an autonomous driving control unit 505, a communication status measurement unit 506, a communication status determination unit 507, a communication status notification unit 508, a remote control receiving unit 509, a remote control driving control unit 510, and a driving control transmission unit 511.

[0061] As shown in FIG. 2, the mobile object storage unit 112 includes an environmental map data storage unit 601 , a driving route storage unit 602 , a driving state storage unit 603 , and a communication state storage unit 604 .

[0062] The SLAM unit 501 creates environmental map data of the surroundings of the mobile object 100, and performs SLAM to estimate the self-position of the mobile object 100, thereby creating the environmental map data. The created environmental map data and the travel route are stored in the mobile object storage unit 112. The SLAM unit 501 may simultaneously create the environmental map data of the surroundings of the mobile object 100 and estimate the self-position of the mobile object 100, for example, by using an image of the surrounding environment captured by the camera unit 102 or distance measurement information of the surrounding environment sensed by the sensor unit 103. The mobile object 100 can move autonomously based on the environmental map data created by the SLAM unit 501 and the estimated self-position.

[0063] In this embodiment, SLAM unit 501 will be described as using only sensor unit 103, but this embodiment is not limited to such an example. SLAM unit 501 may use, for example, only a camera unit, or may use both camera unit 102 and sensor unit 103. In that case, camera unit 102 or sensor unit 103 includes at least one imaging device or sensor.

[0064] The driving route designation unit 502 sets a route along which the mobile object 100 will travel during autonomous driving, using the environmental map data created by the SLAM unit 501. The driving route designation unit 502 may set a driving route, for example, by designating coordinates of the environmental map data created by the SLAM unit 501, or may set a driving route by designating a line connecting coordinates of the environmental map data.

[0065] The traveling state control unit 503 controls the state of the traveling control of the moving body 100 to any one of manual operation traveling, autonomous traveling, or remote operation traveling. The traveling state control unit 503 may control the state of the traveling control of the moving body 100 based on, for example, an operation of a moving body operation unit, an operation from the remote operation terminal 200, or a determination by the moving body 100.

[0066] The manual operation running control unit 504 causes the moving body 100 to run under the control of the moving body operating unit 111 of the moving body 100 when the running state control unit 503 is in the manual operation running state.

[0067] When the driving state control unit 503 is in an autonomous driving state, the autonomous driving control unit 505 calculates a driving route using the environmental map data and the stored driving route, and causes the moving body 100 to drive so as to follow the calculated driving route (i.e., causes the moving body 100 to drive autonomously).

[0068] The communication status measurement unit 506 quickly measures the wireless network status of the mobile unit 100 (e.g., wireless network power intensity, communication speed with the mobile unit operation system unit 300, etc.) when creating an environmental map or during autonomous driving, and stores the measurement results in the mobile unit memory unit 112.

[0069] The communication status determination unit 507 determines the network status (quality of communication connecting to the network 400) based on the measurement result of the communication status measurement unit 506. In other words, the communication status determination unit 507 functions as a "communication quality measurement unit" that acquires an index related to the communication quality (connection quality) of the mobile object 100 to the network 400.

[0070] The communication status notifying section 508 transmits the wireless network status to the remote control terminal 200 in accordance with the wireless network status measured by the communication status measuring section 506 .

[0071] The remote control receiving unit 509 receives control information for the moving body 100 from the remote control terminal 200 when the traveling state control unit 503 is in a remote control state, and outputs control information for controlling the traveling of the moving body 100 to the remote control traveling control unit 510. In addition, the remote control receiving unit 509 outputs information received from the remote control terminal 200 instructing the start or end of remote control traveling to the traveling state control unit 503.

[0072] The remote control driving control unit 510 drives the moving body 100 based on control information for controlling the driving of the moving body 100 received by the remote control receiving unit 509 when the driving state control unit 503 is in a remote control state.

[0073] The driving control transmission unit 511 determines the control to be transmitted depending on the driving control state of the moving body 100 of the driving state control unit 503. For example, when the driving control state of the moving body 100 is manual operation driving, the driving control transmission unit 511 controls the operation of the moving body 100 by the manual operation driving control unit 504. When the driving control state of the moving body 100 is autonomous driving, the operation of the moving body 100 is controlled by the autonomous driving control unit 505. When the driving control state of the moving body 100 is remote operation driving, the operation of the moving body 100 is controlled by the remote operation driving control unit 510.

[0074] The environment map data storage unit 601 accumulates and stores the environment map data created by the SLAM unit 501.

[0075] The travel route storage unit 602 accumulates and stores information on the travel route of the moving body 100 during remote-controlled travel.

[0076] The traveling state storage unit 603 accumulates and stores information on the traveling state of the mobile object 100 .

[0077] The communication status storage unit 604 accumulates and stores the measurement results of the communication status measured by the communication status measurement unit 506 .

[0078] (A-2) Operation of the First Embodiment An example of the operation of the moving body 100 in the remote operation system 10 according to this embodiment (an example of the control method according to the first embodiment) will be described.

[0079] In the remote operation system 10, first the mobile object operation system unit 300 starts up, and then the mobile object 100 and the remote operation terminal 200 each start up.

[0080] First, in order to create environmental map data along which the moving body 100 travels, the moving body operation unit 111 or the remote control terminal 200 is operated to change the moving body 100 to a state in which environmental map data is to be created.

[0081] When the traveling state control unit 503 is operated by the mobile body operation unit 111 or remote operation of the remote control terminal 200 to change the state to one in which environmental map data is created, it rewrites the state of the mobile body 100 stored in the traveling state storage unit 603 of the mobile body memory unit 112 to one in which environmental map data is created.

[0082] When the driving state storage unit 603 rewrites the state of the moving body 100 to a state in which environmental map data is created, the SLAM unit 501 and the manual operation driving control unit 504 or the remote operation driving control unit 510 start operating, and the moving body 100 is driven manually or remotely by the moving body operation unit 111 to create map data.

[0083] FIG. 3 is a flowchart showing an example of the operation of creating map data during manual operation driving of the mobile object control unit 101 in the remote operation system 10 according to the first embodiment.

[0084] As shown in FIG. 3, the manual operation traveling control unit 504 determines whether the traveling state of the moving object 100 is a map creation state (S101).

[0085] If the driving state is not a map creation state (S101 / No), the process waits until the driving state becomes a map creation state, and if the driving state is a map creation state (S101 / Yes), the driving control transmission unit 511 transmits a driving control signal of the manual operation driving control unit 504 to the moving unit 107, enabling the moving body 100 to be driven manually by the moving body operation unit 111, and confirms whether the SLAM unit 501 has started operating (S102).

[0086] If the SLAM unit 501 has not started operating (S102 / No), the process waits until the SLAM unit 501 starts operating. If the SLAM unit 501 has started operating (S102 / Yes), manual operation driving is started (S103), and while the moving body 100 is being manually operated to travel in a place where it is to travel autonomously (S104), the SLAM unit 501 creates map data and estimates its own position using input signals from the camera unit 102 and the sensor unit 103, and continues manual operation driving until the creation of the environmental map data is completed (S105 / No).

[0087] When the creation of the environment map data is completed (S105 / Yes), the manual operation traveling is stopped (S106), and the created environment map data is stored in the environment map data storage unit 601 of the mobile object storage unit 112 (S107).

[0088] The driving state in the driving state storage unit 603 is changed to a stopped state (S108), and the map data creation is completed.

[0089] When the creation of the environmental map data is completed, a travel route is set using the environmental map data in the travel route designation unit 502 by remote operation of the mobile object operation unit 111 or the remote operation terminal 200. When the setting of the travel route is completed, the set travel route is stored in the travel route storage unit 602 of the mobile object storage unit 112.

[0090] When the creation of the environmental map data and the setting of the driving route are completed, the driving state in the driving state storage unit 603 of the moving object memory unit 112 is changed to an autonomous driving state by remote control of the moving object operation unit 111 or the remote control terminal 200.

[0091] When the traveling state of the traveling state storage unit 603 of the moving object storage unit 112 is changed to an autonomous traveling state, the autonomous traveling control unit 505 operates.

[0092] FIG. 4 is a flowchart showing an example of the operation of the mobile object control unit 101 during autonomous driving in the remote operation system 10 according to the first embodiment.

[0093] As shown in FIG. 4, the autonomous driving control unit 505 determines whether the traveling state of the moving object 100 is an autonomous driving state (S201).

[0094] If the driving state is not an autonomous driving state (S201 / No), the process waits until the driving state becomes an autonomous driving state, and if the driving state is an autonomous driving state (S201 / Yes), the process checks whether the environmental map data is stored in the environmental map data storage unit 601 of the mobile object memory unit 112 and whether the driving route is stored in the driving route storage unit 602 of the mobile object memory unit 112 (S202).

[0095] If the environmental map data and the driving route have not been saved (S202 / No), the process returns to determining whether the vehicle is in an autonomous driving state (S201); if the environmental map data and the driving route have been saved (S202 / Yes), the communication status measurement unit 506 starts measuring the network status (S203), the measurement results of the network status are output to the communication status determination unit 507 while being stored in the communication status storage unit 604 of the mobile object memory unit 112, the environmental map data is used in the environmental map data storage unit 601 of the mobile object memory unit 112 to start autonomous driving (S204), the vehicle drives autonomously while generating a route between the driving routes so as to pass through the driving route (S205), and the communication status determination unit 507 makes a determination (S206).

[0096] Various determination methods can be applied as a specific method for measuring the communication status by the communication status measurement unit 506. For example, the communication status measurement unit 506 may measure the radio wave strength of the connected wireless network, may measure the communication speed of the connected wireless network, or may measure the throughput with the mobile operation system unit 300.

[0097] The communication status measurement unit 506 outputs the measurement result to the communication status determination unit 507 and the communication status holding unit 604 in the mobile unit memory unit 112 .

[0098] If the communication status determination unit 507 determines that the network status will deteriorate during autonomous driving (S206 / Yes), the communication status notification unit 508 notifies the remote control terminal that the network status has deteriorated (S207).

[0099] The specific method of the communication status determination unit 507 may be various methods. For example, when the communication status measurement unit 506 measures the radio wave strength of the connected wireless network, the communication status determination unit 507 may determine that the network status has deteriorated when the radio wave strength falls below a threshold (for example, radio wave strength of -67 dBm or less), when the communication status measurement unit 506 measures the communication speed of the connected wireless network, the communication status determination unit 507 may determine that the network status has deteriorated when the communication speed falls below a threshold (for example, communication speed of 2.5 Mbps or less), and when the throughput with the mobile operation system unit 300 is measured, the communication status determination unit 507 may determine that the network status has deteriorated when the throughput falls below a threshold (for example, communication speed of 2.5 Mbps or less). As described above, the communication status determination unit 507 determines that the quality of the network connection has deteriorated when the quality of the network connection falls below a predetermined level based on the above criteria.

[0100] The communication status determination unit 507 outputs the determination result to the communication status notification unit 508 .

[0101] If the entire travel route has been traversed and autonomous travel is complete (S208 / Yes), or if autonomous travel is not complete (S208 / No) and the travel state has changed (209 / YES), the measurement of the network situation, route generation, and autonomous travel are stopped (S210), the travel state is changed to a stopped state (S211), and the process ends. If autonomous travel is not complete (S208 / No) and the travel state has not changed (209 / NO), the process returns to the above-mentioned step S205 and continues autonomous travel.

[0102] When the remote control terminal 200 receives a notification of a worsening network condition from the mobile body 100, the remote control terminal 200 changes the driving state of the driving state storage unit 603 of the mobile body memory unit 112 of the mobile body 100 that sent the notification of the worsening network condition to a remotely controlled driving state in order to remotely control the mobile body 100 that sent the notification of the worsening network condition to drive it.

[0103] When the traveling state in the traveling state storage unit 603 of the moving object storage unit 112 is changed to a remotely controlled traveling state, the remotely controlled traveling control unit 510 operates.

[0104] Next, an example of the operation of the remote-operated traveling control unit 510 in the remote operation system 10 according to this embodiment will be described with reference to FIG.

[0105] FIG. 5 is a flowchart illustrating the flow of operations performed by the moving object control unit 101 during remote control traveling according to this embodiment.

[0106] As shown in FIG. 5, the remote-controlled driving control unit 510 determines whether the driving state of the moving object 100 is a remote-controlled driving state (S301).

[0107] If the driving state is not a remotely controlled driving state (S301 / No), the process waits until the driving state becomes a remotely controlled driving state, and if the driving state becomes a remotely controlled driving state (S301 / Yes), remotely controlled driving is started (S302), and remote control information is received to perform remote driving or change the driving route (S303).

[0108] If the remote control is not completed (S304 / No), the remote control continues, and if the remote driving is completed (S304 / Yes), the remote control driving is stopped (S305), the driving state is changed to a stopped state (S306), and the process ends.

[0109] The driving control transmission unit 511 transmits driving control information according to the driving state of the driving state storage unit 603. When the driving state of the driving state storage unit 603 is autonomous driving, the driving control transmission unit 511 transmits to the moving unit 107 the driving control information output from the autonomous driving control unit 505, and when the driving state of the driving state storage unit 603 is remote-controlled driving, the driving control transmission unit 511 transmits to the moving unit 107 the driving control information output from the autonomous driving control unit 505.

[0110] (A-3) Effects of the First Embodiment The first embodiment can provide the following advantages.

[0111] When the network condition of the moving body 100 of the first embodiment becomes bad, the moving body 100 transmits a notification to the remote control terminal. This allows the operator operating the remote control terminal 200 who has confirmed the notification to be prompted to take various measures for the moving body 100. For example, the network from the moving body 100 to the remote control terminal 200 switches the moving body 100 from an autonomous driving state to a remotely controlled driving state, and performs remotely controlled driving or changes the driving route, so that the moving body 100 moves to a place with good network conditions or changes the driving route to one with good network conditions, thereby preventing the network of the moving body 100 from being disconnected, and thus making it possible to further improve the operability of the remote operation system 10.

[0112] As described above, in the remote operation system 10 of the first embodiment, by notifying the remote remote control terminal 200 (operator OP) according to the wireless network conditions within the range in which each moving body 100 travels autonomously, it becomes possible to continue to control each moving body 100 even if there are locations with poor wireless network environments, and the operation rate of the operation center (remote control terminal 200 and operator OP) can be maintained.

[0113] (B) Second embodiment A second embodiment of the control device, control method, control program, and control system according to the present invention will be described below in detail with reference to the drawings. In this embodiment, an example in which the control device, control program, and control method of the present invention are applied to a mobile object control unit that controls a mobile object will be described. Also, in this embodiment, an example in which the control system of the present invention is applied to a remote operation system will be described.

[0114] (B-1) Configuration of the second embodiment Next, a remote operation system 10A according to a second embodiment will be described.

[0115] The overall configuration of a remote operation system 10A according to the second embodiment can also be shown using the above-mentioned Fig. 1. Note that the reference characters in parentheses in Fig. 1 are reference characters used only in the second embodiment.

[0116] The remote operation system 10A differs from the first embodiment in that the moving body 100 is replaced with a moving body 100A.

[0117] FIG. 6 is a block diagram showing a detailed configuration of a moving object 100A according to the second embodiment.

[0118] The moving object 100A is different from the first embodiment in that the moving object control unit 101 is replaced with a moving object control unit 101A. In Fig. 6, the same or corresponding parts as in Fig. 2 are denoted by the same or corresponding reference numerals.

[0119] In the following, the second embodiment will be described, focusing on the differences from the first embodiment, and a description of the parts that overlap with the first embodiment will be omitted.

[0120] Next, the detailed configuration of the moving body 100A will be described.

[0121] 6, the moving body 100A includes a SLAM unit 501, a driving route designation unit 502, a driving state control unit 503, a manually operated driving control unit 504, an autonomous driving control unit 505, a communication status measurement unit 506, a remote control receiving unit 509, a remote control driving control unit 510, a driving control transmission unit 511, and a map communication status reflection unit 512. That is, the moving body 100A differs from the first embodiment in that the communication status determination unit 507 and the communication status notification unit 508 are excluded and the map communication status reflection unit 512 is added.

[0122] The map communication status reflecting unit 512 reflects the measurement result of the communication status measuring unit 506 in the environment map data held in the environment map data holding unit 601 .

[0123] (B-2) Operation of the Second Embodiment An example of the operation of the mobile object control unit 101A in the remote operation system 10A according to this embodiment (an example of the control method according to the second embodiment) will be described.

[0124] FIG. 7 is a flowchart showing an example of the operation of the mobile object control unit 101A (the autonomous driving control unit 505 and the map communication status reflecting unit 512) in the remote operation system 10A according to the second embodiment.

[0125] The basic operation of the process in the mobile object control unit 101A according to the second embodiment is similar to that of the mobile object control unit 101 described in the first embodiment. The following will mainly describe in detail the operation related to the map communication status reflecting unit 512, which is different from the first embodiment.

[0126] In Fig. 7, steps S205 to S207 are removed from the flow chart of Fig. 4, and step S401 is added, which is different from the first embodiment. In the mobile object control unit 101A according to the second embodiment, the processes other than those described above are the same as those in the first embodiment, so detailed explanations will be omitted.

[0127] If the environmental map data and the travel route are not stored (S202 / No), the mobile control unit 101A returns to determining whether the mobile control unit 101A is in an autonomous travel state (S201), and if the environmental map data and the travel route are stored (S202 / Yes), the mobile control unit 101A starts measuring the network state (S203), stores the measurement result of the network state in the communication state storage unit 604 of the mobile storage unit 112, and starts autonomous travel (S204). The mobile control unit 101A performs autonomous travel while generating a route between the travel routes so as to generate a travel route that passes through the travel route and does not pass through places where the network state is poor, using the environmental map data stored in the environmental map data storage unit 601 of the mobile storage unit 112 and the communication state of the network stored in the communication state storage unit 604 of the mobile storage unit 112 (S401). After that, the mobile control unit 101A proceeds to step S208 as in the first embodiment, and detailed description thereof will be omitted.

[0128] The map communication status reflecting unit 512 may apply various determination methods as a method of reflecting the communication status of the network on the map. For example, when the communication status measuring unit 506 measures the radio wave strength of the connected wireless network, the map communication status reflecting unit 512 may reflect in the environmental map data that the vehicle does not autonomously travel within a certain section of the current position (for example, within 1 m from the current position) when the radio wave strength is below a threshold (for example, radio wave strength of -67 dBm or less), when the communication status measuring unit 506 measures the communication speed of the connected wireless network, the map communication status reflecting unit 512 may reflect in the environmental map data that the vehicle does not autonomously travel within a certain section of the current position (for example, within 1 m from the current position) when the communication speed is below a threshold (for example, communication speed of 2.5 Mbps or less), and when the throughput with the mobile operation system unit 300 is measured, the map communication status reflecting unit 512 may reflect in the environmental map data that the vehicle does not autonomously travel within a certain section of the current position (for example, within 1 m from the current position) when the throughput is below a threshold (for example, communication speed of 2.5 Mbps or less).

[0129] In this embodiment, a method of measuring the communication status during autonomous driving and updating the environmental map data has been described, but the present invention is not limited to this method and various methods can be applied. For example, when creating the environmental map data, the measurement results of the network communication status and the measured position information may be stored in the communication status storage unit 604, a heat map of the network status may be created in advance, and the areas where the network status is poor may be reflected in advance in the environmental map data stored in the environmental map data storage unit 601, or the heat map of the network status created in advance may be updated and reflected in the environmental map data.

[0130] (B-3) Effects of the Second Embodiment According to the second embodiment, in addition to the effects of the first embodiment, the following effects can be achieved.

[0131] In the mobile object control unit 101A according to the second embodiment, the location where the network condition is poor is reflected in the environmental map data, so that the mobile object 100 does not travel through the location where the network condition is poor during autonomous travel. As a result, in the second embodiment, it is possible to prevent the network connection of the mobile object 100A from being interrupted, and to further improve the operability of the remote operation system 10.

[0132] (C) Third embodiment Hereinafter, a third embodiment of a control device, a control method, a control program, and a control system according to the present invention will be described in detail with reference to the drawings.

[0133] In this embodiment, an example will be described in which the control device, the control program, and the control method of the present invention are applied to a mobile object control unit that controls a mobile object. Also, in this embodiment, an example will be described in which the control system of the present invention is applied to a remote operation system.

[0134] (C-1) Configuration of the third embodiment FIG. 8 is a block diagram of the overall configuration of a remote operation system 10B according to this embodiment.

[0135] The basic configuration of the remote operation system 10B according to the third embodiment is similar to the configuration of the remote operation system 10 described in the first embodiment, so in FIG. 8, the same or corresponding parts as those in FIG. 1 are denoted by the same or corresponding symbols.

[0136] In the following, the third embodiment will be described, focusing on the differences from the first embodiment, and a description of the parts that overlap with the first embodiment will be omitted.

[0137] As shown in FIG. 8, the remote operation system 10B of the third embodiment differs from the remote operation system 10 of the first embodiment in that the moving body 100 is replaced with a moving body 100B, and a communication relay moving body 600 is further added.

[0138] 8, the remote operation system 10B includes a mobile object 100B, a communication relay mobile object 600, a remote control terminal 200, and a mobile object operation system unit 300. The mobile object 100B, the remote control terminal 200, and the mobile object operation system unit 300 are connected via a network 400 so as to be able to transmit and receive data to and from each other.

[0139] The communication relay mobile body 600 has the same basic configuration as the mobile body 100B, but differs from the mobile body 100B in that it includes a communication relay unit 120 for relaying a network. In other words, the communication relay mobile body 600 may be another mobile body 100B itself equipped with the communication relay unit 120.

[0140] Next, a mobile object control unit 101B of a remote operation system 10B according to a third embodiment of the present invention will be described with reference to FIG.

[0141] The moving object 100B differs from the first embodiment in that the moving object control unit 101 is replaced with a moving object control unit 101B.

[0142] The basic configuration of the mobile body control unit 101B of the remote operation system 10B in the third embodiment is similar to the configuration of the mobile body control unit 101 of the remote operation system 10 described in the first embodiment, so in Figure 9, the same or corresponding parts as in Figure 1 are given the same symbols or corresponding symbols.

[0143] 9, the mobile object control unit 101B includes a SLAM unit 501, a travel route designation unit 502, a travel state control unit 503, a manually operated travel control unit 504, an autonomous travel control unit 505, a communication situation measurement unit 506, a communication situation determination unit 507, a communication situation notification unit 508, a communication relay mobile object call request unit 513, a remote operation receiving unit 509, a remote operation travel control unit 510, and a travel control transmission unit 511. The mobile object control unit 101B differs from the first embodiment in that it includes the communication relay mobile object call request unit 513.

[0144] When the communication status determination unit 507 determines that the network status is deteriorating (the quality of the network connection is determined to be below a specified level), the communication relay mobile call request unit 513 requests the mobile operation system unit 300 to move the communication relay mobile unit 600 equipped with a network repeater to the position or vicinity of the mobile unit 100B.

[0145] (C-2) Operation of the Third Embodiment An example of the operation of the mobile object control unit 101B in the remote operation system 10B according to this embodiment (an example of the control method according to the third embodiment) will be described.

[0146] 10 and 11 are flowcharts showing an example of the operation of the mobile object control unit 101B (the autonomous driving control unit 505 and the communication relay mobile object call request unit 513) in the remote operation system 10B according to the second embodiment.

[0147] The basic operation of the processing in the mobile object control unit 101B according to the second embodiment is similar to that in the mobile object control unit 101 described in the first embodiment.

[0148] Fig. 10 is different from the first embodiment in that step S207 is removed and steps S501 to S505 are added to the flowchart of Fig. 4. In the mobile object control unit 101B according to the third embodiment, the processes other than those described above are the same as those in the first embodiment, so detailed explanations will be omitted.

[0149] The following will be described in detail, focusing on the operation of the communication relay mobile unit call request unit 513, which is a difference from the first embodiment.

[0150] If the environmental map data and the driving route have not been saved (S202 / No), the mobile object control unit 101B returns to determining whether the mobile object is in an autonomous driving state (S201); if the environmental map data and the driving route have been saved (S202 / Yes), the mobile object control unit 101B starts measuring the network state (S203), starts autonomous driving (S204), and drives autonomously while generating a route between the driving routes so as to pass through the driving route using the environmental map data in the environmental map data storage unit 601 of the mobile object memory unit 112 (S205).

[0151] When the network conditions deteriorate during autonomous driving (S206 / Yes), the mobile control unit 101B requests the mobile operation system unit 300 to move the communication relay mobile unit to the current position of the mobile unit (S501), changes the driving state of the mobile unit 100 to a communication relay mobile unit movement waiting state (S502), stops measuring the network conditions, route generation, and autonomous driving (S211), and returns to determining whether the mobile unit is in an autonomous driving state (S201).

[0152] When it is determined in step S201 that the vehicle is in an autonomous driving state, and the vehicle is not in an autonomous driving state (S201 / No), the process waits until the vehicle becomes in an autonomous driving state.

[0153] FIG. 11 is a flowchart showing an example of the operation of the mobile object control unit 101B (autonomous driving control unit 505 and communication relay mobile object call request unit 513) until the movement of the communication relay mobile object 600 is completed after the above-mentioned step S502.

[0154] After the process of step S502 described above, the autonomous driving control unit 505 determines whether the driving state of the driving state storage unit 603 is a communication relay mobile object movement waiting state (S601).

[0155] If the communication relay mobile object is not in a waiting state to move (S601 / No), the process waits until the communication relay mobile object is in a waiting state to move. If the communication relay mobile object is in a waiting state to move (S601 / Yes), the process waits until the communication relay mobile object has completed its movement (S602 / No). When the movement is completed (S602 / Yes), the process changes the traveling state to an autonomous traveling state (S602), terminates the processing of the flowchart in Figure 11, and returns to the processing of the flowchart in Figure 10 (processing of step S201 (determining whether the traveling state is an autonomous traveling state)).

[0156] When the mobile unit 100 moves to a place where the network condition is good again, a return instruction is sent from the mobile unit operation system unit 300 to the communication relay mobile unit 600 so as to return to the position before the movement, and the communication relay mobile unit 600 returns to the waiting place.

[0157] When the communication relay mobile unit 600 receives the return instruction, it travels back to the position before it moved.

[0158] 12 and 13 are explanatory diagrams showing an example of the operation of the mobile object 100B and the communication relay mobile object 600 in the third embodiment.

[0159] 12 and 13 are diagrams (conceptual diagrams) of the area in which the moving body 100B is located, as viewed from above. In addition, in Fig. 12 and Fig. 13, the current position of the moving body 100B is P1, the route by which the moving body 100B has traveled to P1 (past travel route) is indicated by a dashed arrow, and the route along which the moving body 100B plans to travel in the future (planned travel route) is indicated by a dashed line.

[0160] In Fig. 12, it is assumed that the mobile unit control unit 101B detects a deterioration in communication conditions (detects that the network connection quality has fallen below a predetermined level) when the mobile unit 100B reaches position P1. Then, it is assumed that the mobile unit control unit 101B requests the mobile unit operation system unit 300 to move the communication relay mobile unit 600 to position P2, which is a predetermined distance or less from the current position (position P1) on the past movement route. As a result, as shown in Fig. 13, when the communication relay mobile unit 600 reaches position P2, the range in which radio waves of the network can reach is expanded, and the mobile unit control unit 101B can ensure network connection even if the mobile unit control unit 101B resumes autonomous traveling (traveling along the planned movement route) of the mobile unit control unit 101B.

[0161] The predetermined position P2 may be, for example, a position that is a certain distance away from the current position (position P1) and where no deterioration in the communication situation is detected. It may also be a position that is a certain distance away from the current position (position P1) and where no deterioration in the communication situation is detected.

[0162] In this case, the distance X between the positions P1 and P2 is not limited as long as it is a distance or a positional relationship that does not interfere with relay communication. For example, the distance X may be a distance at which the mobile control unit 101B and the communication relay mobile unit 600 do not come into contact with each other. In addition, the distance X may be set to a certain extent (for example, about 5 m to 10 m) so that a position with a better network connection than the position P1 (the position of the communication relay mobile unit 600) is set to the position P2, or the distance X may be set to a certain extent (for example, about 1 m to 5 m) so that the area where the network is connected is as wide as possible. Furthermore, the distance X may be set to 0 m (that is, in FIG. 13, the communication relay mobile unit 600 is placed below the position P1, or the mobile unit 100 is moved and placed when the communication relay mobile unit 600 approaches). In this case, the mobile unit 100B needs to move to a position where it does not come into contact with the communication relay mobile unit 600.

[0163] (C-3) Effects of the Third Embodiment According to the third embodiment, the following advantages can be achieved compared to the first embodiment.

[0164] In the mobile object control unit 101B according to the third embodiment, the communication relay mobile object 600 is moved to a place where the network condition is poor, and then the autonomous traveling of the mobile object 100 is resumed. As a result, the network condition improves during the autonomous traveling of the mobile object 100B, so that the network of the mobile object 100B is not interrupted, and the operability of the remote operation system 10B can be further improved.

[0165] (D) Fourth embodiment Hereinafter, a fourth embodiment of the control device, control method, control program, and control system according to the present invention will be described in detail with reference to the drawings. In this embodiment, an example in which the control device, control program, and control method of the present invention are applied to a mobile object control unit that controls a mobile object will be described. Also, in this embodiment, an example in which the control system of the present invention is applied to a remote operation system will be described.

[0166] FIG. 14 is a block diagram of the overall configuration of a remote operation system 10C according to the fourth embodiment.

[0167] As shown in Fig. 14, the remote operation system 10C includes N moving objects 100-1, 100-2, ... 100-N, M remote control terminals 200-1, 200-2, ... 200-M, and a moving object operation system unit 300. M and N are any integers independent of each other and equal to or greater than 2. However, in order to efficiently operate the moving object 100 by the remote control terminal 200, M may be an integer smaller than N. That is, in the remote operation system 10C of the fourth embodiment, a plurality of moving objects 100 and remote control terminals 200 are arranged, and any moving object 100 and any remote control terminal 200 can communicate with and be controlled by each other.

[0168] In the remote operation system 10C, N mobile objects 100-1, 100-2, ..., 100-N are connected to M remote control terminals 200-1, 200-2, ..., 200-M via a network 400 so as to be able to transmit and receive data to and from each other. The M remote control terminals 200-1, 200-2, ..., 200-M are each operated by a separate operator OP-1, OP-2, ..., OP-M, and can remotely control each of the mobile objects 100.

[0169] As described above, in the remote operation system 10C of the fourth embodiment, for example, a plurality of moving objects 100 can be simultaneously operated and monitored, and therefore the operating efficiency of the moving objects 100 can be improved.

[0170] (E) Fifth embodiment Hereinafter, a fifth embodiment of the control device, control method, control program, and control system according to the present invention will be described in detail with reference to the drawings. In this embodiment, an example in which the control device, control program, and control method of the present invention are applied to a mobile object control unit that controls a mobile object will be described. Also, in this embodiment, an example in which the control system of the present invention is applied to a remote operation system will be described.

[0171] FIG. 15 is a block diagram of the overall configuration of a remote operation system 10D according to the fifth embodiment.

[0172] In the fourth embodiment, the mobile body 100B is configured to be connected to the network 400 via one communication relay mobile body 600, but in the remote operation system 10D of the fifth embodiment, a plurality of communication relay mobile bodies 600 are wirelessly relayed (for example, relayed by wireless multi-hop communication) to connect the mobile body 100B to the network 400. In addition, in Fig. 15, the mobile body 100B and the remote control terminal 200 are configured to be multiple units, but these devices may be configured to be one unit.

[0173] 15, the remote operation system 10D of the fifth embodiment includes, for example, N mobile objects 100B-1, 100B-2, ... 100B-N, M remote control terminals 200-1, 200-2, ... 200-M, a mobile object operation system unit 300, and L communication relay mobile objects 600-1, 600-2, ... 600-L. M, N, and L are any integers independent of each other and equal to or greater than 2. However, in order to efficiently operate the mobile object 100 by the remote control terminal 200, M may be an integer smaller than N and L.

[0174] N mobile objects 100B-1, 100B-2, ... 100B-N and L communication relay mobile objects 600-1, 600-2, ... 600-L are connected to M remote control terminals 200-1, 200-2, ... 200-M via a network 400 so as to be able to transmit and receive data to and from each other. The M remote control terminals 200-1, 200-2, ... 200-M are each operated by a separate operator OP-1, OP-2, ... OP-M, and can remotely control each of the mobile objects 100B. In this embodiment, the remote control terminal 200 may be one (there may also be one operator OP).

[0175] FIG. 16 is an explanatory diagram showing an example of the operation of the mobile object 100B-1 and the communication relay mobile object 600 in the fifth embodiment.

[0176] Fig. 16 is a diagram (conceptual diagram) of the area in which the mobile object control unit 101D is arranged, as viewed from above. In Fig. 16, the current position of the mobile object 100B-1 is P7, the route that the mobile object 100B-1 has taken to P7 (past route of travel) is indicated by a dashed arrow, and the route that the mobile object 100B-1 plans to take in the future (planned route of travel) is indicated by a dashed line.

[0177] 16, positions P1 to P7 are shown on the past movement route of the moving object 100B-1. It is assumed that the moving object 100B-1 has moved to the positions P1 to P7 in this order.

[0178] 16, communication relaying mobile objects 600-1 to 600-3 are arranged along the past movement route of the moving object 100B-1. As seen from the moving object 100B-1, the communication relaying mobile objects 600-1 to 600-3 are arranged in this order. In FIG. 16, the positions of the communication relaying mobile objects 600-1 to 600-3 are P5, P3, and P1, respectively.

[0179] 16, the mobile object 100B-1 is in a state of connecting to the network 400 by relaying (by wireless communication) the communication relay mobile objects 600-1 to 600-3. The wireless communication method (relay method) used by the mobile object 100B-1 and the communication relay mobile objects 600-1 to 600-3 is not limited, and may be, for example, by multi-hop wireless communication (for example, multi-hop wireless communication by 920 MHz band wireless communication).

[0180] Next, a process of requesting the mobile operation system unit 300 to move the communication relay mobile objects 600-1 to 600-3 while the mobile object 100B-1 is moving to the positions P1 to P7 will be described.

[0181] Here, when the mobile unit 100B-1 reaches position P1, a deterioration in the network connection state is detected (network connection quality is detected to be below a predetermined level), and the mobile unit operation system unit 300 is requested to place the communication relay mobile unit 600-1 at position P2, and the mobile unit 100B-1 reaches position P2. When the communication relay mobile unit 600-1 has completed moving to position P2, the mobile unit 100B-1 connects to the network 400 via the communication relay mobile unit 600-1. The process of requesting the mobile unit operation system unit 300 to place (move) the communication relay mobile unit 600 is the same as in the third embodiment, and therefore a detailed description thereof will be omitted.

[0182] Thereafter, the mobile unit 100B-1 resumes autonomous traveling (traveling along the planned travel route) from position P1, and when it reaches position P3, it detects a deterioration in the network connection state again, and requests the mobile unit operation system unit 300 to place the communication relay mobile unit 600-2 at position P4, which it assumes has been reached. When the communication relay mobile unit 600-2 has completed its movement to position P4, the mobile unit 100B-1 connects to the network 400 via the communication relay mobile units 600-2 and 600-1.

[0183] Thereafter, the mobile unit 100B-1 resumes autonomous traveling (traveling along the planned travel route) from position P3, and when it reaches position P5, it detects a deterioration in the network connection state again, and requests the mobile unit operation system unit 300 to place the communication relay mobile unit 600-3 at position P6, which it assumes has been reached. When the communication relay mobile unit 600-3 has completed its movement to position P6, the mobile unit 100B-1 connects to the network 400 via the communication relay mobile units 600-1 to 600-3. Thereafter, the mobile unit 100B-1 resumes autonomous traveling (traveling along the planned travel route) from position P5, and reaches position P7, resulting in the state shown in FIG. 16.

[0184] As described above, in the remote operation system 10D of the fifth embodiment, by moving multiple mobile bodies 100B and multiple communication relay mobile bodies 600 in coordination with each other, it is possible to dynamically construct an area with good network conditions, thereby further improving the operating efficiency of the mobile body 100B compared to the third and fourth embodiments.

[0185] (F) Other embodiments The present invention is not limited to the above-described embodiments, and may include modified embodiments such as those exemplified below.

[0186] (F-1) In the above embodiment, the preferred embodiment of the present invention has been described in detail with reference to the attached drawings, but the present invention is not limited to such an example. It is clear that a person having ordinary knowledge in the technical field to which the present invention pertains can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present invention.

[0187] (F-2) In the mobile body 100 (mobile body control unit 101) of the first embodiment, a method of measuring the communication status during autonomous driving and determining the network status has been described, but the method is not limited to this method and various methods can be applied. For example, in the mobile body 100 (mobile body control unit 101), when creating environmental map data, the measurement results of the network communication status and the measured position information are stored in the communication status storage unit 604, a heat map of the network status is created in advance, and a notification is given when the mobile body 100 moves to a place where the network status in the heat map is bad by autonomous driving, or a heat map of the network status is created in advance, and a notification is given when the mobile body 100 moves to a place where the network status is bad by autonomous driving while updating the heat map of the network status during autonomous driving. [Explanation of symbols]

[0188] 10, 10A, 10B, 10C, 10D... remote operation system, 100, 100A, 100B... mobile body, 102... camera unit, 103... sensor unit, 104... display unit, 105... speaker, 106... microphone, 107... mobile unit, 108... mobile body LAN hub unit, 109... mobile body communication unit, 110... AI unit, 111... mobile body operation unit, 112... mobile body memory unit, 113... mobile body battery unit, 200... remote control terminal, 201... remote control terminal control unit, 202... camera unit, 203... sensor unit, 204... display unit, 205... speaker, 206... microphone, 207... remote control unit, 208... remote control terminal LAN hub unit, 209... remote control terminal communication unit, 2 10... remote control terminal memory unit, 211... power supply unit, 300... mobile operation system unit, 400... network, 501... SLAM unit, 502... driving route designation unit, 503... driving state control unit, 504... manual operation driving control unit, 505... autonomous driving control unit, 506... communication status measurement unit, 507... communication status determination unit, 508... communication status notification unit, 509... remote control receiving unit, 510... remote control driving control unit, 511... driving control transmitting unit, 512... map communication status reflection unit, 513... communication relay mobile unit call request unit, 600... communication relay mobile unit, 601... environmental map data storage unit, 602... driving route storage unit, 603... driving status storage unit, 604... communication status storage unit

Claims

1. A control device for controlling an operation of a moving object, An autonomous driving control unit that autonomously drives the moving body along a predetermined route; a remote-controlled driving control unit that switches the driving of the moving body from the autonomous driving by the autonomous driving control unit to a remote-controlled driving by a remote terminal operated by an operator, and drives the moving body based on an input operation from the remote terminal; a communication quality measurement unit that measures a communication quality of communication connected to a network during the autonomous operation traveling of the moving body, A predetermined operation is performed when the communication quality measured by the communication quality measurement unit is equal to or lower than a predetermined value. A control device comprising:

2. 2. The control device according to claim 1, further comprising a communication quality communication unit that notifies the remote terminal of the communication quality when the communication quality measured by the communication quality measurement unit is equal to or lower than a predetermined value.

3. The control device according to claim 1, further comprising a map communication quality reflection unit that reflects the measurement results of the communication quality measurement unit mounted on the mobile body in environmental map data held by the mobile body based on the measurement results of the communication quality measurement unit mounted on the mobile body.

4. The control device according to claim 3 , characterized in that the map communication quality reflection unit reflects in the environmental map data locations where the communication quality is below a predetermined level, thereby preventing the moving body from traveling therethrough during the autonomous traveling.

5. The control device according to claim 1, further comprising a communication relay mobile unit call request means for requesting an external device to move another mobile unit equipped with a communication relay means for relaying wireless communication of the mobile unit when the communication quality measured by the communication quality measurement unit is below a predetermined level.

6. 6. The control device according to claim 5, wherein said communication relay mobile unit call request means moves said other mobile unit to a current position of said mobile unit or to a position at a predetermined distance or less from said mobile unit.

7. A computer mounted on a control device that controls the operation of a moving object, An autonomous driving control unit that autonomously drives the moving body along a predetermined route; a remote-controlled driving control unit that switches the driving of the moving body from the autonomous driving by the autonomous driving control unit to a remote-controlled driving by a remote terminal operated by an operator, and drives the moving body based on an input operation from the remote terminal; A communication quality measurement unit that measures a communication quality of communication connected to a network during the autonomous operation traveling of the moving body; and functioning as a means for performing a predetermined operation when the communication quality measured by the communication quality measuring unit is equal to or lower than a predetermined level. A control program comprising:

8. A control method performed by a control device for controlling an operation of a moving object, comprising: The control device includes an autonomous driving control unit, a remote control driving control unit, and a communication quality measurement unit, The autonomous driving control unit causes the moving body to autonomously drive along a predetermined route, the remote-operated travel control unit switches the travel of the moving body from the autonomous travel by the autonomous travel control unit to remote-operated travel by a remote terminal operated by an operator, and causes the moving body to travel based on an input operation from the remote terminal; The communication quality measurement unit measures communication quality during the autonomous operation traveling of the moving body, A predetermined operation is performed when the communication quality measured by the communication quality measurement unit is equal to or lower than a predetermined value. A control method comprising:

9. A control system comprising a mobile object and a remote terminal operated by an operator, the mobile object being equipped with a control device for controlling an operation of the mobile object, The control device includes: An autonomous driving control unit that autonomously drives the moving body along a predetermined route; a remote-controlled driving control unit that switches the driving of the moving body from the autonomous driving by the autonomous driving control unit to remote-controlled driving by the remote terminal operated by the operator, and drives the moving body based on an input operation from the remote terminal; a communication quality measurement unit that measures a communication quality of communication connected to a network during the autonomous operation traveling of the moving body, A predetermined operation is performed when the communication quality measured by the communication quality measurement unit is equal to or lower than a predetermined value. A control system comprising:

Images