Display device, communication system, display control method, and program

Abstract

The present invention provides a display device, a communication system, a display control method, and a program that provide a display screen that allows the user to easily select the destination of a moving object. [Solution] In the communication system 1, the display device 50 for performing a predetermined operation on a mobile body 10 (control device 30) moving within a predetermined base acquires an image of a part of the base taken by the mobile body 10, and displays the acquired image as information related to setting the destination for the autonomous movement of the mobile body 10.

Description

Technical Field

[0001] The present disclosure relates to a display device, a communication system, a display control method and a program.

Background Art

[0002] Robots that are installed at bases such as factories and warehouses and can autonomously move within the bases are known. Such robots are used, for example, as inspection robots or service robots, and can perform tasks such as inspection of equipment inside the base instead of workers.

[0003] Furthermore, robots that autonomously move to a specified destination by designating the robot's destination based on location information or a name specified by a user are already known. For example, Patent Document 1 discloses the content of designating a moving destination of a moving body based on position information arbitrarily selected by a user, and causing an autonomous mobile robot to autonomously move to the designated moving destination.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0005] However, in conventional methods, when setting the destination of a moving body, the user specifies a character string such as position information or a keyword, which poses a problem that it is difficult to appropriately grasp the content of the presented information.

Means for Solving the Problem

[0006] To solve the above-mentioned problems, the invention according to claim 1 is a display device for performing a predetermined operation on a mobile body moving within a predetermined base, comprising: acquisition means for acquiring a captured image of a part of the base taken by the mobile body; and display control means for displaying the acquired captured image as information related to setting the destination of the autonomous movement of the mobile body. [Effects of the Invention]

[0007] The present invention provides a display screen that allows the user to easily select the destination of a moving object. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram shows an example of the overall configuration of a communication system. [Figure 2] This figure shows a schematic example of the configuration of a mobile unit. [Figure 3] This figure shows an example of the hardware configuration of a mobile device. [Figure 4] This figure shows an example of the hardware configuration of a display device. [Figure 5] This diagram shows an example of the functional configuration of a communication system. [Figure 6] This is a conceptual diagram showing an example of a destination candidate management table. [Figure 7] This is a conceptual diagram showing an example of a route information management table. [Figure 8] This is a conceptual diagram showing an example of a map information management table. [Figure 9] This is a sequence diagram showing an example of the process for registering a candidate destination. [Figure 10] This figure shows an example of the user interface. [Figure 11] This is a sequence diagram showing an example of manual operation of a mobile object. [Figure 12] This flowchart shows an example of the process for registering destination candidates based on the movement status of a moving object. [Figure 13]It is a sequence diagram showing an example of destination setting processing for a moving body. [Figure 14] It is a diagram showing an example of a selection screen. [Figure 15] It is a diagram showing an example of a selection screen. [Figure 16] It is a flowchart showing an example of autonomous movement processing for a moving body. [Figure 17] It is a diagram showing an example of an operation screen during autonomous movement of a moving body. [Figure 18] It is a diagram showing an example of an operation screen during autonomous movement of a moving body. [Figure 19] It is a diagram showing a first modification of the selection screen. [Figure 20] It is a diagram showing a second modification of the selection screen. [Figure 21] It is a diagram showing a third modification of the selection screen. [Figure 22] It is a diagram showing a fourth modification of the selection screen. [Figure 23] It is a diagram showing a fifth modification of the selection screen. [Figure 24] It is a diagram showing an example of the overall configuration of a communication system according to a first modification of the embodiment. [Figure 25] It is a diagram showing an example of the functional configuration of a communication system according to the first modification of the embodiment. [Figure 26] It is a sequence diagram showing an example of destination candidate registration processing according to the first modification of the embodiment. [Figure 27] It is a sequence diagram showing an example of destination setting processing for a moving body according to the first modification of the embodiment. [Figure 28] It is a diagram showing an example of the overall configuration of a communication system according to a second modification of the embodiment. [Figure 29] It is a sequence diagram showing an example of destination setting processing for a moving body according to the second modification of the embodiment. [Figure 30] It is a diagram showing an example of an image search screen. [Figure 31] It is a diagram for schematically explaining an example of a movement route of a moving body. [Figure 32]This is a conceptual diagram showing an example of a location management table. [Figure 33] This is a conceptual diagram showing an example of an area information management table. [Figure 34] This figure shows an example of a settings screen. [Figure 35] This diagram shows an example of the functional configuration of a communication system. [Modes for carrying out the invention]

[0009] The embodiments for carrying out the invention will be described below with reference to the drawings. In the description of the drawings, the same elements will be denoted by the same reference numeral, and redundant explanations will be omitted.

[0010] ●Embodiment● ● System Configuration Figure 1 shows an example of the overall configuration of a communication system. The communication system 1 shown in Figure 1 is a system that allows a user to remotely control a mobile object 10 that is moving within a predetermined location.

[0011] Communication system 1 includes a mobile device 10 and a display device 50 located at a predetermined location. The mobile device 10 and the display device 50 constituting communication system 1 can communicate via a communication network 100. The communication network 100 is constructed using the Internet, a mobile communication network, a LAN (Local Area Network), etc. In addition to wired communication, the communication network 100 may also include wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0012] Mobile unit 10 is a robot installed at a target site and capable of autonomous movement within that site. This autonomous movement includes the robot learning (machine learning) of previously traveled routes within the target site and using the results of this learning to move autonomously within the site. Furthermore, autonomous movement may also involve autonomous movement within the site following a pre-set route or using techniques such as line tracing. In addition, mobile unit 10 can also be moved by manual operation from a remote user. That is, mobile unit 10 can move within the target site while switching between autonomous movement and manual operation by the user. Furthermore, mobile unit 10 can perform predetermined tasks such as inspection, maintenance, transportation, or light work while moving within the target site. Here, mobile unit 10 refers to a robot in a broad sense, and any robot capable of both autonomous movement and movement via remote operation by a user is acceptable. Mobile unit 10 also includes automobiles that can switch between autonomous driving and manual operation via remote control. Furthermore, the mobile body 10 also includes aircraft such as drones, multicopters, or unmanned aerial vehicles.

[0013] Furthermore, the target locations where the mobile unit 10 will be installed include, for example, outdoor locations such as offices, factories, construction sites, substations, farms, fields, or cultivated land or disaster sites, or indoor locations such as offices, schools, factories, warehouses, commercial facilities, hospitals or nursing homes. In other words, the target locations can be any locations where there is a need to have the mobile unit 10 take over tasks that were previously performed manually.

[0014] The display device 50 is located at a management location different from the target location and is a computer such as a notebook PC (Personal Computer) used by an operator (user) who performs predetermined operations on the mobile device 10. The operator, at the management location such as an office, uses the operation screen displayed on the display device 50 to perform operations on the mobile device 10, such as moving it or performing predetermined tasks. For example, the operator remotely controls the mobile device 10 while viewing an image of the target location displayed on the display device 50.

[0015] Figure 1 shows an example where one mobile unit 10 and one display device 50 are connected via a communication network 100. However, the display device 50 may be configured to connect to multiple mobile units 10 located at a single target site, or to connect to mobile units 10 located at different target sites. Also, Figure 1 shows an example where the display device 50 is located at a remote management site different from the target site where the mobile units 10 are installed. However, the display device 50 may be located within the target site where the mobile units 10 are installed. Furthermore, the display device 50 is not limited to a notebook PC, but may be, for example, a desktop PC, tablet terminal, smartphone, or wearable device.

[0016] Conventionally, when an operator sets the destination of an autonomously moving object, the operator must specify an arbitrary string such as the name or location information of the destination. For example, if the person who sets the string and the person who sets the destination are different, the person who sets the destination may have difficulty properly understanding the content of the information provided because they do not know the name of the destination.

[0017] Therefore, the communication system 1 automatically takes photographs of the area around the mobile body 10 using the camera 12 mounted on the mobile body 10, depending on the movement state of the mobile body 10, and registers the captured images in advance as candidates for destinations such as the mobile body 10's destination. The system then displays the captured images indicating the registered candidates for destinations on the display device 50. In this way, the communication system 1 can provide a display screen that allows the user to easily select the destination of the mobile body 10.

[0018] ● Configuration of the mobile unit Next, the specific configuration of the mobile body 10 will be explained using Figure 2. Figure 2 is a diagram showing an example of a schematic configuration of the mobile body. Note that the configuration of the mobile body 10 shown in Figure 2 may have components added or removed as needed.

[0019] The mobile body 10 shown in Figure 2 comprises a housing 11 equipped with a control device 30 for controlling the processing or operation of the mobile body 10, a shooting device 12, a support member 13, a display 14, a moving mechanism 15 (15a, 15b) for moving the mobile body 10, and a movable arm 16 for causing the mobile body 10 to perform a predetermined task (operation). Of these, the housing 11 is located in the body portion of the mobile body 10 and houses the control device 30, etc., for controlling the processing or operation of the mobile body 10.

[0020] The imaging device 12 photographs subjects such as people, objects, or landscapes located at the base where the mobile body 10 is installed and acquires captured images. The imaging device 12 is a digital camera (general imaging device) capable of acquiring planar images, such as a digital SLR camera or a compact digital camera. The imaging device 12 may also be a special imaging device capable of acquiring a 360° panoramic image. The special imaging device is, for example, a 360° panoramic imaging device for photographing a subject and obtaining two hemispherical images that will form the basis of a 360° panoramic image. Furthermore, the special imaging device may also be, for example, a wide-angle camera or stereo camera capable of acquiring a wide-angle image (panoramic image) with a field of view greater than a predetermined value. The special imaging device may also be an imaging device capable of photographing a subject and acquiring a hemispherical image. In other words, the special imaging device is an imaging means capable of acquiring an image taken using a lens with a focal length shorter than a predetermined value. Here, the special imaging device is an example of an imaging device that acquires a special image taken in all directions within the base. The special imaging device may be configured to generate a special image that captures all directions within the base by rotating a general imaging device that captures normal planar images and combining the obtained images. In either method, the special imaging device generates the final captured image, the special image, by stitching together multiple shooting results through image processing.

[0021] Furthermore, the mobile unit 10 may be configured to include multiple imaging devices 12. In this case, the mobile unit 10 may be configured to include both a special imaging device that captures a part of the base in all directions and a general imaging device that can capture a part of the subject captured by the special imaging device to acquire a planar image. In the following description, an example will be given in which the mobile unit 10 includes both a special imaging device 12a and a general imaging device 12b as imaging devices 12. The images captured by the imaging devices 12 may be videos, still images, or both. In addition, the images captured by the imaging devices 12 may include audio data along with the image data.

[0022] The support member 13 is a component for installing (fixing) the imaging device 12 to the mobile body 10 (housing 11). The support member 13 may be a pole or the like fixed to the housing 11, or it may be a base fixed to the housing 11. The support member 13 may also be a movable component that can adjust the imaging direction (orientation) and position (height) of the imaging device 12.

[0023] The moving mechanism 15 is a unit that moves the mobile body 10 and consists of wheels, a driving motor, a driving encoder, a steering motor, and a steering encoder, etc. Since the movement control of the mobile body 10 is based on existing technology, a detailed explanation will be omitted. For example, the mobile body 10 receives a driving instruction from an operator (display device 50), and the moving mechanism 15 moves the mobile body 10 based on the received driving instruction. The moving mechanism 15 may be bipedal or single-wheeled. Furthermore, the shape of the mobile body 10 is not limited to the vehicle type shown in Figure 2; for example, it may be a bipedal humanoid, a form mimicking a living creature, a form mimicking a specific character, etc.

[0024] The movable arm 16 has an operating means that enables additional movements other than the movement of the mobile body 10. As shown in Figure 2, the movable arm 16 is equipped with an operating means, for example, a hand at the tip of the movable arm 16 for grasping objects such as parts. The mobile body 10 can perform predetermined tasks (movements) by rotating or deforming the movable arm 16. In addition to the above configuration, the mobile body 10 may also have various sensors capable of detecting information about its surroundings. Examples of such sensors include barometers, thermometers, photometers, motion sensors, gas sensors, odor sensors, or illuminometers.

[0025] ● Hardware configuration Next, the hardware configuration of the device or terminal constituting the communication system according to the embodiment will be described using Figures 3 and 4. Note that the hardware configuration of the device or terminal shown in Figures 3 and 4 may have components added or removed as needed.

[0026] ○Hardware configuration of the mobile device○ Figure 3 shows an example of the hardware configuration of a mobile device. The mobile device 10 is equipped with a control device 30 that controls the processing or operation of the mobile device 10. As described above, the control device 30 is housed inside the housing 11 of the mobile device 10. The control device 30 may be provided outside the housing 11 of the mobile device 10, or it may be provided as a separate device from the mobile device 10.

[0027] The control device 30 includes a CPU (Central Processing Unit) 301, ROM (Read Only Memory) 302, RAM (Random Access Memory) 303, HDD (Hard Disk Drive) 304, media interface 305, input / output interface 306, audio input / output interface 307, network interface 308, short-range communication circuit 309, antenna 309a for the short-range communication circuit 309, external device connection interface 311, and bus line 310.

[0028] The CPU 301 controls the entire mobile unit 10. The CPU 301 is a computing device that realizes each function of the mobile unit 10 by reading programs or data stored in the ROM 302 or HD (Hard Disk) 304a, etc., onto the RAM 303 and executing processing.

[0029] ROM302 is a non-volatile memory that can retain programs or data even when the power is turned off. RAM303 is a volatile memory used as a work area for the CPU301, etc. HDD304 controls the reading or writing of various data to HD304a according to the control of CPU301. HD304a stores various data such as programs. Media I / F305 controls the reading or writing (storage) of data to recording media 305a such as USB (Universal Serial Bus) memory, memory card, optical disk, or flash memory.

[0030] The input / output interface 306 is an interface for inputting and outputting characters, numbers, various instructions, etc., to and from various external devices. The input / output interface 306 controls the display of various information such as cursors, menus, windows, characters, or images on the display 14, such as an LCD (Liquid Crystal Display). The display 14 may be a touch panel display equipped with input means. In addition to the display 14, the input / output interface 306 may also be connected to input means such as a mouse or keyboard. The sound input / output interface 307 is a circuit that processes the input and output of sound signals between the microphone 307a and the speaker 307b according to the control of the CPU 301. The microphone 307a is a type of built-in sound collection means that inputs sound signals according to the control of the CPU 301. The speaker 307b is a type of playback means that outputs sound signals according to the control of the CPU 301.

[0031] Network I / F 308 is a communication interface that communicates (connects) with other devices or equipment via the communication network 100. Network I / F 308 is a communication interface such as a wired or wireless LAN. Near-field communication circuit 309 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). External device connection I / F 311 is an interface for connecting other devices to the control device 30.

[0032] The bus line 310 is an address bus, data bus, etc., for electrically connecting the above components, and transmits address signals, data signals, and various control signals. The CPU 301, ROM 302, RAM 303, HDD 304, media I / F 305, input / output I / F 306, sound input / output I / F 307, network I / F 308, short-range communication circuit 309, and external device connection I / F 311 are interconnected via the bus line 310.

[0033] Furthermore, the control device 30 is connected to the drive motor 101, actuator 102, acceleration / direction sensor 103, GPS (Global Positioning System) sensor 104, special imaging device 12a, general imaging device 12b, battery 120, and obstacle detection sensor 106 via the external device connection I / F 311.

[0034] The drive motor 101 rotates the movement mechanism 15 based on commands from the CPU 301 to move the mobile body 10 along the ground. The actuator 102 deforms the movable arm 16 based on commands from the CPU 301. The acceleration / direction sensor 103 is a sensor such as an electronic magnetic compass, gyrocompass, and acceleration sensor that detects the Earth's magnetic field. The GPS sensor 104 receives GPS signals from GPS satellites. The battery 120 is a unit that supplies the total power required for the mobile body 10. Note that the battery 120 may include an external battery that provides auxiliary power from an external source, in addition to the battery built into the mobile body 10. The obstacle detection sensor 106 is a detection sensor that detects obstacles in the surroundings when the mobile body 10 is moving. The obstacle detection sensor 106 is, for example, an image sensor such as a stereo camera or a camera equipped with an area sensor in which photoelectric conversion elements are arranged in a planar manner, or a distance measuring sensor such as a TOF (Time Of Flight) sensor, LIDAR (Light Detection and Ranging) sensor, radar sensor, laser rangefinder, ultrasonic sensor, depth camera, or depth sensor.

[0035] ○Hardware configuration of the display device○ Figure 4 shows an example of the hardware configuration of a display device. Each hardware component of the display device 50 is indicated by a code in the 500 series. The display device 50 is built by a computer and includes, as shown in Figure 4, a CPU 501, ROM 502, RAM 503, HD 504, HDD controller 505, display 506, external device connection I / F 507, network I / F 508, bus line 510, keyboard 511, pointing device 512, audio input / output I / F 513, microphone 514, speaker 515, camera 516, DVD-RW (Digital Versatile Disk Rewritable) drive 517, and media I / F 519.

[0036] Of these components, the CPU 501 controls the operation of the entire display device 50. The ROM 502 stores programs used to drive the CPU 501, such as the IPL (Initial Program Loader). The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The display 506 may be a touch panel display equipped with input means. The display 506 is also just one example of a display unit. The display unit as the display 506 may be an external device with display functions connected to the display device 50. In this case, the display unit may be, for example, an external display such as an IWB (Interactive White Board), or a projection surface (for example, the ceiling or wall of a management base, the windshield of a car, etc.) onto which images from a PJ (Projector) or HUD (Head-Up Display) connected as an external device are projected. The external device connection I / F 507 is an interface for connecting various external devices. The network I / F 508 is an interface for data communication using the communication network 100. The bus line 510 is an address bus or data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 4.

[0037] Furthermore, the keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting or executing various instructions, selecting a processing target, or moving a cursor, etc. Note that the input means may not be limited to the keyboard 511 and the pointing device 512, but may also be a touch panel or an audio input device, etc. The audio input / output I / F 513 is a circuit that processes the input and output of audio signals between the microphone 514 and the speaker 515 according to the control of the CPU 501. The microphone 514 is a type of built-in sound collection means for inputting sound. The speaker 515 is a type of built-in output means for outputting audio signals. The camera 516 is a type of built-in imaging means for capturing images of a subject and obtaining image data. Note that the microphone 514, speaker 515, and camera 516 may be external devices instead of being built into the display device 50. The DVD-RW drive 517 controls the reading or writing of various data to the DVD-RW 518, which is an example of a removable recording medium. The removable recording medium is not limited to DVD-RW; it may also be DVD-R or Blu-ray® Disc, etc. The media interface 519 controls the reading or writing (storage) of data to or from the recording medium 521, such as flash memory.

[0038] The above programs may be distributed as installable or executable files recorded on a computer-readable recording medium. Examples of recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray Disc, SD card, or USB memory. The recording media can also be provided domestically or internationally as a program product. For example, the display device 50 implements the display control method according to the present invention when the program according to the present invention is executed.

[0039] ●Functional Configuration Next, the functional configuration of the communication system according to the embodiment will be described using Figures 5 to 8. Figure 5 is a diagram showing an example of the functional configuration of the communication system. Note that Figure 5 shows the devices or terminals shown in Figure 1 that are related to the processing or operation described later.

[0040] ○Functional configuration of the mobile unit (control device)○ First, the functional configuration of the control device 30 that controls the processing or operation of the mobile body 10 will be explained using Figure 5. The control device 30 includes a transmitting / receiving unit 31, a determination unit 32, an image capture control unit 33, a state detection unit 34, a location information acquisition unit 35, a destination candidate acquisition unit 36, a route information generation unit 37, a route information management unit 38, a destination setting unit 39, a movement control unit 40, an image recognition processing unit 41, a mode setting unit 42, an autonomous movement processing unit 43, a manual operation processing unit 44, a task execution unit 45, a map information management unit 46, an image processing unit 47, a learning unit 48, and a storage / reading unit 49. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 3 operating according to instructions from the CPU 301 in accordance with a control device program deployed on the RAM 303. The control device 30 also has a storage unit 3000 constructed from a ROM 302, HD 304a, or recording medium 305a shown in Figure 3.

[0041] The transmitting / receiving unit 31 is mainly implemented by the processing of the network interface 308 by the CPU 301, and transmits and receives various data or information with other devices or terminals via the communication network 100.

[0042] The decision unit 32 is implemented by the processing of the CPU 301 and performs various decisions. The shooting control unit 33 is mainly implemented by the processing of the CPU 301 on the external device connection I / F 311 and controls the shooting process for the shooting device 12. The shooting control unit 33, for example, instructs the shooting process for the special shooting device 12a and the general shooting device 12b. The shooting control unit 33 also acquires the captured images obtained from the shooting process by the special shooting device 12a and the general shooting device 12b.

[0043] The state detection unit 34 is mainly implemented by the processing of the CPU 301 to the external device connection I / F 311, and detects the state of the mobile body 10 or its surroundings using various sensors. The state detection unit 34 measures the distance to objects (obstacles) present around the mobile body 10 using, for example, an obstacle detection sensor 106, and outputs it as distance data. The state detection unit 34 may also be configured to obtain position by applying SLAM (Simultaneous Localization and Mapping) to the distance data measured using, for example, the obstacle detection sensor 106, and comparing it with an environmental map. Here, SLAM is a technology that can perform self-position estimation and environmental map creation simultaneously. Furthermore, the state detection unit 34 detects the direction the mobile body 10 is facing using, for example, an acceleration / direction sensor 103.

[0044] The location information acquisition unit 35 is implemented by the CPU 301 processing the external device connection I / F 311 and acquires location information indicating the current position of the moving object 10 using the GPS sensor 104. For example, the location information acquisition unit 35 uses the GPS sensor 104 to acquire coordinate information indicating the latitude and longitude of the current position of the moving object 10.

[0045] The destination candidate acquisition unit 36 ​​is mainly implemented by the CPU 301 and acquires destination candidate images that indicate candidate destinations for the mobile body 10. Specifically, the destination candidate acquisition unit 36 ​​acquires captured images taken by the shooting control unit 33 that show a portion of the area within the base where the mobile body 10 is located, as destination candidate images.

[0046] The route information generation unit 37 is mainly implemented by the CPU 301 and generates route information indicating the movement path of the mobile body 10. Based on the location of the destination candidate selected by the operator of the mobile body 10, the route information generation unit 37 generates route information indicating the path from the current position to the final destination. The method of generating the route information may include, for example, connecting each waypoint from the current position to the final destination with a straight line, or using information on obstacles obtained from captured images or the state detection unit 34 to minimize travel time while avoiding obstacles.

[0047] The route information management unit 38 is mainly implemented by the processing of the CPU 301 and manages the route information generated by the route information generation unit 37 using the route information management DB 3002.

[0048] The destination setting unit 39 is mainly implemented by the CPU 301 and sets the destination of the mobile object 10. For example, based on the current position of the mobile object 10 acquired by the position information acquisition unit 35 and the route information generated by the route information generation unit 37, the destination setting unit 39 sets the destination to be reached from among the destination candidates selected by the operator of the mobile object 10.

[0049] The movement control unit 40 is mainly implemented by the processing of the CPU 301 to the external device connection I / F 311, and controls the movement of the mobile body 10 by driving the movement mechanism 15. The movement control unit 40 moves the mobile body 10 in response to drive commands from, for example, the autonomous movement processing unit 43 or the manual operation processing unit 44.

[0050] The image recognition processing unit 41 is mainly implemented by the CPU 301 and performs image recognition processing on the captured images acquired by the shooting control unit 33. For example, the image recognition processing unit 41 detects whether or not a specific subject is present in the acquired captured image through image recognition. A specific subject may be, for example, an obstacle on or around the path of the moving object 10, an intersection such as a crossroads or L-shaped road, or a sign or signal installed at a base.

[0051] The mode setting unit 42 is mainly implemented by the CPU 301 and sets an operation mode that indicates the operation of moving the mobile body 10. The mode setting unit 42 sets either an autonomous movement mode in which the mobile body 10 moves autonomously or a manual operation mode in which the mobile body 10 is moved by the operator's manual operation.

[0052] The autonomous movement processing unit 43 is mainly implemented by the CPU 301 and controls the autonomous movement process of the mobile body 10. For example, the autonomous movement processing unit 43 outputs a drive command to the movement control unit 40 for the mobile body 10 to follow the movement path indicated in the path information generated by the path information generation unit 37.

[0053] The manual operation processing unit 44 is mainly implemented by the CPU 301 and controls the manual operation of the mobile body 10. For example, the manual operation processing unit 44 outputs a drive command for the mobile body 10 to the movement control unit 40 in response to a manual operation command transmitted from the display device 50.

[0054] The task execution unit 45 is mainly implemented by the processing of the CPU 301 and executes a predetermined task by the mobile body 10 in response to a request from the operator. The predetermined task executed by the task execution unit 45 includes, for example, imaging processing for inspecting equipment within the base, or light work using the movable arm 16.

[0055] The map information management unit 46 is primarily implemented by the processing of the CPU 301 and uses the map information management DB 3003 to manage map information showing the environmental map of the target location where the mobile unit 10 is installed. The map information management unit 46 manages map information showing, for example, environmental maps downloaded from an external server or environmental maps created by applying SLAM.

[0056] The image processing unit 47 is mainly implemented by the CPU 301 and generates the display image to be shown on the display device 50. The image processing unit 47 generates the display image to be shown on the display device 50 by processing the captured image acquired by the capture control unit 33, for example.

[0057] The learning unit 48 is mainly implemented by the processing of the CPU 301 and learns the movement path for autonomous movement of the mobile body 10. The learning unit 48 performs imitation learning (machine learning) of the movement path for autonomous movement based on, for example, captured images acquired during movement in manual operation mode by the manual operation processing unit 44 and detection data by the state detection unit 34. The autonomous movement processing unit 43 performs autonomous movement of the mobile body 10 based on, for example, the learning data which is the result of imitation learning by the learning unit 48.

[0058] The memory / read unit 49 is mainly implemented by the CPU 301, and stores various data (or information) in the memory unit 3000 and reads various data (or information) from the memory unit 3000.

[0059] ○Destination candidate management table Figure 6 is a conceptual diagram showing an example of a destination candidate management table. The destination candidate management table is a table for managing destination candidates that indicate potential destinations for the mobile object 10. The storage unit 3000 has a destination candidate management DB 3001 constructed, which consists of a destination candidate management table like the one shown in Figure 6.

[0060] The destination candidate management table manages destination candidate data for each base ID that identifies the base where the mobile object 10 is installed. This data is associated with a candidate ID that identifies a destination candidate, location information indicating the location of the destination candidate, and image data of a portion of the base that is the destination candidate. The location information is represented by coordinate information including latitude and longitude indicating the location of the destination candidate within the base. If the mobile object 10 is an aircraft such as a drone, the location information includes information such as the aircraft's speed, attitude, or altitude in addition to the coordinate information indicating latitude and longitude. The image data refers to the image data file taken by the mobile object 10 at the location of the destination candidate within the base. Here, the destination candidate is an example of a candidate destination for the mobile object 10. The candidate destination for the mobile object 10 includes not only the destination of the mobile object 10, but also, for example, candidate locations to be excluded from the mobile object 10's travel route.

[0061] ○ Route information management table Figure 7 is a conceptual diagram showing an example of a route information management table. The route information management table is a table for managing route information that shows the movement path of the mobile object 10. The storage unit 3000 has a route information management DB 3002 constructed, which is composed of route information management tables like the one shown in Figure 7.

[0062] The route information management table manages route information that identifies the movement path of the mobile body 10 and associates it with a base ID that identifies the base where the mobile body 10 is installed. Of this information, the route information shows the future movement path of the mobile body 10 in order of future destinations. The route information is generated by the route information generation unit 37 when the mobile body 10 starts moving.

[0063] ○Map Information Management Table Figure 8 is a conceptual diagram showing an example of a map information management table. The map information management table is a table for managing map information, which is an environmental map of the target location where the mobile unit 10 is installed. The storage unit 3000 has a map information management DB 3003 constructed, which is composed of map information management tables like the one shown in Figure 8.

[0064] The map information management table manages map information that associates the location ID and location name that identify the target location where the mobile unit 10 is installed, with the storage location of the environmental map for the target location. Of these, the storage location is, for example, destination information for accessing the storage area within the mobile unit 10 where the environmental map is stored or an external server, and is indicated by a URL (Uniform Resource Locator) or URI (Uniform Resource Identifier).

[0065] Here, using Figures 31 to 33, an overview of the target site where the mobile unit 10 is installed will be described. Figure 31 is a diagram that schematically illustrates an example of the mobile unit's movement path. The movement path of the mobile unit 10 shows, for example, the route from arbitrary site locations (W0, W1, W2, ...) on the target site, as shown in Figure 31, to the designated destination. Any site location on the target site is represented by XY coordinates that indicate the coordinate position on the map data showing the entire target site.

[0066] Figure 32 is a conceptual diagram showing an example of a base location management table. The base location management table is a table for managing base location information that indicates a predetermined base location within a target base. A base location management table like the one shown in Figure 32 is included, for example, in the map information management DB3003.

[0067] The base location management table manages base location information, which is associated with each base ID that identifies the base where the mobile unit 10 is installed. This information includes a base location ID that identifies a predetermined location within the target base, location information indicating the target base location, and an adjacent base ID indicating an adjacent base location. Of these, the base location ID is identification information that identifies any base location (W0, W1, W2, ...) on the target base. The adjacent base ID indicates a base location ID that identifies an adjacent base location. The base location management table stores multiple locations on the mobile unit 10's travel path within the target base at predetermined intervals. Any base location on the target base is represented by XY coordinates that indicate the coordinate position on the map data representing the entire target base. The mobile unit 10 moves to its destination based on the travel path set using the location information and adjacent base IDs indicated in the base location information.

[0068] Figure 33 is a conceptual diagram showing an example of an area information management table. An area information management table is used to manage area information, for example, that indicates the area of ​​a target location divided by task. An area information management table like the one shown in Figure 33 is included, for example, in the map information management DB3003.

[0069] The area information management table manages area information associated with each base ID that identifies the base where the mobile unit 10 is installed. This information includes an area ID and area name that identify a predetermined area within the target base, as well as a reference location for determining the area's location. The reference location is the location information that the mobile unit 10 uses to identify an area within the target base. The reference location is identified, for example, by the locations of two bases along the mobile unit 10's travel path.

[0070] For example, the target base shown in Figure 31 is divided into four areas (Area 1 to Area 4). For example, if we want to move the mobile object 10 to Area 1, the movement path is set to a route with base location W8, which is the area reference position in Area 1, as the destination (W0→W1→W2→W3→W4→W8).

[0071] ○ Learning data management database Here, the learning data management DB 3004 built in the memory unit 3000 will be described. The learning data management DB 3004 manages the learning data, which is the result of the learning unit 48 learning the autonomous movement path of the mobile body 10. For example, the learning data management DB 3004 stores captured image data and various detection data acquired from the mobile body 10, and also stores learning data, which is the result of imitation learning (machine learning) for each location or for each mobile body 10. The mobile body 10 performs autonomous movement based on the learning data stored in the learning data management DB 3004. Note that the captured image data taken by the special imaging device 12a or the general imaging device 12b may include PTZ (Pan-Tilt-Zoom) parameters to specify the shooting direction of the special imaging device 12a or the general imaging device 12b. Captured image data including PTZ parameters is stored in the memory unit 3000 (learning data management DB 3004) of the mobile body 10. Furthermore, the PTZ parameters may be stored in the storage unit 3000 in association with the location information of the candidate destination, and coordinate data (x,y,θ) indicating the attitude of the mobile body 10 when the captured image data of the candidate destination was acquired may also be stored in the storage unit 3000 at the same time. This makes it possible to correct the attitude of the mobile body 10 using the PTZ parameters and coordinate data (x,y,θ) if there is a discrepancy between the actual stopping position of the mobile body 10 and the destination. Some data, such as the data of the autonomous movement path (GPS trajectory) of the mobile body 10 and the captured image data of the candidate destination used for display on the display device 50, may be stored in a cloud computing service such as AWS (Amazon Web Services) (trademark).

[0072] ○Functional Configuration of Display Device○ Next, the functional configuration of the display device 50 will be described using Figure 5. The display device 50 includes a transmitting / receiving unit 51, a receiving unit 52, a display control unit 53, a judgment unit 54, a manual operation command generation unit 55, an autonomous movement request information generation unit 56, an image processing unit 57, and a storage / reading unit 59. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 501 in accordance with a display device program deployed on the RAM 503. The display device 50 also has a storage unit 5000 constructed from the ROM 502, HD 504, or recording medium 521 shown in Figure 4.

[0073] The transmitting / receiving unit 51 is mainly implemented by the processing of the network interface 508 by the CPU 501, and transmits and receives various data or information with other devices or terminals.

[0074] The reception unit 52 is mainly implemented by the CPU 501 processing the keyboard 511 or pointing device 512, and accepts various selections or inputs from the user. The display control unit 53 is mainly implemented by the CPU 501, and displays various screens on a display unit such as the display 506. The decision unit 54 is implemented by the CPU 501, and makes various decisions.

[0075] The manual operation command generation unit 55 is mainly implemented by the CPU 501 and generates manual operation commands to move the mobile body 10 manually in response to the operator's input. The autonomous movement request information generation unit 56 is mainly implemented by the CPU 501 and generates autonomous movement request information to move the mobile body 10 autonomously. The autonomous movement request information generation unit 56 transmits autonomous movement request information to the mobile body 10, for example, based on information of a candidate destination selected by the operator.

[0076] The image processing unit 57 is mainly implemented by the CPU 501 and generates display images to be shown on a display unit such as the display 506. The image processing unit 57 generates display images to be shown on the display device 50 by processing, for example, the captured images acquired by the mobile device 10. The communication system 1 only needs to have the functionality of either the image processing unit 47 of the mobile device 10 or the image processing unit 57 of the display device 50.

[0077] The memory / read unit 59 is mainly implemented by the CPU 501, and stores various data (or information) in the memory unit 5000 and reads various data (or information) from the memory unit 5000.

[0078] ●Processing or operation of the embodiment ○Destination candidate registration process○ Next, the processing or operation of the communication system according to the embodiment will be described using Figures 9 to 23. First, the process of registering destination candidates for the mobile body 10 will be described using Figures 9 to 12. Figure 9 is a sequence diagram showing an example of the destination candidate registration process.

[0079] First, the display device 50 starts operating the mobile device 10 in response to a predetermined input operation by the operator (step S11). Then, the transmitting / receiving unit 51 transmits an operation start request to the mobile device 10 (step S12). As a result, the transmitting / receiving unit 31 of the control device 30 installed in the mobile device 10 receives the operation start request transmitted from the display device 50.

[0080] Next, the shooting control unit 33 of the control device 30 starts the shooting process using the special shooting device 12a and the general shooting device 12b (step S13). The shooting control unit 33 acquires the shooting image data captured by the special shooting device 12a and the general shooting device 12b. Here, the image acquired by the shooting process in step S13 is a moving image. Then, the transmitting / receiving unit 31 transmits the shooting image data acquired in step S13 to the display device 50 (step S14). As a result, the transmitting / receiving unit 51 of the display device 50 receives the shooting image data transmitted from the moving body 10.

[0081] Next, the display control unit 53 of the display device 50 causes the operation screen 200, which includes the captured image data received in step S14, to be displayed on a display unit such as the display 506 (step S15). Figure 10 is a diagram showing an example of the operation screen. The operation screen 200 shown in Figure 10 is a display screen for the operator to remotely control the mobile device 10.

[0082] The operation screen 200 includes a base display area 210 that displays the captured image (planar image) taken by the general imaging device 12b, which is the captured image data received in step S14, and a base display area 230 that displays the captured image (special image) taken by the special imaging device 12a, which is the captured image data received in step S14. Of these, the special image displayed in the base display area 230 is, as described above, an image that captures the entire base in all directions, such as a 360-degree image, a wide-angle image, or a hemispherical image, taken by the special imaging device 12a. The special image may also be an image obtained by the special imaging device 12a, which generates an image that captures all directions by combining images taken while rotating the general imaging device. In addition, the base display area 230 displays a moving object display image 235 that shows the presence of the moving object 10 on the special image, superimposed on the special image. The operation screen 200 also displays coordinate information represented by latitude (Lat) and longitude (Lon) as position information indicating the current position of the moving object 10. Furthermore, if the mobile object 10 is an aircraft such as a drone, the operation screen 200 may be configured to display information such as the aircraft's speed, attitude, or altitude, in addition to the latitude and longitude coordinate information. The operation screen 200 may also display images captured by the special imaging device 12a and the general imaging device 12b as live streaming video delivered in real time via a computer network such as the Internet.

[0083] The operation screen 200 also includes operation icons 250 for remotely controlling the mobile unit 10. The operation icons 250 include direction indicator buttons that are pressed when requesting horizontal movement of the mobile unit 10 (forward, backward, right rotation, left rotation). The operator can remotely control the mobile unit 10 by selecting the operation icons 250 while viewing special images such as a planar image displayed in the base display area 210 and a 360-degree image displayed in the base display area 230. Although the explanation describes an example in which the movement of the mobile unit 10 is remotely controlled by accepting a selection of the operation icons 250 displayed on the operation screen 200, the movement of the mobile unit 10 may also be performed using a keyboard or a dedicated controller such as a gamepad with a joystick.

[0084] Furthermore, the operation screen 200 includes a destination candidate registration button 270, which is pressed when registering a destination candidate using a planar image displayed in the base display area 210 or a special image displayed in the base display area 230. In the following description, an example will be given in which a destination candidate is registered using a special image displayed in the base display area 230 when the operator presses the destination candidate registration button 270. The same process is performed when the operator registers a destination using a planar image displayed in the base display area 210 when the operator presses the destination candidate registration button 270. The operation screen 200 also includes a destination setting button 290, which is pressed when setting a destination for the mobile unit 10.

[0085] Next, the display device 50 uses the operation screen 200 displayed in step S15 to perform manual operation of the mobile body 10 (step S16). Details of the process in step S16 will be described later.

[0086] Next, when the operator presses the destination candidate registration button 270 on the operation screen 200, the reception unit 52 accepts the registration of a destination candidate for the area shown in the special image displayed in the base display area 230 (step S17). Then, the transmitting / receiving unit 51 transmits a destination candidate registration request to the mobile body 10 (step S18). This destination registration request is, for example, a request to photograph the area shown in the special image displayed in the base display area 230. As a result, the control device 30 of the mobile body 10 receives the destination candidate registration request transmitted from the display device 50.

[0087] Next, the position information acquisition unit 35 of the control device 30 acquires position information indicating the current position of the mobile body 10 using a GPS sensor 306 or the like (step S19). Specifically, the position information acquisition unit 35 acquires coordinate information including latitude and longitude indicating the current position of the mobile body 10. If the mobile body 10 is an aircraft such as a drone, the position information acquisition unit 35 acquires information such as the speed, attitude, or altitude of the aircraft as position information in addition to the coordinate information including latitude and longitude. Furthermore, the destination candidate acquisition unit 36 ​​acquires a captured image taken by the special photographic device 12a at the current position of the mobile body 10 as a destination candidate image indicating a destination candidate (step S20). Here, the captured image acquired as a destination candidate image by the destination candidate acquisition unit 36 ​​is a still image.

[0088] Then, the storage and reading unit 49 stores the destination candidate data, including the location information acquired in step S19 and the captured image acquired in step S20, in the destination candidate management DB 3001 (see Figure 6) (step S21). The storage and reading unit 49 registers the acquired location information and captured image as destination candidate data, each associated with a candidate ID that identifies the destination candidate.

[0089] In this way, the communication system 1 displays the images captured by the mobile body 10 on the display device 50 of the operator remotely controlling the mobile body 10, allowing the operator to remotely control the mobile body 10 while checking the surrounding situation in real time. Furthermore, the communication system 1 can pre-register images of areas within a base that can serve as candidate destinations for autonomous movement of the mobile body 10, in response to the operator's input during manual operation of the mobile body 10.

[0090] ○Manual operation processing Next, the manual operation process for the mobile body 10 in step S16 will be described in detail using Figures 11 and 12. Figure 11 is a sequence diagram showing an example of the manual operation process for the mobile body.

[0091] First, the receiving unit 52 of the display device 50 accepts the operator's input operation to select an operation icon 250 on the operation screen 200 displayed in step S15 (step S31). The manual operation command generation unit 55 generates a manual operation command according to the content of the operation icon 250 selected in step S31 (step S32). In the example in Figure 10, the operator has selected the "Forward (↑)" icon, so in this case, the manual operation command generation unit 55 generates a command to move the mobile body 10 forward. Then, the transmitting / receiving unit 51 transmits the manual operation command generated in step S32 to the mobile body 10 (step S33). As a result, the control device 30 of the mobile body 10 receives the manual operation command transmitted from the display device 50. If the operator selects the "Backward (↓)" icon while the mobile body 10 is moving forward during the operator's input operation of the operation icon 250, the captured image may be switched to the rear view of the mobile body 10, and the mobile body 10 may be made to move backward (reverse) from that point. Furthermore, the transmission of manual operation commands from the display device 50 to the mobile device 10 may be configured to be performed via a managed cloud platform such as AWS IoT Core™.

[0092] Next, the mode setting unit 42 of the control device 30 sets the operating mode of the mobile body 10 to manual operation mode (step S34). Then, the manual operation processing unit 44 outputs a drive command based on the manual operation command received in step S33 to the movement control unit 40. The movement control unit 40 executes the movement process of the mobile body 10 in response to the drive command from the manual operation processing unit 44 (step S35). The learning unit 48 also performs imitation learning (machine learning) of the movement path in response to the manual operation by the manual operation processing unit 44 (step S36). For example, the learning unit 48 performs imitation learning of the movement path based on the captured images acquired during the movement operation in manual operation mode by the manual operation processing unit 44 and the detection data from the state detection unit 34. The learning unit 48 may be configured to perform imitation learning of the movement path using only the captured images acquired during manual operation, or it may be configured to perform imitation learning of the movement path using both the captured images and the detection data from the state detection unit 34. Furthermore, the captured images used for imitation learning by the learning unit 48 may be captured images acquired during autonomous movement in autonomous movement mode by the autonomous movement processing unit 43.

[0093] Next, the mode setting unit 42 of the control device 30 sets the operating mode of the mobile body 10 to manual operation mode (step S34). Then, the manual operation processing unit 44 outputs a drive command based on the manual operation command received in step S33 to the movement control unit 40. The movement control unit 40 executes the movement process of the mobile body 10 in response to the drive command from the manual operation processing unit 44 (step S35). The learning unit 48 also performs imitation learning (machine learning) of the movement path in response to the manual operation by the manual operation processing unit 44 (step S36). For example, the learning unit 48 performs imitation learning of the movement path based on captured images acquired during movement in manual operation mode by the manual operation processing unit 44 and detection data from the state detection unit 34.

[0094] Then, the mobile unit 10 performs a destination candidate registration process while moving (step S37). Figure 12 is a flowchart showing an example of the destination candidate registration process according to the movement state of the mobile unit.

[0095] First, the determination unit 32 of the control device 30 determines whether or not a predetermined task has been executed by the task execution unit 45 (step S51). Specifically, the task execution unit 45 executes a predetermined task by the mobile body 10 in response to, for example, a task execution request from an operator or a pre-set schedule. The determination unit 32 then determines whether or not the predetermined task can be executed by the task execution unit 45. Here, the predetermined task includes, for example, a photography process for inspecting equipment within the base, or light work using the movable arm 16.

[0096] For example, when the mobile unit 10 moves within the base and enters the area to be inspected, it performs the task of checking the items to be inspected, such as meters and valves. At this time, the mobile unit 10 stops moving and performs operations such as taking pictures to acquire inspection images, and this action can be used as a trigger to register a candidate destination. In addition, the mobile unit 10 can also use the registered name of the inspection item or task entered in the settings screen for setting the captured inspection images, as shown in Figure 34, as the registered name of the candidate destination.

[0097] If the determination unit 32 determines that the predetermined task has been executed by the task execution unit 45 (YES in step S51), it proceeds to step S56. On the other hand, if the determination unit 32 determines that the predetermined task has not been executed by the task execution unit 45 (NO in step S51), it proceeds to step S52.

[0098] Next, the determination unit 32 determines whether or not the movement of the mobile body 10 has stopped (step S52). Specifically, the determination unit 32 determines that the movement of the mobile body 10 has stopped if it detects that the drive control of the mobile mechanism 15 by the movement control unit 40 has stopped. If the determination unit 32 determines that the movement of the mobile body 10 has stopped (YES in step S52), it proceeds to step S56. On the other hand, if the determination unit 32 determines that the movement of the mobile body 10 has not stopped (is still moving) (NO in step S52), it proceeds to step S53.

[0099] Next, the determination unit 32 determines whether or not an intersection has been detected around the mobile body 10 (step S53). Specifically, the image recognition processing unit 41 performs image recognition processing on the captured image acquired by the special imaging device 12a or the general imaging device 12b. The determination unit 32 then determines whether or not an intersection has been detected in the captured image based on the processing by the image recognition processing unit 41. The mobile body 10 can acquire images from different directions along its movement path at once, regardless of the direction of movement, by acquiring special images captured in all directions at a distinctive point such as an intersection. Here, an intersection is an example of a specific subject detected by the image recognition processing unit 41. In addition to intersections, specific subjects may also be obstacles on or around the movement path of the mobile body 10, or signs or signals installed at a base. Furthermore, specific subjects differ depending on the type of mobile body 10, such as when the mobile body 10 is traveling on a road or when a mobile body 10 such as a drone is flying, and information on specific subjects to be detected is set in advance for the mobile body 10.

[0100] If the determination unit 32 determines that it has detected an intersection around the moving object 10 (YES in step S53), it proceeds to step S56. On the other hand, if the determination unit 32 determines that it has not detected an intersection around the moving object 10 (NO in step S52), it proceeds to step S54.

[0101] Next, the determination unit 32 determines whether the current position of the mobile body 10 is close to a destination candidate registered in the destination candidate management DB 3001 (step S54). Specifically, the determination unit 32 refers to the location information of the destination candidates stored in the destination candidate management DB 3001 and determines whether there is a destination sequence close to the current position of the mobile body 10, which is acquired by the location information acquisition unit 35. Here, the determination of whether or not it is close to a destination candidate is made, for example, by the difference in coordinate values ​​between the location of a pre-set destination candidate and the current position of the mobile body 10. Alternatively, the determination of whether or not it is close to a destination candidate may be made using image recognition between a captured image corresponding to the destination candidate (destination candidate image) and a captured image at the current position of the mobile body 10.

[0102] If the determination unit 32 determines that the current location of the mobile body 10 is close to a destination candidate registered in the destination candidate management DB 3001 (YES in step S54), it proceeds to step S56. On the other hand, if the determination unit 32 determines that the current location of the mobile body 10 is far from a destination candidate registered in the destination candidate management DB 3001 (NO in step S54), it proceeds to step S55.

[0103] Next, the determination unit 32 determines whether the direction indicated by the manual operation command transmitted from the display device 50 has been changed (step S55). Specifically, the determination unit 32 determines whether the direction of movement indicated by the manual operation command has been changed based on the manual operation command received in step 33. The determination unit 32 may also determine that the direction of movement has been changed, for example, if the manual operation command indicates a characteristic operation. Characteristic operations include, for example, a certain amount of steering operation or accelerator or brake operation performed when driving through intersections or curves.

[0104] If the determination unit 32 determines that the operating direction indicated by the manual operation command transmitted from the display device 50 has been changed (YES in step S55), it proceeds to step S56. On the other hand, if the determination unit 32 determines that the operating direction indicated by the manual operation command transmitted from the display device 50 has not been changed (NO in step S55), it terminates the process.

[0105] Then, in step S56, the position information acquisition unit 35 of the control device 30 acquires position information indicating the current position of the mobile body 10 using a GPS sensor 306 or the like. The destination candidate acquisition unit 36 ​​also acquires a captured image taken by the special photographing device 12a at the current position of the mobile body 10 as a destination candidate image indicating a destination candidate for the mobile body 10 (step S57).

[0106] Then, the storage and reading unit 49 stores the destination candidate data, including the location information acquired in step S19 and the captured image acquired in step S20, in the destination candidate management DB 3001 (see Figure 6) (step S58). The storage and reading unit 49 registers the acquired location information and captured image as destination candidate data, each associated with a candidate ID that identifies the destination candidate.

[0107] In this way, the mobile body 10 can automatically take photographs of its surroundings using the camera 12 mounted on the mobile body 10, based on predetermined judgment criteria corresponding to the mobile body 10's movement state, and can pre-register photographs indicating candidate destinations for the mobile body 10. The judgment criteria used by the judgment unit 32 shown in steps S51 to S55 are not limited to these and are set appropriately according to the content of the manual operation of the mobile body 10 and the environment surrounding the mobile body 10. The judgment criteria used by the judgment unit 32 include, for example, changes in the state of the mobile body 10 that the operator should be aware of when selecting a destination for the mobile body 10, or changes in the status of the base or the environment within the base.

[0108] The mobile device 10 may be configured to automatically register a destination candidate if, for example, the image captured by the camera 12 contains a characteristic string of characters. In this case, the mobile device 10 performs OCR (Optical Character Recognition) on the image captured by the camera 12 during movement, and if a characteristic string of characters is recognized, it registers the recognized string of characters and the captured image as destination candidate data. This makes it easier for the mobile device 10 to search for data, as, if the destination candidate is a chemical plant, the tank number or contents are often written on the tank, and keywords (strings) such as the name or contents are registered together with the captured image.

[0109] Furthermore, the mobile device 10 may refer to the learning data management DB 3004, etc., and if it moves away from a coordinate position it has previously traveled within the base, it may be configured to automatically register the image taken at the corresponding location as a destination candidate. In addition, the mobile device 10 may refer to the map information management DB 3003, and if information indicating a characteristic landmark and its coordinates is stored, it may be configured to automatically register the image taken when it reaches or passes near the stored landmark as a destination candidate. In this case, if the name of the landmark is stored in the map information management DB 3003, the mobile device 10 may be configured to register the name stored along with the image as destination candidate data.

[0110] Furthermore, although Figure 12 illustrates the process performed when the mobile unit 10 is manually operated, the destination candidate registration process shown in Figure 12 may also be performed when the mobile unit 10 is moving autonomously. In addition, the destination candidate registration process shown in Figures 9 to 12 is just one example of the process for registering candidate destinations for the mobile unit 10, and the process may be performed not only for the purpose of registering candidate destinations for the mobile unit 10, but also, for example, for the purpose of registering candidate locations to be excluded from the mobile unit 10's movement path.

[0111] ○Destination setting process○ Next, using Figures 13 to 18, we will explain the process of setting the destination of the mobile body 10 using the destination candidates registered by the above process. Figure 13 is a sequence diagram showing an example of the process of setting the destination of the mobile body.

[0112] First, the receiving unit 52 of the display device 50 accepts the selection of the destination setting button 290 based on the operator's input operation on the operation screen 200 (step S71). Next, the transmitting / receiving unit 51 sends a destination candidate acquisition request to the mobile device 10, indicating that it is requesting the acquisition of data indicating destination candidates (step S72). This destination candidate acquisition request includes a base ID that identifies the base where the mobile device 10 is installed. As a result, the transmitting / receiving unit 31 of the control device 30 equipped with the mobile device 10 transmits the destination candidate acquisition request sent from the display device 50.

[0113] Next, the storage / reading unit 49 of the control device 30 searches the destination candidate management DB 3001 (see Figure 6) using the base ID indicated in the destination acquisition request received in step S72 as a search key, and reads destination candidate data associated with the same base ID as the received base ID (step S73). Then, the transmitting / receiving unit 31 transmits the destination candidate data read in step S73 to the requesting display device 50 (step S74). This destination candidate data includes a set of multiple candidate IDs, captured images, and location information read in step S73. As a result, the transmitting / receiving unit 51 of the display device 50 acquires the destination candidate data by receiving it from the mobile body 10.

[0114] Next, the display control unit 53 of the display device 50 causes the selection screen 400, which includes the destination candidate data received in step S74, to be displayed on a display unit such as the display 506 (step S75). Specifically, the image processing unit 57 generates the selection screen 400 based on the received destination candidate data. The display control unit 53 then displays the selection screen 400 generated by the image processing unit 57. The generation process of the selection screen 400 may also be performed by the image processing unit 47 of the control device 30. In this case, the image processing unit 47 generates the selection screen 400 based on the target candidate data read in step S73. Then, in step S74, the transmitting and receiving unit 31 transmits the screen data of the selection screen 400, which includes the destination candidate data generated by the image processing unit 47, to the display device 50.

[0115] Figures 14 and 15 show examples of selection screens. The selection screen 400 shown in Figures 14 and 15 is a display screen for the operator to select the destination to which the mobile object 10 will move.

[0116] The selection screen 400 includes an image display area 410 in which multiple captured images (destination candidate images) 415 (415a to 417f) included in the destination candidate data received in step S74 are displayed for selection. The selection screen 400 also includes an OK button 430 to be pressed when selection is complete, and a Cancel button 435 to be pressed to cancel the selection process. The operator selects a captured image 415 displayed in the image display area 410 using an input means such as a pointing device 512.

[0117] The reception unit 52 of the display device 50 accepts the selection of a destination candidate image for setting the area indicated by the captured image as the destination, when the operator selects the captured image 415 displayed in the image display area 410 (step S76). The selection screen 400 shown in Figure 15 shows, for example, the state in which captured images 415a, 415b, and 415c have been selected by the operator. When the captured image 415 displayed in the image display area 410 is selected, the selected captured image is displayed in the selection result display area 420 on the selection screen 400, as shown in Figure 15. The selection result display area 420 shows the selected captured images in the order in which they were selected by the operator (No. 1 to No. 3). For example, the operator selects a captured image that shows an area indicating the position where the mobile body 10 will perform a predetermined task, or a captured image that shows an area indicating a branching point on the route, such as an intersection within the base, when moving the mobile body 10 to the position where the predetermined task will be performed, as the destination for the mobile body 10.

[0118] In this way, the display device 50 uses captured images 415 (destination candidate images) that have been previously captured by the mobile body 10 and indicate candidate destinations for the mobile body 10, to select the area indicated by the captured images 415 as the destination for the mobile body 10. This improves the operability of selecting the destination for the mobile body 10 compared to conventional methods that require the operator to specify location information or a string of characters such as a keyword.

[0119] Next, when the selection of a candidate destination image is accepted in step S76 and the operator presses the OK button 430, the autonomous movement request information generation unit 56 of the display device 50 generates autonomous movement request information (step S77). This autonomous movement request information includes a candidate ID associated with the captured image 415 whose selection was accepted in step S76, and information indicating the order in which the selections were accepted in step S76.

[0120] Next, the transmitting / receiving unit 51 transmits the autonomous movement request information generated in step S77 to the mobile body 10 (step S78). As a result, the transmitting / receiving unit 31 of the control device 30 installed in the mobile body 10 receives the autonomous movement request information transmitted from the display device 50. Then, the mobile body 10 starts the autonomous movement process based on the autonomous movement request information received in step S78 (step S79).

[0121] ○ Autonomous movement processing Next, the autonomous movement process of the mobile body 10 in step S79 will be explained in detail using Figures 16 to 18. Figure 16 is a flowchart showing an example of the autonomous movement process of the mobile body.

[0122] First, the mode setting unit 42 of the control device 30 provided in the mobile body 10 sets the operating mode of the mobile body 10 to autonomous movement mode (step S91) when autonomous movement request information is received in step S78. Next, the route information generation unit 37 generates route information indicating the autonomous movement route of the mobile body 10 based on the autonomous movement request information received in step S78 (step S92). Specifically, the route information generation unit 37 generates a movement route that causes the mobile body 10 to autonomously move to the area indicated by the captured images 415 in the order in which the operator selected the captured images 415, based on the candidate ID indicated in the received autonomous movement request information and the order in which the captured images 415 identified by the candidate ID were selected. Note that the route information generation unit 37 is not limited to the order in which the captured images 415 were selected, but may also be configured to generate route information based on the information of the shooting position of the captured images 415 stored in the destination candidate management DB 3001, for example, so that the distance or travel time of the movement route based on the selected captured images 415 is shortened. Then, the route information management unit 38 stores the route information generated in step S92 in the route information management DB 3002 (see Figure 7) in association with the base ID received in step S72 (step S93).

[0123] Next, the location information acquisition unit 35 acquires location information indicating the current location of the mobile body 10 (step S94). The destination setting unit 39 sets the destination of the mobile body 10 based on the current location of the mobile body 10 acquired by the location information acquisition unit 35 and the route information generated in step S92 (step S95). Specifically, the destination setting unit 39 sets the location indicated by the next destination to be reached from among a plurality of destinations identified by the candidate ID shown in the generated route information as the destination. For example, if autonomous movement is about to begin, the destination setting unit 39 sets the location indicated by the destination identified by the first candidate ID shown in the route information as the destination. Then, the destination setting unit 39 generates a movement route from the acquired current location of the mobile body 10 to the set destination. The method of generating the movement route by the destination setting unit 39 is to connect the current location and the destination with a straight line, or to minimize the movement time while avoiding obstacles using captured images or obstacle information obtained by the state detection unit 34.

[0124] Furthermore, the generation of a travel route by the destination setting unit 39 is not limited to using location information of destination candidates (travel destinations) registered in the destination candidate management DB 3001. It may also be configured to identify the location shown in the captured image 415 set as the travel destination through image recognition processing by the image processing unit 47 using the captured image taken by the shooting device 12. In addition, the destination setting unit 39 may generate a travel route to the set travel destination using, for example, the coordinates of the base locations shown in the base location management table (see Figure 31) and the area information management table (see Figure 32). In this case, the destination setting unit 39 may be configured to generate the shortest travel route by identifying adjacent base locations using the adjacent IDs stored in the base location management table and using a general route search method (such as Dijkstra's algorithm or A-STAR algorithm).

[0125] Then, the movement control unit 40 moves the mobile body 10 to the destination, which is set to follow the movement path generated in step S92. In this case, the movement control unit 40 autonomously moves the mobile body 10 in response to a drive command from the autonomous movement processing unit 43. The autonomous movement processing unit 43 performs autonomous movement based on learning data, which is the result of imitation learning by the learning unit 48, for example (step S96).

[0126] Then, the movement control unit 40 terminates processing when the moving body 10 reaches its final destination (YES in step S97). On the other hand, the movement control unit 40 continues autonomous movement processing by repeating the process from step S94 until the moving body 10 reaches its final destination (NO in step S97).

[0127] In this way, the mobile unit 10 can autonomously move to a location shown in the captured image 415 selected by the operator as its destination. Furthermore, when operating in autonomous movement mode, the mobile unit 10 can perform autonomous movement using generated route information and learned data acquired during manual operation mode.

[0128] Here, using Figures 17 and 18, an example of the operation screen displayed on the display device 50 during the autonomous movement of the mobile body 10 will be described. In the following description, the differences from the configuration of the operation screen 200 shown in Figure 10 will be explained. The operation screen 600A shown in Figure 17 includes a status display area 610A that shows the movement status of the mobile body 10, and a base display area 630A that displays images captured by the mobile body 10. The status display area 610A displays a destination candidate image indicating the destination set in step S95 and the current position of the mobile body 10, indicating that the mobile body 10 is heading towards the location shown in the destination candidate image. In addition, the base display area 630A displays a special image acquired by the special imaging device 12a, similar to the base display area 230 of the operation screen 200 shown in Figure 10. Also, in the base display area 630A, similar to the base display area 230 of the operation screen 200, a mobile body display image 635A indicating the presence of the mobile body 10 on the special image is superimposed on the special image. Furthermore, in the base display area 630A, a destination display image 640 indicating the set destination is superimposed and displayed at a position on the special image corresponding to the destination set in step S95.

[0129] Furthermore, the operation screen 600A includes a stop button 690 which is pressed when the autonomous movement of the mobile body 10 is to be interrupted. When the operator presses the stop button 690, the mobile body 10 interrupts the autonomous movement processing by the autonomous movement processing unit 43 in response to the stop request from the display device 50 and stops the mobile body 10.

[0130] Furthermore, the operation screen 600B shown in Figure 18 is a display screen that is shown when the mobile body 10 is autonomously moving toward multiple destinations. The operation screen 600B shown in Figure 18 includes a status display area 610B that shows the movement status of the mobile body 10, and a base display area 630B that displays images captured by the mobile body 10. The status display area 610B displays destination candidate images corresponding to the multiple destination sequences included in the route information generated in step S92, and the current position of the mobile body 10, indicating which destination the mobile body 10 is currently heading toward. In addition, the base display area 630B, similar to the base display area 630A, displays special images acquired by the special imaging device 12a, and a mobile body display image 635B that shows the presence of the mobile body 10 on the special image is superimposed on the special image. Furthermore, the operation screen 600B, similar to the operation screen 600A, includes a stop button 690 that is pressed when the autonomous movement of the mobile body 10 is interrupted.

[0131] Furthermore, in the base display area 630B, destination display images 650 (650a, 650b) representing each destination series are superimposed and displayed at positions on the special image corresponding to the multiple destination series included in the route information generated in step S92. In the example in Figure 18, destination display images 650a and 650b representing destination series No. 2 and No. 3 are displayed, respectively.

[0132] In this way, the display device 50 displays in real time the autonomous movement status of the mobile body 10, which is heading towards the location shown in the captured image 415 selected by the operator. This allows the operator to confirm the current location and movement status of the mobile body 10 even while the mobile body 10 is moving autonomously.

[0133] ○A variation of the selection screen○ Next, a modified example of the selection screen 400 to be displayed on the display device 50 will be described using Figures 19 to 23. Figure 19 is a diagram showing modified example 1 of the selection screen. The selection screen 400A shown in Figure 19 differs from the selection screen 400 shown in Figures 14 and 15 in the configuration of the image display area 410A.

[0134] The image display area 410A displays one special image 450 from the captured image data included in the destination candidate data received in step S74. The operator searches for a location to select as a destination by changing the display position of the special image, in which a part of the base is captured in all directions, in the image display area 410A. The display position of the special image 450 displayed in the image display area 410A is changed by swiping on the screen or by input to the display position change buttons 451 (451a~451d).

[0135] Furthermore, the operator selects a destination by specifying a predetermined position on the special image 450 displayed in the image display area 410A using an input means such as a pointing device 512. The reception unit 52 of the display device 50 accepts the selection of a position on the special image 450 specified by the operator when the operator specifies a predetermined position on the special image 450 displayed in the image display area 410A. The image display area 410A displays a specified position image 455, which indicates the position on the special image 450 specified by the operator, superimposed on the special image 450. Also, the selection screen 400A displays candidate destination images corresponding to the position on the special image 450 specified by the operator in the selection result display area 420, similar to the selection screen 400 shown in Figure 15.

[0136] Furthermore, the selection screen 400A includes a file import button 457 that is pressed when selecting or changing a special image 450 to be displayed in the image display area 410A. When the operator selects the file import button 457 and specifies a predetermined image file, the display control unit 53 displays a different special image from the captured image data received in step 74 in the image display area 410A.

[0137] In this way, the selection screen 400A allows the operator to select a destination for the mobile body 10 using a special image 450 captured by the mobile body 10, enabling the operator to select a predetermined position within the entire surrounding area captured in a single special image 450 as a candidate destination. Furthermore, the display device 50 can improve the operability of selecting a destination for the mobile body 10 by allowing the operator to select a predetermined position within the area captured in the special image 450 as a candidate destination for the mobile body 10.

[0138] Figure 20 shows a modified example of the selection screen, part 2. The selection screen 400B shown in Figure 20 includes, in addition to the configuration of the selection screen 400, a map image 470 that shows the location where the destination candidate image selected by the operator was taken.

[0139] The selection screen 400B includes a map display button 460 and the map image 470, which are pressed when the map image 470 is displayed. The map image 470 includes a current location display image 471 that shows the current location of the moving object 10, and a display selection button 475 that is pressed when zooming in or out on the displayed map. The map image 470 also includes shooting location display images 473 (473a to 473c) that show the shooting locations of the captured images 415 selected by the operator on the map. In the example in Figure 20, the shooting location display images 473a to 473c are displayed at positions on the map corresponding to the shooting locations of captured images No. 1 to No. 3.

[0140] When the display device 50 receives a selection of the map display button 460 from the reception unit 52, it sends a map display request to the mobile device 10. The mobile device 10 then reads the map image data corresponding to the location where the mobile device 10 is located, stored in the map information management DB 3003, and sends it to the requesting display device 50. The display control unit 53 of the display device 50 then displays the received map image data with the current location display image 471 and the shooting location display image 473 superimposed on it.

[0141] Thus, when the operator selects the destination of the mobile object 10, the selection screen 400B displays the location on the map corresponding to the area indicated by the selected captured image, allowing the operator to visually understand where the area indicated by the selected captured image is located within the base, thereby improving the operability of selecting the destination of the mobile object 10. Alternatively, the selection screen 400B may be configured to display the current location display image 471 and the captured location display image 473 superimposed on a simplified map that does not accurately display distance scales, etc. In this case, the selection screen 400B can provide a UI that makes it easier for the operator to understand the mobile object 10's movement path, reducing the difficulty in understanding the mobile object 10's movement path due to the connection of the captured locations indicated by the captured location display image 473 and differences in map resolution.

[0142] Figure 21 shows a third modified example of the selection screen. The selection screen 400C shown in Figure 21 and the selection screen 400A shown in Figure 19 differ from the selection screen 400 shown in Figure 15 in the configuration of the selection result display area 480.

[0143] The selection result display area 480 includes information on the time required to move the mobile body 10 to the position corresponding to the captured image 415 selected by the operator. In the example in Figure 21, it is shown that the time required to reach the area indicated by captured image No. 1 from the current position of the mobile body 10 is 1 minute and 5 seconds, the time required to reach the area indicated by captured image No. 2 from the area indicated by captured image No. 1 is 1 minute and 40 seconds, and the time required to reach the area indicated by captured image No. 3 from the area indicated by captured image No. 2 is 1 minute. The selection result display area 480 also displays the total time required to move to the areas indicated by all captured images 415 selected by the operator (3 minutes and 45 seconds in the example in Figure 21).

[0144] Either the image processing unit 47 of the mobile body 10 or the image processing unit 57 of the display device 50 estimates the time required to move between two points, corresponding to the current position of the mobile body 10 or the position of the region indicated by a plurality of captured images 415 selected by the operator, based on preset information regarding the mobile body 10's movement speed and relative distance on the base. Then, either the image processing unit 47 or the image processing unit 57 generates a selection screen 400C containing the estimated time required information, and the display control unit 53 displays the generated selection screen 400C on a display unit such as the display 506.

[0145] Furthermore, if the mobile body 10 performs a predetermined task in the area indicated by the selected captured image 415, the selection screen 400C may be configured to display not only the travel time but also the total time required to perform the task.

[0146] In this way, the selection screen 400C displays the travel time required for the moving body 10 to move to the area indicated by the captured image 415 selected by the operator, allowing the operator to appropriately decide whether or not to select a destination based on the travel time, and the order in which to select multiple destinations.

[0147] Figure 22 shows a modified example of the selection screen, part 4. The selection screen 400D shown in Figure 22 differs from the selection screen 400 shown in Figure 14 in the configuration of the image display area 410.

[0148] The image display area 410 of the selection screen 400D displays additional information 490 (490a, 490b, 490c, 490e) associated with each of the displayed captured images 415 (415a to 415f). The additional information 490 includes strings indicating the name of the structure or equipment that is the subject in the captured image 415, or the condition of the intersection, slope, or road surface. In the example in Figure 22, additional information 490a indicating "Meter A5" is displayed associated with captured image 415a, and additional information 490b indicating "Tank B9" is displayed associated with captured image 415b. Additionally, additional information 490c indicating "Building C" is displayed associated with captured image 415c, and additional information 490e indicating "Building Y" is displayed associated with captured image 415e.

[0149] Either the image processing unit 47 of the mobile device 10 or the image processing unit 57 of the display device 50 identifies information corresponding to the shooting location of the captured image 415 by image recognition processing of the captured image 415 based on pre-set information about the base and images of subjects present at the base. Then, either the image processing unit 47 or the image processing unit 57 generates a selection screen 400E that associates the identified information with the corresponding captured image 415 as additional information, and the display control unit 53 displays the generated selection screen 400E on a display unit such as the display 506. Here, if, in the destination candidate registration process described above, the name of the destination candidate or a string (keyword) identified by OCR processing is registered in the destination candidate management DB 3001 in association with the captured image data, the selection screen 400E may be configured to display the name or keyword of the destination candidate corresponding to the captured image as additional information.

[0150] Thus, the selection screen 400D displays the captured image 415 (destination candidate image) which shows the candidate destination for the moving object 10, along with additional information 490 such as the name of the subject in the captured image 415. This allows the operator to make a more appropriate decision compared to selecting the destination for the moving object 10 by looking only at the captured image 415. In this case, in the process of setting the destination for the moving object 10 in step S95, the destination setting unit 39 is not limited to a configuration that generates a movement path using the location information of the destination candidate (destination) registered in the destination candidate management DB 3001, but may also be configured to generate a movement path by identifying the position in the captured image 415 from the information of the subject in the captured image 415 identified by the image recognition process.

[0151] Figure 23 shows a modified example of the selection screen 5. The selection screen 400E shown in Figure 23 is an example of a screen that is displayed when the captured images 415 (415a to 417f) displayed in the image display area 410 can be used for purposes other than selecting the destination of the mobile body 10, as information related to setting the movement path for the autonomous movement of the mobile body.

[0152] The selection screen 400E includes a destination selection button 440, which is pressed when selecting a destination for the mobile body 10 using the captured image 415 displayed in the image display area 410 as described above, and an exclusion route selection button 445, which is pressed when selecting an exclusion route when moving the mobile body 10 using the captured image 415 displayed in the image display area 410. Here, the selection of an exclusion route is a process for setting locations to be excluded from the movement path of the mobile body 10 when the mobile body 10 is autonomously moved, that is, locations where the movement of the mobile body 10 is restricted. When selecting an exclusion route, the operator checks, for example, obstacles or the condition of the road surface within the base shown in the captured image 415, and selects the captured image 415 showing locations where the mobile body 10 does not want to move as an exclusion route.

[0153] The operator can select an image corresponding to the specified selection mode by selecting either the destination selection button 440 or the excluded route selection button 445, and then selecting a captured image 415 displayed in the image display area 410. Furthermore, by using the selection screen 400E to combine destination selection and excluded route selection, the operator can select an image that moves the moving object 10 to a predetermined destination without moving the excluded route.

[0154] Thus, the selection screen 400E allows the captured images 415, which have been pre-registered through the registration process of candidate destinations for the mobile body 10, to be used for purposes other than selecting the destination of the mobile body 10. This improves the operability of the operator in selecting destinations for the mobile body 10, including excluded routes.

[0155] ● Effects of the embodiment As explained above, the communication system 1 can improve operability when allowing the operator to set the destination of the mobile body 10 by pre-registering captured images 415, which are still images showing a part of the base area, as candidate destinations for the mobile body 10, and displaying the registered captured images 415 on the display device 50 to allow the operator to select the destination of the mobile body 10. In addition, the communication system 1 can set a travel path for the mobile body 10 that includes multiple destinations according to the order in which they are selected by the operator by allowing the operator to select from multiple captured images 415 displayed on the display device 50 in the order in which they were selected.

[0156] Furthermore, the communication system 1 can reduce the effort required to register potential destinations by automatically acquiring captured images 415 of areas within the bases to be registered as potential destinations, depending on the movement status of the mobile body 10. In addition, the communication system 1 can provide a display screen (selection screen 400, etc.) that allows for easy setting of the movement path when the mobile body 10 is moving autonomously by displaying the captured images 415, which have been automatically acquired according to the movement status of the mobile body 10, on the display device 50.

[0157] ● Modified examples of embodiments ○Example 1○ Next, a modified example 1 of the communication system according to the embodiment will be described with reference to Figures 24 and 27. Note that components and functions identical to those in the above embodiment are denoted by the same reference numerals, and their descriptions are omitted. The communication system 1A according to Modified Example 1 is a system that manages data of destination candidates for the mobile body 10A using an information processing device 90.

[0158] Figure 24 shows an example of the overall configuration of a communication system according to Modification 1 of the embodiment. In addition to the configuration of the embodiment described above, the communication system 1A according to Modification 1 includes an information processing device 90 that can communicate with a mobile body 10A and a display device 50A via a communication network 100.

[0159] The information processing device 90 is a server computer for managing communication between the mobile device 10A and the display device 50A, performing various controls on the mobile device 10A, and generating various display screens to be shown on the display device 50A. The information processing device 90 may consist of one server computer or multiple server computers. Furthermore, although the information processing device 90 is described as a server computer located in a cloud environment, it may also be a server located in an on-premises environment. Hereinafter, the hardware configuration of the information processing device 90 is the same as that of the display device 50 shown in Figure 4. For convenience, the hardware configuration of the information processing device 90 will be described below using the 900 series of reference numerals for the configuration shown in Figure 4.

[0160] Figure 25 is a diagram showing an example of the functional configuration of a communication system according to Modification 1 of the embodiment. The configuration of the display device 50A according to Modification 1 shown in Figure 25 is the same as the configuration of the display device 50 shown in Figure 5. Furthermore, the control device 30A that controls the processing or operation of the mobile body 10A according to Modification 1 has a configuration that excludes the functions of the map information management unit 46 and the destination candidate management DB 3001, map information management DB 3003, and learning data management DB 3004 that were built in the storage unit 3000 from the control device 30 shown in Figure 5.

[0161] The information processing device 90 includes a transmitting / receiving unit 91, a map information management unit 92, and a storage / reading unit 99. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 901 in accordance with a program for the information processing device deployed on the RAM 903. The information processing device 90 also has a storage unit 9000 constructed from a ROM 902, HD 904, or recording medium 921, as shown in Figure 4.

[0162] The transmitting / receiving unit 91 is mainly implemented by the processing of the network interface 908 by the CPU 901, and transmits and receives various data or information with other devices or terminals.

[0163] The map information management unit 92 is primarily implemented by the processing of the CPU 901 and uses the map information management DB 9002 to manage map information showing the environmental map of the target location where the mobile unit 10 is installed. The map information management unit 92 manages map information showing, for example, environmental maps downloaded from an external server or environmental maps created by applying SLAM.

[0164] The memory / read unit 99 is primarily implemented by the CPU 901, and stores various data (or information) in the memory unit 9000 and reads various data (or information) from the memory unit 9000. The memory unit 9000 also contains a destination candidate management DB 9001, a map information management DB 9002, and a learning data management DB 9003. Of these, the destination candidate management DB 9001 is composed of the destination candidate management table shown in Figure 6. The map information management DB 9002 is composed of the map information management table shown in Figure 8. Furthermore, the learning data management DB 9003 has the same configuration as the learning data management DB 3004 constructed in the memory unit 3000 of the mobile body 10 described above.

[0165] Figure 26 is a sequence diagram showing an example of the destination candidate registration process according to Modification 1 of the Embodiment. Note that the processes in steps S101 to S107 are the same as the processes in steps S11 to S17 shown in Figure 11, so their explanation is omitted.

[0166] In step S108, the transmitting / receiving unit 51 of the display device 50A transmits a destination candidate registration request to the information processing device 90. As a result, the transmitting / receiving unit 91 of the information processing device 90 receives the destination candidate registration request transmitted from the display device 50A.

[0167] Next, the transmitting / receiving unit 91 of the information processing device 90 transmits a data acquisition request necessary for registering a destination candidate to the mobile body 10A (step S109). As a result, the transmitting / receiving unit 31 of the control device 30A equipped on the mobile body 10A receives the data acquisition request transmitted from the information processing device 90.

[0168] Next, the location information acquisition unit 35 of the control device 30A transmits a data acquisition request necessary for registering a destination candidate to the mobile body 10 (step S109). As a result, the transmitting / receiving unit 31 of the control device 30A, which is installed in the mobile body 10A, receives the data acquisition request transmitted from the information processing device 90.

[0169] Next, the position information acquisition unit 35 of the control device 30A acquires position information indicating the current position of the mobile body 10A using a GPS sensor 306 or the like (step S110). Specifically, the position information acquisition unit 35 acquires coordinate information including latitude and longitude indicating the current position of the mobile body 10. If the mobile body 10 is an aircraft such as a drone, the position information acquisition unit 35 acquires information such as the speed, attitude, or altitude of the aircraft as position information in addition to the coordinate information including latitude and longitude. The destination candidate acquisition unit 36 ​​acquires a captured image taken by the special imaging device 12a at the current position of the mobile body 10A as a destination candidate image indicating a destination candidate for the mobile body 10 (step S111). Then, the transmitting / receiving unit 31 transmits the position information acquired in step S110 and the captured image acquired in step S11 to the requesting information processing device 90. As a result, the transmitting / receiving unit 91 of the information processing device 90 receives the position information and captured image transmitted from the mobile body 10A.

[0170] Then, the storage / reading unit 99 stores the destination candidate data, including the location information and captured image received in step S111, in the destination candidate management DB 9001 (step S112). The storage / reading unit 99 registers the received location information and captured image as destination candidate data, each associated with a candidate ID that identifies the destination candidate. Note that, as with the embodiment described above, the destination candidate registration process shown in Figure 26 is an example of the registration process for the destination of the mobile body 10A, and the process may be performed not only for the purpose of registering destination candidates for the mobile body 10A, but also, for example, for the purpose of registering candidates for locations to be excluded from the mobile body 10A's travel route.

[0171] Figure 27 is a sequence diagram showing an example of the process for setting the destination of a mobile body according to a modified example of the embodiment 1. First, the receiving unit 52 of the display device 50A accepts the selection of the destination setting button 290 by an input operation on the operator's operation screen 200, similar to step S71 in Figure 13 (step S131). Next, the transmitting / receiving unit 51 sends a destination candidate acquisition request to the information processing device 90, indicating a request to acquire data indicating destination candidates (step S132). This destination candidate acquisition request includes a base ID that identifies the base where the mobile body 10A is installed. As a result, the transmitting / receiving unit 91 of the information processing device 90 transmits the destination candidate acquisition request sent from the display device 50A.

[0172] Next, the storage and reading unit 99 of the information processing device 90 searches the destination candidate management DB 9001 using the location ID indicated in the destination candidate acquisition request received in step S132 as a search key, and reads destination candidate data associated with the same location ID as the received location ID (step S133). Then, the transmitting and receiving unit 91 transmits the destination candidate data read in step S133 to the requesting display device 50A (step S134). This destination candidate data includes a set of multiple candidate IDs, captured images, and location information read in step S133. As a result, the display device 50A receives the destination candidate data transmitted from the information processing device 90.

[0173] Next, the display control unit 53 of the display device 50A displays a selection screen 400, as shown in Figures 14 and 15, which includes the destination candidate data received in step S134, on a display unit such as the display 506 (step S135). The processing in steps S135 to S139 is the same as the processing in steps S75 to S79 ​​shown in Figure 13, so the explanation is omitted. As with the embodiment described above, the destination setting process shown in Figure 27 is an example of the process for setting the destination of the mobile body 10A, and the processing may be performed not only for setting the destination of the mobile body 10A, but also, for example, for setting exclusion routes to be excluded from the travel path of the mobile body 10A.

[0174] Thus, even when the information processing device 90 registers and manages destination candidate data, the communication system 1A according to Modification 1 can display a selection screen 400 showing destination candidate images on the display device 50A, allowing the operator to select the destination of the mobile body 10A. This improves the operability when allowing the operator to select the destination of the mobile body 10A. Furthermore, by registering and centrally managing destination candidate data using the information processing device 90, the communication system 1A can select a destination using images acquired from different mobile bodies 10A, thus improving the convenience for the operator in setting the destination of the mobile body 10.

[0175] ○Differentiation Example 2○ Next, a modified example 2 of the communication system according to the embodiment will be described with reference to Figures 28 to 30. Note that components and functions identical to those in the above embodiment are denoted by the same reference numerals, and their descriptions are omitted. The communication system 1B according to modified example 2 is a system capable of displaying image data provided by an external service N on a display device 50B, thereby enabling the mobile body 10B to select a destination.

[0176] Figure 28 shows an example of the overall configuration of a communication system according to Modification 2 of the embodiment. In addition to the configuration of the embodiment described above, the communication system 1B according to Modification 2 includes an external service N that can communicate with the mobile body 10B and the display device 50B via the communication network 100.

[0177] External service N is an external system that provides various services via the communication network 100. External service N may include, for example, a cloud service such as a storage service (or online storage) that stores data files on the cloud, or a search service that uses a search engine via the internet.

[0178] The display device 50B performs destination selection using images by running a pre-installed dedicated application or accessing a website using a web browser and acquiring image data provided by an external service. Furthermore, since the communication system 1B can acquire images taken at various locations from the external service N without being limited to the location of the subject, the mobile body 10B is not limited to a mobile body that moves within a limited base as in the above embodiment, but can also be applied to a mobile body that moves over a wide range without being limited in its movement range. In this case, the mobile body 10B is preferably used as an automated vehicle, for example, as shown in Figure 28. If the mobile body 10B is an automobile, the display device 50B may be a car navigation system installed in the automobile, or a smartphone used by an operator riding in the automobile.

[0179] Figure 29 is a sequence diagram showing an example of the process for setting a destination for a mobile object according to Modification 2 of the embodiment. Note that the communication system 1B according to Modification 2 is a system that sets a destination using images provided by an external service N, so the destination candidate registration process (see Figure 9) shown in the above embodiment is unnecessary.

[0180] First, the display control unit 53 of the display device 50B accesses the external service N using a dedicated application or a web browser to display the image search screen 800 on a display unit such as the display 506 (step S201). Figure 30 shows an example of the image search screen. The image search screen 800 shown in Figure 30 is a display screen for searching for images that can be obtained by using the external service N. The image search screen 800 is a service provision screen provided by the external service N or an arbitrary website, etc.

[0181] The image search screen 800 includes a keyword input area 810 for entering a predetermined keyword and a search button 815 which is pressed when performing an image search using the keyword entered in the keyword input area 810. The image search screen 800 also includes a location information input area 830 for entering a predetermined location information, a map reference button 831 which is pressed when displaying a map to be used to specify the location information, and a search button 835 which is pressed when performing an image search using the location information entered in the location information input area 830 or the location information specified using the map referenced from the map reference button 831. Furthermore, the image search screen 800 includes a cancel button 805 which is pressed when canceling the image search.

[0182] The reception unit 52 of the display device 50B accepts input of keywords or location information for the image search screen 800 (step S202). When the reception unit 52 accepts the operator's selection for the search button 815 or the search button 835, the transmitting / receiving unit 51 sends an image acquisition request to the external service N (step S203). This image acquisition request includes the input data accepted in step S202.

[0183] Next, the external service N performs an image search based on the input data transmitted from the display device 50B and transmits the resulting image data to the requesting display device 50B (step S204). As a result, the transmitting / receiving unit 51 of the display device 50B receives the image data transmitted from the external service N.

[0184] Then, the transmitting and receiving unit 51 of the display device 50B causes a selection screen 400, as shown in Figures 14 and 15, which includes the destination candidate data received in step S204, to be displayed on a display unit such as the display 506 (step S205). In this case, the image display area 410 of the selection screen 400 displays image data transmitted from the external service N. Note that the processing in steps S206 to S209 is the same as the processing in steps S76 to S79 ​​shown in Figure 13, so the explanation is omitted.

[0185] Thus, the communication system 1B according to Modification 2 can select the destination of the mobile body 10B using image data provided by the external service N, eliminating the need to register candidate destinations using images captured by the mobile body 10B, thereby improving the convenience of selecting the destination of the mobile body 10B. Furthermore, since the communication system B can obtain images of various locations from the external service N through image search, it can set the destination of the mobile body 10B without being limited by the mobile body 10B's movement range.

[0186] Figure 35 is a diagram showing an example of the functional configuration of a communication system. In comparison with the functional configuration of the communication system shown in Figure 25, the display device 50 has the same configuration as the display device 50 shown in Figure 25. The control device 30B that controls the processing or operation of the mobile body 10B has the same configuration as the control device 30A shown in Figure 25, but with the route information generation unit 37, route information management unit 38, destination setting unit 39, learning unit 48, and the route information management DB 3002 that was built in the storage unit 3000 removed.

[0187] In the communication system 1B shown in Figure 35, the information processing device 90 supports cloud computing services such as AWS (trademark), and the communication system 1B allows the display device 50 and the mobile unit 10B (control device 30B) to communicate via the information processing device 90 as indicated by arrows a and b. In addition, the functions of the route information generation unit 37, route information management unit 38, destination setting unit 39, learning unit 48, and route information management DB 3002, which were removed from the control device 30A, have been transferred to the information processing device 90. In other words, the information processing device 90 has a transmitting / receiving unit 91, a map information management unit 92, a route information generation unit 93, a route information management unit 94, a destination setting unit 95, and a learning unit 96. Furthermore, the storage unit 9000 of the information processing device 90 contains a destination candidate management DB 9001, a map information management DB 9002, a learning data management DB 9003, and a route information management DB 9004. The functions of the above-mentioned components transferred from the control device 30A (Figure 25) to the information processing device 90 are the same as those described in Figure 25, so their explanation will be omitted.

[0188] As described above, in the communication system 1B, communication between the display device 50 and the mobile device 10B (control device 30B) is performed via an information processing device 90 that supports cloud computing services. By using authentication processing by the cloud computing service during communication, the security of manual operation commands from the display device 50 and captured image data from the mobile device 10B can be enhanced. Furthermore, by placing the data generation and management functions in the information processing device 90 (cloud service), it becomes possible to share the same data at multiple locations, and it becomes possible to flexibly support not only P2P (Peer to Peer) communication (one-to-one direct communication) but also one-to-multi-location communication.

[0189] ●Summary● As described above, the display device according to one embodiment of the present invention is a display device 50 (50A, 50B) for performing a predetermined operation on a mobile body 10 (10A, 10B) moving within a predetermined location, and acquires a captured image 415 of a part of the location taken by the mobile body 10 (10A, 10B), and displays the acquired captured image 415 as information related to setting the destination of the autonomous movement of the mobile body 10 (10A, 10B). As a result, the display device 50 (50A, 50B) can provide a display screen that allows the user to easily select the destination of the mobile body 10 (10A, 10B).

[0190] Furthermore, the display device according to one embodiment of the present invention acquires captured images 415 of a portion of the base area taken by the mobile body 10 (10A, 10B) according to the movement state of the mobile body 10 (10A, 10B). As a result, the display device 50 (50A, 50B) can display the captured images 415 that have been automatically acquired according to the movement state of the mobile body 10 (10A, 10B), thereby providing a display screen (selection screen 400, etc.) that allows for easy setting of the movement path when the mobile body 10 (10A, 10B) is moving autonomously.

[0191] Furthermore, the display device according to one embodiment of the present invention acquires captured images 415 when the mobile body 10 (10A, 10B) performs a predetermined task. The display device 50 (50A, 50B) also acquires captured images 415 when the operation of the mobile body 10 (10A, 10B) stops. Furthermore, the display device 50 (50A, 50B) acquires captured images 415 when it detects a specific subject on the movement path of the mobile body 10 (10A, 10B). In addition, the display device 50 (50A, 50B) receives manual operation on the mobile body 10 (10A, 10B) and acquires captured images 415 when the direction of operation of the received manual operation is changed. As a result, the display device 50 (50A, 50B) can display captured images 415 that have been automatically acquired based on predetermined criteria corresponding to the movement state of the mobile body 10 (10A, 10B), thereby providing a display screen (selection screen 400, etc.) that allows for easy setting of the movement path when the mobile body 10 (10A, 10B) is moving autonomously.

[0192] Furthermore, a communication system according to one embodiment of the present invention is a communication system 1(1A,1B) comprising a display device 50(50A,50B) for performing predetermined operations on a mobile body 10(10A,10B) moving within a predetermined location, and a mobile body 10(10A,10B) capable of communicating with the display device 50(50A,50B). The mobile body 10(10A,10B) moves in response to a manual operation command indicating a manual operation received by the display device 50(50A,50B), and acquires a captured image 415 taken at the current position of the mobile body 10(10A,10B) according to the movement state of the mobile body 10(10A,10B) by manual operation. As a result, the communication system 1 (1A, 1B) can reduce the effort required to register potential destinations by automatically acquiring captured images 415 of areas within the base to be registered as potential destinations, depending on the manual operation status of the mobile unit 10 (10A, 10B).

[0193] ●Additional Information● Each function of the embodiment described above can be realized by one or more processing circuits. Here, "processing circuit" in this embodiment includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), SOCs (System on a chip), GPUs (Graphics Processing Units), and conventional circuit modules designed to execute each of the functions described above.

[0194] Furthermore, the various tables in the embodiments described above may be generated by the learning effect of machine learning, and tables may not be used if the data of each related item is classified by machine learning. Here, machine learning is a technique for enabling computers to acquire human-like learning abilities, and it refers to a technique in which a computer autonomously generates algorithms necessary for judgments such as data identification from pre-incorporated training data, and applies these to new data to make predictions. The learning method for machine learning may be supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, or deep learning, or a combination of these learning methods may be used, and the learning method for machine learning is not limited.

[0195] While a display device, communication system, display control method, and program according to one embodiment of the present invention have been described so far, the present invention is not limited to the embodiments described above. It can be modified to the extent that a person skilled in the art can conceive of it, such as by adding, changing, or deleting other embodiments, and any embodiment that achieves the function and effect of the present invention is included within the scope of the present invention. [Explanation of Symbols]

[0196] 1,1A,1B Communication System 100 Communication Networks 10,10A,10B Mobile object 30, 30A, 30B control devices 31 Transmitter / Receiver 32 Judgment Department 35 Location information acquisition section 36. Destination candidate acquisition unit (an example of a second acquisition method) 37 Route Information Generation Unit 40 Movement Control Unit 41 Image Recognition Processing Unit 42 Mode setting section 43 Autonomous Mobile Processing Unit 44 Manual operation processing unit (an example of a movement processing means) 45 Task Execution Unit 48 Learning Department 3000 Memory Unit (An example of a memory means) 50,50A,50B display device 51 Transmitting / receiving unit (Example of acquisition means, Example of first acquisition means) 52 Reception area (an example of an operation reception means) 53 Display Control Unit (An example of display control means) 56 Autonomous movement request information generation unit 90 Information Processing Equipment 91 Transmitter / Receiver 9000 Memory Unit (An example of a memory device) 200 Operation screen 400, 400A, 400B, 400C, 400D, 400E Selection Screen 415 Photographed Images 450 Special Images 490 Additional Information 600A,600B operation screen 800 Image Search Screen

Claims

1. A display device for performing a predetermined operation on a mobile object moving within a predetermined location, Acquisition means for acquiring captured images of a portion of the area within the base captured by the mobile body, A display control means that displays the acquired captured image as information related to setting the destination of the autonomous movement of the moving object, A display device equipped with the following features.

2. The display device according to claim 1, wherein the acquisition means acquires an image of a portion of the area within the base captured by the moving body, depending on the movement state of the moving body.

3. The display device according to claim 1 or 2, wherein the acquisition means acquires the captured image when the moving body performs a predetermined task.

4. The display device according to any one of claims 1 to 3, wherein the acquisition means acquires the captured image when the movement of the moving body stops.

5. The display device according to any one of claims 1 to 4, wherein the acquisition means acquires the captured image when a specific subject is detected on the movement path of the moving body.

6. The display device according to claim 5, wherein the specific subject is an intersection on the movement path of the moving body.

7. A display device according to any one of claims 1 to 6, further, The system includes an operation receiving means for receiving manual operations on the aforementioned moving body, The acquisition means is a display device that acquires the captured image taken when the direction of the manual operation received by the operation reception means is changed.

8. The display device according to any one of claims 1 to 7, wherein the acquisition means acquires the captured image taken in response to the user's request to take a picture.

9. A communication system comprising a display device for performing a predetermined operation on a mobile body moving within a predetermined location, and the mobile body capable of communicating with the display device, A first acquisition means for acquiring captured images of a portion of the area within the base captured by the mobile body, A display control means that displays the acquired captured image as information related to setting the destination of the autonomous movement of the moving object, A communication system equipped with [the following features].

10. A communication system according to claim 9, The aforementioned display device is The system includes an operation receiving means for receiving manual operations on the aforementioned moving body, The aforementioned moving body is A movement processing means for moving the movable body in response to a manual operation command indicating a manual operation received by the operation receiving means, A second acquisition means for acquiring the captured image taken at the current position of the moving body according to the movement state of the moving body by the manual operation, A communication system equipped with [the following features].

11. A communication system according to claim 9 or 10, further, The system includes a storage means for storing the captured image acquired by the second acquisition means and the shooting position of the captured image in association with each other. The display control means is a communication system that displays the stored captured image.

12. A communication system according to claim 11, The second acquisition means is a communication system that, when the moving body approaches the location where the stored image was taken, acquires the image taken at the current location of the moving body.

13. A display control method performed by a display device for performing a predetermined operation on a moving object moving within a predetermined location, The acquisition step involves acquiring captured images of a portion of the area within the base captured by the mobile device, A display control step that displays the acquired captured image as information related to setting the destination of the autonomous movement of the moving object, A display control method that executes [this].

14. A program that causes a computer to execute the display control method described in claim 13.

Images