Display device, communication system, display control method, and program

Abstract

To improve operability of a user when selecting a moving destination of a moving body.SOLUTION: A display device for performing a predetermined operation to a moving body acquires a photographed image 415, displays the acquired photographed image 415 so as to be selectable by a user, and receives selection to the displayed photographed image 415. Then, the display device sets the moving destination of the mobile object on the basis of the area indicated by the photographed image 415 whose selection is received.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] The present disclosure relates to a display device, a communication system, a display control method, and a program. [Background technology]

[0002] Robots that are installed at bases such as factories and warehouses and can move autonomously within the bases are known. Such robots are used, for example, as inspection robots or service robots, and can perform tasks such as inspecting equipment within the bases in place of workers.

[0003] Also, a robot that autonomously moves to a destination designated by a user based on location information or a name is already known. For example, Patent Document 1 discloses a method in which a user designates a destination of a moving body based on location information arbitrarily selected by the user, and an autonomous mobile robot autonomously moves to the designated destination. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-94743 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional method, when setting the destination of a moving object, the user must specify a character string such as location information or a keyword, making it difficult to select a desired destination. [Means for solving the problem]

[0006] In order to solve the above-mentioned problem, the invention of claim 1 is a display device for performing predetermined operations on a moving body moving within a predetermined base, and is equipped with an acquisition means for acquiring a photographed image showing a partial area within the base, a display control means for displaying the acquired photographed image so that it can be selected by a user, and an acceptance means for accepting a selection from the displayed photographed image, and is a display device that sets a movement path for the moving body based on the area shown in the photographed image for which the selection has been accepted. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the operability for the user when selecting the destination of a moving object. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a communication system. [Figure 2] FIG. 1 is a diagram illustrating an example of a schematic configuration of a moving body. [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a mobile object. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device. [Figure 5] FIG. 1 illustrates an example of a functional configuration of a communication system. [Figure 6] FIG. 10 is a conceptual diagram illustrating an example of a destination candidate management table. [Figure 7] FIG. 10 is a conceptual diagram illustrating an example of a route information management table. [Figure 8] FIG. 2 is a conceptual diagram illustrating an example of a map information management table. [Figure 9] FIG. 10 is a sequence diagram illustrating an example of a process for registering a destination candidate. [Figure 10] FIG. 10 is a diagram illustrating an example of an operation screen. [Figure 11] FIG. 10 is a sequence diagram illustrating an example of a manual operation process of a moving object. [Figure 12] 10 is a flowchart illustrating an example of a process for registering destination candidates according to the movement state of a mobile object. [Figure 13] FIG. 10 is a sequence diagram illustrating an example of a process for setting a destination of a moving body. [Figure 14] FIG. 10 is a diagram illustrating an example of a selection screen. [Figure 15] FIG. 10 is a diagram illustrating an example of a selection screen. [Figure 16] 10 is a flowchart illustrating an example of an autonomous movement process of a moving object. [Figure 17] FIG. 10 is a diagram showing an example of an operation screen when the moving body moves autonomously. [Figure 18] FIG. 10 is a diagram showing an example of an operation screen when the moving body moves autonomously. [Figure 19] FIG. 10 is a diagram showing a first modified example of the selection screen. [Figure 20] FIG. 10 is a diagram showing a second modified example of the selection screen. [Figure 21] FIG. 10 is a diagram showing a third modified example of the selection screen. [Figure 22] FIG. 10 is a diagram showing a fourth modified example of the selection screen. [Figure 23] FIG. 10 is a diagram showing a fifth modified example of the selection screen. [Figure 24] FIG. 10 is a diagram illustrating an example of the overall configuration of a communication system according to a first modified example of the embodiment. [Figure 25] FIG. 10 is a diagram illustrating an example of a functional configuration of a communication system according to a first modified example of the embodiment. [Figure 26] FIG. 10 is a sequence diagram illustrating an example of a process of registering a destination candidate according to a first modified example of the embodiment. [Figure 27] FIG. 10 is a sequence diagram illustrating an example of a process for setting a destination of a moving object according to a first modified example of the embodiment. [Figure 28] FIG. 10 is a diagram illustrating an example of the overall configuration of a communication system according to a second modification of the embodiment. [Figure 29] FIG. 10 is a sequence diagram illustrating an example of a process for setting a destination of a moving object according to a second modification of the embodiment. [Figure 30] FIG. 10 is a diagram illustrating an example of an image search screen. [Figure 31] FIG. 1 illustrates an example of a functional configuration of a communication system. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated explanations will be omitted.

[0010] ●Embodiment● ●System configuration Fig. 1 is a diagram showing an example of the overall configuration of a communication system. The communication system 1 shown in Fig. 1 is a system that allows a user to remotely control a mobile object 10 that moves within a predetermined base.

[0011] The communication system 1 includes a mobile object 10 and a display device 50 located at a predetermined base. The mobile object 10 and the display device 50 constituting the communication system 1 can communicate via a communication network 100. The communication network 100 is constructed using the Internet, a mobile communication network, a LAN (Local Area Network), etc. Note that the communication network 100 may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0012] The mobile body 10 is a robot installed at a target base and capable of autonomously moving within the target base. This autonomous movement includes imitation learning (machine learning) of previously traveled routes within the target base and autonomously moving within the target base using the results of imitation learning. Furthermore, autonomous movement may also be moving within the target base autonomously according to a preset movement route or autonomously moving within the target base using technology such as line tracing. Furthermore, the mobile body 10 can also be moved by manual operation from a remote user. That is, the mobile body 10 can move within the target base while switching between autonomous movement and manual operation by a user. Furthermore, the mobile body 10 can perform predetermined tasks, such as inspection, maintenance, transportation, or light work, while moving within the target base. Here, the mobile body 10 refers to a robot in a broad sense, and may be a robot capable of both autonomous movement and movement remotely controlled by a user. The mobile body 10 also includes a vehicle that can switch between automatic driving and manual operation by remote control. Furthermore, the mobile object 10 also includes an air vehicle such as a drone, a multicopters, or an unmanned air vehicle.

[0013] The target base where the mobile object 10 is installed may be, for example, an outdoor base such as a business office, factory, construction site, substation, farm, rice field, plantation, cultivated land, or disaster site, or an indoor base such as an office, school, factory, warehouse, commercial facility, hospital, or nursing home. In other words, the target base may be any base where there is a need to have the mobile object 10 take on tasks that have traditionally been performed manually.

[0014] The display device 50 is a computer such as a notebook PC (Personal Computer) that is located at a management base different from the target base and is used by an operator (user) who performs predetermined operations on the mobile object 10. The operator, at the management base such as an office, uses an operation screen displayed on the display device 50 to perform operations such as moving the mobile object 10 or performing predetermined tasks. For example, the operator remotely controls the mobile object 10 while viewing an image of the target base displayed on the display device 50.

[0015] 1 shows an example in which one mobile object 10 and one display device 50 are connected via a communication network 100, the display device 50 may be configured to be connectable to multiple mobile objects 10 located at one target base, or may be configured to be connectable to mobile objects 10 located at different target bases. Also, while FIG. 1 shows an example in which the display device 50 is located at a management base in a remote location different from the target base where the mobile object 10 is installed, the display device 50 may be configured to be located within the target base where the mobile object 10 is installed. Furthermore, the display device 50 is not limited to a notebook PC, and may be, for example, a desktop PC, a tablet terminal, a smartphone, or a wearable terminal.

[0016] Conventionally, when an operator sets the destination of an autonomously moving body, the operator is required to specify an arbitrary character string such as the name of the destination location or location information. For example, in cases where the person who sets the character string is different from the person who sets the destination, there has been a problem in that the person who sets the destination has difficulty in appropriately selecting the destination of the moving body as the destination because they do not know the name of the destination, etc.

[0017] Therefore, while the moving object 10 is moving, the communication system 1 uses the image capturing device 12 mounted on the moving object 10 to capture images of the surroundings of the moving object 10, in addition to the location information of the moving object 10, and registers the captured images in advance as candidate destinations such as the destination of the moving object 10. Then, the communication system 1 displays the captured images showing the registered candidate destinations on the display device 50, and allows the operator to select a destination for the moving object 10. This allows the communication system 1 to improve operability when allowing the user to select a destination for the moving object 10.

[0018] ●Configuration of moving body Next, a specific configuration of the moving body 10 will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of a schematic configuration of the moving body. Note that components may be added or deleted from the configuration of the moving body 10 shown in Fig. 2 as needed.

[0019] 2 includes a housing 11 equipped with a control device 30 that controls the processing or operation of the moving body 10, an imaging device 12, a support member 13, a display 14, a movement mechanism 15 (15a, 15b) for moving the moving body 10, and a movable arm 16 for causing the moving body 10 to perform a predetermined task (operation). Of these, the housing 11 is located in the body of the moving body 10 and has built-in control device 30 and the like that controls the processing or operation of the moving body 10.

[0020] The camera 12 captures images of subjects such as people, objects, or landscapes located at a base where the mobile unit 10 is installed. The camera 12 is a digital camera (general camera) capable of capturing planar images, such as a digital single-lens reflex camera or a compact digital camera. The camera 12 may also be a special-purpose camera capable of capturing a 360° panoramic image. The special-purpose camera is, for example, a spherical camera that captures a subject and obtains two hemispherical images that form the basis of the spherical (panoramic) image. The special-purpose camera may also be, for example, a wide-angle camera or a stereo camera that can capture a wide-angle image (panoramic image) with a field of view equal to or greater than a predetermined value. The special-purpose camera may also be a camera capable of capturing a hemispherical image of a subject. In other words, the special-purpose camera is a camera means capable of capturing images captured using a lens with a focal length shorter than a predetermined value. Here, the special-purpose camera is an example of a camera that captures special images captured in all directions within a base. The special photography device may be configured to, for example, rotate a general photography device that takes ordinary planar images and synthesize the resulting images to generate a special image that captures the entire area within the base. In either method, the special photography device stitches together multiple photographic results using image processing (stitching process) to generate the final special image.

[0021] Furthermore, the moving body 10 may be configured to include multiple image capturing devices 12. In this case, the moving body 10 may be configured to include both a special image capturing device that captures images of a portion of the base in all directions and a general image capturing device that can capture a planar image of a portion of a subject captured by the special image capturing device. In the following description, an example will be described in which the moving body 10 includes both a special image capturing device 12a and a general image capturing device 12b as the image capturing devices 12. Note that the captured images captured by the image capturing device 12 may be video or still images, or may be both video and still images. Furthermore, the captured images captured by the image capturing device 12 may include audio data along with image data.

[0022] The support member 13 is a member for installing (fixing) the image capturing device 12 on the moving body 10 (housing 11). The support member 13 may be a pole or the like fixed to the housing 11, or may be a base fixed to the housing 11. The support member 13 may also be a movable member that can adjust the image capturing direction (orientation) and position (height) of the image capturing device 12.

[0023] The movement mechanism 15 is a unit that moves the moving body 10, and is composed of wheels, a travel motor, a travel encoder, a steering motor, a steering encoder, etc. Controlling the movement of the moving body 10 is an existing technology, so a detailed explanation will be omitted. However, the moving body 10 receives a travel instruction from an operator (display device 50), for example, and the movement mechanism 15 moves the moving body 10 based on the received travel instruction. The movement mechanism 15 may be a bipedal foot type or a single-wheel type. Furthermore, the shape of the moving body 10 is not limited to the vehicle type shown in FIG. 2, and may be, for example, a bipedal human type, a form imitating a living creature, or a form imitating a specific character.

[0024] The movable arm 16 has an operating means that enables additional operations other than the movement of the movable body 10. As shown in FIG. 2, the movable arm 16 is provided with, for example, a hand at the tip of the movable arm 16 as an operating means for grasping an object such as a part. The movable body 10 can perform a predetermined task (operation) by rotating or deforming the movable arm 16. In addition to the above configuration, the movable body 10 may also have various sensors that can detect information about the surroundings of the movable body 10. The various sensors are, for example, sensor devices such as a barometer, thermometer, photometer, human presence sensor, gas sensor, odor sensor, or illuminance meter.

[0025] ●Hardware configuration Next, the hardware configuration of the device or terminal constituting the communication system according to the embodiment will be described with reference to Figures 3 and 4. Note that components may be added or deleted from the hardware configuration of the device or terminal shown in Figures 3 and 4 as needed.

[0026] ○Mobile hardware configuration○ 3 is a diagram showing an example of the hardware configuration of a mobile object. The mobile object 10 includes a control device 30 that controls the processing or operation of the mobile object 10. As described above, the control device 30 is provided inside the housing 11 of the mobile object 10. Note that the control device 30 may be provided outside the housing 11 of the mobile object 10, or may be provided as a device separate from the mobile object 10.

[0027] The control device 30 includes a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, an HDD (Hard Disk Drive) 304, a media I / F (Interface) 305, an input / output I / F 306, an audio input / output I / F 307, a network I / F 308, a short-range communication circuit 309, an antenna 309a of the short-range communication circuit 309, an external device connection I / F 311, and a bus line 310.

[0028] The CPU 301 performs overall control of the moving body 10. The CPU 301 is a calculation device that implements each function of the moving body 10 by reading programs or data stored in the ROM 302 or HD (Hard Disk) 304a, etc., onto the RAM 303 and executing the processing.

[0029] The ROM 302 is a non-volatile memory that can retain programs or data even when the power is turned off. The RAM 303 is a volatile memory used as a work area for the CPU 301, etc. The HDD 304 controls the reading and writing of various data from and to the HD 304a under the control of the CPU 301. The HD 304a stores various data such as programs. The media I / F 305 controls the reading and writing (storage) of data from and to a recording medium 305a such as a USB (Universal Serial Bus) memory, a memory card, an optical disk, or a flash memory.

[0030] The input / output I / F 306 is an interface for inputting and outputting characters, numerical values, various instructions, etc., to and from various external devices, etc. The input / output I / F 306 controls the display of various information, such as a cursor, menu, window, characters, or images, on the display 14, such as an LCD (Liquid Crystal Display). The display 14 may be a touch panel display equipped with input means. In addition to the display 14, input means such as a mouse or keyboard may be connected to the input / output I / F 306. The sound input / output I / F 307 is a circuit that processes the input and output of sound signals between the microphone 307a and the speaker 307b under the control of the CPU 301. The microphone 307a is a type of built-in sound collection means that inputs sound signals under the control of the CPU 301. The speaker 307b is a type of playback means that outputs sound signals under the control of the CPU 301.

[0031] The network I / F 308 is a communication interface that communicates (connects) with other devices or apparatuses via the communication network 100. The network I / F 308 is, for example, a communication interface such as a wired or wireless LAN. The short-range communication circuit 309 is a communication circuit such as NFC (Near Field communication) or Bluetooth (registered trademark). The external device connection I / F 311 is an interface for connecting other devices to the control device 30.

[0032] The bus line 310 is an address bus, a data bus, etc. for electrically connecting the above-mentioned components, and transmits address signals, data signals, various control signals, etc. The CPU 301, the ROM 302, the RAM 303, the HDD 304, the media I / F 305, the input / output I / F 306, the sound input / output I / F 307, the network I / F 308, the short-range communication circuit 309, and the external device connection I / F 311 are connected to each other via the bus line 310.

[0033] Furthermore, the control device 30 is connected to a drive motor 101, an actuator 102, an acceleration / direction sensor 103, a GPS (Global Positioning System) sensor 104, a special imaging device 12a, a general imaging device 12b, a battery 120, and an obstacle detection sensor 106 via an external device connection I / F 311.

[0034] Based on commands from the CPU 301, the drive motor 101 rotates the movement mechanism 15 to move the mobile object 10 along the ground. Based on commands from the CPU 301, the actuator 102 deforms the movable arm 16. The acceleration / direction sensor 103 is a sensor such as an electronic magnetic compass, a gyrocompass, and an acceleration sensor that detects geomagnetism. The GPS sensor 104 receives GPS signals from GPS satellites. The battery 120 is a unit that supplies the power required for the entire mobile object 10. The battery 120 may be one built into the mobile object 10 itself, or it may include an external battery that serves as an external auxiliary power supply. The obstacle detection sensor 106 is a detection sensor that detects obstacles in the surrounding area as the mobile object 10 moves. The obstacle detection sensor 106 is, for example, an image sensor such as a stereo camera or a camera equipped with an area sensor in which photoelectric conversion elements are arranged in a planar form, or a ranging sensor such as a TOF (Time Of Flight) sensor, a LIDAR (Light Detection and Ranging) sensor, a radar sensor, a laser range finder, an ultrasonic sensor, a depth camera, or a depth sensor.

[0035] ○Display device hardware configuration○ Fig. 4 is a diagram showing an example of the hardware configuration of a display device. Each piece of hardware configuration of display device 50 is indicated by a reference numeral in the 500 series. Display device 50 is constructed by a computer, and as shown in Fig. 4, includes a CPU 501, a ROM 502, a RAM 503, a HD 504, a HDD controller 505, a display 506, an external device connection I / F 507, a network I / F 508, a bus line 510, a keyboard 511, a pointing device 512, an audio input / output I / F 513, a microphone 514, a speaker 515, a camera 516, a DVD-RW (Digital Versatile Disk Rewritable) drive 517, and a media I / F 519.

[0036] Of these, the CPU 501 controls the overall operation of the display device 50. The ROM 502 stores programs used to drive the CPU 501, such as an IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, menus, windows, characters, or images. The display 506 may be a touch panel display equipped with input means. The display 506 is an example of a display unit. The display unit as the display 506 may be an external device equipped with a display function and connected to the display device 50. In this case, the display unit may be, for example, an external display such as an IWB (Interactive White Board), or a projection unit (for example, a ceiling or wall of a management center, a windshield of an automobile, etc.) onto which an image is projected from a PJ (Projector) or HUD (Head-Up Display) connected as an external device. The external device connection I / F 507 is an interface for connecting various external devices. The network I / F 508 is an interface for data communication using the communication network 100. The bus line 510 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 501 shown in FIG. 4.

[0037] The keyboard 511 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting or executing various instructions, selecting a processing target, moving a cursor, etc. The input means may be not only the keyboard 511 and the pointing device 512, but also a touch panel, an audio input device, etc. The audio input / output I / F 513 is a circuit that processes the input and output of audio signals between the microphone 514 and the speaker 515 under the control of the CPU 501. The microphone 514 is a type of built-in sound collection means that inputs audio. The speaker 515 is a type of built-in output means that outputs audio signals. The camera 516 is a type of built-in imaging means that captures an image of a subject and obtains image data. The microphone 514, the speaker 515, and the camera 516 may be external devices rather than built into the display device 50. The DVD-RW drive 517 controls the reading and writing of various data from and to a DVD-RW 518, which is an example of a removable recording medium. The removable recording medium is not limited to DVD-RW, but may be DVD-R or Blu-ray (registered trademark) Disc, etc. Media I / F 519 controls reading and writing (storing) of data from and to recording medium 521 such as flash memory.

[0038] Each of the above programs may be recorded on a computer-readable recording medium as an installable or executable file and distributed. Examples of the recording medium include a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), a Blu-ray Disc, an SD card, or a USB memory. The recording medium may be provided domestically or internationally as a program product. For example, the display device 50 realizes the display control method according to the present invention by executing the program according to the present invention.

[0039] ●Function configuration Next, the functional configuration of the communication system according to the embodiment will be described with reference to Fig. 5 to Fig. 8. Fig. 5 is a diagram showing an example of the functional configuration of the communication system. Fig. 5 shows devices or terminals shown in Fig. 1 that are related to the processing or operation described below.

[0040] ○ Functional configuration of mobile unit (control device) ○ First, referring to FIG. 5, the functional configuration of the control device 30 that controls the processing or operation of the moving object 10 will be described. The control device 30 includes a transmitter / receiver 31, a determination unit 32, a photography control unit 33, a status detection unit 34, a location information acquisition unit 35, a destination candidate acquisition unit 36, a route information generation unit 37, a route information management unit 38, a destination setting unit 39, a movement control unit 40, an image recognition processing unit 41, a mode setting unit 42, an autonomous movement processing unit 43, a manual operation processing unit 44, a task execution unit 45, a map information management unit 46, an image processing unit 47, a learning unit 48, and a storage / readout unit 49. Each of these units is a function or a means for performing the function, realized by one of the components shown in FIG. 3 operating in response to an instruction from the CPU 301 in accordance with a control device program loaded on the RAM 303. The control device 30 also includes a storage unit 3000 constructed using the ROM 302, HD 304a, or recording medium 305a shown in FIG. 3.

[0041] The transmitting / receiving unit 31 is mainly realized by the processing of the CPU 301 for the network I / F 308, and transmits and receives various data or information to and from other devices or terminals via the communication network 100.

[0042] The determination unit 32 is realized by processing of the CPU 301 and performs various determinations. The photography control unit 33 is mainly realized by processing of the CPU 301 for the external device connection I / F 311 and controls the photography process for the photography device 12. The photography control unit 33 instructs the photography process for the special photography device 12a and the general photography device 12b, for example. In addition, the photography control unit 33 acquires photographed images obtained by the photography process by the special photography device 12a and the general photography device 12b, for example.

[0043] The state detection unit 34 is mainly realized by processing of the CPU 301 on the external device connection I / F 311, and detects the state of the mobile object 10 or the surroundings of the mobile object 10 using various sensors. The state detection unit 34 measures the distance to an object (obstacle) present around the mobile object 10 using, for example, the obstacle detection sensor 106, and outputs the distance data. The state detection unit 34 may also be configured to acquire a position by matching the distance data measured using, for example, the obstacle detection sensor 106 with an environmental map using simultaneous localization and mapping (SLAM). SLAM is a technology that enables self-location estimation and environmental map creation simultaneously. Furthermore, the state detection unit 34 detects the direction in which the mobile object 10 is facing using, for example, the acceleration / direction sensor 103.

[0044] The location information acquisition unit 35 is realized by processing of the CPU 301 on the external device connection I / F 311, and acquires location information indicating the current location of the mobile object 10 using the GPS sensor 104. The location information acquisition unit 35 acquires coordinate information indicating the latitude and longitude of the current location of the mobile object 10 using, for example, the GPS sensor 104.

[0045] The destination candidate acquisition unit 36 ​​is mainly realized by processing of the CPU 301, and acquires destination candidate images that indicate candidate destinations to which the moving object 10 is to move. Specifically, the destination candidate acquisition unit 36 ​​acquires, as the destination candidate images, images that are captured by the imaging control unit 33 and show a partial area within the base where the moving object 10 is located.

[0046] The route information generation unit 37 is mainly realized by processing of the CPU 301, and generates route information that indicates a travel route of the mobile body 10. The route information generation unit 37 generates route information that indicates a route from the current position to the final destination based on the positions of candidate destinations selected by the operator of the mobile body 10. The route information can be generated, for example, by connecting each waypoint from the current position to the final destination with a straight line, or by using information about obstacles obtained from captured images or the status detection unit 34 to minimize travel time while avoiding obstacles.

[0047] The route information management unit 38 is mainly realized by the processing of the CPU 301 , and manages the route information generated by the route information generation unit 37 using the route information management DB 3002 .

[0048] The destination setting unit 39 is mainly realized by the processing of the CPU 301, and sets the destination of the moving body 10. The destination setting unit 39 sets the destination to which the moving body 10 is currently heading as the destination from among destination candidates selected by the operator of the moving body 10, based on, for example, the current position of the moving body 10 acquired by the position information acquisition unit 35 and the route information generated by the route information generation unit 37.

[0049] The movement control unit 40 is mainly realized by processing of the CPU 301 for the external device connection I / F 311, and controls the movement of the moving object 10 by driving the moving mechanism 15. The movement control unit 40 moves the moving object 10 in response to a drive command from, for example, the autonomous movement processing unit 43 or the manual operation processing unit 44.

[0050] The image recognition processing unit 41 is mainly realized by the processing of the CPU 301, and performs image recognition processing on the captured image acquired by the photography control unit 33. The image recognition processing unit 41 detects, for example, by image recognition, whether or not a specific subject is captured in the acquired captured image. The specific subject is, for example, an obstacle on or around the movement path of the mobile object 10, an intersection such as a crossroads or an L-shaped road, or a sign or traffic light installed at a base.

[0051] The mode setting unit 42 is mainly realized by the processing of the CPU 301, and sets the operation mode that indicates the operation of moving the moving body 10. The mode setting unit 42 sets either an autonomous movement mode in which the moving body 10 moves autonomously, or a manual operation mode in which the moving body 10 is moved by manual operation by the operator.

[0052] The autonomous movement processing unit 43 is mainly realized by the processing of the CPU 301, and controls the autonomous movement processing of the moving body 10. The autonomous movement processing unit 43 outputs a drive command for the moving body 10 to the movement control unit 40 so that the moving body 10 follows the movement route indicated in the route information generated by the route information generation unit 37, for example.

[0053] The manual operation processing unit 44 is mainly realized by the processing of the CPU 301, and controls the manual operation processing of the moving body 10. The manual operation processing unit 44 outputs a drive command for the moving body 10 to the movement control unit 40 in response to a manual operation command transmitted from the display device 50, for example.

[0054] The task execution unit 45 is mainly realized by the processing of the CPU 301, and executes a predetermined task by the mobile object 10 in response to a request from the operator. Here, the predetermined task executed by the task execution unit 45 includes, for example, a photographing process for inspecting equipment within a base, or light work using the movable arm 16.

[0055] The map information management unit 46 is mainly realized by processing of the CPU 301, and manages map information indicating an environmental map of a target base where the mobile object 10 is installed, using the map information management DB 3003. The map information management unit 46 manages, for example, map information indicating an environmental map downloaded from an external server or the like, or an environmental map created by applying SLAM.

[0056] The image processing unit 47 is mainly realized by the processing of the CPU 301, and generates a display image to be displayed on the display device 50. The image processing unit 47 generates a display image to be displayed on the display device 50, for example, by processing the captured image acquired by the imaging control unit 33.

[0057] The learning unit 48 is mainly realized by the processing of the CPU 301, and learns a movement route for autonomous movement of the moving object 10. The learning unit 48 performs imitation learning (machine learning) of a movement route for autonomous movement, for example, based on captured images acquired during movement operation in manual operation mode by the manual operation processing unit 44 and detection data by the state detection unit 34. The autonomous movement processing unit 43 performs autonomous movement of the moving object 10, for example, based on learning data that is the result of imitation learning by the learning unit 48.

[0058] The storage / readout unit 49 is mainly realized by the processing of the CPU 301 , and stores various data (or information) in the storage unit 3000 and reads out various data (or information) from the storage unit 3000 .

[0059] Destination candidate management table Fig. 6 is a conceptual diagram showing an example of a destination candidate management table. The destination candidate management table is a table for managing destination candidates that indicate potential destinations for the mobile object 10. A destination candidate management DB 3001 that is configured using the destination candidate management table shown in Fig. 6 is created in the storage unit 3000.

[0060] The destination candidate management table manages destination candidate data, which associates, for each base ID identifying a base where the mobile body 10 is installed, a candidate ID identifying the destination candidate, location information indicating the location of the destination candidate, and captured image data of a portion of the base that is the destination candidate. Among these, the location information is represented by coordinate information including latitude and longitude indicating the location of the destination candidate within the base. Note that, if the mobile body 10 is an aircraft such as a drone, the location information includes not only coordinate information indicating latitude and longitude, but also information such as the speed, attitude, or altitude of the aircraft. Furthermore, the captured image data indicates an image data file captured by the mobile body 10 at the location of the destination candidate within the base. Here, the destination candidate is an example of a candidate destination for the mobile body 10. The candidate destination for the mobile body 10 includes not only candidate destinations for the mobile body 10 but also, for example, candidate locations to be excluded from the travel route of the mobile body 10.

[0061] Route information management table Fig. 7 is a conceptual diagram showing an example of a route information management table. The route information management table is a table for managing route information indicating the movement route of the mobile object 10. A route information management DB 3002 configured by the route information management table shown in Fig. 7 is constructed in the storage unit 3000.

[0062] The route information management table manages, for each base ID that identifies the base where the mobile object 10 is installed, a route ID that identifies the movement route of the mobile object 10 and route information that indicates the movement route of the mobile object 10, in association with each other. Of these, the route information indicates the future movement route of the mobile object 10 in the order of future destinations. The route information is generated by the route information generation unit 37 when the mobile object 10 starts moving.

[0063] Map information management table Fig. 8 is a conceptual diagram showing an example of a map information management table. The map information management table is a table for managing map information, which is an environmental map of a target base where the mobile object 10 is installed. A map information management DB 3003 configured by the map information management table shown in Fig. 8 is constructed in the storage unit 3000.

[0064] The map information management table manages map information in which a base ID and base name for identifying a target base where the mobile object 10 is installed are associated with a storage location of an environmental map of the target base. The storage location is, for example, destination information for accessing a storage area in the mobile object 10 or an external server where the environmental map is stored, and is indicated by a URL (Uniform Resource Locator) or a URI (Uniform Resource Identifier).

[0065] Learning data management database Here, the learning data management DB 3004 constructed in the storage unit 3000 will be described. The learning data management DB 3004 manages learning data, which is the result of learning of the autonomous movement path of the moving object 10 by the learning unit 48. The learning data management DB 3004 accumulates, for example, photographed image data and various detection data acquired from the moving object 10, and stores learning data, which is the result of imitation learning (machine learning) for each base or each moving object 10. The moving object 10 moves autonomously based on the learning data stored in the learning data management DB 3004. Note that the photographed image data captured by the special imaging device 12a or the general imaging device 12b may include PTZ (Pan-Tilt-Zoom) parameters for specifying the imaging direction, etc., of the special imaging device 12a or the general imaging device 12b. The photographed image data including the PTZ parameters is stored (saved) in the storage unit 3000 (learning data management DB 3004) of the moving object 10. Furthermore, the PTZ parameters may be linked to the position information of the destination candidate and stored in the storage unit 3000, and coordinate data (x, y, θ) indicating the attitude of the moving object 10 when the captured image data of the destination candidate was acquired may be simultaneously stored in the storage unit 3000. This makes it possible to correct the attitude of the moving object 10 using the PTZ parameters and the coordinate data (x, y, θ) when a deviation occurs in the actual stopping position of the moving object 10 relative to the destination. Note that some data, such as data on the autonomous movement path (GPS trajectory) of the moving object 10 and the captured image data of the destination candidate used for display on the display device 50, may be stored in a cloud computing service such as AWS (Amazon Web Services) (trademark).

[0066] ○Functional configuration of the display device○ Next, the functional configuration of the display device 50 will be described with reference to Fig. 5. The display device 50 has a transmitting / receiving unit 51, a receiving unit 52, a display control unit 53, a determining unit 54, a manual operation command generating unit 55, an autonomous movement request information generating unit 56, an image processing unit 57, and a storing / reading unit 59. Each of these units is a function or a means for performing the function, which is realized when any of the components shown in Fig. 4 operates in response to an instruction from the CPU 501 in accordance with a display device program loaded on the RAM 503. The display device 50 also has a storage unit 5000 constructed using the ROM 502, HD 504, or recording medium 521 shown in Fig. 4.

[0067] The transmitting / receiving unit 51 is mainly realized by the processing of the CPU 501 for the network I / F 508, and transmits and receives various data or information to and from other devices or terminals.

[0068] The reception unit 52 is mainly realized by the processing of the CPU 501 using the keyboard 511 or the pointing device 512, and receives various selections or inputs from the user. The display control unit 53 is mainly realized by the processing of the CPU 501, and displays various screens on a display unit such as the display 506. The determination unit 54 is mainly realized by the processing of the CPU 501, and makes various determinations.

[0069] The manual operation command generation unit 55 is mainly realized by the processing of the CPU 501, and generates a manual operation command for manually moving the moving body 10 in response to an input operation by the operator. The autonomous movement request information generation unit 56 is mainly realized by the processing of the CPU 501, and generates autonomous movement request information for autonomously moving the moving body 10. The autonomous movement request information generation unit 56 transmits autonomous movement request information to be sent to the moving body 10 based on, for example, information on a destination candidate selected by the operator.

[0070] The image processing unit 57 is mainly realized by processing of the CPU 501, and generates a display image to be displayed on a display unit such as the display 506. The image processing unit 57 processes, for example, a captured image acquired by the moving object 10, to generate a display image to be displayed on the display device 50. Note that the communication system 1 may be provided with the functions of either the image processing unit 47 of the moving object 10 or the image processing unit 57 of the display device 50.

[0071] The storage / readout unit 59 is mainly realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 5000 and reads out various data (or information) from the storage unit 5000 .

[0072] Processing or operation of the embodiment ○Destination candidate registration process○ Next, the processing or operation of the communication system according to the embodiment will be described with reference to Fig. 9 to Fig. 23. First, the processing of registering destination candidates for the moving object 10 will be described with reference to Fig. 9 to Fig. 12. Fig. 9 is a sequence diagram showing an example of the processing of registering destination candidates.

[0073] First, the display device 50 starts operating the moving object 10 in response to a predetermined input operation by the operator (step S11). Then, the transmitting / receiving unit 51 transmits an operation start request to the moving object 10 (step S12). As a result, the transmitting / receiving unit 31 of the control device 30 included in the moving object 10 receives the operation start request transmitted from the display device 50.

[0074] Next, the photography control unit 33 of the control device 30 starts photography processing using the special photography device 12a and the general photography device 12b (step S13). The photography control unit 33 acquires photographed image data photographed by the special photography device 12a and the general photography device 12b. Here, the images acquired by the photography processing in step S13 are moving images. Then, the transmission / reception unit 31 transmits the photographed image data acquired in step S13 to the display device 50 (step S14). As a result, the transmission / reception unit 51 of the display device 50 receives the photographed image data transmitted from the moving object 10.

[0075] Next, the display control unit 53 of the display device 50 displays an operation screen 200 including the captured image data received in step S14 on a display unit such as the display 506 (step S15). Fig. 10 is a diagram showing an example of the operation screen. The operation screen 200 shown in Fig. 10 is a display screen used by the operator to remotely control the moving object 10.

[0076] The operation screen 200 includes a base display area 210 displaying a photographed image (planar image) captured by the general camera 12b, which is the photographed image data received in step S14, and a base display area 230 displaying a photographed image (special image) captured by the special camera 12a, which is the photographed image data received in step S14. The special image displayed in the base display area 230 is an image captured in all directions within the base, such as a celestial sphere image, a wide-angle image, or a hemispherical image, captured by the special camera 12a, as described above. The special image may also be an image obtained by the special camera 12a, which generates an image capturing all directions by combining images captured while rotating the general camera. The base display area 230 also displays a moving object display image 235, which indicates the presence of the moving object 10 on the special image, superimposed on the special image. The operation screen 200 also displays coordinate information represented by latitude (Lat) and longitude (Lon) as location information indicating the current location of the moving object 10. If the moving object 10 is an air vehicle such as a drone, the operation screen 200 may be configured to display information such as the speed, attitude, or altitude of the air vehicle in addition to the coordinate information of latitude and longitude. Also, the operation screen 200 may display images captured by the special imaging device 12a and the general imaging device 12b as live streaming video that is distributed in real time via a computer network such as the Internet.

[0077] The operation screen 200 also includes operation icons 250 for remotely controlling the moving object 10. The operation icons 250 include direction buttons that are pressed when requesting movement of the moving object 10 in a horizontal direction (forward, backward, right turn, left turn). The operator can remotely control the moving object 10 by selecting the operation icons 250 while viewing the planar image displayed in the base display area 210 and the special image such as a celestial sphere image displayed in the base display area 230. Note that, although an example will be described in which the movement of the moving object 10 is remotely controlled by accepting a selection of the operation icons 250 displayed on the operation screen 200, the movement of the moving object 10 may also be controlled using a dedicated controller such as a keyboard or a game pad equipped with a joystick.

[0078] Furthermore, the operation screen 200 includes a destination candidate registration button 270 that is pressed when registering a destination candidate using the planar image displayed in the base display area 210 or the special image displayed in the base display area 230. In the following explanation, an example will be described in which the operator presses the destination candidate registration button 270 to register a destination candidate using the special image displayed in the base display area 230. Note that the same processing is also performed when the operator presses the destination candidate registration button 270 to register a destination using the planar image displayed in the base display area 210. Furthermore, the operation screen 200 includes a destination setting button 290 that is pressed when setting a destination for the mobile object 10.

[0079] Next, the display device 50 uses the operation screen 200 displayed in step S15 to manually operate the moving object 10 (step S16). Details of the processing in step S16 will be described later.

[0080] Next, when the operator presses the destination candidate registration button 270 on the operation screen 200, the reception unit 52 receives the destination candidate registration for the area appearing in the special image displayed in the base display area 230 (step S17). Then, the transmission / reception unit 51 transmits a destination candidate registration request to the mobile object 10 (step S18). This destination registration request is, for example, a request to photograph the area appearing in the special image displayed in the base display area 230. As a result, the control device 30 included in the mobile object 10 receives the destination candidate registration request transmitted from the display device 50.

[0081] Next, the position information acquisition unit 35 of the control device 30 acquires position information indicating the current position of the mobile object 10 using the GPS sensor 306 or the like (step S19). Specifically, the position information acquisition unit 35 acquires coordinate information including latitude and longitude indicating the current position of the mobile object 10. Note that if the mobile object 10 is an air vehicle such as a drone, the position information acquisition unit 35 acquires information such as the speed, attitude, or altitude of the air vehicle as position information in addition to the coordinate information including latitude and longitude. Furthermore, the destination candidate acquisition unit 36 ​​acquires, as destination candidate images indicating destination candidates, images captured by the special imaging device 12a at the current position of the mobile object 10 (step S20). Here, the captured images acquired by the destination candidate acquisition unit 36 ​​as destination candidate images are still images.

[0082] Then, the storage / readout unit 49 stores the destination candidate data, including the location information acquired in step S19 and the photographed images acquired in step S20, in the destination candidate management DB 3001 (see FIG. 6) (step S21). The storage / readout unit 49 registers the acquired location information and photographed images as destination candidate data, each associated with a candidate ID that identifies the destination candidate.

[0083] In this way, the communication system 1 displays the images captured by the moving body 10 on the display device 50 of the operator remotely operating the moving body 10, thereby enabling the operator to remotely control the moving body 10 while checking in real time the situation around the moving body 10. Furthermore, the communication system 1 can pre-register, in accordance with input operations by the operator when manually operating the moving body 10, captured images of areas within a base that are potential destinations when the moving body 10 is moved autonomously.

[0084] ○Manual operation processing Next, the manual operation process for the moving object 10 in step S16 will be described in detail with reference to Figures 11 and 12. Figure 11 is a sequence diagram showing an example of the manual operation process for the moving object.

[0085] First, the reception unit 52 of the display device 50 receives a selection of the operation icon 250 on the operation screen 200 displayed in step S15 through an input operation by the operator (step S31). The manual operation command generation unit 55 generates a manual operation command according to the content of the operation icon 250 selected in step S31 (step S32). In the example of FIG. 10, the operator has selected the "forward (↑)" icon, so in this case the manual operation command generation unit 55 generates a command for moving the moving object 10 forward. Then, the transmission / reception unit 51 transmits the manual operation command generated in step S32 to the moving object 10 (step S33). As a result, the control device 30 provided in the moving object 10 receives the manual operation command transmitted from the display device 50. Note that, in the input operation of the operation icon 250 by the operator, if the operator selects the "reverse (↓)" icon while the moving object 10 is moving forward, the captured image may be switched to a screen behind the moving object 10, and the moving object 10 may be caused to reverse (go backward) from that point on. Furthermore, the transmission of the manual operation command from the display device 50 to the mobile object 10 may be configured to be performed via a managed cloud platform such as AWS IoT Core (trademark), for example.

[0086] Next, the mode setting unit 42 of the control device 30 sets the operation mode of the moving object 10 to the manual operation mode (step S34). Then, the manual operation processing unit 44 outputs a drive command based on the manual operation command received in step S33 to the movement control unit 40. The movement control unit 40 executes movement processing of the moving object 10 in accordance with the drive command from the manual operation processing unit 44 (step S35). Furthermore, the learning unit 48 performs imitation learning (machine learning) of a movement path in accordance with the manual operation by the manual operation processing unit 44 (step S36). The learning unit 48 performs imitation learning of the movement path based on, for example, captured images acquired during movement operation in the manual operation mode by the manual operation processing unit 44 and detection data by the state detection unit 34. Note that the learning unit 48 may be configured to perform imitation learning of the movement path using only captured images acquired during manual operation, or may be configured to perform imitation learning of the movement path using both the captured images and detection data by the state detection unit 34. Furthermore, the captured images used for imitation learning by the learning unit 48 may be captured images acquired during autonomous movement by the autonomous movement processing unit 43 in the autonomous movement mode.

[0087] Next, the mode setting unit 42 of the control device 30 sets the operation mode of the moving object 10 to the manual operation mode (step S34). Then, the manual operation processing unit 44 outputs a drive command based on the manual operation command received in step S33 to the movement control unit 40. The movement control unit 40 executes movement processing of the moving object 10 in accordance with the drive command from the manual operation processing unit 44 (step S35). Furthermore, the learning unit 48 performs imitation learning (machine learning) of a movement path in accordance with the manual operation by the manual operation processing unit 44 (step S36). The learning unit 48 performs imitation learning of the movement path, for example, based on captured images acquired during movement operation in the manual operation mode by the manual operation processing unit 44 and detection data by the state detection unit 34.

[0088] Then, the moving body 10 executes a process of registering destination candidates while moving (step S37). Fig. 12 is a flowchart showing an example of a process of registering destination candidates according to the moving state of the moving body.

[0089] First, the determination unit 32 of the control device 30 determines whether or not a predetermined task has been executed by the task execution unit 45 (step S51). Specifically, the task execution unit 45 executes a predetermined task by the mobile object 10, for example, in response to a task execution request from an operator or a preset schedule. Then, the determination unit 32 determines whether or not the predetermined task can be executed by the task execution unit 45. Here, the predetermined task includes, for example, a photographing process for inspecting equipment within a base, light work using the movable arm 16, etc.

[0090] If the determination unit 32 determines that the task execution unit 45 has executed the predetermined task (YES in step S51), the process proceeds to step S56. On the other hand, if the determination unit 32 determines that the task execution unit 45 has not executed the predetermined task (NO in step S51), the process proceeds to step S52.

[0091] Next, the determination unit 32 determines whether the movement of the moving body 10 has stopped (step S52). Specifically, when the determination unit 32 detects that the drive control of the movement mechanism 15 by the movement control unit 40 has stopped, the determination unit 32 determines that the movement of the moving body 10 has stopped. When the determination unit 32 determines that the movement of the moving body 10 has stopped (YES in step S52), the determination unit 32 shifts the process to step S56. On the other hand, when the determination unit 32 determines that the movement of the moving body 10 has not stopped (moving is in progress) (NO in step S52), the determination unit 32 shifts the process to step S53.

[0092] Next, the determination unit 32 determines whether an intersection has been detected around the moving object 10 (step S53). Specifically, the image recognition processing unit 41 performs image recognition processing on the captured image acquired by the special imaging device 12a or the general imaging device 12b. Then, the determination unit 32 determines whether an intersection has been detected in the captured image through processing by the image recognition processing unit 41. Here, an intersection is an example of a specific object detected by the image recognition processing unit 41. The specific object may be an intersection, an obstacle on or around the moving path of the moving object 10, or a sign or signal installed at a base. Furthermore, the specific object differs depending on the type of moving object 10, such as when the moving object 10 is traveling on a road surface or when the moving object 10 is flying, such as a drone, and information about the specific object to be detected is set in advance in the moving object 10.

[0093] If the determination unit 32 determines that an intersection has been detected around the moving object 10 (YES in step S53), the process proceeds to step S56. On the other hand, if the determination unit 32 determines that an intersection has not been detected around the moving object 10 (NO in step S52), the process proceeds to step S54.

[0094] Next, the determination unit 32 determines whether the current location of the mobile object 10 is close to a destination candidate registered in the destination candidate management DB 3001 (step S54). Specifically, the determination unit 32 references the location information of the destination candidate stored in the destination candidate management DB 3001 and determines whether there is a destination series that is close to the current location of the mobile object 10 acquired by the location information acquisition unit 35. Here, the determination of whether the destination candidate is close to the destination candidate is made, for example, based on the difference in coordinate values ​​between the location of a preset destination candidate and the current location of the mobile object 10. Note that the determination of whether the destination candidate is close to the destination candidate may also be made using image recognition between a captured image corresponding to the destination candidate (a destination candidate image) and an image captured at the current location of the mobile object 10.

[0095] If the determination unit 32 determines that the current location of the moving object 10 is close to a destination candidate registered in the destination candidate management DB 3001 (YES in step S54), the process proceeds to step S56. On the other hand, if the determination unit 32 determines that the current location of the moving object 10 is far from a destination candidate registered in the destination candidate management DB 3001 (NO in step S54), the process proceeds to step S55.

[0096] Next, the determination unit 32 determines whether or not the direction indicated in the manual operation command transmitted from the display device 50 has changed (step S55). Specifically, the determination unit 32 determines whether or not the movement direction indicated in the manual operation command has changed, based on the manual operation command received in step S33.

[0097] If the determination unit 32 determines that the operation direction indicated in the manual operation command transmitted from the display device 50 has been changed (YES in step S55), the process proceeds to step S56. On the other hand, if the determination unit 32 determines that the operation direction indicated in the manual operation command transmitted from the display device 50 has not been changed (NO in step S55), the process ends.

[0098] Then, in step S56, the location information acquisition unit 35 of the control device 30 acquires location information indicating the current location of the moving object 10 using the GPS sensor 306 or the like. In addition, the destination candidate acquisition unit 36 ​​acquires images captured by the special imaging device 12a at the current location of the moving object 10 as destination candidate images indicating destination candidates for the moving object 10 (step S57).

[0099] Then, the storage / readout unit 49 stores the destination candidate data, including the location information acquired in step S19 and the photographed images acquired in step S20, in the destination candidate management DB 3001 (see FIG. 6) (step S58). The storage / readout unit 49 registers the acquired location information and photographed images as destination candidate data, each associated with a candidate ID that identifies the destination candidate.

[0100] In this way, the moving body 10 can automatically capture images of the surroundings of the moving body 10 using the imaging device 12 mounted on the moving body 10 based on predetermined criteria corresponding to the moving state of the moving body 10, and can pre-register captured images indicating candidate destinations for the moving body 10. Note that the criteria by the determination unit 32 shown in steps S51 to S55 are not limited to these and can be set appropriately depending on the details of manual operation of the moving body 10 and the environment surrounding the moving body 10. The criteria by the determination unit 32 include, for example, changes in the state of the moving body 10 that the operator wants to be aware of when selecting a destination for the moving body 10, or changes in the status of a base or the environment within the base. Furthermore, while FIG. 12 has been described as processing performed when the moving body 10 is manually operated, the destination candidate registration processing shown in FIG. 12 may also be performed when the moving body 10 is autonomously moving. Furthermore, the process of registering destination candidates shown in Figures 9 to 12 is an example of a process of registering candidate destinations for the mobile body 10, and the process may be performed not only for the purpose of registering candidate destinations for the mobile body 10, but also for the purpose of registering candidate locations to be excluded from the travel route of the mobile body 10, for example.

[0101] ○Destination setting process○ Next, a process for setting a destination of the moving body 10 using the destination candidates registered by the above-described process will be described with reference to Figures 13 to 18. Figure 13 is a sequence diagram showing an example of a process for setting a destination of a moving body.

[0102] First, the reception unit 52 of the display device 50 receives selection of the destination setting button 290 through an input operation by the operator on the operation screen 200 (step S71). Next, the transmission / reception unit 51 transmits a destination candidate acquisition request to the mobile object 10, requesting acquisition of data indicating destination candidates (step S72). This destination candidate acquisition request includes a base ID that identifies the base where the mobile object 10 is installed. As a result, the transmission / reception unit 31 of the control device 30 included in the mobile object 10 transmits the destination candidate acquisition request transmitted from the display device 50.

[0103] Next, the storage / readout unit 49 of the control device 30 searches the destination candidate management DB 3001 (see FIG. 6) using the base ID indicated in the destination acquisition request received in step S72 as a search key, and reads out destination candidate data associated with the same base ID as the received base ID (step S73). The transmission / reception unit 31 then transmits the destination candidate data read out in step S73 to the requesting display device 50 (step S74). This destination candidate data includes sets of the multiple candidate IDs, captured images, and location information read out in step S73. As a result, the transmission / reception unit 51 of the display device 50 receives and acquires the destination candidate data transmitted from the mobile object 10.

[0104] Next, the display control unit 53 of the display device 50 causes a display unit such as the display 506 to display a selection screen 400 including the destination candidate data received in step S74 (step S75). Specifically, the image processing unit 57 generates the selection screen 400 based on the received destination candidate data. Then, the display control unit 53 causes the selection screen 400 generated by the image processing unit 57 to be displayed. Note that the process of generating the selection screen 400 may be performed by the image processing unit 47 of the control device 30. In this case, the image processing unit 47 generates the selection screen 400 based on the destination candidate data read out in step S73. Then, in step S74, the transmission / reception unit 31 transmits screen data of the selection screen 400 including the destination candidate data generated by the image processing unit 47 to the display device 50.

[0105] 14 and 15 are diagrams showing an example of a selection screen. The selection screen 400 shown in Fig. 14 and 15 is a display screen for the operator to select a destination to which the moving object 10 will move.

[0106] The selection screen 400 includes an image display area 410 in which a plurality of photographed images (destination candidate images) 415 (415a to 417f) included in the destination candidate data received in step S74 are displayed in a selectable manner. The selection screen 400 also includes an OK button 430 to be pressed when the selection is complete, and a cancel button 435 to be pressed to cancel the selection process. The operator uses input means such as a pointing device 512 to select one of the photographed images 415 displayed in the image display area 410.

[0107] When the operator selects a photographed image 415 displayed in the image display area 410, the reception unit 52 of the display device 50 receives a selection of a destination candidate image for setting the area indicated by the photographed image as a destination (step S76). The selection screen 400 shown in FIG. 15 shows a state in which, for example, photographed images 415a, 415b, and 415c have been selected by the operator. When the photographed image 415 displayed in the image display area 410 is selected, the selection screen 400 displays the selected photographed image in a selection result display area 420 as shown in FIG. 15. The selection result display area 420 displays the selected photographed images in the order (No. 1 to No. 3) selected by the operator.

[0108] In this way, the display device 50 uses a captured image 415 (candidate destination image) that shows candidate destinations for the moving body 10, which has been captured in advance by the moving body 10, to select the area shown in the captured image 415 as the destination for the moving body 10, thereby improving operability in selecting the destination for the moving body 10 compared to conventional methods that require the operator to specify location information or a string of characters such as a keyword.

[0109] Next, in step S76, when the selection of the destination candidate image is accepted and the operator presses the OK button 430, the autonomous movement request information generation unit 56 of the display device 50 generates autonomous movement request information (step S77). This autonomous movement request information includes the candidate ID associated with the photographed image 415 whose selection was accepted in step S76, and information indicating the order in which the selection was accepted in step S76.

[0110] Next, the transmitting / receiving unit 51 transmits the autonomous movement request information generated in step S77 to the moving object 10 (step S78). As a result, the transmitting / receiving unit 31 of the control device 30 included in the moving object 10 receives the autonomous movement request information transmitted from the display device 50. Then, the moving object 10 starts autonomous movement processing based on the autonomous movement request information received in step S78 (step S79).

[0111] ○Autonomous movement processing Next, the autonomous movement processing of the moving body 10 in step S79 will be described in detail with reference to Figures 16 to 18. Figure 16 is a flowchart showing an example of the autonomous movement processing of the moving body.

[0112] First, when the autonomous movement request information is received in step S78, the mode setting unit 42 of the control device 30 included in the moving object 10 sets the operation mode of the moving object 10 to the autonomous movement mode (step S91). Next, the route information generation unit 37 generates route information indicating an autonomous movement route of the moving object 10 based on the autonomous movement request information received in step S78 (step S92). Specifically, the route information generation unit 37 generates a movement route that causes the moving object 10 to autonomously move to an area indicated by the photographed images 415 in accordance with the order in which the operator selected the photographed images 415, based on the candidate IDs indicated in the received autonomous movement request information and the order in which the photographed images 415 identified by the candidate IDs were selected. Note that the route information generation unit 37 is not limited to the order in which the photographed images 415 were selected, and may be configured to generate route information based on information on the shooting positions of the photographed images 415 stored in the destination candidate management DB 3001, for example, so as to shorten the distance or travel time of the movement route based on the selected photographed images 415. Then, the route information management unit 38 stores the route information generated in step S92 in the route information management DB 3002 (see FIG. 7) in association with the base ID received in step S72 (step S93).

[0113] Next, the position information acquisition unit 35 acquires position information indicating the current position of the moving object 10 (step S94). The destination setting unit 39 sets the destination of the moving object 10 based on the current position of the moving object 10 acquired by the position information acquisition unit 35 and the route information generated in step S92 (step S95). Specifically, the destination setting unit 39 sets, as the destination, the position indicated by the next destination to be traveled from among multiple destinations identified by candidate IDs indicated in the generated route information. For example, when autonomous movement is about to start, the destination setting unit 39 sets, as the destination, the position indicated by the destination identified by the first candidate ID indicated in the route information. Then, the destination setting unit 39 generates a travel route from the acquired current position of the moving object 10 to the set destination. The travel route generated by the destination setting unit 39 may be a method of connecting the current position and the destination with a straight line, or a method of minimizing travel time while avoiding obstacles using information about obstacles obtained from captured images or the state detection unit 34. The generation of the travel route by the destination setting unit 39 is not limited to a configuration that uses the location information of the destination candidates (destinations) registered in the destination candidate management DB 3001, but may also be a configuration that identifies the position appearing in the captured image 415 set as the destination through image recognition processing by the image processing unit 47 using the captured image taken by the photographing device 12.

[0114] Then, the movement control unit 40 moves the moving object 10 to the destination that is set to pass through the movement route generated in step S92. In this case, the movement control unit 40 moves the moving object 10 autonomously in response to a drive command from the autonomous movement processing unit 43. The autonomous movement processing unit 43 executes autonomous movement based on learning data that is a result of imitation learning by the learning unit 48, for example (step S96).

[0115] Then, when the moving object 10 reaches the final destination (YES in step S97), the movement control unit 40 ends the process. On the other hand, the movement control unit 40 continues the autonomous movement process by repeating the process from step S94 until the moving object 10 reaches the final destination (NO in step S97).

[0116] In this way, the moving body 10 can move autonomously with the position shown in the captured image 415 selected by the operator as the destination of the moving body 10. Furthermore, when operating in the autonomous movement mode, the moving body 10 can move autonomously using the generated route information and learning data learned in the manual operation mode.

[0117] Here, an example of an operation screen displayed on the display device 50 while the moving object 10 is autonomously moving will be described with reference to FIGS. 17 and 18. In the following description, differences from the configuration of the operation screen 200 shown in FIG. 10 will be described. The operation screen 600A shown in FIG. 17 includes a status display area 610A showing the movement status of the moving object 10 and a base display area 630A showing images captured by the moving object 10. The status display area 610A displays a destination candidate image showing the destination set in step S95 and the current position of the moving object 10, indicating that the moving object 10 is heading to the location shown in the destination candidate image. Furthermore, the base display area 630A displays a special image acquired by the special imaging device 12a, similar to the base display area 230 of the operation screen 200 shown in FIG. 10. Furthermore, the base display area 630A displays a moving object display image 635A showing the presence of the moving object 10 on the special image, superimposed on the special image, similar to the base display area 230 of the operation screen 200. Furthermore, in the base display area 630A, a destination display image 640 indicating the set destination is superimposed and displayed on the special image at a position corresponding to the destination set in step S95.

[0118] The operation screen 600A also includes a stop button 690 that is pressed when suspending the autonomous movement of the moving body 10. When the operator presses the stop button 690, the moving body 10 suspends the autonomous movement processing by the autonomous movement processing unit 43 in response to a stop request from the display device 50, and stops the moving body 10.

[0119] 18 is a display screen displayed when the moving object 10 is autonomously moving toward multiple destinations. The operation screen 600B shown in FIG. 18 includes a status display area 610B that shows the moving status of the moving object 10 and a base display area 630B that displays images captured by the moving object 10. The status display area 610B displays destination candidate images corresponding to multiple destination series included in the route information generated in step S92 and the current position of the moving object 10, thereby indicating which destination the moving object 10 is currently heading toward. Similarly to the base display area 630A, the base display area 630B displays a special image acquired by the special imaging device 12a, and a moving object display image 635B that indicates the presence of the moving object 10 on the special image is superimposed on the special image. Similarly to the operation screen 600A, the operation screen 600B also includes a stop button 690 that can be pressed to interrupt the autonomous movement of the moving object 10.

[0120] Additionally, in the base display area 630B, destination display images 650 (650a, 650b) indicating each destination series are superimposed and displayed at positions on the special image corresponding to the multiple destination series included in the route information generated in step S92. In the example of Fig. 18, destination display images 650a, 650b indicating destination series No. 2 and No. 3 are displayed, respectively.

[0121] In this way, the display device 50 displays in real time the status of the autonomous movement of the moving body 10, whose destination is the location shown in the captured image 415 selected by the operator, thereby allowing the operator to check the current location and movement status of the moving body 10 even while the moving body 10 is moving autonomously.

[0122] ○Selection screen changes○ Next, modified examples of the selection screen 400 displayed on the display device 50 will be described with reference to Figures 19 to 23. Figure 19 is a diagram showing Modified Example 1 of the selection screen. The selection screen 400A shown in Figure 19 differs from the selection screen 400 shown in Figures 14 and 15 in the configuration of the image display area 410A.

[0123] Image display area 410A displays one special image 450 from the captured image data included in the destination candidate data received in step S74. The operator searches for a location to select as a destination by changing the display position in image display area 410A of a special image in which part of a base is captured in all directions. The display position of special image 450 displayed in image display area 410A can be changed by swiping on the screen or by accepting an input operation on display position change buttons 451 (451a to 451d).

[0124] 15, the operator selects a destination by specifying a predetermined position on the special image 450 displayed in the image display area 410A using input means such as the pointing device 512. The reception unit 52 of the display device 50 receives the selection of the specified position on the special image 450 when the operator specifies a predetermined position on the special image 450 displayed in the image display area 410A. The image display area 410A displays a specified position image 455 indicating the position on the special image 450 specified by the operator, superimposed on the special image 450. The selection screen 400A also displays, in the selection result display area 420, destination candidate images corresponding to the position on the special image 450 specified by the operator, similar to the selection screen 400 shown in FIG.

[0125] Furthermore, selection screen 400A includes file import button 457, which is pressed when selecting or changing special image 450 to be displayed in image display area 410A. When the operator selects file import button 457 and specifies a predetermined image file, display control unit 53 displays a different special image from the captured image data received in step 74 in image display area 410A.

[0126] In this way, the selection screen 400A allows the operator to select the destination of the moving body 10 using the special image 450 captured by the moving body 10, and therefore allows the operator to select a predetermined position within the entire surrounding area captured in one special image 450 as a candidate destination. Furthermore, the display device 50 allows the operator to select a predetermined position within the area captured in the special image 450 as a candidate destination of the moving body 10, thereby improving operability when selecting the destination of the moving body 10.

[0127] Fig. 20 is a diagram showing a modified example 2 of the selection screen. In addition to the configuration of the selection screen 400, the selection screen 400B shown in Fig. 20 includes a map image 470 indicating the location where the destination candidate image selected by the operator was photographed.

[0128] The selection screen 400B includes a map image 470 and a map display button 460 that is pressed to display the map image 470. The map image 470 includes a current position display image 471 that indicates the current position of the moving object 10, and a display selection button 475 that is pressed to enlarge or reduce the displayed map. The map image 470 also includes shooting position display images 473 (473a to 473c) that indicate on the map the shooting positions of the photographed images 415 selected by the operator. In the example of FIG. 20, the shooting position display images 473a to 473c are displayed at positions on the map that correspond to the shooting positions of the photographed images No. 1 to No. 3, respectively.

[0129] When the reception unit 52 receives the selection of the map display button 460, the display device 50 transmits a map display request to the mobile object 10. The mobile object 10 then reads out map image data corresponding to the base where the mobile object 10 is located, which is stored in the map information management DB 3003, and transmits it to the requesting display device 50. The display control unit 53 of the display device 50 then displays a current position display image 471 and a shooting position display image 473, each superimposed on the received map image data.

[0130] In this way, when the operator selects the destination of the moving body 10, the selection screen 400B displays the location on the map corresponding to the area indicated by the selected photographed image, allowing the operator to visually understand where within the base the area indicated by the selected photographed image is located, thereby improving operability in selecting the destination of the moving body 10.

[0131] Fig. 21 is a diagram showing selection screen variation 3. Selection screen 400C shown in Fig. 21 differs from selection screen 400A shown in Fig. 19 in the configuration of selection result display area 480 in selection screen 400 shown in Fig. 15.

[0132] The selection result display area 480 includes information on the time required to move the moving object 10 to the position corresponding to the photographed image 415 selected by the operator. In the example of Fig. 21, the time required to reach the area indicated by the No. 1 photographed image from the current position of the moving object 10 is 1 minute 5 seconds, the time required to reach the area indicated by the No. 1 photographed image from the area indicated by the No. 2 photographed image is 1 minute 40 seconds, and the time required to reach the area indicated by the No. 2 photographed image from the area indicated by the No. 3 photographed image is 1 minute. The selection result display area 480 also displays the total time required to move to the areas indicated by all of the photographed images 415 selected by the operator (3 minutes 45 seconds in the example of Fig. 21).

[0133] Either the image processing unit 47 of the moving object 10 or the image processing unit 57 of the display device 50 estimates the time required to move between two points corresponding to the current position of the moving object 10 or the positions of the areas indicated by the multiple captured images 415 selected by the operator, based on information on the preset moving speed of the moving object 10 and the relative distance at the base. Then, either the image processing unit 47 or the image processing unit 57 generates a selection screen 400C including information on the estimated required time, and the display control unit 53 displays the generated selection screen 400C on a display unit such as the display 506.

[0134] In addition, when the moving body 10 performs a specified task in the area indicated by the selected captured image 415, the selection screen 400C may be configured to display not only the travel time but also the required time including the time required to perform the task.

[0135] In this way, the selection screen 400C displays the travel time required for the moving body 10 to move to the area indicated by the captured image 415 selected by the operator, thereby allowing the operator to appropriately determine whether or not to select a destination based on the travel time, and the order in which to select multiple destinations if multiple destinations are selected.

[0136] 22 is a diagram showing a fourth variation of the selection screen. Selection screen 400D shown in FIG. 22 differs from selection screen 400 shown in FIG.

[0137] The image display area 410 of the selection screen 400D displays, for each of the displayed photographed images 415 (415a to 415f), additional information 490 (490a, 490b, 490c, 490e) indicating the characteristics of the subject appearing in the photographed image 415 in association with the image. The additional information 490 includes a character string indicating the name of the structure or equipment that is the subject appearing in the photographed image 415, or the condition of an intersection, a slope, or a road surface. In the example of FIG. 22, additional information 490a indicating "Meter A5" is displayed in association with the photographed image 415a, and additional information 490b indicating "Tank B9" is displayed in association with the photographed image 415b. Furthermore, additional information 490c indicating "Building C" is displayed in association with the photographed image 415c, and additional information 490e indicating "Building Y" is displayed in association with the photographed image 415e.

[0138] Either the image processing unit 47 of the moving body 10 or the image processing unit 57 of the display device 50 performs image recognition processing on the photographed image 415 based on information about a preset base and an image of a subject present at the base, thereby identifying information corresponding to the photographed position of the photographed image 415. Then, either the image processing unit 47 or the image processing unit 57 generates a selection screen 400E in which the identified information is associated with the corresponding photographed image 415 as additional information, and the display control unit 53 displays the generated selection screen 400E on a display unit such as the display 506.

[0139] In this way, the selection screen 400D displays, as additional information 490, the name and other characteristics of the subject appearing in the photographed image 415 together with the photographed image 415 (candidate destination image) indicating candidate destinations for the moving object 10, thereby enabling the operator to make an appropriate decision compared to selecting the destination of the moving object 10 by looking only at the photographed image 415. In this case, in the process of setting the destination of the moving object 10 in step S95, the destination setting unit 39 is not limited to a configuration in which it generates a travel route using position information of the candidate destinations (destinations) registered in the candidate destination management DB 3001, but may also be a configuration in which it generates a travel route by identifying a position appearing in the photographed image 415 from information about the subject appearing in the photographed image 415 identified by image recognition processing.

[0140] Fig. 23 is a diagram showing modified example 5 of the selection screen. The selection screen 400E shown in Fig. 23 is an example of a screen that is displayed when the captured images 415 (415a to 417f) displayed in the image display area 410 can be used for purposes other than selecting the destination of the moving body 10 as information related to setting a travel route for the autonomous movement of the moving body.

[0141] The selection screen 400E includes a destination selection button 440 that is pressed when selecting a destination for the moving object 10 using the captured image 415 displayed in the image display area 410 as described above, and an exclusion route selection button 445 that is pressed when selecting an exclusion route for the moving object 10 using the captured image 415 displayed in the image display area 410. Here, the selection of an exclusion route is, for example, a process for setting locations to be excluded from the movement path of the moving object 10, i.e., locations where the movement of the moving object 10 is restricted, when the moving object 10 is caused to move autonomously. In selecting an exclusion route, the operator, for example, checks obstacles captured in the captured image 415 or the condition of the road surface within the base, and selects, as the exclusion route, a captured image 415 that shows a location where the operator does not want the moving object 10 to move.

[0142] The operator can select an image corresponding to the specified selection mode by selecting the destination selection button 440 or the exclusion route selection button 445 and then selecting the captured image 415 displayed in the image display area 410. Furthermore, by using the selection screen 400E to simultaneously select a destination and an exclusion route, the operator can select an image that will move the moving object 10 to a specified destination without moving along the exclusion route.

[0143] In this way, the selection screen 400E can use the captured image 415 that has been pre-registered through the registration process of candidate destinations for the moving body 10 for purposes other than selecting the destination of the moving body 10, thereby improving the operability for the operator in selecting destinations, including the destination of the moving body 10 and excluded routes, etc.

[0144] Effect of the embodiment As described above, the communication system 1 can improve operability when having the operator set the destination of the moving object 10 by registering in advance the captured images 415, which are still images showing a partial area of ​​a base by the moving object 10, as candidates for the destination of the moving object 10, and displaying the registered captured images 415 on the display device 50 to allow the operator to select the destination of the moving object 10. Furthermore, the communication system 1 can set a movement route for the moving object 10 that includes multiple destinations according to the order of selection by having the operator select from the multiple captured images 415 displayed on the display device 50 in order.

[0145] Furthermore, the communication system 1 can reduce the effort required to register candidate destinations by automatically acquiring captured images 415 of areas within a base to be registered as candidate destinations in accordance with the movement state of the mobile object 10. Furthermore, the communication system 1 can provide a display screen (selection screen 400, etc.) that can easily set a movement route for the mobile object 10 during autonomous movement by displaying the captured images 415 automatically acquired in accordance with the movement state of the mobile object 10 on the display device 50.

[0146] Modified examples of the embodiment ○Variation 1○ Next, a first modification of the communication system according to the embodiment will be described with reference to Figures 24 and 27. The same configurations and functions as those of the above embodiment are denoted by the same reference numerals, and the description thereof will be omitted. The communication system 1A according to the first modification is a system in which an information processing device 90 manages data on destination candidates for a moving object 10A.

[0147] 24 is a diagram showing an example of the overall configuration of a communication system according to Modification 1 of the embodiment. In addition to the configuration of the above-described embodiment, the communication system 1A according to Modification 1 includes an information processing device 90 that can communicate with a moving object 10A and a display device 50A via a communication network 100.

[0148] The information processing device 90 is a server computer that manages communications between the mobile object 10A and the display device 50A, performs various controls on the mobile object 10A, and generates various display screens to be displayed on the display device 50A. The information processing device 90 may be configured with a single server computer or multiple server computers. The information processing device 90 will be described as a server computer existing in a cloud environment, but may also be a server existing in an on-premises environment. The hardware configuration of the information processing device 90 has the same configuration as the display device 50 shown in FIG. 4. For convenience, the hardware configuration of the information processing device 90 will be described below using reference numerals in the 900 series for the configuration shown in FIG. 4.

[0149] Fig. 25 is a diagram illustrating an example of the functional configuration of a communication system according to Modification 1 of the embodiment. The configuration of a display device 50A according to Modification 1 shown in Fig. 25 is similar to the configuration of the display device 50 shown in Fig. 5. Furthermore, a control device 30A that controls the processing or operation of a moving object 10A according to Modification 1 has a configuration obtained by removing the function of the map information management unit 46 and the destination candidate management DB 3001, map information management DB 3003, and learning data management DB 3004 that are constructed in the storage unit 3000 from the control device 30 shown in Fig. 5.

[0150] The information processing device 90 has a transmission / reception unit 91, a map information management unit 92, and a storage / readout unit 99. Each of these units is a function or a means for performing the function that is realized when any of the components shown in Fig. 4 operates in response to an instruction from the CPU 901 in accordance with a program for the information processing device loaded on the RAM 903. The information processing device 90 also has a storage unit 9000 constructed by the ROM 902, HD 904, or recording medium 921 shown in Fig. 4.

[0151] The transmitting / receiving unit 91 is mainly realized by the processing of the CPU 901 for the network I / F 908, and transmits and receives various data or information to and from other devices or terminals.

[0152] The map information management unit 92 is mainly realized by the processing of the CPU 901, and manages map information indicating an environmental map of a target base where the mobile object 10 is installed, using a map information management DB 9002. The map information management unit 92 manages, for example, map information indicating an environmental map downloaded from an external server or the like, or an environmental map created by applying SLAM.

[0153] The storage / readout unit 99 is mainly realized by the processing of the CPU 901, and stores various data (or information) in the storage unit 9000 and reads out various data (or information) from the storage unit 9000. The storage unit 9000 also has a destination candidate management DB 9001, a map information management DB 9002, and a learning data management DB 9003. Of these, the destination candidate management DB 9001 is configured by the destination candidate management table shown in FIG. 6. The map information management DB 9002 is configured by the map information management table shown in FIG. 8. The learning data management DB 9003 has the same configuration as the learning data management DB 3004 configured in the storage unit 3000 of the above-mentioned mobile object 10.

[0154] 26 is a sequence diagram showing an example of a process of registering a destination candidate according to Modification 1 of the embodiment. Note that the processes of steps S101 to S107 are similar to the processes of steps S11 to S17 shown in FIG. 11, and therefore, description thereof will be omitted.

[0155] In step S108, the transmitter / receiver 51 of the display device 50A transmits a destination candidate registration request to the information processing device 90. As a result, the transmitter / receiver 91 of the information processing device 90 receives the destination candidate registration request transmitted from the display device 50A.

[0156] Next, the transmitter / receiver 91 of the information processing device 90 transmits a data acquisition request required for registering destination candidates to the mobile object 10A (step S109). As a result, the transmitter / receiver 31 of the control device 30A included in the mobile object 10A receives the data acquisition request transmitted from the information processing device 90.

[0157] Next, the location information acquisition unit 35 of the control device 30A transmits a data acquisition request required for registering the destination candidate to the mobile object 10 (step S109). As a result, the transceiver unit 31 of the control device 30A provided in the mobile object 10A receives the data acquisition request transmitted from the information processing device 90.

[0158] Next, the position information acquisition unit 35 of the control device 30A acquires position information indicating the current position of the moving object 10A using the GPS sensor 306 or the like (step S110). Specifically, the position information acquisition unit 35 acquires coordinate information including latitude and longitude indicating the current position of the moving object 10. If the moving object 10 is an air vehicle such as a drone, the position information acquisition unit 35 acquires information such as the speed, attitude, or altitude of the air vehicle as position information in addition to the coordinate information including latitude and longitude. Furthermore, the destination candidate acquisition unit 36 ​​acquires images captured by the special imaging device 12a at the current position of the moving object 10A as destination candidate images indicating destination candidates for the moving object 10 (step S111). Then, the transmission / reception unit 31 transmits the position information acquired in step S110 and the captured images acquired in step S11 to the requesting information processing device 90. As a result, the transmission / reception unit 91 of the information processing device 90 receives the position information and captured images transmitted from the moving object 10A.

[0159] Then, the storage / readout unit 99 stores the destination candidate data including the location information and captured images received in step S111 in the destination candidate management DB 9001 (step S112). The storage / readout unit 99 registers the received location information and captured images as destination candidate data, each associated with a candidate ID that identifies the destination candidate. As in the above-described embodiment, the destination candidate registration process shown in FIG. 26 is an example of a process for registering candidates for the destinations of the moving object 10A, and the process may be performed not only for the purpose of registering candidates for the destinations of the moving object 10A, but also for the purpose of registering candidates for locations to be excluded from the movement route of the moving object 10A, for example.

[0160] FIG. 27 is a sequence diagram showing an example of a process for setting a destination of a moving object according to the first modification of the embodiment. First, similar to step S71 of FIG. 13, the reception unit 52 of the display device 50A receives selection of the destination setting button 290 through an input operation on the operation screen 200 by the operator (step S131). Next, the transmission / reception unit 51 transmits a destination candidate acquisition request to the information processing device 90, requesting acquisition of data indicating destination candidates (step S132). This destination candidate acquisition request includes a base ID that identifies the base where the moving object 10A is installed. As a result, the transmission / reception unit 91 of the information processing device 90 transmits the destination candidate acquisition request transmitted from the display device 50A.

[0161] Next, the storage / readout unit 99 of the information processing device 90 searches the destination candidate management DB 9001 using the base ID indicated in the destination candidate acquisition request received in step S132 as a search key, and reads out destination candidate data associated with the same base ID as the received base ID (step S133). Then, the transmission / reception unit 91 transmits the destination candidate data read out in step S133 to the requesting display device 50A (step S134). This destination candidate data includes sets of the multiple candidate IDs, captured images, and location information read out in step S133. As a result, the display device 50A receives the destination candidate data transmitted from the information processing device 90.

[0162] Next, the display control unit 53 of the display device 50A causes a display unit such as the display 506 to display a selection screen 400 as shown in Figs. 14 and 15, including the destination candidate data received in step S134 (step S135). Note that the subsequent processes of steps S135 to S139 are similar to the processes of steps S75 to S79 ​​shown in Fig. 13, respectively, and therefore description thereof will be omitted. Note that, as in the above-described embodiment, the destination setting process shown in Fig. 27 is an example of a process for setting a destination of the moving object 10A, and the process may be performed not only for setting a destination of the moving object 10A, but also for the purpose of, for example, setting an exclusion rule to exclude the moving object 10A from the moving route of the moving object 10A.

[0163] In this way, even when the information processing device 90 registers and manages destination candidate data, the communication system 1A according to the first modification can display a selection screen 400 or the like showing destination candidate images on the display device 50A to allow the operator to select a destination for the moving body 10A, thereby improving operability when allowing the operator to select a destination for the moving body 10A. Furthermore, by registering and centrally managing destination candidate data using the information processing device 90, the communication system 1A can select a destination using captured images acquired from different moving bodies 10A, thereby improving the convenience for the operator in setting the destination of the moving body 10.

[0164] ○Variation 2○ Next, a second modification of the communication system according to the embodiment will be described with reference to Figures 28 to 30. The same configurations and functions as those of the above embodiment are denoted by the same reference numerals, and the description thereof will be omitted. The communication system 1B according to the second modification is a system that displays image data provided from an external service N on a display device 50B, and enables a destination to be selected for a moving object 10B.

[0165] 28 is a diagram illustrating an example of the overall configuration of a communication system according to Modification 2 of the embodiment. In addition to the configuration of the above-described embodiment, the communication system 1B according to Modification 2 includes an external service N that can communicate with a moving object 10B and a display device 50B via a communication network 100.

[0166] The externally provided service N is an external system that provides various services via the communication network 100. The externally provided service N provides, for example, a cloud service called a storage service (or online storage) that stores data files on the cloud, or a search service that uses a search engine connected to the Internet.

[0167] The display device 50B executes a pre-installed dedicated application or accesses a website using a web browser to acquire image data provided by an external service, thereby selecting a destination using the image. Furthermore, since the communication system 1B can acquire images taken at various locations from the external service N without limiting the location of the subject, the mobile object 10B is not limited to a mobile object moving within a limited base as in the above-described embodiment, but can also be applied to a mobile object that moves over a wide area without a limited range of movement. In this case, the mobile object 10B is preferably used in an automobile, for example, as shown in FIG. 28. If the mobile object 10B is an automobile, the display device 50B may be a car navigation device installed in the automobile, a smartphone used by an operator in the automobile, or the like.

[0168] 29 is a sequence diagram showing an example of a process for setting a destination of a moving object according to Modification 2 of the embodiment. Note that the communication system 1B according to Modification 2 is a system for setting a destination using an image provided by an external service N, and therefore does not require the process for registering destination candidates (see FIG. 9) as shown in the above embodiment.

[0169] First, the display control unit 53 of the display device 50B accesses the externally provided service N using a dedicated application or a web browser, thereby displaying an image search screen 800 on a display unit such as the display 506 (step S201). FIG. 30 is a diagram showing an example of the image search screen. The image search screen 800 shown in FIG. 30 is a display screen for searching for images that can be obtained by using the externally provided service N. The image search screen 800 is a service provision screen provided by the externally provided service N, an arbitrary website, or the like.

[0170] Image search screen 800 includes a keyword input area 810 for inputting a predetermined keyword, and a search button 815 that is pressed when performing an image search using the keyword input in keyword input area 810. Image search screen 800 also includes a location information input area 830 for inputting predetermined location information, a map reference button 831 that is pressed when displaying a map to be referenced for specifying location information, and a search button 835 that is pressed when performing an image search using the location information input in location information input area 830 or the location information specified using the map referenced from map reference button 831. Image search screen 800 also includes a cancel button 805 that is pressed when canceling the image search.

[0171] The reception unit 52 of the display device 50B receives an input of a keyword or location information on the image search screen 800 (step S202). When the reception unit 52 receives the operator's selection of the search button 815 or the search button 835, the transmission / reception unit 51 transmits an image acquisition request to the external service N (step S203). This image acquisition request includes the input data received in step S202.

[0172] Next, the external service N performs an image search based on the input data transmitted from the display device 50B, and transmits the image data as a search result to the requesting display device 50B (step S204). As a result, the transmitting / receiving unit 51 of the display device 50B receives the image data transmitted from the external service N.

[0173] Then, the transmitter / receiver 51 of the display device 50B causes a display unit such as the display 506 to display a selection screen 400 as shown in Figs. 14 and 15, which includes the destination candidate data received in step S204 (step S205). In this case, the image data transmitted from the external service N is displayed in the image display area 410 of the selection screen 400. Note that the subsequent processes of steps S206 to S209 are similar to the processes of steps S76 to S79 ​​shown in Fig. 13, respectively, and therefore will not be described again.

[0174] In this way, the communication system 1B according to the second modification can select a destination for the moving body 10B using image data provided by the external service N, which eliminates the need to register candidate destinations using images taken by the moving body 10B, thereby improving convenience in selecting a destination for the moving body 10B. Furthermore, the communication system B can acquire images of various locations from the external service N by image search, which allows the setting of a destination for the moving body 10B without being limited by the movement range of the moving body 10B.

[0175] Fig. 31 is a diagram showing an example of the functional configuration of a communication system. In comparison with the functional configuration of the communication system shown in Fig. 25, the display device 50 has the same configuration as the display device 50 shown in Fig. 25. The control device 30B that controls the processing or operation of the moving object 10B has a configuration similar to that of the control device 30A shown in Fig. 25, except that the route information generation unit 37, the route information management unit 38, the destination setting unit 39, the learning unit 48, and the route information management DB 3002 constructed in the storage unit 3000 are removed.

[0176] In the communication system 1B shown in FIG. 31, an information processing device 90 supports a cloud computing service such as AWS (trademark), and in the communication system 1B, a display device 50 and a moving body 10B (control device 30B) communicate with each other via the information processing device 90, as indicated by arrows a and b. The functions of the route information generation unit 37, route information management unit 38, destination setting unit 39, learning unit 48, and route information management DB 3002, which are removed from the control device 30A, have been transferred to the information processing device 90. That is, the information processing device 90 includes a transmission / reception unit 91, a map information management unit 92, a route information generation unit 93, a route information management unit 94, a destination setting unit 95, and a learning unit 96. A destination candidate management DB 9001, a map information management DB 9002, a learning data management DB 9003, and a route information management DB 9004 are configured in a storage unit 9000 of the information processing device 90. The functions of the above-mentioned units transferred from the control device 30A (FIG. 25) to the information processing device 90 are the same as those described in FIG. 25 and the like, and therefore will not be described again.

[0177] As described above, in communication system 1B, communication between display device 50 and mobile object 10B (control device 30B) is performed via information processing device 90 compatible with cloud computing services. In such information processing device 90, by utilizing authentication processing by the cloud computing service during communication, it is possible to improve the security of manual operation commands from display device 50 and captured image data from mobile object 10B. Furthermore, by placing the generation and management functions of each data in information processing device 90 (cloud service), it becomes possible to share the same data at multiple locations, and it becomes possible to flexibly handle not only P2P (Peer to Peer) communication (one-to-one direct communication) but also one-to-multiple-location communication.

[0178] ●Summary● As described above, a display device according to one embodiment of the present invention is a display device 50 (50A, 50B) for performing a predetermined operation on a moving object 10 (10A, 10B), and acquires a captured image 415, displays the acquired captured image 415 so that the user can select it, and accepts a selection from the displayed captured image 415. The display device 50 (50A, 50B) then sets a destination for the moving object 10 (10A, 10B) based on the area indicated by the captured image 415 for which the selection has been accepted. This allows the display device 50 (50A, 50B) to improve user operability when selecting a destination for the moving object 10 (10A, 10B).

[0179] Furthermore, a display device according to an embodiment of the present invention displays the acquired photographed image 415 as an image showing candidate destinations for the moving body 10 (10A, 10B). As a result, the display device 50 (50A, 50B) uses the acquired photographed image 415 to select an area shown in the photographed image 415 as the destination for the moving body 10, thereby improving operability for selecting the destination for the moving body 10 (10A, 10B) compared to conventional methods in which an operator specifies location information or a character string such as a keyword. Furthermore, on the display device 50 (50A, 50B), the image showing candidate destinations for the moving body 10 (10A, 10B) is an image showing candidate destinations for the moving body 10 (10A, 10B). As a result, the display device 50 (50A, 50B) can improve the operability of selecting a destination for the moving body 10 (10A, 10B) by using the acquired photographed image 415, which is a destination candidate image, to select the area indicated by the photographed image 415 as the destination for the moving body 10.

[0180] Furthermore, the display device according to one embodiment of the present invention autonomously moves the moving object 10 (10A, 10B) according to a movement route based on the area indicated by the photographed image 415 for which selection has been accepted. As a result, the display device 50 (50A, 50B) can autonomously move the moving object 10 (10A, 10B) based on the movement route of the moving object 10 set based on the area indicated by the selected photographed image 415. Furthermore, the display device 50 (50A, 50B) accepts a selection from the multiple displayed photographed images 415, and autonomously moves the moving object 10 (10A, 10B) according to a movement route corresponding to the order in which the selection of the multiple photographed images 415 for which selection has been accepted was accepted. As a result, the display device 50 (50A, 50B) can autonomously move the moving object 10 (10A, 10B) based on the movement route of the moving object 10 including multiple destinations corresponding to the order of selection by having the operator select from the multiple displayed photographed images 415 in order.

[0181] Furthermore, the display device according to one embodiment of the present invention displays the travel time required to move to the area indicated by the selected photographed image 415. In this way, the display device 50 (50A, 50B) displays the time required for the moving object 10 to move to the area indicated by the selected photographed image 415, allowing the operator to appropriately determine whether or not to select a destination based on the travel time, and the order in which to select multiple destinations.

[0182] Furthermore, in a display device according to one embodiment of the present invention, the acquired captured image is a special image 450 (e.g., a spherical image) captured in all directions within a predetermined base where the moving object 10 (10A, 10B) is moving, and the display device 50 (50A, 50B) accepts a selection of a predetermined position included in the displayed special image 450, and autonomously moves the moving object 10 (10A, 10B) according to a movement path based on the position on the special image 450 whose selection was accepted. This allows the display device 50 (50A, 50B) to allow the operator to select a predetermined position within the area captured in the special image 450 as a candidate destination for the moving object 10 (10A, 10B), thereby improving operability when selecting a destination for the moving object 10 (10A, 10B).

[0183] Furthermore, the display device according to one embodiment of the present invention displays additional information 490 indicating the characteristics of the subject appearing in the acquired photographed image 415 together with the photographed image 415. In this way, the display device 50 (50A, 50B) displays the characteristics of the subject appearing in the photographed image 415, such as the name, as additional information 490 together with the photographed image 415, thereby allowing the operator to make an appropriate decision compared to when selecting the destination of the moving object 10 by looking only at the photographed image 415.

[0184] Furthermore, a communication system according to an embodiment of the present invention is a communication system 1 (1A, 1B) including a display device 50 (50A, 50B) for performing a predetermined operation on the moving object 10 (10A, 10B), and the moving object 10 (10A, 10B) capable of communicating with the display device 50 (50A, 50B). The moving object 10 (10A, 10B) generates route information indicating a movement route of the moving object 10 (10A, 10B) based on the shooting position of the photographed image 415 whose selection has been accepted, and causes the moving object 10 (10A, 10B) to autonomously move based on the movement route indicated in the generated route information. This allows the communication system 1 (1A, 1B) to autonomously move the moving object 10 (10A, 10B) based on the movement route generated based on the photographed image 415 selected by the operator.

[0185] ●Additional Information● Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in the present embodiment includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices designed to perform each function described above, such as an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field-programmable gate array (FPGA), a system-on-a-chip (SOC), a graphics processing unit (GPU), and a conventional circuit module.

[0186] Furthermore, the various tables in the above-described embodiments may be generated by the learning effects of machine learning, and tables may not be used by classifying data for each associated item using machine learning. Here, machine learning refers to a technology that allows a computer to acquire human-like learning capabilities, in which the computer autonomously generates algorithms necessary for judgments such as data classification from previously acquired learning data and applies these algorithms to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods. Any learning method for machine learning is acceptable.

[0187] So far, we have described a display device, a communication system, a display control method, and a program according to one embodiment of the present invention, but the present invention is not limited to the above-described embodiment, and other embodiments can be added, modified, or deleted, etc., within the scope that can be conceived by a person skilled in the art, and any aspect is included in the scope of the present invention as long as it achieves the functions and effects of the present invention. [Explanation of symbols]

[0188] 1,1A,1B communication system 100 Communication Network 10,10A,10B Mobile object 30, 30A, 30B Control device 31 Transmitter / Receiver 32 Judgment Department 35 Location information acquisition section 36 Destination candidate acquisition department 37 Route information generation unit (an example of route information generation means) 40 Movement control unit (an example of movement control means) 41 Image recognition processing section 42 Mode setting section 43 Autonomous movement processing unit 44 Manual operation processing section 45 Task execution unit 48 Learning Department 3000 Memory unit (an example of memory means) 50,50A,50B display device 51 Transmitting / receiving unit (an example of an acquiring means, an example of a transmitting means) 52 Reception unit (an example of reception means) 53 Display control unit (an example of display control means) 56 Autonomous movement request information generation unit (an example of a generation means) 90 Information processing equipment 91 Transmitter / Receiver 9000 Memory unit (an example of memory means) 200 Operation screen 400, 400A, 400B, 400C, 400D, 400E selection screen 415 images taken 450 Special Images 490 Additional Information 600A,600B operation screen 800 Image search screen

Claims

1. A display device for performing a predetermined operation on a moving object, an acquisition means for acquiring a photographed image; a display control means for displaying the acquired photographed images in a manner selectable by a user; a receiving means for receiving a selection from the displayed photographed image, A display device that sets the destination of the moving object based on the area indicated by the captured image for which the selection has been accepted.

2. The display device according to claim 1 , wherein the display control means displays the acquired photographed image as an image showing candidate destinations of the moving object.

3. The display device according to claim 2 , wherein the image showing the candidate destinations of the mobile object is an image showing candidate destinations of the mobile object.

4. The display device according to claim 1 , wherein the moving object is autonomously moved along a movement path based on the area indicated by the captured image for which the selection has been accepted.

5. the accepting means accepts a selection from the displayed plurality of photographed images; The display device according to claim 1 , wherein the moving object is autonomously moved along a movement path corresponding to an order in which the selection of the plurality of captured images is accepted.

6. The display device according to claim 1 , wherein the display control means displays a travel time required to travel to the area indicated by the captured image, the selection of which has been accepted by the accepting means.

7. the captured image is a special image captured in all directions within the base where the moving object is moving, the accepting means accepts a selection of a predetermined position included in the displayed special image; The display device according to claim 1 , wherein the moving object is autonomously moved along a movement path based on the position on the special image where the selection has been accepted.

8. The display device according to claim 1 , wherein the display control means displays a map image indicating the photographing position of the acquired photographed image together with the photographed image.

9. The display device according to claim 1 , wherein the display control means displays additional information indicating characteristics of a subject appearing in the acquired photographic image together with the photographic image.

10. The display device according to any one of claims 1 to 9, further comprising: a generation means for generating autonomous movement request information for requesting the moving object to start autonomous movement based on the captured image for which the selection has been accepted; a transmitting means for transmitting the generated autonomous movement request information to the moving body, A display device that causes the moving body to move autonomously in response to the transmitted autonomous movement request information.

11. The display device according to claim 1 , wherein the display control means displays a current position of the moving body moving to an area indicated by the captured image for which the selection has been accepted, when the moving body is moving autonomously.

12. A communication system including a display device for performing a predetermined operation on a mobile object, and the mobile object capable of communicating with the display device, an acquisition means for acquiring a photographed image; a display control means for displaying the acquired photographed images in a manner selectable by a user; a receiving means for receiving a selection from the displayed photographed image, A communication system that sets the destination of the moving object based on the area indicated by the captured image for which the selection has been accepted.

13. 13. The communication system of claim 12, The moving body is a route information generating means for generating route information indicating a moving route of the moving object based on the photographing position of the photographed image for which the selection has been accepted; a movement control means for autonomously moving the moving object based on the movement route indicated in the generated route information; A communication system comprising:

14. 14. A communication system according to claim 12 or 13, a storage means for storing the captured image taken by the moving body, The display control means displays the captured image stored in the storage means.

15. A display control method executed by a display device for performing a predetermined operation on a moving object, comprising: an acquisition step of acquiring a captured image; a display control step of displaying the acquired photographed images in a manner selectable by a user; a receiving step of receiving a selection for the displayed photographed image; A display control method for setting a destination of the moving object based on an area indicated by the captured image for which the selection has been accepted.

16. A program for causing a computer to execute the display control method according to claim 15.

Images