Display system, communication system, display control method, and program

Abstract

To allow users to appropriately determine switching between autonomous movement and manual operation of a mobile body.SOLUTION: A display system for performing a predetermined operation on a mobile body 10 (10A, 10B, or 10C) includes: operation reception means (for example, a mode switching button 900) for receiving a switching operation of switching between a manual operation mode in which the mobile body 10 (10A, 10B, or 10C) is moved by manual operation and an autonomous movement mode in which the mobile body 10 (10A, 10B, or 10C) is moved by autonomous movement; and display control means (for example, a display control unit 53) for displaying notification information indicating accuracy of the autonomous movement.SELECTED DRAWING: Figure 13

Description

Technical Field

[0001] The present disclosure relates to a display system, a communication system, a display control method, and a program.

Background Art

[0002] Robots that are installed at bases such as factories and warehouses and can autonomously move within the base are known. Such robots can be used, for example, as inspection robots or service robots, and can perform tasks such as inspecting facilities within the base instead of workers.

[0003] Also, a system is known in which a robot that can autonomously move within a base can be manually operated by a user located remotely, according to the state of the robot, the situation of the base, or the intended use. For example, Patent Document 1 discloses that the autonomous driving and remote control of an unmanned vehicle are switched by the unmanned vehicle itself based on the mixing ratio of the driving environment based on distance measurement data and the communication environment of the remote control device, and the result is presented to the user.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the conventional method has a problem that it is difficult for a user to appropriately judge the switching when the user wants to switch between the autonomous movement and the manual operation of a moving object such as a robot.

Means for Solving the Problems

[0006] To solve the above-mentioned problems and achieve the objectives, the present inventiona display system for performing predetermined operations on a moving object, comprising: an operation receiving means for receiving a switching operation to switch between a manual operation mode in which the moving object is moved by manual operation and an autonomous movement mode in which the moving object is moved by autonomous movement; and a display control means for displaying notification information indicating the accuracy of the autonomous movement. The autonomous movement is characterized by being autonomous movement based on learning, and the notification information is information indicating the learning accuracy of the autonomous movement. . [Effects of the Invention]

[0007] According to the present invention, the user can easily decide whether to switch between autonomous movement and manual operation of the mobile object. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram shows an example of the overall configuration of a communication system. [Figure 2] This figure shows a schematic example of the configuration of a mobile unit. [Figure 3] This figure shows an example of the hardware configuration of a mobile device. [Figure 4] This figure shows an example of the hardware configuration of a display device. [Figure 5] This diagram shows an example of the functional configuration of a communication system. [Figure 6] This is a conceptual diagram showing an example of a map information management table. [Figure 7] This is a conceptual diagram showing an example of a destination series management table. [Figure 8] This is a conceptual diagram showing an example of a route information management table. [Figure 9] This is a sequence diagram showing an example of motion control processing for a moving object. [Figure 10] This is a sequence diagram showing an example of the process leading up to the start of movement of a moving object. [Figure 11A] This figure shows an example of a route input screen. [Figure 11B] This figure shows an example of a route input screen. [Figure 12] This sequence diagram shows an example of the process for switching between autonomous movement and manual operation of a mobile object using an operation screen. [Figure 13] It is a diagram showing an example of an operation screen. [Figure 14] It is a diagram showing an example of an operation screen. [Figure 15A] It is a diagram showing an example of an operation screen. [Figure 15B] It is a diagram showing an example of an operation screen. [Figure 16] It is a flowchart showing an example of a switching process between an autonomous movement mode and a manual operation mode in a moving body. [Figure 17] It is a flowchart showing an example of an autonomous movement process of a moving body. [Figure 18] It is a sequence diagram showing an example of a manual operation process of a moving body. [Figure 19] It is a diagram showing an example of an operation command input screen. [Figure 20A] It is a diagram showing Modification Example 1 of an operation screen. [Figure 20B] It is a diagram showing Modification Example 1 of an operation screen. [Figure 21] It is a diagram showing Modification Example 2 of an operation screen. [Figure 22] It is a diagram showing Modification Example 3 of an operation screen. [Figure 23] It is a diagram showing Modification Example 4 of an operation screen. [Figure 24] It is a diagram showing Modification Example 5 of an operation screen. [[ID= / / ID=41]] [Figure 25] It is a diagram showing Modification Example 6 of an operation screen. [Figure 26] It is a diagram showing an example of a functional configuration of a communication system according to Modification Example 1 of an embodiment. [Figure 27] It is a sequence diagram showing an example of a switching process between autonomous movement and manual operation of a moving body using an operation screen according to Modification Example 1 of an embodiment. [Figure 28] It is a diagram showing an example of an overall configuration of a communication system according to Modification Example 2 of an embodiment. [Figure 29] It is a diagram showing an example of a functional configuration of a communication system according to Modification Example 2 of an embodiment. [Figure 30] It is a sequence diagram showing an example of a process until the start of movement of a moving body according to Modification Example 2 of an embodiment. [Figure 31] This is a sequence diagram showing an example of the switching process between autonomous movement and manual operation of a mobile body using an operation screen according to a modified example 2 of the embodiment. [Figure 32] This diagram shows an example of the functional configuration of a communication system. [Modes for carrying out the invention]

[0009] The embodiments for carrying out the invention will be described below with reference to the drawings. In the description of the drawings, the same elements will be denoted by the same reference numeral, and redundant explanations will be omitted.

[0010] ●Embodiment● ● System Configuration Figure 1 shows an example of the overall configuration of a communication system. The communication system 1 shown in Figure 1 is a system that allows a user to remotely control a mobile object 10 that is moving within a predetermined location.

[0011] Communication system 1 includes a mobile device 10 and a display device 50 located at a predetermined location. The mobile device 10 and the display device 50 constituting communication system 1 can communicate via a communication network 100. The communication network 100 is constructed using the Internet, a mobile communication network, a LAN (Local Area Network), etc. In addition to wired communication, the communication network 100 may also include wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0012] Mobile unit 10 is a robot installed at a target site and capable of autonomous movement within that site. This autonomous movement includes the robot learning (machine learning) of previously traveled routes within the target site and using the results of this learning to move autonomously within the site. Furthermore, autonomous movement may also involve autonomous movement within the site following a pre-set route or using techniques such as line tracing. In addition, mobile unit 10 can also be moved by manual operation from a remote user. That is, mobile unit 10 can move within the target site while switching between autonomous movement and manual operation by the user. Furthermore, mobile unit 10 can perform predetermined tasks such as inspection, maintenance, transportation, or light work while moving within the target site. Here, mobile unit 10 refers to a robot in a broad sense, and any robot capable of both autonomous movement and movement via remote operation by a user is acceptable. Mobile unit 10 also includes automobiles that can switch between autonomous driving and manual operation via remote control. Furthermore, the mobile body 10 also includes aircraft such as drones, multicopters, or unmanned aerial vehicles.

[0013] Furthermore, the target locations where the mobile unit 10 will be installed include, for example, outdoor locations such as offices, factories, construction sites, substations, farms, fields, or cultivated land or disaster sites, or indoor locations such as offices, schools, factories, warehouses, commercial facilities, hospitals or nursing homes. In other words, the target locations can be any locations where there is a need to have the mobile unit 10 take over tasks that were previously performed manually.

[0014] The display device 50 is located at a management location different from the target location and is a computer such as a notebook PC (Personal Computer) used by an operator (user) who performs predetermined operations on the mobile device 10. The operator, at the management location such as an office, uses the operation screen displayed on the display device 50 to perform operations on the mobile device 10, such as moving it or performing predetermined tasks. For example, the operator remotely controls the mobile device 10 while viewing an image of the target location displayed on the display device 50.

[0015] Figure 1 shows an example where one mobile unit 10 and one display device 50 are connected via a communication network 100. However, the display device 50 may be configured to connect to multiple mobile units 10 located at a single target site, or to connect to mobile units 10 located at different target sites. Also, Figure 1 shows an example where the display device 50 is located at a remote management site different from the target site where the mobile units 10 are installed. However, the display device 50 may be located within the target site where the mobile units 10 are installed. Furthermore, the display device 50 is not limited to a notebook PC, but may be, for example, a desktop PC, tablet terminal, smartphone, or wearable device.

[0016] Traditionally, for example, if a mobile vehicle became unable to move due to an obstacle while autonomously moving, the operator would manually perform a recovery operation to return it to autonomous movement. However, the information presented to the operator in the conventional manner made it difficult for the operator to accurately decide whether to switch from manual control back to autonomous movement. Furthermore, when autonomous movement was being performed through learning during manual control, if the learning results could not be appropriately utilized due to changes in the environment such as weather or buildings within the base, it was difficult for the operator to decide whether to switch back to manual control to perform learning again. In other words, when the operator wanted to switch between autonomous movement and manual control of the mobile vehicle, the conventional method made it difficult for the operator to make an appropriate decision on the switch.

[0017] Therefore, the communication system 1 displays notification information indicating the accuracy of the autonomous movement of the mobile unit 10 on the display device 50 used by the operator remotely controlling the mobile unit 10, making it easy for the operator to decide whether or not to switch between autonomous movement and manual operation. Furthermore, since the communication system 1 allows switching between autonomous movement and manual operation of the mobile unit 10 using the operation screen displayed on the display device 50, it is possible to improve the user's operability when switching between autonomous movement and manual operation of the mobile unit 10. In addition, even for mobile units 10 that learn movement paths using manual operation, the communication system 1 allows the operator to make a more appropriate judgment about the necessity of learning through manual operation.

[0018] ● Configuration of the mobile unit Next, the specific configuration of the mobile body 10 will be explained using Figure 2. Figure 2 is a diagram showing an example of a schematic configuration of the mobile body. Note that the configuration of the mobile body 10 shown in Figure 2 may have components added or removed as needed.

[0019] The mobile body 10 shown in Figure 2 comprises a housing 11 equipped with a control device 30 for controlling the processing or operation of the mobile body 10, a shooting device 12, a support member 13, a display 14, a moving mechanism 15 (15a, 15b) for moving the mobile body 10, and a movable arm 16 for causing the mobile body 10 to perform a predetermined task (operation). Of these, the housing 11 is located in the body portion of the mobile body 10 and houses the control device 30, etc., for controlling the processing or operation of the mobile body 10.

[0020] The shooting device 12 photographs subjects such as people, objects, or landscapes located at the base where the mobile body 10 is installed and acquires captured images. The shooting device 12 is a digital camera (general shooting device) capable of acquiring planar images (detailed images), such as a digital SLR camera or a compact digital camera. The captured images acquired by the shooting device 12 may be video, still images, or both. The captured images acquired by the shooting device 12 may also include audio data along with the image data. Furthermore, the shooting device 12 may be a wide-angle shooting device capable of acquiring a 360° panoramic image. The wide-angle shooting device is, for example, a 360° panoramic shooting device for photographing a subject and obtaining two hemispherical images that will form the basis of a 360° panoramic image. Furthermore, the wide-angle shooting device may be, for example, a wide-angle camera or stereo camera capable of acquiring wide-angle images with a field of view greater than or equal to a predetermined value. That is, the wide-angle shooting device is a shooting means capable of acquiring images (360° images, wide-angle images) taken using a lens with a focal length shorter than a predetermined value. Furthermore, the mobile unit 10 may be configured to include multiple imaging devices 12. In this case, the mobile unit 10 may be configured to include both a wide-angle imaging device and a general imaging device capable of capturing a portion of the subject captured by the wide-angle imaging device to obtain a detailed image (planar image).

[0021] The support member 13 is a component for installing (fixing) the imaging device 12 to the mobile body 10 (housing 11). The support member 13 may be a pole or the like fixed to the housing 11, or it may be a base fixed to the housing 11. The support member 13 may also be a movable component that can adjust the imaging direction (orientation) and position (height) of the imaging device 12.

[0022] The moving mechanism 15 is a unit that moves the mobile body 10 and consists of wheels, a driving motor, a driving encoder, a steering motor, and a steering encoder, etc. Since the movement control of the mobile body 10 is based on existing technology, a detailed explanation will be omitted. For example, the mobile body 10 receives a driving instruction from an operator (display device 50), and the moving mechanism 15 moves the mobile body 10 based on the received driving instruction. The moving mechanism 15 may be bipedal or single-wheeled. Furthermore, the shape of the mobile body 10 is not limited to the vehicle type shown in Figure 2; for example, it may be a bipedal humanoid, a form mimicking a living creature, a form mimicking a specific character, etc.

[0023] The movable arm 16 has an operating means that enables additional movements other than the movement of the mobile body 10. As shown in Figure 2, the movable arm 16 is equipped with an operating means, for example, a hand at the tip of the movable arm 16 for grasping objects such as parts. The mobile body 10 can perform predetermined tasks (movements) by rotating or deforming the movable arm 16. In addition to the above configuration, the mobile body 10 may also have various sensors capable of detecting information about its surroundings. Examples of such sensors include barometers, thermometers, photometers, motion sensors, gas sensors, odor sensors, or illuminometers.

[0024] ● Hardware configuration Next, the hardware configuration of the device or terminal constituting the communication system according to the embodiment will be described using Figures 3 and 4. Note that the hardware configuration of the device or terminal shown in Figures 3 and 4 may have components added or removed as needed.

[0025] ○Hardware configuration of the mobile device○ Figure 3 shows an example of the hardware configuration of a mobile device. The mobile device 10 is equipped with a control device 30 that controls the processing or operation of the mobile device 10. As described above, the control device 30 is housed inside the housing 11 of the mobile device 10. The control device 30 may be provided outside the housing 11 of the mobile device 10, or it may be provided as a separate device from the mobile device 10.

[0026] The control device 30 includes a CPU (Central Processing Unit) 301, ROM (Read Only Memory) 302, RAM (Random Access Memory) 303, HDD (Hard Disk Drive) 304, media interface 305, input / output interface 306, audio input / output interface 307, network interface 308, short-range communication circuit 309, antenna 309a for the short-range communication circuit 309, external device connection interface 311, and bus line 310.

[0027] The CPU 301 controls the entire mobile unit 10. The CPU 301 is a computing device that realizes each function of the mobile unit 10 by reading programs or data stored in the ROM 302 or HD (Hard Disk) 304a, etc., onto the RAM 303 and executing processing.

[0028] ROM302 is a non-volatile memory that can retain programs or data even when the power is turned off. RAM303 is a volatile memory used as a work area for the CPU301, etc. HDD304 controls the reading or writing of various data to HD304a according to the control of CPU301. HD304a stores various data such as programs. Media I / F305 controls the reading or writing (storage) of data to recording media 305a such as USB (Universal Serial Bus) memory, memory card, optical disk, or flash memory.

[0029] The input / output interface 306 is an interface for inputting and outputting characters, numbers, various instructions, etc., to and from various external devices. The input / output interface 306 controls the display of various information such as cursors, menus, windows, characters, or images on the display 14, such as an LCD (Liquid Crystal Display). The display 14 may be a touch panel display equipped with input means. In addition to the display 14, the input / output interface 306 may also be connected to input means such as a pointing device such as a mouse or a keyboard. The sound input / output interface 307 is a circuit that processes the input and output of sound signals between the microphone 307a and the speaker 307b according to the control of the CPU 301. The microphone 307a is a type of built-in sound collection means that inputs sound signals according to the control of the CPU 301. The speaker 307b is a type of playback means that outputs sound signals according to the control of the CPU 301.

[0030] Network I / F 308 is a communication interface that communicates (connects) with other devices or equipment via the communication network 100. Network I / F 308 is a communication interface such as a wired or wireless LAN. Near-field communication circuit 309 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). External device connection I / F 311 is an interface for connecting other devices to the control device 30.

[0031] The bus line 310 is an address bus, data bus, etc., for electrically connecting the above components, and transmits address signals, data signals, and various control signals. The CPU 301, ROM 302, RAM 303, HDD 304, media I / F 305, input / output I / F 306, sound input / output I / F 307, network I / F 308, short-range communication circuit 309, and external device connection I / F 311 are interconnected via the bus line 310.

[0032] Furthermore, the control device 30 is connected to the drive motor 101, actuator 102, acceleration / direction sensor 103, GPS (Global Positioning System) sensor 104, imaging device 12, battery 120, and obstacle detection sensor 106 via the external device connection I / F 311.

[0033] The drive motor 101 rotates the movement mechanism 15 based on commands from the CPU 301 to move the mobile body 10 along the ground. The actuator 102 deforms the movable arm 16 based on commands from the CPU 301. The acceleration / direction sensor 103 is a sensor such as an electronic magnetic compass, gyrocompass, and acceleration sensor that detects the Earth's magnetic field. The GPS sensor 104 receives GPS signals from GPS satellites. The battery 120 is a unit that supplies the total power required for the mobile body 10. Note that the battery 120 may include an external battery that provides auxiliary power from an external source, in addition to the battery built into the mobile body 10. The obstacle detection sensor 106 is a detection sensor that detects obstacles in the surroundings when the mobile body 10 is moving. The obstacle detection sensor 106 is, for example, an image sensor such as a stereo camera or a camera equipped with an area sensor in which photoelectric conversion elements are arranged in a planar manner, or a distance measuring sensor such as a TOF (Time Of Flight) sensor, LIDAR (Light Detection and Ranging) sensor, radar sensor, laser rangefinder, ultrasonic sensor, depth camera, or depth sensor.

[0034] ○Hardware configuration of the display device○ Figure 4 shows an example of the hardware configuration of a display device. Each hardware component of the display device 50 is indicated by a code in the 500 series. The display device 50 is built by a computer and includes, as shown in Figure 4, a CPU 501, ROM 502, RAM 503, HD 504, HDD controller 505, display 506, external device connection I / F 507, network I / F 508, bus line 510, keyboard 511, pointing device 512, audio input / output I / F 513, microphone 514, speaker 515, camera 516, DVD-RW (Digital Versatile Disk Rewritable) drive 517, and media I / F 519.

[0035] Of these components, the CPU 501 controls the operation of the entire display device 50. The ROM 502 stores programs used to drive the CPU 501, such as the IPL (Initial Program Loader). The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The display 506 may be a touch panel display equipped with input means. The display 506 is also just one example of a display unit. The display unit as the display 506 may be an external device with display functions connected to the display device 50. In this case, the display unit may be, for example, an external display such as an IWB (Interactive White Board), or a projection surface (for example, the ceiling or wall of a management base, the windshield of a car, etc.) onto which images from a PJ (Projector) or HUD (Head-Up Display) connected as an external device are projected. The external device connection I / F 507 is an interface for connecting various external devices. The network I / F 508 is an interface for data communication using the communication network 100. The bus line 510 is an address bus or data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 4.

[0036] Furthermore, the keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting or executing various instructions, selecting a processing target, or moving a cursor, etc. Note that the input means may not be limited to the keyboard 511 and the pointing device 512, but may also be a touch panel or an audio input device, etc. Also, the input means such as the keyboard 511 and the pointing device 512 may be an external UI (User Interface) of the display device 50. The sound input / output I / F 513 is a circuit that processes the input and output of sound signals between the microphone 514 and the speaker 515 according to the control of the CPU 501. The microphone 514 is a type of built-in sound collection means for inputting sound. The speaker 515 is a type of built-in output means for outputting sound signals. The camera 516 is a type of built-in imaging means for capturing images of a subject and obtaining image data. Note that the microphone 514, speaker 515 and camera 516 may be external devices instead of being built into the display device 50. The DVD-RW drive 517 controls the reading or writing of various data to the DVD-RW 518, which is an example of a removable recording medium. Note that the removable recording medium is not limited to DVD-RW, but may also be DVD-R or Blu-ray® Disc, etc. The media interface 519 controls the reading or writing (storage) of data to the recording medium 521, such as flash memory.

[0037] The above programs may be distributed as installable or executable files recorded on a computer-readable recording medium. Examples of recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray Disc, SD card, or USB memory. The recording media can also be provided domestically or internationally as a program product. For example, the display device 50 implements the display control method according to the present invention when the program according to the present invention is executed.

[0038] ●Functional Configuration Next, the functional configuration of the communication system according to the embodiment will be described using Figures 5 to 8. Figure 5 is a diagram showing an example of the functional configuration of the communication system. Note that Figure 5 shows the devices or terminals shown in Figure 1 that are related to the processing or operation described later.

[0039] ○Functional configuration of the mobile unit (control device)○ First, using Figure 5, the functional configuration of the control device 30 that controls the processing or operation of the mobile body 10 will be described. The control device 30 includes a transmitting / receiving unit 31, a judgment unit 32, an image capture control unit 33, a state detection unit 34, a map information management unit 35, a destination sequence management unit 36, a self-position estimation unit 37, a route information generation unit 38, a route information management unit 39, a destination setting unit 40, a movement control unit 41, a mode setting unit 42, an autonomous movement processing unit 43, a manual operation processing unit 44, an accuracy calculation unit 45, an image generation unit 46, a learning unit 47, and a storage / reading unit 49. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 3 operating according to instructions from the CPU 301 in accordance with a control device program deployed on the RAM 303. The control device 30 also has a storage unit 3000 constructed from a ROM 302, HD 304a, or recording medium 305a shown in Figure 3.

[0040] The transmitting / receiving unit 31 is mainly implemented by the processing of the network interface 308 by the CPU 301, and transmits and receives various data or information with other devices or terminals via the communication network 100.

[0041] The decision unit 32 is implemented by the processing of the CPU 301 and performs various decisions. The shooting control unit 33 is mainly implemented by the processing of the CPU 301 on the external device connection I / F 311 and controls the shooting process for the shooting device 12. For example, the shooting control unit 33 instructs the shooting device 12 to perform the shooting process. The shooting control unit 33 also acquires the captured image obtained by the shooting process performed by the shooting device 12.

[0042] The state detection unit 34 is mainly implemented by the processing of the CPU 301 to the external device connection I / F 311, and detects the state of the mobile body 10 or the surroundings of the mobile body 10 using various sensors. The state detection unit 34 measures the distance to objects (obstacles) present around the mobile body 10 using, for example, an obstacle detection sensor 106, and outputs it as distance data.

[0043] Furthermore, the state detection unit 34 detects the position of the moving object 10, for example, using a GPS sensor 104. Specifically, the state detection unit 34 uses the GPS sensor 104 to acquire the position on the environmental map stored in the map information management DB 3001. Alternatively, the state detection unit 34 may be configured to acquire the position by applying SLAM (Simultaneous Localization and Mapping) to distance data measured using an obstacle detection sensor 106, etc., and comparing it with the environmental map. Here, SLAM is a technology that can perform self-position estimation and environmental map creation simultaneously.

[0044] Furthermore, the state detection unit 34 detects the direction the moving object 10 is facing, for example, using an acceleration / direction sensor 103.

[0045] The map information management unit 35 is primarily implemented by the processing of the CPU 301 and uses the map information management DB 3001 to manage map information showing the environmental map of the target location where the mobile unit 10 is installed. The map information management unit 35 manages map information showing, for example, environmental maps downloaded from an external server or environmental maps created by applying SLAM.

[0046] The destination sequence management unit 36 ​​is primarily implemented by the CPU 301 and manages the destination sequence on the mobile body 10's travel path using the destination sequence management DB 3002. The destination sequence includes the final destination (goal) and multiple waypoints (sub-goals) to the final destination on the mobile body 10's travel path. The destination sequence is data identified by location information, such as latitude and longitude, which indicate a location (coordinate value) on a map. The destination sequence can be obtained, for example, by remotely controlling the mobile body 10. The specification method may also be, for example, by using a GUI (Graphical User Interface) on an environmental map.

[0047] The self-position estimation unit 37 is mainly implemented by the CPU 301 and estimates the current position (self-position) of the moving object 10 based on the position information detected by the state detection unit 34 and the direction information indicating the direction the moving object 10 is facing. The self-position estimation unit 37 uses methods such as the Extended Kalman Filter (EKF) to estimate the current position (self-position).

[0048] The route information generation unit 38 is mainly implemented by the processing of the CPU 301 and generates route information that shows the movement path of the mobile object 10. The route information generation unit 38 uses the current position (self position) of the mobile object 10 estimated by the self position estimation unit 37 and the destination sequence managed by the destination sequence management unit 36 ​​to set the final destination (goal) and multiple waypoints (subgoals), and generates route information that shows the path from the current position to the final destination. The method of generating the route information may be, for example, connecting each waypoint from the current position to the final destination with a straight line, or using information on obstacles obtained from captured images or the state detection unit 34 to minimize the movement time while avoiding obstacles.

[0049] The route information management unit 39 is mainly implemented by the processing of the CPU 301 and manages the route information generated by the route information generation unit 38 using the route information management DB 3003.

[0050] The destination setting unit 40 is mainly implemented by the CPU 301 and sets the destination for the mobile object 10. For example, based on the current position (self position) of the mobile object 10 estimated by the self position estimation unit 37, the destination setting unit 40 sets the destination to be reached (current goal) or waypoint (sub-goal) from the destination sequence managed by the destination sequence management unit 36 ​​as the destination. The method of setting the destination may be, for example, setting the destination sequence that is closest to the current position (self position) of the mobile object 10 among the destination sequences that have not yet been reached (for example, those with a status of "not yet reached"), or setting the destination sequence with the smallest data index among the destination sequences that have not yet been reached.

[0051] The movement control unit 41 is mainly implemented by the processing of the CPU 301 to the external device connection I / F 311, and controls the movement of the mobile body 10 by driving the movement mechanism 15. The movement control unit 41 moves the mobile body 10 in response to drive commands from, for example, the autonomous movement processing unit 43 or the manual operation processing unit 44.

[0052] The mode setting unit 42 is mainly implemented by the CPU 301 and sets an operation mode that indicates the operation of moving the mobile body 10. The mode setting unit 42 sets either an autonomous movement mode in which the mobile body 10 moves autonomously or a manual operation mode in which the mobile body 10 is moved by the operator's manual operation. For example, the mode setting unit 42 switches between the autonomous movement mode and the manual operation mode in response to a switching request transmitted from the display device 50.

[0053] The autonomous movement processing unit 43 is mainly implemented by the CPU 301 and controls the autonomous movement process of the mobile body 10. For example, the autonomous movement processing unit 43 outputs a drive command to the movement control unit 41 for the mobile body 10 to follow the movement path indicated in the path information generated by the path information generation unit 38.

[0054] The manual operation processing unit 44 is mainly implemented by the CPU 301 and controls the manual operation of the mobile body 10. For example, the manual operation processing unit 44 outputs a drive command for the mobile body 10 to the movement control unit 41 in response to a manual operation command transmitted from the display device 50.

[0055] The accuracy calculation unit 45 is mainly implemented by the CPU 301 and calculates the accuracy of the autonomous movement of the mobile body 10. Here, the accuracy of the autonomous movement of the mobile body 10 is information that indicates the degree of confidence that the mobile body 10 is capable of autonomous movement, and the higher the calculated value, the more it indicates that the mobile body 10 is capable of autonomous movement. For example, the calculation of the accuracy of autonomous movement can be done by lowering the value if the likelihood is low based on the likelihood value of the self-position estimated by the self-position estimation unit 37, lowering the value if the variance is large using the variance of various sensors, lowering the value if the elapsed movement time in autonomous movement mode is long using the movement elapsed time which is the operating state by the autonomous movement processing unit 43, lowering the value as the distance increases depending on the distance between the destination sequence and the mobile body 10, or lowering the value if there are many obstacles according to the information of obstacles detected by the state detection unit 34.

[0056] The image generation unit 46 is mainly implemented by the processing of the CPU 301 and generates display images to be displayed on the display device 50. For example, the image generation unit 46 generates a route image that shows the destination sequence managed by the destination sequence management unit 36 ​​on the captured image captured by the shooting control unit 33. The image generation unit 46 draws the generated route image on the movement path of the mobile body 10 on the captured image data acquired by the shooting control unit 33. The method for drawing the route image on the captured image data is, for example, a method of drawing the route image by transforming it into a perspective projection based on the self-position (current position) of the mobile body 10 estimated by the self-position estimation unit 37, the installation position of the shooting device 12, and the field of view of the captured image data. The captured image data may also include PTZ (Pan-Tilt-Zoom) parameters to specify the shooting direction of the shooting device 12. The captured image data including PTZ parameters is stored in the storage unit 3000 of the mobile body 10. Furthermore, the PTZ parameters may be stored in the memory unit 3000 in association with the location information of the destination candidates, i.e., the final destination (goal) and multiple waypoints (sub-goals) to the final destination that make up the destination sequence. Alternatively, the memory unit 3000 may simultaneously store coordinate data (x,y,θ) indicating the attitude of the mobile body 10 when the captured image data of the destination candidate was acquired, along with the location information of the destination candidate. This makes it possible to correct the attitude of the mobile body 10 using the PTZ parameters and coordinate data (x,y,θ) if there is a discrepancy between the actual stopping position of the mobile body 10 and the destination. Note that some data, such as the data of the autonomous movement path (GPS trajectory) of the mobile body 10 and the captured image data of the destination candidate used for display on the display device 50, may be stored in a cloud computing service such as AWS (Amazon Web Services) (trademark).

[0057] Furthermore, the image generation unit 46 draws, for example, the current position (self-position) of the moving object 10 estimated by the self-position estimation unit 37 and the destination sequence managed by the destination sequence management unit 36 ​​onto the environment map managed by the map information management unit 35. The method of drawing onto the environment map is, for example, a method using position information such as longitude and latitude from GPS, or a method using coordinate information obtained by SLAM.

[0058] The learning unit 47 is mainly implemented by the processing of the CPU 301 and learns the movement path for autonomous movement of the mobile body 10. The learning unit 47 performs imitation learning (machine learning) of the movement path for autonomous movement based on, for example, captured images acquired during movement in manual operation mode by the manual operation processing unit 44 and detection data by the state detection unit 34. The autonomous movement processing unit 43 performs autonomous movement of the mobile body 10 based on, for example, the learning data which is the result of imitation learning by the learning unit 47.

[0059] The memory / read unit 49 is mainly implemented by the CPU 301, and stores various data (or information) in the memory unit 3000 and reads various data (or information) from the memory unit 3000.

[0060] ○Map Information Management Table Figure 6 is a conceptual diagram showing an example of a map information management table. The map information management table is a table for managing map information, which is an environmental map of the target location where the mobile unit 10 is installed. The storage unit 3000 has a map information management DB 3001 constructed, which is composed of map information management tables like the one shown in Figure 6.

[0061] The map information management table manages map information that associates the location ID and location name that identify the target location where the mobile unit 10 is installed, with the storage location of the environmental map for the target location. Of these, the storage location is, for example, destination information for accessing the storage area within the mobile unit 10 where the environmental map is stored or an external server, and is indicated by a URL (Uniform Resource Locator) or URI (Uniform Resource Identifier).

[0062] ○ Destination series management table Figure 7 is a conceptual diagram showing an example of a destination series management table. The destination series management table is a table for managing destination series that include the final destination or multiple waypoints on the travel path to identify the travel path of the mobile body 10. The storage unit 3000 has a destination series management DB 3002 constructed, which consists of a destination series management table like the one shown in Figure 7.

[0063] The destination series management table manages information by associating a series ID that identifies a destination series, location information that indicates the location of the destination series on the environmental map, and status information that indicates the movement status of the mobile unit 10 with respect to the destination series, for each base ID that identifies the base where the mobile unit 10 is installed and a route ID that identifies the movement path of the mobile unit 10. Of these, the location information is represented by latitude and longitude coordinate information that indicates the location of the mobile unit 10 on the environmental map of the destination series. The status indicates whether or not the mobile unit 10 has reached the destination series. Statuses include, for example, "Arrived," "Current Destination," and "Not Arrived." The status is updated according to the current location and movement status of the mobile unit 10.

[0064] ○ Route information management table Figure 8 is a conceptual diagram showing an example of a route information management table. The route information management table is a table for managing route information that shows the movement path of the mobile object 10. The storage unit 3000 has a route information management DB 3003 constructed, which is composed of route information management tables like the one shown in Figure 8.

[0065] The route information management table manages route information that identifies the movement path of the mobile body 10 and associates it with a base ID that identifies the base where the mobile body 10 is installed. Of this information, the route information shows the future movement path of the mobile body 10 in the order of the sequence of destinations to which it will move in the future. The route information is generated by the route information generation unit 38 when the movement of the mobile body 10 begins.

[0066] ○Functional Configuration of Display Device○ Next, the functional configuration of the display device 50 will be described using Figure 5. The display device 50 includes a transmitting / receiving unit 51, a receiving unit 52, a display control unit 53, a judgment unit 54, a sound output unit 55, and a storage / reading unit 59. Each of these units is a function or means of functioning, realized by the operation of any of the components shown in Figure 4 by instructions from the CPU 501 according to a display device program deployed on the RAM 503. The display device 50 also has a storage unit 5000 constructed from the ROM 502, HD 504, or recording medium 521 shown in Figure 4.

[0067] The transmitting / receiving unit 51 is mainly implemented by the processing of the network interface 508 by the CPU 501, and transmits and receives various data or information with other devices or terminals.

[0068] The reception unit 52 is mainly implemented by the CPU 501 processing the keyboard 511 or pointing device 512, and accepts various selections or inputs from the user. The display control unit 53 is mainly implemented by the CPU 501, and displays various screens on a display unit such as the display 506. The judgment unit 54 is implemented by the CPU 501, and makes various judgments. The sound output unit 55 is mainly implemented by the CPU 501 processing the sound input / output I / F 513, and outputs audio signals such as warning sounds from the speaker 515 according to the state of the mobile body 10.

[0069] The memory / read unit 59 is mainly implemented by the CPU 501, and stores various data (or information) in the memory unit 5000 and reads various data (or information) from the memory unit 5000.

[0070] ●Processing or operation of the embodiment ○Movement control processing○ Next, the processing or operation of the communication system according to the embodiment will be described using Figures 9 to 21. First, Figure 9 will be used to give a general overview of the overall flow of the movement operation of the mobile body 10. Figure 9 is a sequence diagram showing an example of the movement control processing of the mobile body. Details of each process shown in Figure 9 will be explained later using Figures 10 to 19.

[0071] First, the mobile unit 10 sets its current destination using the destination setting unit 40 (step S1). In this case, the destination setting unit 40 sets the destination based on the location and status of the destination series stored in the destination series management DB 3002 (see Figure 7). The mobile unit 10 then starts moving to the destination set in step S1, following the travel path indicated in the travel path information generated by the travel information generation unit 38 (step S2). While moving along the travel path set in step S1, the mobile unit 10 performs self-position estimation using the self-position estimation unit 37 and sets the destination, which is the nearest destination until reaching the final destination, using the destination setting unit 40 (step S3).

[0072] Next, the display device 50 displays an operation screen for operating the mobile body 10 on a display unit such as the display 506, based on various data or information transmitted from the mobile body 10 as it moves within the target site (step S4). If the mobile body 10 switches between autonomous movement and manual operation based on a request from the display device 50 (YES in step S5), the process moves to step S6. On the other hand, if the mobile body 10 does not switch between autonomous movement and manual operation (NO in step S5), the process moves to step S7. In step S6, the mobile body 10 switches its operating mode using the mode setting unit 42 and moves based on the corresponding operating mode (autonomous movement mode or manual operation mode).

[0073] Then, when the mobile body 10 reaches the final destination indicated in the route information generated by the route information generation unit 38 (YES in step S7), it terminates processing and stops at the final destination. On the other hand, the mobile body 10 continues processing from step S3 until it reaches the final destination indicated in the route information (NO in step S7). In addition, the mobile body 10 may be configured to temporarily suspend its movement or terminate the movement processing operation midway if a certain amount of time has elapsed since the start of movement, if an obstacle is detected on the movement path, or if a stop command is received from the operator, even if it has not reached the final destination.

[0074] ○Processing until the start of movement of the mobile object Next, the process leading up to the start of movement of the mobile body 10 will be explained using Figures 10 to 11. Figure 10 is a sequence diagram showing an example of the process leading up to the start of movement of the mobile body.

[0075] First, the transmitting / receiving unit 51 of the display device 50 transmits a route input request to the mobile body 10, indicating that it is requesting input of the mobile body 10's movement path, in response to a predetermined input operation by the operator (step S11). This route input request includes a base ID that identifies the base where the mobile body 10 is located. As a result, the transmitting / receiving unit 31 of the control device 30 equipped with the mobile body 10 receives the route input request transmitted from the display device 50.

[0076] Next, the map information management unit 35 of the control device 30 searches the map information management DB 3001 (see Figure 6) using the base ID received in step S11 as a search key, and reads the map information associated with the same base ID as the received base ID via the storage / reading unit 49 (step S12). Here, as shown in Figure 6, the map information management DB 3001 contains the storage locations of environmental maps downloaded in advance from an external server or the like, or environmental maps created by remotely controlling the mobile unit 10 using SLAM. The map information management unit 35 accesses the storage locations indicated in the read map information and reads the corresponding map image data.

[0077] Next, the transmitting / receiving unit 31 transmits map image data corresponding to the map information read in step S12 to the requesting display device 50 (step S13). As a result, the transmitting / receiving unit 51 of the display device 50 receives the map image data transmitted from the mobile device 10.

[0078] Next, the display control unit 53 of the display device 50 causes the route input screen 200, which includes the map image data received in step S13, to be displayed on a display unit such as the display 506 (step S14). Figure 11 is a diagram showing an example of the route input screen. The route input screen 200 shown in Figure 11 is a display screen for the operator to input the route on which they want to move the mobile object 10.

[0079] The route input screen 200 displays the map image data received in step S13. The route input screen 200 also includes a display selection button 205, which is pressed to zoom in or out of the displayed map image, and a "Complete" button 210, which is pressed to complete the route input process.

[0080] As shown in Figure 11A, the route input screen 200 displays destination sequence 250a when the operator selects a predetermined location on the map image using an input means such as a pointing device 512. The operator selects locations on the map image while looking at the map image displayed on the route input screen 200. As a result, as shown in Figure 11B, the route input screen 200 displays multiple destination sequences 250a to 250h corresponding to the locations selected by the operator.

[0081] Then, as shown in Figure 11B, when the operator selects a predetermined location on the map image and presses the "Complete" button 210, the reception unit 52 accepts the input of destination sequences 250a to 250h (step S15). The transmitting / receiving unit 51 transmits destination sequence data indicating the destination sequences 250a to 250h received in step S15 to the mobile device 10 (step S16). This destination sequence data includes location information indicating the locations on the map image of destination sequences 250a to 250h input in step S15. As a result, the transmitting / receiving unit 31 of the control device 30 equipped with the mobile device 10 receives the destination sequence data transmitted from the display device 50.

[0082] Next, the destination series management unit 36 ​​of the control device 30 stores the destination series data received in step S16 in the destination series management DB 3002 (see Figure 7) via the storage / reading unit 49, associating it with the base ID received in step S11 (step S17). The destination series management unit 36 ​​identifies the multiple destination series (for example, destination series 250a to 250h) shown in the received destination series data by their series IDs and stores location information indicating the location of the corresponding destination series on the map image, associating it with each series ID.

[0083] Next, the self-position estimation unit 37 estimates the current position of the moving object 10 (step S18). Specifically, the self-position estimation unit 37 estimates the self-position (current position) of the moving object 10 using a method such as an extended Kalman filter, which uses position information indicating the position of the moving object 10 detected by the state detection unit 34 and direction information indicating the direction the moving object 10 is facing.

[0084] Next, the route information generation unit 38 generates route information indicating the movement path of the mobile body 10 based on the self-position estimated in step S18 and the destination sequence data received in step S16 (step S19). Specifically, the route information generation unit 38 sets the final destination (goal) and multiple waypoints (sub-goals) of the mobile body 10 using the current position (self-position) of the mobile body 10 estimated in step S18 and the destination sequence data received in step S16. Then, the route information generation unit 38 generates route information indicating the movement path of the mobile body 10 from the current position to the final destination. For example, the route information generation unit 38 identifies the movement path by connecting each waypoint from the current position to the final destination with a straight line, or by using information about obstacles obtained from captured images or the state detection unit 34 to minimize the movement time while avoiding obstacles. The route information management unit 39 then stores the route information generated by the route information generation unit 38 in the route information management DB 3003 (see Figure 8) via the storage and reading unit 49, associating it with the generated route ID.

[0085] Next, the destination setting unit 40 sets the destination of the mobile body 10 based on the current position of the mobile body 10 estimated in step S18 and the route information generated in step S19 (step S20). Specifically, the destination setting unit 40 sets the destination to be reached now (current goal) from the destination sequence shown in the generated route information, based on the estimated current position of the mobile body 10 (self position). For example, the destination setting unit 40 sets the destination sequence that is closest to the current position of the mobile body 10 (self position) among the destination sequences that have not yet been reached (for example, those with a status of "not yet reached") as the destination of the mobile body 10. Then, the movement control unit 41 starts the process of moving the mobile body 10 to the destination set in step S20 (step S21). In this case, the movement control unit 41 autonomously moves the mobile body 10 in response to a drive command from the autonomous movement processing unit 43.

[0086] In this way, the communication system 1 can autonomously move the mobile body 10 based on a travel path generated according to the destination sequence entered by the operator. In step S15, an example was described in which the destination sequence is selected by selecting a position on the map image displayed on the route input screen 200. However, the route input screen 200 may also be configured to display multiple past captured images, which are learning data from the learning unit 47, and allow the operator to select a destination sequence corresponding to the shooting location of the captured image by selecting one of the displayed captured images. In this case, the destination sequence data includes information that identifies the selected captured image instead of location information. The destination sequence management DB 3002 stores the image identification information instead of location information.

[0087] ○ Operator's control of movement of a moving object Next, using Figures 12 to 19, we will explain the control process for the mobile body 10 while it is moving via remote control by the operator. Figure 12 is a sequence diagram showing an example of the process of switching between autonomous movement and manual operation of the mobile body using the operation screen. Figure 12 shows an example where the mobile body 10 has started autonomous movement within the base by the process shown in Figure 10.

[0088] First, the accuracy calculation unit 45 of the control device 30 equipped with the mobile body calculates the autonomous movement accuracy of the mobile body 10 (step S31). The accuracy calculation unit 45 calculates the autonomous movement accuracy based, for example, on the route information generated by the route information generation unit 38 and the current position of the mobile body 10 estimated by the self-position estimation unit 37. The autonomous movement accuracy of the mobile body 10 is information that indicates the degree of confidence (confidence level) of whether the mobile body 10 is capable of autonomous movement, and the higher the calculated value, the more it indicates that the mobile body 10 is capable of autonomous movement. Alternatively, the accuracy calculation unit 45 may calculate the autonomous movement accuracy based, for example, on the learning data from the learning unit 47 and the current position of the mobile body 10 estimated by the self-position estimation unit 37. In this case, the autonomous movement accuracy of the mobile body 10 is information that indicates the learning accuracy regarding autonomous movement.

[0089] Furthermore, the accuracy calculation unit 45 may calculate the autonomous movement accuracy by lowering the value when the likelihood of the self-position estimated by the self-position estimation unit 37 is low, or by lowering the value when the variance is large, using the variance of various sensors, etc. Furthermore, the accuracy calculation unit 45 may calculate the autonomous movement accuracy by lowering the value when the elapsed movement time of the autonomous movement mode is long, for example, using the elapsed movement time which is the operating state by the autonomous movement processing unit 43, or by lowering the value when the distance is large, depending on the distance between the destination sequence and the moving body 10. Also, the accuracy calculation unit 45 may calculate the autonomous movement accuracy by lowering the value when there are many obstacles, for example, according to the information of obstacles detected by the state detection unit 34.

[0090] The shooting control unit 33 performs shooting processing using the shooting device 12 while moving within the base (step S32). The image generation unit 46 generates a virtual route image to be displayed on the captured image acquired by the shooting processing in step S32 (step S33). The route image is generated based, for example, on the current position of the moving object 10 estimated by the self-position estimation unit 37, and the position information and status for each destination series stored in the destination series management DB 3002. The image generation unit 46 also generates a captured display image in which the route image generated in step S33 is drawn on the captured image taken in step S32 (step S34).

[0091] Furthermore, the image generation unit 46 generates a map display image (step S35) on the map image read out in step S12, which includes a current location display image showing the current location (self-location) of the moving object 10 estimated by the self-location estimation unit 37, and a sequence image showing the destination sequence received in step S16.

[0092] Note that the order of steps S31 to S35 may be reversed, or they may be performed in parallel. The mobile unit 10 continuously performs steps S31 to S35 while moving from base to base. Through the processes from steps S31 to S35, the mobile unit 10 generates various information to inform the operator whether the autonomous movement of the mobile unit 10 is working correctly.

[0093] Next, the transmitting / receiving unit 31 transmits notification information indicating the autonomous movement accuracy calculated in step S31, the captured display image data generated in step S34, and the map display image data generated in step S35 to the display device 50 (step S36). As a result, the transmitting / receiving unit 51 of the display device 50 receives the notification information, captured display image data, and map display image data transmitted from the mobile body 10.

[0094] Next, the display control unit 53 of the display device 50 displays the operation screen 400 on a display unit such as the display 106 (step S37). Figure 13 is a diagram showing an example of the operation screen. The operation screen 400 shown in Figure 13 is an example of a GUI for an operator to remotely control the mobile unit 10.

[0095] The operation screen 400 includes a map display image area 600 for displaying map display image data received in step S36, a shooting display image area 700 for displaying shooting display image data received in step S36, a notification information display area 800 for displaying notification information received in step S36, and a mode switching button 900 for accepting a switching operation to switch between autonomous movement mode and manual operation mode.

[0096] Of these, the map display image displayed in the map display image area 600 includes a current location display image 601 showing the current position of the mobile object 10, sequence images 611, 613, and 615 showing the sequence of destinations that constitute the mobile object 10's travel path, and a trajectory display image showing the trajectory of the mobile object 10's travel path, all superimposed on the map image. The map display image area 600 also includes a display selection button 605 that is pressed when zooming in or out of the displayed map image.

[0097] Series images 611, 613, and 615 display the destination series on the map image in a way that allows the operator to identify the movement history showing the locations already reached by the mobile object 10, the current destination, and future destinations. Of these, series image 611 shows the destination series that the mobile object 10 has already reached. Series image 613 shows the destination series that is the current destination of the mobile object 10. Furthermore, series image 615 shows the destinations that the mobile object 10 has not yet reached (future destinations). Series images 611, 613, and 615 are generated in the processing of step S35 based on the status of the destination series stored in the destination series management DB 3002.

[0098] The captured display image displayed in the captured display image area 700 includes route images 711, 713, and 715 that virtually show the movement path of the mobile body 10 generated in step S33. Route images 711, 713, and 715 display destination sequences corresponding to the locations of bases captured in the captured image, in a way that the operator can identify them as movement history showing locations already reached by the mobile body 10, the current destination, and future destinations. Of these, route image 711 shows the destination sequence that the mobile body 10 has already reached. Route image 713 shows the destination sequence that is the current destination of the mobile body 10. Furthermore, route image 715 shows the destination (future destination) that the mobile body 10 has not yet reached. Route images 711, 713, and 715 are generated in step S33 based on the status of the destination sequence stored in the destination sequence management DB 3002. Here, the map image and captured image are examples of images showing bases where the mobile body 10 is installed. Furthermore, the map display image shown in the map display image area 600 and the captured image displayed in the captured image area 700 are examples of base display images showing the movement path of the mobile body 10 in an image indicating a base. In addition, the captured image display area 700 may display images captured by the camera 12 as live streaming video delivered in real time via a computer network such as the Internet.

[0099] The notification information display area 800 displays information regarding the autonomous movement accuracy indicated in the notification information received in step S36. The notification information display area 800 includes a numerical display area 810 that displays the autonomous movement accuracy information as a numerical value (%), and a degree display area 830 that discretizes the numerical value indicating the autonomous movement accuracy and displays it as the degree of autonomous movement. The numerical display area 810 shows the numerical value of the autonomous movement accuracy calculated in the processing of step S31. The degree display area 830 sets a predetermined threshold for the numerical value of the autonomous movement accuracy and indicates the degree of autonomous movement accuracy ("high, medium, low") according to the value. Here, the numerical value indicating the autonomous movement accuracy shown in the numerical display area 810 and the degree of autonomous movement shown in the degree display area 830 are examples of notification information indicating the accuracy of autonomous movement. The notification information display area 800 only needs to include at least one of the numerical display area 810 and the degree display area 830.

[0100] The mode switching button 900 is an example of an operation receiving means that accepts a switching operation to switch between autonomous movement mode and manual operation mode. By selecting the mode switching button 900 using a predetermined input means, the operator can switch between the autonomous movement mode and manual operation mode of the mobile unit 10.

[0101] In the example shown in Figure 13, the operation screen 400 shows that the mobile object 10 is autonomously moving with the position of the sequence image 613 and the path image 713 as its current destination. The operation screen 400 also shows that the current autonomous movement accuracy of the mobile object 10 is "93.8%", which is a relatively high autonomous movement accuracy.

[0102] Figure 14 shows the state after the mobile unit 10 has moved from the state shown in Figure 13. In the operation screen 400 shown in Figure 14, the positions of the sequence image 613 and route image 713, which indicate the current destination, have changed because the mobile unit 10 has moved from the state shown in Figure 13. Also, in the operation screen 400 shown in Figure 14, the current autonomous movement accuracy of the mobile unit 10 is "87.9%", which is a decrease in the value of the autonomous movement accuracy from the state shown in Figure 13, and the degree of autonomous movement accuracy has also changed from "high" to "medium". By viewing the status of the bases shown in the map display image and base display image shown on the operation screen 400, as well as the change in autonomous movement accuracy shown in the notification information display area, the operator can decide whether or not to switch the mobile unit 10 between autonomous movement and manual operation.

[0103] Returning to Figure 12, the reception unit 52 accepts the selection of the mode switching button 900 on the operation screen 400 in response to the operator's input operation using an input means such as the pointing device 512 (step S38). For example, when the operator selects the mode switching button 900 (displaying "Switch to manual operation") in the state shown in Figure 15(A), the display of the mode switching button 900 (displaying "Resume autonomous movement") as shown in Figure 15(B) changes to that of the mode switching button 900 (displaying "Resume autonomous movement") as shown in Figure 15(B). In this case, the operator selects the mode switching button 900 in order to switch the operation mode of the mobile body 10 from autonomous movement mode to manual operation mode.

[0104] Then, the transmitting / receiving unit 51 transmits a mode switching request to the mobile body 10, indicating that it requests the mobile body 10 to switch between autonomous movement mode and manual operation mode (step S39). As a result, the transmitting / receiving unit 31 of the control device 30 installed in the mobile body 10 receives the mode switching request transmitted from the display device 50.

[0105] Next, in response to receiving the mode switching request in step S39, the control device 30 performs mode switching processing for the mobile body 10 (step S40).

[0106] (Switching between autonomous movement and manual operation) Here, we will explain in detail the mode switching process in step S40 using Figure 16. Figure 16 is a flowchart showing an example of the switching process between autonomous movement mode and manual operation mode in a mobile device.

[0107] First, if the control device 30 receives a mode switching request transmitted from the display device 50 by the transmitting / receiving unit 31 (YES in step S51), it proceeds to step S52. On the other hand, the control device 30 continues the processing in step S51 until it receives a mode switching request (NO in step S51).

[0108] Next, if the received mode switching request indicates a switch to manual operation mode (YES in step S52), the mode setting unit 42 moves the process to step S53. The movement control unit 41 stops the autonomous movement process of the mobile body 10 in response to a command to stop the autonomous movement process from the autonomous movement processing unit 43 (step S53). Then, the mode setting unit 42 switches the operation of the mobile body 10 from autonomous movement mode to manual operation mode (step S54). Then, the movement control unit 41 executes the movement of the mobile body 10 by manual operation in response to a drive command from the manual operation processing unit 44 (step S55).

[0109] On the other hand, if the received mode switching request does not indicate a switch to manual operation mode, that is, if it indicates a switch to autonomous movement mode, that is, if the switching request indicates a switch to autonomous movement mode (NO in step S52), the mode setting unit 42 moves the process to step S56. The mode setting unit 42 switches the operation of the mobile body 10 from manual operation mode to autonomous movement mode (step S56). Then, the movement control unit 41 executes the movement of the mobile body 10 by autonomous movement in response to the drive command from the autonomous movement processing unit 43 (step S57).

[0110] In this way, the display device 50 can enable the operator to appropriately determine whether to switch between autonomous movement and manual operation by displaying an operation screen 400 that includes notification information indicating the autonomous movement accuracy of the mobile body 10. Furthermore, the display device 50 can improve the operability when the operator switches between autonomous movement and manual operation by enabling the operator to switch between autonomous movement and manual operation using a mode switching button 900 on the operation screen 400 that displays the notification information indicating the autonomous movement accuracy. In addition, the mobile body 10 can perform movement control according to the operator's request by switching between the operating modes, autonomous movement mode and manual operation mode, in response to a switching request transmitted from the display device 50.

[0111] Furthermore, the mobile unit 10 may be configured not only to switch operating modes in response to switching requests transmitted from the display device 50, but also to switch from autonomous movement mode to manual operation mode when, for example, the value of the autonomous movement accuracy calculated by the accuracy calculation unit 45 falls below a predetermined threshold.

[0112] Furthermore, the display device 50 may not only display the operation screen 400 but also provide means for notifying the operator of the degree of autonomous movement accuracy. For example, the sound output unit 55 of the display device 50 may be configured to output a warning sound from the speaker 515 when the value of the autonomous movement accuracy falls below a predetermined threshold. Alternatively, the display device 50 may be configured to vibrate input means such as a controller used for manual operation of the mobile body when the value of the autonomous movement accuracy falls below a predetermined threshold.

[0113] Furthermore, the display device 50 may not directly display the autonomous movement accuracy on the operation screen 400, but may instead display a predetermined message based on the numerical value or degree of autonomous movement accuracy as notification information. In this case, the operation screen 400 may, for example, display a message prompting the operator to switch to manual operation if the numerical value or degree of autonomous movement accuracy falls below a predetermined threshold. Alternatively, the operation screen 400 may, for example, display a message prompting the operator to switch from manual operation to autonomous movement if the numerical value or degree of autonomous movement accuracy exceeds a predetermined threshold.

[0114] ○ Autonomous movement processing Next, the autonomous movement process of the mobile body 10, which is performed by the process shown in step S57, will be explained using Figure 17. Figure 17 is a flowchart showing an example of the autonomous movement process of the mobile body.

[0115] First, the destination setting unit 40 of the control device 30 equipped with the mobile body 10 sets the destination of the mobile body 10 based on the current position of the mobile body 10 estimated by the self-position estimation unit 37 and the route information stored in the route information management DB 3003 (see Figure 8) (step S71). Specifically, the destination setting unit 40 sets the position indicated by the destination series closest to the current position of the mobile body 10 estimated by the self-position estimation unit 37, from among the destination series indicated in the route information stored in the route information management DB 3003, as the destination. In the example shown in Figure 7, the position of the destination series with the status of the current destination, series ID "P003", is set as the destination. Then, the destination setting unit 40 generates a travel route to the set destination. The method of generating the travel route by the destination setting unit 40 is to connect the current position and the destination with a straight line, or to minimize the travel time while avoiding obstacles using captured images or obstacle information obtained by the state detection unit 34.

[0116] Then, the movement control unit 41 moves the mobile body 10 to the destination, which is set to follow the movement path generated in step S71. In this case, the movement control unit 41 autonomously moves the mobile body 10 in response to a drive command from the autonomous movement processing unit 43. The autonomous movement processing unit 43, for example, executes autonomous movement based on the learning data which is the result of imitation learning by the learning unit 47 (step S72).

[0117] Then, the movement control unit 41 terminates processing when the moving body 10 reaches its final destination or when autonomous movement by the autonomous movement processing unit 43 is interrupted (YES in step S73). Interruption of autonomous movement occurs, for example, when the mode setting unit 42 switches from autonomous movement mode to manual mode in response to a request to switch from autonomous movement mode to manual operation mode, as shown in Figure 16. On the other hand, the movement control unit 41 continues the autonomous movement processing in step S72 (NO in step S73) until it detects that the moving body 10 has reached its final destination or that autonomous movement by the autonomous movement processing unit 43 has been interrupted.

[0118] Thus, when operating in an autonomous movement mode set in response to a switching request from the operator, the mobile unit 10 can perform autonomous movement using the generated route information and the learned data acquired during manual operation mode. Furthermore, by learning about autonomous movement using various data acquired in manual operation mode, the mobile unit 10 can perform autonomous movement using the learned data and improve the accuracy of the mobile unit 10's autonomous movement.

[0119] ○Manual operation processing Next, the manual operation process of the mobile body 10 performed by the process shown in step S55 will be described using Figures 18 and 19. Figure 18 is a sequence diagram showing an example of the manual operation process of the mobile body.

[0120] First, the reception unit 52 of the display device 50 receives a manual operation command in response to the operator's input to the operation command input screen 450 as shown in Figure 19 (step S91). Figure 19 is a diagram showing an example of the operation command input screen. The operation command input screen 450 shown in Figure 19 displays icons for remotely controlling the mobile body 10. The operation command input screen 450 is displayed on the operation screen 400, for example, when the operation mode of the mobile body 10 is set to manual operation mode. The operation command input screen 450 includes a movement instruction key 455 that is pressed when requesting horizontal movement of the mobile body 10 (forward, backward, right turn, left turn), and a speed bar 457 that displays the movement speed indicating the state of the mobile body 10's movement speed. When the operator remotely controlling the mobile body 10 using the display device 50 selects a movement instruction key 455, the reception unit 52 receives a manual operation command for the selected movement instruction key 455.

[0121] In Figure 19, an example is shown in which the movement of the mobile body 10 is remotely controlled by accepting a selection for the movement instruction key 455 displayed on the operation command input screen 450. However, the movement of the mobile body 10 may be controlled using a keyboard or a dedicated controller such as a gamepad equipped with a joystick. Furthermore, if the operator selects "backward (↓)" while the mobile body 10 is moving forward during the operation of the movement instruction key 455, the captured image may be switched to a rear view of the mobile body 10, and the mobile body 10 may be made to move backward (reverse) from that point. In addition, the transmission of manual operation commands from the display device 50 to the mobile body 10 may be performed via a managed cloud platform such as AWS IoT Core (trademark).

[0122] Next, the transmitting / receiving unit 51 transmits the manual operation command received in step S91 to the mobile body 10 (step S92). As a result, the transmitting / receiving unit 31 of the control device 30 on the mobile body 10 receives the manual operation command transmitted from the display device 50. The manual operation processing unit 44 of the control device 30 then outputs a drive command based on the manual operation command received in step S92 to the movement control unit 41. The movement control unit 41 executes the movement process of the mobile body 10 in response to the drive command from the manual operation processing unit 44 (step S93). The learning unit 47 also performs imitation learning (machine learning) of the movement path corresponding to the manual operation by the manual operation processing unit 44 (step S94). For example, the learning unit 47 performs imitation learning of the movement path related to autonomous movement based on the captured images acquired during the movement operation in manual operation mode by the manual operation processing unit 44 and the detection data from the state detection unit 34. The learning unit 47 may be configured to perform imitation learning of the movement path using only the captured images acquired during manual operation, or it may be configured to perform imitation learning of the movement path using both the captured images and the detection data from the state detection unit 34. Furthermore, the captured images used for imitation learning by the learning unit 47 may be captured images acquired during autonomous movement in the autonomous movement mode by the autonomous movement processing unit 43.

[0123] Thus, when operating in a manual operation mode set in response to a switching request from the operator, the mobile unit 10 can move in accordance with manual operation commands from the operator. Furthermore, the mobile unit 10 can learn about autonomous movement using various data such as captured images acquired in manual operation mode.

[0124] ○A modified version of the operation screen○ Next, a modified example of the operation screen 400 displayed on the display device 50 will be described using Figures 20 to 25. Figure 20 is a diagram showing modified example 1 of the operation screen. In addition to the configuration of the operation screen 400, the operation screen 400A shown in Figure 20 displays notification information indicating autonomous movement accuracy in the map display image area 600 and the captured image display area 700.

[0125] The map display image displayed in the map display image area 600 of the operation screen 400A includes, in addition to the configuration displayed in the map display image area 600 of the operation screen 400, an accuracy display image 660 indicating the degree of autonomous movement accuracy on the map image. Similarly, the captured display image displayed in the captured display image area 700 of the operation screen 400A includes, in addition to the configuration displayed in the captured display image area 700 of the operation screen 400, an accuracy display image 760 indicating the degree of autonomous movement accuracy on the captured image. The accuracy display images 660 and 760 indicate the degree of autonomous movement accuracy by the size of the circle. For example, the circle size decreases as autonomous movement accuracy increases, and the circle size increases as autonomous movement accuracy decreases, indicating uncertainty in autonomous movement or self-position. Here, the accuracy display image 660 and the accuracy display image 760 are examples of notification information indicating the accuracy of autonomous movement. Furthermore, the accuracy display images 660 and 760 may be configured to indicate the degree of autonomous movement accuracy by changing the color of the circle according to the degree of autonomous movement accuracy.

[0126] The accuracy display image 660 is generated by drawing it on the map image in step S35 based on the numerical value of autonomous movement accuracy calculated by the accuracy calculation unit 45. Similarly, the accuracy display image 760 is generated by drawing it on the captured image in step S34 based on the numerical value of autonomous movement accuracy calculated by the accuracy calculation unit 45. The operation screen 400A displays a map display image with the accuracy display image 660 superimposed on the map image, and a captured display image with the accuracy display image 760 superimposed on the captured image.

[0127] In this way, the operation screen 400A displays an image indicating the autonomous movement accuracy on the map image and the captured image, allowing the operator to intuitively understand the current autonomous movement accuracy of the mobile body 10 while observing the movement status of the mobile body 10.

[0128] Figure 21 shows a modified example of the operation screen, part 2. In addition to the configuration of operation screen 400, operation screen 400B shown in Figure 21 displays notification information indicating autonomous movement accuracy in the map display image area 600 and the captured image display area 700, similar to operation screen 400A.

[0129] The map display image displayed in the map display image area 600 of the operation screen 400B includes, in addition to the configuration displayed in the map display image area 600 of the operation screen 400, an accuracy display image 670 indicating the degree of autonomous movement accuracy on the map image. Similarly, the captured display image displayed in the captured display image area 700 of the operation screen 400B includes, in addition to the configuration displayed in the captured display image area 700 of the operation screen 400, an accuracy display image 770 indicating the degree of autonomous movement accuracy on the captured image. The accuracy display images 670 and 770 display the degree of autonomous movement accuracy using contour lines. For example, the accuracy display images 670 and 770 show the degree of autonomous movement accuracy at each location on the map image and captured image using contour lines. Here, the accuracy display image 670 and the accuracy display image 770 are examples of notification information indicating the accuracy of autonomous movement. Furthermore, the accuracy display images 670 and 770 may be configured to indicate the degree of autonomous movement accuracy by changing the color of the contour lines according to the degree of autonomous movement accuracy.

[0130] The accuracy display image 670 is generated by drawing it on the map image in step S35 based on the numerical value of autonomous movement accuracy calculated by the accuracy calculation unit 45. Similarly, the accuracy display image 770 is generated by drawing it on the captured image in step S34 based on the numerical value of autonomous movement accuracy calculated by the accuracy calculation unit 45. The operation screen 400B displays a map display image with the accuracy display image 670 superimposed on the map image, and a captured display image with the accuracy display image 770 superimposed on the captured image.

[0131] In this way, the operation screen 400B displays contour lines indicating autonomous movement accuracy on the map image and captured image, making it clear which areas have low autonomous movement accuracy and visually assisting the operator in taking routes with high autonomous movement accuracy during manual operation. Furthermore, when using machine learning or the like to improve autonomous movement performance with each manual operation, the communication system 1 can expand the area where autonomous movement is possible by accumulating learning data when the operator looks at the contour line map indicating autonomous movement accuracy and manually moves to areas with low autonomous movement accuracy.

[0132] Figure 22 shows a modified version 3 of the operation screen. The operation screen 400C shown in Figure 22 has the same configuration as the operation screen 400, but in addition, the notification information display area 800 displays the degree of autonomous movement accuracy using facial images of different levels.

[0133] The notification information display area 800 of the operation screen 400C includes a degree display area 835 that shows the degree of autonomous movement as a facial image, in addition to the configuration displayed in the notification information display area 800 of the operation screen 400. Similar to the degree display area 830, the degree display area 835 discretizes a numerical value indicating autonomous movement accuracy and displays it as the degree of autonomous movement. The degree display area 835 sets a predetermined threshold for the numerical value of autonomous movement accuracy and switches the facial expression of the facial image according to the numerical value of autonomous movement accuracy calculated by the accuracy calculation unit 45. Here, the facial image shown in the degree display area 835 is an example of notification information indicating the accuracy of autonomous movement. Note that the degree display area 835 is not limited to a facial image, and may be configured to display a predetermined illustration or other image that allows the operator to recognize the degree of autonomous movement accuracy in stages.

[0134] Figure 23 shows a modified example of the operation screen, part 4. The operation screen 400D shown in Figure 23 displays the autonomous movement accuracy using the color of the screen frame, in addition to the configuration of the operation screen 400.

[0135] The operation screen 400D includes, in addition to the configuration of the operation screen 400, a screen frame display area 430 that converts the degree of autonomous movement accuracy into color and displays it as a screen frame. The screen frame display area 430 changes the color of the screen frame according to the degree of autonomous movement accuracy. The screen frame display area 430 sets a predetermined threshold for the autonomous movement accuracy value and changes the color of the screen frame according to the value of autonomous movement accuracy calculated by the accuracy calculation unit 45. For example, the screen frame display area 430 displays the screen frame in red when the autonomous movement accuracy is low, and in blue when the autonomous movement accuracy is high. Here, the color of the screen frame shown in the screen frame display area 430 is an example of notification information indicating the accuracy of autonomous movement. Note that the operation screen 400D may also be configured to change the color of the entire operation screen according to the degree of autonomous movement accuracy, not just the screen frame.

[0136] Figure 24 shows a modified example of the operation screen 5. In addition to the configuration of the operation screen 400, the operation screen 400E shown in Figure 24 displays the direction that the moving object 10 should face when manually operating in the map display image area 600 and the captured image display area 700.

[0137] The map display image displayed in the map display image area 600 of the operation screen 400E includes, in addition to the configuration displayed in the map display image area 600 of the operation screen 400, a direction indicator image 690 on the map image that shows the direction the mobile body 10 should face when manually operated with an arrow. Similarly, the shooting display image displayed in the shooting display image area 700 of the operation screen 400E includes, in addition to the configuration displayed in the shooting display image area 700 of the operation screen 400, a direction indicator image 790 on the shooting image that shows the direction the mobile body 10 should face when manually operated with an arrow. The direction to face when manually operated is, for example, a direction that indicates an area with high autonomous movement accuracy, and a direction that guides the mobile body 10 to a position where autonomous movement is likely to resume. Note that the direction indicator images 690 and 790 are not limited to displays using arrows, but any display configuration that allows the operator to identify the direction the mobile body 10 should face when manually operated is acceptable.

[0138] In this way, the operation screen 400E displays the direction in which the mobile object 10 should move during manual operation on the map image and the captured image, allowing the operator to visually understand the direction in which the mobile object 10 should move.

[0139] Figure 25 shows a modified version 6 of the operation screen. The operation screen 400F shown in Figure 25 does not display the map display image area 600 that was displayed in each of the operation screens described above, but instead displays the shooting display image area 700, the notification information display area 800, and the mode switching button 900.

[0140] Of these, the image displayed in the image display area 700 of the operation screen 400F displays the accuracy display image 760 shown on the operation screen 400B and the direction display image 690 shown on the operation screen 400E on the image. Also, unlike the operation screens described above, the image displayed on the operation screen 400F does not display the route images 711, 713, and 715 on the image. The notification information display area 800 and the mode switching button 900 have the same configuration as shown on the operation screen 400.

[0141] In this way, the operation screen 400F displays at least the captured image taken by the mobile body 10 and notification information indicating the autonomous movement accuracy of the mobile body 10, allowing the operator to understand the movement status of the mobile body 10 using only the minimum necessary information. In addition, the operation screen 400F may be configured to display, in addition to or instead of the elements shown in Figure 25, the elements displayed in the captured image display area 700 and the notification information display area 800 of each operation screen described above.

[0142] ● Effects of the embodiment As described above, the communication system 1 displays notification information indicating the autonomous movement accuracy of the mobile unit 10 using numerical values ​​or images on the operation screen used by the operator, making it easy for the operator to decide whether to switch between autonomous movement and manual operation. Furthermore, the communication system can improve the operability when the operator switches between autonomous movement and manual operation by allowing the operator to switch between autonomous movement and manual operation using the mode switching button 900 on the operation screen where the notification information indicating the autonomous movement accuracy is displayed.

[0143] Furthermore, the communication system 1 can switch between the autonomous movement mode and manual operation mode of the mobile body 10 in response to the operator's switching request, thereby enabling control of the switching between autonomous movement and manual operation of the mobile body 10 according to the operator's request. In addition, the communication system 1 allows the operator to appropriately determine the necessity of manual operation for the mobile body 10, which is learning about autonomous movement using captured images etc. acquired in manual operation mode.

[0144] Here, each of the above-described operation screens should be configured to display at least notification information indicating the autonomous movement accuracy of the mobile body 10 and a mode switching button 900 that accepts switching operations between autonomous movement mode and manual operation mode. Of these, the mode switching button 900 may be replaced by an input means such as the keyboard 511 of the display device 50 instead of being displayed on the operation screen. The communication system 1 may also be configured to include an external input means such as a dedicated button outside the display device 50 that accepts switching operations between autonomous movement mode and manual operation mode. In these cases, the input means such as the keyboard 511 of the display device 50 or the external input means such as a dedicated button outside the display device 50 is an example of an operation acceptance means. Furthermore, a display device 50 that displays an operation screen including the mode switching button 900, or a display device 50 that accepts switching operations using an input means such as a keyboard 511, or a system including the display device 50 and an external input means such as a dedicated button is an example of a display system in this embodiment. Furthermore, the operation reception means may include not only means for switching between autonomous movement mode and manual operation mode using a mode switching button 900, but also means for receiving operations to perform predetermined control of the mobile body 10.

[0145] ● Modified examples of embodiments ○Example 1○ Next, a modified example 1 of the communication system according to the embodiment will be described with reference to Figures 26 and 27. Note that components and functions identical to those in the above embodiment are denoted by the same reference numerals, and their descriptions are omitted. The communication system 1A according to Modified Example 1 is an example in which the calculation of the autonomous movement accuracy of the mobile body 10A and the generation of various display images to be displayed on the operation screen 400, etc., are performed by the display device 50A.

[0146] Figure 26 is a diagram showing an example of the functional configuration of a communication system according to Modification 1 of the embodiment. The display device 50A according to Modification 1 shown in Figure 26 includes, in addition to the configuration of the display device 50 shown in Figure 5, a precision calculation unit 56 and an image generation unit 57.

[0147] The accuracy calculation unit 56 is mainly implemented by the CPU 501 and calculates the accuracy of the autonomous movement of the mobile body 10A. The image generation unit 57 is mainly implemented by the CPU 501 and generates a display image to be shown on the display device 50A. The accuracy calculation unit 56 and the image generation unit 57 have the same configuration as the accuracy calculation unit 45 and the image generation unit 46 shown in Figure 5, respectively. Accordingly, the control device 30A that controls the processing or operation of the mobile body 10A according to the modified example 1 has a configuration that does not include the functions of the accuracy calculation unit 45 and the image generation unit 46.

[0148] Figure 27 is a sequence diagram showing an example of the switching process between autonomous movement and manual operation of a mobile body using the operation screen according to Modification 1 of the Embodiment. Similar to Figure 12, Figure 27 shows an example where the mobile body 10A has started autonomous movement within the base by the process shown in Figure 10.

[0149] First, the shooting control unit 33 of the control device 30A equipped with the mobile unit 10A performs shooting processing using the shooting device 12 while moving within the base (step S101). Then, the transmitting / receiving unit 31 transmits to the display device 50A the shooting image data captured in step S101, the map image data read out in step S12, the route information stored in the route information management DB 3003, the location information indicating the current location (self-location) of the mobile unit 10A estimated by the self-location estimation unit 37, and the learning data from the learning unit 47 (step S102). As a result, the transmitting / receiving unit 51 of the display device 50A receives various data and information transmitted from the mobile unit 10A.

[0150] Next, the accuracy calculation unit 56 of the display device 50A calculates the autonomous movement accuracy of the mobile body 10A (step S103). The accuracy calculation unit 45 calculates the autonomous movement accuracy based, for example, on the route information and position information received in step S102. Alternatively, the accuracy calculation unit 56 may calculate the autonomous movement accuracy based, for example, on the learning data and position information received in step S102.

[0151] Next, the image generation unit 57 generates a route image to be displayed on the captured image received in step S102 (step S104). The route image is generated, for example, based on the location information received in step S102, and the location information and status for each destination series indicated in the route information received in step S102. The image generation unit 57 also generates a captured display image on the captured image received in step S102, with the route image generated in step S104 drawn on it (step S105). Furthermore, the image generation unit 57 generates a map display image on the map image received in step S102, with a current location display image indicating the current location (self-location) of the mobile body 10A as indicated in the location information received in step S102, and a series image indicating the destination series as indicated in the route information received in step S102 drawn on it (step S106).

[0152] The details of the processes in steps S103, S104, S105, and S106 are the same as those in steps S31, S33, S34, and S35 shown in Figure 12, respectively. The order of the processes in steps S103 to S106 may be reversed, or they may be performed in parallel. The display device 50A receives the captured image data and the like transmitted from the mobile body 10A by the process in step S102, and continuously executes the processes in steps S103 to S106.

[0153] Next, the display control unit 53 displays the operation screen 400 shown in Figure 13, etc., on a display unit such as the display 106 (step S107). The display control unit 53 displays the information calculated or generated in steps S103 to S106 on the operation screen 400. Note that the display control unit 53 is not limited to the operation screen 400, but may be configured to display any of the operation screens 400A to 400F described above. The processing in steps S108 to S110 is the same as the processing in steps S38 to S40 shown in Figure 12, so the explanation is omitted.

[0154] Thus, even when the communication system 1A according to Modified Example 1 calculates the autonomous movement accuracy and generates various display screens in the display device 50A, the operation screen 400 including notification information indicating the autonomous movement accuracy can be displayed in the display device 50A, making it easy for the operator to decide whether to switch between autonomous movement and manual operation.

[0155] ○Differentiation Example 2○ Next, a modified example 2 of the communication system according to the embodiment will be described with reference to Figures 28 to 31. Note that components and functions identical to those in the above embodiment are denoted by the same reference numerals, and their descriptions are omitted. The communication system 1B according to modified example 2 is an example in which the information processing device 90 performs the calculation of the autonomous movement accuracy of the mobile body 10B and the generation of various display images to be shown on the operation screen 400, etc.

[0156] Figure 28 shows an example of the overall configuration of a communication system according to a modified example 2 of the embodiment. In addition to the configuration of the embodiment described above, the communication system 1B according to modified example 2 includes an information processing device 90 that can communicate with a mobile body 10B and a display device 50B via a communication network 100.

[0157] The information processing device 90 is a server computer that manages communication between the mobile device 10B and the display device 50B, performs various controls on the mobile device 10B, and generates various display screens to be displayed on the display device 50B. The information processing device 90 may consist of one server computer or multiple server computers. Furthermore, although the information processing device 90 is described as a server computer located in a cloud environment, it may also be a server located in an on-premises environment. Hereinafter, the hardware configuration of the information processing device 90 is the same as that of the display device 50 shown in Figure 4. For convenience, the hardware configuration of the information processing device 90 will be described below using the 900 series of reference numerals for the configuration shown in Figure 4.

[0158] Figure 29 is a diagram showing an example of the functional configuration of a communication system according to Modification 2 of the embodiment. The configuration of the display device 50B according to Modification 2 shown in Figure 29 is the same as the configuration of the display device 50 shown in Figure 5. Furthermore, the control device 30B that controls the processing or operation of the mobile body 10B according to Modification 2 does not have the functions of the map information management unit 35, the accuracy calculation unit 45, and the image generation unit 46, nor does it have the map information management DB 3001 that was built in the storage unit 3000.

[0159] The information processing device 90 includes a transmitting / receiving unit 91, a map information management unit 92, an accuracy calculation unit 93, an image generation unit 94, and a storage / reading unit 99. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 901 in accordance with an information processing device program deployed on the RAM 903. The information processing device 90 also has a storage unit 9000 constructed from a ROM 902, HD 904, or recording medium 921 shown in Figure 4.

[0160] The transmitting / receiving unit 91 is mainly implemented by the processing of the network interface 908 by the CPU 901, and transmits and receives various data or information with other devices or terminals.

[0161] The map information management unit 92 is primarily implemented by the processing of the CPU 901 and uses the map information management DB 9001 to manage map information showing the environmental map of the target location where the mobile unit 10B is installed. The map information management unit 92 manages map information showing, for example, environmental maps downloaded from an external server or environmental maps created by applying SLAM.

[0162] The accuracy calculation unit 93 is mainly implemented by the CPU 901 and calculates the accuracy of the autonomous movement of the mobile body 10B. The image generation unit 94 is mainly implemented by the CPU 301 and generates a display image to be displayed on the display device 50B. The accuracy calculation unit 93 and the image generation unit 94 have the same configuration as the accuracy calculation unit 45 and the image generation unit 46 shown in Figure 5, respectively.

[0163] The storage / reading unit 99 is primarily implemented by the CPU 901, and stores various data (or information) in the storage unit 9000 and reads various data (or information) from the storage unit 9000. The storage unit 9000 also contains a map information management database 9001. The map information management database 9001 is composed of the map information management tables shown in Figure 6.

[0164] Figure 30 is a sequence diagram showing an example of the process up to the start of movement of a mobile body according to a modified example 2 of the embodiment. First, the transmitting / receiving unit 51 of the display device 50B transmits a route input request to the information processing device 90 indicating that it is requesting input of the movement route of the mobile body 10 (step S201) in response to a predetermined input operation by the operator. This route input request includes a base ID that identifies the base where the mobile body 10B is located. As a result, the transmitting / receiving unit 91 of the information processing device 90 receives the route input request transmitted from the display device 50B.

[0165] Next, the map information management unit 92 of the information processing device 90 searches the map information management DB 9001 (see Figure 6) using the base ID received in step S201 as a search key, and reads out map information associated with the same base ID as the received base ID via the storage / reading unit 99 (step S202). The map information management unit 92 accesses the storage location indicated in the read map information and reads out the corresponding map image data.

[0166] Next, the transmitting / receiving unit 91 transmits map image data corresponding to the map information read in step S202 to the requesting display device 50B (step S203). As a result, the transmitting / receiving unit 51 of the display device 50B receives the map image data transmitted from the information processing device 90.

[0167] Next, the display control unit 53 of the display device 50B displays the route input screen 200 (see Figure 11), which includes the map image data received in step S203, on a display unit such as the display 506 (step S204). When the operator selects a predetermined location on the map image and presses the "Complete" button 210, the reception unit 52 accepts the input of destination sequences 250a to 250h, similar to step S15 in Figure 12 (step S205). The transmitting / receiving unit 51 transmits destination sequence data indicating the destination sequences 250a to 250h received in step S205 to the information processing device 90 (step S206). This destination sequence data includes location information indicating the location on the map image of the destination sequences 250a to 250h input in step S205. The transmitting / receiving unit 91 of the information processing device 90 transmits (transfers) the destination sequence data transmitted from the display device 50B to the mobile device 10B (step S207). As a result, the transmitting / receiving unit 31 of the control device 30B equipped with the mobile unit 10B receives the destination sequence data transmitted from the display device 50B. The processing of steps S208 to S212 thereafter is the same as the processing of steps S17 to S21 shown in Figure 10, so the explanation is omitted.

[0168] Figure 31 is a sequence diagram showing an example of the switching process between autonomous movement and manual operation of a mobile body using an operation screen according to a modified example of the embodiment 2. Similar to Figure 12, Figure 31 shows an example where the mobile body 10B has started autonomous movement within the base by the process shown in Figure 10.

[0169] First, the shooting control unit 33 of the control device 30B equipped with the mobile body 10B performs shooting processing using the shooting device 12 while moving within the base (step S231). Then, the transmitting / receiving unit 31 transmits to the information processing device 90 the shooting image data captured in step S231, route information stored in the route information management DB 3003, location information indicating the current position (self position) of the mobile body 10B estimated by the self position estimation unit 37, and learning data from the learning unit 47 (step S232). As a result, the transmitting / receiving unit 91 of the information processing device 90 receives various data and information transmitted from the mobile body 10B.

[0170] Next, the accuracy calculation unit 93 of the information processing device 90 calculates the autonomous movement accuracy of the mobile body 10B (step S233). The accuracy calculation unit 45 calculates the autonomous movement accuracy based, for example, on the route information and position information received in step S232. Alternatively, the accuracy calculation unit 56 may calculate the autonomous movement accuracy based, for example, on the learning data and position information received in step S232.

[0171] Next, the image generation unit 94 generates a route image to be displayed on the captured image received in step S232 (step S234). The route image is generated, for example, based on the location information received in step S232, and the location information and status for each destination series indicated in the route information received in step S232. The image generation unit 57 then generates a captured display image on the captured image received in step S232, with the route image generated in step S234 drawn on it (step S235). Furthermore, the image generation unit 94 generates a map display image on the map image read out in step S202, with a current location display image indicating the current location (self-location) of the mobile body 10B as indicated in the location information received in step S232, and a series image indicating the destination series as indicated in the route information received in step S232 drawn on it (step S236).

[0172] The details of the processes in steps S233, S234, S235, and S236 are the same as those in steps S31, S33, S34, and S35 shown in Figure 12, respectively. The order of the processes in steps S233 to S236 may be reversed, or they may be performed in parallel. The information processing device 90 receives the captured image data and the like transmitted from the mobile body 10 by the process in step S232, and continuously executes the processes in steps S233 to S236.

[0173] Next, the transmitting / receiving unit 91 transmits to the display device 50B notification information indicating the autonomous movement accuracy calculated in step S233, the captured display image data generated in step S235, and the map display image data generated in step S236 (step S237). As a result, the transmitting / receiving unit 51 of the display device 50B receives the notification information, captured display image data, and map display image data transmitted from the information processing device 90.

[0174] Next, the display control unit 53 of the display device 50B causes the operation screen 400 shown in Figure 13, etc., to be displayed on a display unit such as the display 106 (step S238). The display control unit 53 displays the data and information received in step S237 on the operation screen 400. Note that the display control unit 53 is not limited to the operation screen 400, but may be configured to display any of the operation screens 400A to 400F described above.

[0175] Next, similar to step S38 in Figure 12, the reception unit 52 accepts the selection of the mode switching button 900 on the operation screen 400 in response to an input operation by the operator using an input means such as the pointing device 512 (step S239). The transmitting / receiving unit 51 transmits a mode switching request to the information processing device 90 indicating that it requests a switch between the autonomous movement mode and manual operation mode of the mobile body 10B (step S240). The transmitting / receiving unit 91 of the information processing device 90 then transmits (transfers) the mode switching request sent from the display device 50B to the mobile body 10B (step S241). As a result, the transmitting / receiving unit 31 of the control device 30B equipped with the mobile body 10B receives the mode switching request sent from the display device 50B. Then, in response to the receipt of the mode switching request in step S241, the control device 30B executes the mode switching process for the mobile body 10B as shown in Figure 16 (step S242).

[0176] Thus, even when the information processing device 90 calculates the autonomous movement accuracy and generates various display screens, the communication system 1B according to the modified example 2 can display an operation screen 400 containing notification information indicating the autonomous movement accuracy on the display device 50B, allowing the operator to easily decide whether to switch between autonomous movement and manual operation.

[0177] Figure 32 is a diagram showing an example of the functional configuration of the communication system. In comparison with the functional configuration of the communication system shown in Figure 29, the display device 50 has the same configuration as the display device 50 shown in Figure 29. The control device 30C, which controls the processing or operation of the mobile body 10C, has the same configuration as the control device 30B shown in Figure 29, but with the destination series management unit 36, route information generation unit 38, route information management unit 39, and the destination series management DB 3002 and route information management DB 3003 that were built into the storage unit 3000 removed.

[0178] In the communication system 1C shown in Figure 32, the information processing device 90 supports cloud computing services such as AWS (trademark), and the communication system 1C allows the display device 50 and the mobile device 10C (control device 30C) to communicate via the information processing device 90 as indicated by arrows a and b. In addition, the functions of the destination series management unit 36, route information generation unit 38, route information management unit 39, destination series management DB 3002, and route information management DB 3003, which were removed from the control device 30B, have been transferred to the information processing device 90. In other words, the information processing device 90 has a transmitting / receiving unit 91, a map information management unit 92, an accuracy calculation unit 93, an image generation unit 94, a destination series management unit 95, a route information generation unit 96, and a route information management unit 97. Furthermore, the storage unit 9000 of the information processing device 90 contains the map information management DB 9001, the destination series management DB 9002, and the route information management DB 9003. The functions of the above-mentioned components transferred from the control device 30B (Figure 29) to the information processing device 90 are the same as those described in Figure 29, so their explanation will be omitted.

[0179] As described above, in the communication system 1C, communication between the display device 50 and the mobile device 10C (control device 30C) is performed via an information processing device 90 that supports cloud computing services. By using authentication processing by the cloud computing service during communication, the security of manual operation commands from the display device 50 and captured image data from the mobile device 10C can be enhanced. Furthermore, by placing the data generation and management functions in the information processing device 90 (cloud service), it becomes possible to share the same data at multiple locations, and it becomes possible to flexibly support not only P2P (Peer to Peer) communication (one-to-one direct communication) but also one-to-multi-location communication.

[0180] ●Summary● As described above, the display system according to one embodiment of the present invention is a display system for performing predetermined operations on a mobile body 10 (10A, 10B, 10C), and includes an operation receiving means (for example, a mode switching button 900) that receives a switching operation to switch between a manual operation mode in which the mobile body 10 (10A, 10B, 10C) is moved by manual operation and an autonomous movement mode in which the mobile body 10 (10A, 10B, 10C) is moved by autonomous movement, and a display control unit 53 (an example of a display control means) that displays notification information indicating the accuracy of autonomous movement. As a result, the display system according to one embodiment of the present invention makes it easy for the user to decide whether to switch between autonomous movement and manual operation, and improves the operability when the user switches between autonomous movement and manual operation.

[0181] Furthermore, when a switching operation to switch between manual operation mode and autonomous movement mode is received, the display system according to one embodiment of the present invention transmits a switching request between autonomous movement mode and manual operation mode to the mobile body 10 (10A, 10B, 10C), and switches the autonomous movement mode and manual operation mode of the mobile body 10 (10A, 10B, 10C) based on the transmitted switching request. In this way, the display system according to one embodiment of the present invention can control the switching between autonomous movement and manual operation of the mobile body 10 (10A, 10B, 10C) in response to the user's request.

[0182] Furthermore, in the display system according to one embodiment of the present invention, the notification information indicating the accuracy of autonomous movement is information indicating the learning accuracy of autonomous movement, and when the mobile body 10 (10A, 10B, 10C) switches from autonomous movement mode to manual operation mode, it becomes possible to perform learning for autonomous movement. As a result, the display system according to one embodiment of the present invention allows the operator to appropriately determine the necessity of learning for autonomous movement by manual operation.

[0183] Furthermore, a communication system according to one embodiment of the present invention is a communication system 1 (1A, 1B, 1C) comprising a display system that performs predetermined operations on a mobile body 10 (10A, 10B, 10C), and the mobile body 10 (10A, 10B, 10C), wherein the mobile body 10 (10A, 10B, 10C) receives a request from the display system to switch between autonomous movement mode and manual operation mode, sets either autonomous movement mode or manual operation mode based on the received switching request, and executes movement processing of the mobile body 10 (10A, 10B, 10C) based on the set mode. As a result, the communication system 1 (1A, 1B, 1C) can control the movement of the mobile body 10 (10A, 10B, 10C) in accordance with the user's request by having the mobile body 10 (10A, 10B, 10C) switch between autonomous movement mode and manual operation mode in response to the switching request sent from the display system.

[0184] Furthermore, in a communication system according to one embodiment of the present invention, when the manual operation mode is set, the mobile body 10 (10A, 10B, 10C) learns a movement path related to autonomous movement and calculates the accuracy of autonomous movement based on the learned data. Also, when the autonomous movement mode is set, the mobile body 10 (10A, 10B, 10C) moves autonomously based on the learned data. As a result, the communication system 1 (1A, 1B, 1C) can learn about autonomous movement using various data acquired in the manual operation mode of the mobile body 10 (10A, 10B, 10C), thereby enabling autonomous movement of the mobile body 10 (10A, 10B, 10C) using the learned data, and can also improve the accuracy of autonomous movement of the mobile body 10 (10A, 10B, 10C).

[0185] ●Additional Information● Each function of the embodiment described above can be realized by one or more processing circuits. Here, "processing circuit" in this embodiment includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), SOCs (System on a chip), GPUs (Graphics Processing Units), and conventional circuit modules designed to execute each of the functions described above.

[0186] Furthermore, the various tables in the embodiments described above may be generated by the learning effect of machine learning, and tables may not be used if the data of each related item is classified by machine learning. Here, machine learning is a technique for enabling computers to acquire human-like learning abilities, and it refers to a technique in which a computer autonomously generates algorithms necessary for judgments such as data identification from pre-incorporated training data, and applies these to new data to make predictions. The learning method for machine learning may be supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, or deep learning, or a combination of these learning methods may be used, and the learning method for machine learning is not limited.

[0187] While a display system, communication system, display control method, and program relating to one embodiment of the present invention have been described so far, the present invention is not limited to the embodiments described above. It can be modified to the extent that a person skilled in the art can conceive of it, such as by adding, changing, or deleting other embodiments, and any embodiment that achieves the function and effect of the present invention is included within the scope of the present invention. [Explanation of symbols]

[0188] 1,1A,1B,1C Communication System 100 Communication Networks 10,10A,10B,10C Mobile 30, 30A, 30B, 30C control devices 31 Transmitting and Receiving Unit (Example of Switching Request Receiving Means, Example of Precision Transmission Means) 37 Self-position estimation unit (an example of a self-position estimation means) 38 Route information generation unit (an example of route information generation means) 42 Mode setting section (an example of a mode setting means) 43. Autonomous movement processing unit (an example of movement processing means) 44 Manual operation processing unit (an example of a movement processing means) 45. Accuracy calculation unit (an example of a second accuracy calculation means) 46 Image generation unit 47. Learning Section (Examples of Learning Methods) 50,50A,50B display device 51 Transmitting / receiving unit (Example of switching request transmission means, example of acquisition means) 52 Reception Department 53 Display Control Unit (An example of display control means) 56. Accuracy Calculation Unit (An Example of the First Accuracy Calculation Means) 57 Image generation unit (an example of acquisition means) 90 Information Processing Equipment 91 Transmitter / Receiver 93 Accuracy Calculation Unit 94 Image generation unit 200 Route Input Screen 250 destination series 400, 400A, 400B, 400C, 400D, 400E, 400F Operation Screen 600 Map display image area 611,613,615 series images 650,660 precision display image 700 Captured display image area Route images 711, 713, 715 750,760 precision display image 800 Notification information display area 900 Mode switching button (an example of an operation acceptance method)

Claims

1. A display system for performing predetermined operations on a moving object, An operation receiving means that receives a switching operation to switch between a manual operation mode in which the mobile body is moved by manual operation and an autonomous movement mode in which the mobile body is moved by autonomous movement, A display control means for displaying notification information indicating the accuracy of the autonomous movement, Equipped with, The aforementioned autonomous movement is autonomous movement based on learning, The aforementioned notification information is information indicating the learning accuracy of the autonomous movement. Display system.

2. A display system for performing predetermined operations on a moving object, An operation receiving means that receives a switching operation to switch between a manual operation mode in which the mobile body is moved by manual operation and an autonomous movement mode in which the mobile body is moved by autonomous movement, A first accuracy calculation means calculates the accuracy of the autonomous movement based on position information indicating the current position of the moving body and path information indicating the movement path of the moving body, A display control means for displaying notification information indicating the accuracy of the autonomous movement, Equipped with, The display control means displays the accuracy of the autonomous movement calculated by the first accuracy calculation means as the notification information. Display system.

3. The display system according to claim 1, which enables learning for autonomous movement when the mobile body switches from the autonomous movement mode to the manual operation mode.

4. A display system according to claim 2, The first accuracy calculation means is a display system that calculates the accuracy of autonomous movement based on the position information, the path information, and learning data relating to the autonomous movement of the moving object.

5. The display control means displays a numerical value indicating the accuracy of the autonomous movement as notification information, according to claim 2 or 4.

6. The display control means displays an image indicating the accuracy of the autonomous movement as notification information, according to claim 2, 4, or 5.

7. The display system according to claim 6, further, The system includes an acquisition means for acquiring a base display image that shows the movement path of the moving object in an image indicating a base, The display control means is a display system that displays an image indicating the accuracy of the autonomous movement as notification information on the acquired base display image.

8. A display system according to any one of claims 1 to 7, further comprising: When the switching operation is accepted, the system includes a switching request transmission means that transmits a request to the mobile body to switch between the autonomous movement mode and the manual operation mode. A display system that switches between the autonomous movement mode and the manual operation mode of the mobile body based on the transmitted switching request.

9. A communication system comprising the display system described in claim 8 and the mobile body, The aforementioned moving body is A switching request receiving means that receives the switching request transmitted from the display system, A mode setting means that sets either the autonomous movement mode or the manual operation mode based on the received switching request, A movement processing means that performs movement processing of the moving body based on the set mode, A communication system equipped with [the following features].

10. A communication system according to claim 9, The aforementioned moving body further, A self-position estimation means for estimating the current position of the moving object, Route information generation means for generating route information indicating the movement path of the moving object, A second accuracy calculation means calculates the accuracy of autonomous movement based on the estimated current location information and the generated path information, The system includes a precision transmission means for transmitting notification information indicating the calculated precision of autonomous movement to the display system, The display control means of the display system is a communication system that displays the notification information transmitted from the mobile body.

11. A communication system according to claim 10, The aforementioned moving body further, When the manual operation mode is set by the mode setting means, the system includes a learning means for learning the movement path related to autonomous movement. The second accuracy calculation means is a communication system that calculates the accuracy of autonomous movement based on the learning data obtained by the learning means.

12. The communication system according to claim 11, wherein the movement processing means moves the moving body by autonomous movement based on the learning data from the learning means when the autonomous movement mode is set by the mode setting means.

13. A display control method executed by a display system that performs a predetermined operation on a moving object, An operation reception step that receives a switching operation to switch between a manual operation mode in which the mobile body is moved by manual operation and an autonomous movement mode in which the mobile body is moved by autonomous movement, A display control step that displays notification information indicating the accuracy of the autonomous movement, Execute The aforementioned autonomous movement is autonomous movement based on learning, The aforementioned notification information is information indicating the learning accuracy of the autonomous movement. Display control method.

14. A display control method executed by a display system that performs a predetermined operation on a moving object, An operation reception step that receives a switching operation to switch between a manual operation mode in which the mobile body is moved by manual operation and an autonomous movement mode in which the mobile body is moved by autonomous movement, A first accuracy calculation step calculates the accuracy of the autonomous movement based on position information indicating the current position of the moving body and path information indicating the movement path of the moving body. A display control step that displays notification information indicating the accuracy of the autonomous movement, the display control step that displays the accuracy of the autonomous movement calculated by the first accuracy calculation step as the notification information, A display control method that performs the following.

15. A display system that performs a predetermined operation on a moving object, An operation reception step that receives a switching operation to switch between a manual operation mode in which the mobile body is moved by manual operation and an autonomous movement mode in which the mobile body is moved by autonomous movement, A display control step that displays notification information indicating the accuracy of the autonomous movement, To execute The aforementioned autonomous movement is autonomous movement based on learning, The aforementioned notification information is information indicating the learning accuracy of the autonomous movement. program.

16. A display system that performs a predetermined operation on a moving object, An operation reception step that receives a switching operation to switch between a manual operation mode in which the mobile body is moved by manual operation and an autonomous movement mode in which the mobile body is moved by autonomous movement, A first accuracy calculation step calculates the accuracy of the autonomous movement based on position information indicating the current position of the moving body and path information indicating the movement path of the moving body. A display control step that displays notification information indicating the accuracy of the autonomous movement, the display control step that displays the accuracy of the autonomous movement calculated by the first accuracy calculation step as the notification information, A program that executes the command.

Images