Mobile apparatus, information processing apparatus, image capturing method, and recording medium

Abstract

A mobile apparatus to capture an image of an object at a preset position in autonomous travel includes an image processing unit to perform image processing on first image data to output coordinates of the object in the first image data. The first image data includes an image of the object and an image of surrounding of the object captured at the preset position in autonomous travel. The mobile apparatus further includes an image-capture condition correction unit to correct a condition of image capturing of the object based on the coordinates of the object in the first image data output by the image processing unit, and an image-capture control unit to capture an image of the object under the condition corrected by the image-capture condition correction unit.

Description

[DESCRIPTION][Title of Invention]MOBILE APPARATUS, INFORMATION PROCESSING APPARATUS, IMAGE CAPTURING METHOD, AND RECORDING MEDIUM [Technical Field]

[0001] The present disclosure relates to a mobile apparatus, an information processing apparatus, an image capturing method, and a recording medium.[Background Art]

[0002] Mobile apparatuses installed on a site such as a factory or a warehouse to autonomously travel the site are known.Such a mobile apparatus is used as, for example, an inspection robot or a service robot and may be referred to as a mobile robot. The mobile robot can perform a task such as inspecting a facility at a preset position in the site instead of a worker.Such tasks of inspecting a facility at a preset position in a site include capturing an image of an object to be inspected (for example, see Patent Literature (PTL) 1). PTL 1 discloses a technique of causing a mobile robot to sequentially travel inspection points following a travel route set in advance and temporarily stop at an inspection point to capture an image of an object to be inspected.[Citation List][Patent Literature]

[0003] [PTL 1]Japanese Unexamined Patent Application Publication No. 2022-058164[Summary of Invention][Technical Problem]

[0004] In the related art, the object may fall outside the angle of view of the mobile robot. It is difficult for the mobile robot to accurately stop at the position (location) set in advance due to errors in detecting the position and posture of the mobile robot. Therefore, even when, for example, the conditions for capturing an image of the object to be inspected are the same, the object may be out of the angle of view or may be significantly shifted from the center of the angle of view depending on the stop position and posture of the mobile robot and the direction of the optical axis of the image-capturing device.

[0005] In view of the foregoing, an object of the present disclosure is to provide a technique for a mobile apparatus to reliably capture an image of an object even when the mobile apparatus does not accurately stop at a preset position due to an error in the position or posture of the mobile apparatus.[Solution to Problem]

[0006] In one aspect, mobile apparatus to capture an image of an object at a preset position in autonomous travel includes an image processing unit to perform image processing on first image data to output coordinates of the object in the first image data. The first image data includes an image of the object and an image of surrounding of the object captured at the preset position in autonomous travel. The mobile apparatus further includes an imagecapture condition correction unit to correct a condition of image capturing of the object based on the coordinates of the object in the first image data output by the image processing unit, and an image-capture control unit to capture an image of the object under the condition corrected by the image-capture condition correction unit.Another aspect concerns an information processing apparatus to communicate via a network with a mobile apparatus that captures an image of an object at a preset position in autonomous travel. The information processing apparatus includes a communication unit to receive first image data including an image of the object and an image of surrounding of the object captured at the preset position in autonomous travel, an image processing unit to perform image processing on the first image data to output coordinates of the object in the first image data, and an image-capture condition correction unit to correct a condition of image capturing of the object based on the coordinates of the object in the first image data output by the image processing unit. The communication unit transmits, to the mobile apparatus, a request to capture the object under a corrected condition corrected by the image-capture condition correction unit.Another aspect concerns a method for capturing an image of an object at a preset position with a mobile apparatus performing autonomous travel. The method includes performing image processing on first image data to output coordinates of the object in the first image data including an image of the object and an image of surrounding of the object captured at the preset position in autonomous travel, correcting a condition for image capturing of the object based on the coordinates of the object in the output first image data, and capturing an image of the object under the corrected condition.Another aspect concerns a recording medium storing a plurality of program codes which, when executed by one or more processors, causes the processors to perform the method described above.[Advantageous Effects of Invention]

[0007] The present disclosure provides a technique for the mobile apparatus to reliably capture an image of an object even when the apparatus does not accurately stop at a preset position due to an error in the position or posture of the apparatus.[Brief Description of Drawings]

[0008] A more complete appreciation of embodiments of the present disclosure and many of the attendant advantages and features thereof can be readily obtained and understood from the following detailed description with reference to the accompanying drawings.[FIG. 1]FIG. 1 is a diagram illustrating a process in which a mobile robot according to an embodiment of the present disclosure captures an image of an inspected object.[FIG. 2]FIG. 2 is a diagram illustrating an example of an overall configuration of a communication system according to some embodiments.[FIG. 3A]FIGS. 3A is an exterior perspective view of an example of the mobile robot.[FIG. 3B]FIG. 3B is an exterior perspective view of an example of the mobile robot.[FIG. 4]FIG. 4 is a block diagram illustrating an example of a hardware configuration of the mobile robot.[FIG. 5]FIG. 5 is a block diagram illustrating an example of a hardware configuration of a display apparatus.[FIG. 6]FIG. 6 is a block diagram illustrating an example of a functional configuration of a communication system according to a first embodiment.[FIG. 7]FIG. 7 is a conceptual diagram illustrating an example of a destination-candidate management table.[FIG. 8]FIG. 8 is a conceptual diagram illustrating an example of a route-information management table.[FIG. 9]FIG. 9 is a conceptual diagram illustrating an example of a map -information management table.[FIG. 10]FIG. 10 is a conceptual diagram illustrating an example of a correction-information management table.[FIG. 11]FIG. 11 is a conceptual diagram illustrating an example of an image data management table. [FIG. 12]FIG. 12 is a sequence diagram illustrating an example process of registering a destination candidate.[FIG. 13]FIG. 13 is a diagram illustrating an example of an operation screen.[FIG. 14]FIG. 14 is a sequence diagram illustrating an example process of a manual operation for the mobile robot.[FIG. 15]FIG. 15 is a flowchart of an example process of registering a destination candidate according to the travel state of the mobile robot.[FIG. 16]FIG. 16 is a sequence diagram illustrating an example process of setting a destination of the mobile robot.[FIG. 17]FIG. 17 is a diagram illustrating an example of a selection screen.[FIG. 18]FIG. 18 is a diagram illustrating another example of the selection screen.[FIG. 19]FIG.19 is a flowchart of an example process of autonomous travel performed by the mobile robot.[FIG. 20]FIG. 20 is a flowchart of an example process in which the mobile robot captures a high- magnification image of an inspected object and an equal magnification (lx) image of a facility in registration of a destination candidate under the control of an operator.[FIG. 21 A]FIG. 21 A illustrates an example of the high-magnification image stored in registration of a destination candidate.[FIG. 21B]FIG. 2 IB illustrates an example of the lx image stored in registration of the destination candidate.[FIG. 22]FIG. 22 is a flowchart of an example process in which a size determination unit determines the qualities around the center of a lx image.[FIG. 23]FIG. 23 is a flowchart of an example process in which the mobile robot captures an image of an inspected object in autonomous travel.[FIG. 24]FIG. 24 is a diagram illustrating calculation of change amounts in pan and tilt in step S 112 inFIG. 23.[FIG. 25]FIG. 25 is a flowchart illustrating in detail an example process of template matching in step SI 14 in FIG. 23.[FIG. 26A]FIG. 26A illustrates an example of images used in description of the process of FIG. 25.[FIG. 26B]FIG. 26B illustrates another example of images used in description of the process of FIG. 25.[FIG. 26C]FIG. 26C illustrates another example of images used in description of the process of FIG. 25.[FIG. 26D]FIG. 26D illustrates another example of images used in description of the process of FIG. 25.[FIG. 26E]FIG. 26E illustrates another example of images used in description of the process of FIG. 25.[FIG. 27]FIG. 27 is a diagram illustrating the process of step S122 in FIG. 25.[FIG. 28]FIG. 28 is a diagram illustrating an example of an inspection result display screen displayed on the display apparatus by the operator.[FIG. 29]FIG. 29 is a diagram illustrating an example of a correction screen displayed by the display apparatus.[FIG. 30]FIG. 30 is a flowchart of an example process performed by the display apparatus when the operator corrects the image of the inspected object.[FIG. 31]FIG. 31 is a flowchart of an example process in which an image processing unit cuts out a template image using correction information.[FIG. 32]FIG. 32 is a flowchart of another example process in which the image processing unit cuts out a template image using the correction information.[FIG. 33]FIG. 33 is a block diagram illustrating an example of a functional configuration of a communication system according to a second embodiment.[FIG. 34]FIG. 34 is a flowchart of an example process in which a mobile robot of FIG. 33 captures a high-magnification image of an inspected object and a lx image in registration of a destination candidate under the control of the operator.[FIG. 35]FIG. 35 is a sequence diagram illustrating an example process in which the mobile robot captures an image of an inspected object in autonomous travel.[FIG. 36]FIG. 36 is a block diagram illustrating an example of a hardware configuration of a head mounted display (HMD) used as the display apparatus.[FIG. 37]FIG. 37 is a diagram illustrating an example of image data displayed by the HMD used as the display apparatus.The accompanying drawings are intended to depict embodiments of the present disclosure and should not be interpreted to limit the scope thereof. The accompanying drawings are not to be considered as drawn to scale unless explicitly noted. Also, identical or similar reference numerals designate identical or similar components throughout the several views. [Description of Embodiments]

[0009] In describing embodiments illustrated in the drawings, specific terminology is employed for the sake of clarity. However, the disclosure of this specification is not intended to be limited to the specific terminology so selected and it is to be understood that each specific element includes all technical equivalents that have a similar function, operate in a similar manner, and achieve a similar result.Referring now to the drawings, a mobile apparatus (mobile robot), an information processing apparatus, and an image capturing method performed by the mobile apparatus according to embodiments of the present disclosure are described below. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0010] First EmbodimentOverview of Process by Mobile RobotA mobile robot according to the present embodiment detects an object to be inspected from image data captured at that point by template matching when reaching a preset inspection point. The object to be inspected may be referred to as “inspected object” or “target object” in the following description. The mobile apparatus corrects the image-capture condition based on the acquired coordinates of the inspected object, and captures an image of the inspected object under the corrected image-capture condition.

[0011] FIG. 1 is a diagram illustrating a process in which the mobile robot according to the present embodiment captures an image of an inspected object.(1) The mobile robot 10 has reached a preset inspection point. In FIG. 1, an inspected object150 is present in front of the mobile robot 10. The mobile robot 10 captures an equal magnification (lx) image 153 (an example of first image data) of a facility at the inspection point.(2) The mobile robot 10 cuts (extracts) an image of the inspected object from a prestored lx image 151 of the facility including the inspected object 150. The mobile robot 10 sets the image extracted from the prestored lx image 151 to be a template image 152. The lx image151 is an example of second image data in which the inspected object and the surroundings of the inspected object are captured. It is assumed that the position of the inspected object 150 in the lx image 151 is known. The template image 152 may be cut out in advance.(3) The mobile robot 10 performs template matching of the lx image 153 with the template image 152.(4) The mobile robot 10 calculates coordinates at which the matching rate is maximum.(5) The mobile robot 10 changes the pan and tilt so that the optical axis of a camera 111 is directed to the coordinates. Accordingly, the inspected object 150 is captured at the center of the angle of view.(6) The mobile robot 10 sets the camera 111 to have a zoom ratio stored in advance and captures the inspected object to generate an image 154 of the inspected object.

[0012] As described above, the mobile robot according to the present embodiment stops at a preset position in the execution of inspection and searches for an inspected object by template matching from a wide-angle lx image. Accordingly, the mobile robot determines the optimal image capturing direction and acquires an enlarged image of the inspected object.

[0013] TermsLocation information is information specifying the location of the mobile robot 10 with reference to the origin, and posture is information specifying the direction of the mobile robot 10 with reference to a reference direction. A posture may be a posture in a three-dimensional space or a two-dimensional space.

[0014] Autonomous travel refers to an action in which the mobile robot 10 autonomously travels on or along a travel route traveled in the teaching travel. Teaching travel refers to the travel, under the manual operation of a controller by the operator, of the mobile robot 10 on the travel route on which the mobile robot 10 is to autonomously travel so that the mobile robot 10 stores the travel route.

[0015] An inspection point is a position set in advance for the mobile robot 10 to execute an inspection task. In the present embodiment, an inspection task is described as an example, but the task may be any task including capturing an image of an object at a predetermined position. The inspected object may be any object and may be a person present in the same place not limited to an object.

[0016] The inspected object is an object to be inspected. In the present embodiment, the inspected object is an object whose image is to be captured. Various things may be the inspected objects. For example, the inspected object is the state of a part such as a meter or a valve of a plant, or the status of people entering.

[0017] The image-capture conditions are generally the capturing directions of pan and tilt. Further, other conditions such as the zoom ratio, f-number (the ratio of the focal length to the diameter of the entrance pupil), exposure, shutter speed, whether to perform autofocus, and whether toperform auto white balance may be included. Moving the camera to the left and right is called “panning,” and moving the camera up and down is called “tilting.” In the present embodiment, the term “pan” is also used as an angle in the horizontal direction, and the term “tilt” is also used as an angle in the vertical direction.

[0018] Image processing refers to processing for outputting the coordinates of an inspected object from a lx image using the template image. In the present embodiment, template matching is described as an example of image processing, but a model generated by machine learning may be used.

[0019] System ConfigurationFIG. 2 is a diagram illustrating an example overall configuration of a communication system. The communication system 1 illustrated in FIG. 2 includes the mobile robot 10, which receives remote control by an operator to travel in a certain site. The certain site where the mobile robot 10 is located may be referred to as an operation site.

[0020] The communication system 1 includes the mobile robot 10 located at the operation site, a route management server 50, and a display apparatus 60. The mobile robot 10, the route management server 50, and the display apparatus 60 communicate with each other via a communication network 100. The route management server 50 may be omitted.The communication network 100 includes, for example, the Internet, a mobile communication network, a local area network (LAN), or combinations thereof. The communication network 100 may include, in addition to a wired network, a wireless network in compliance with such as third generation (3G), fourth generation (4G), and fifth generation (5G) for mobile communications; Wireless Fidelity (WI-FI); Worldwide Interoperability for Microwave Access (WIMAX); and Long Term Evolution (LTE).

[0021] The mobile robot 10 is a robot installed in the operation site to autonomously travel from one location to another location at the operation site. The autonomous travel includes operation of autonomously travelling the operation site using the result of imitation learning (machine learning) of the routes traveled at the operation site in the past. The autonomous travel may be operation of autonomously travelling the operation site on a travel route set in advance, or operation of autonomously travelling the operation site using a technology such as line tracing. The mode in which the mobile robot 10 travels by autonomous travel may be called an “autonomous travel mode” in the following description. Further, the mobile robot 10 may travel under the manual remote control by an operator at a remote location. The mode in which the mobile robot 10 travels under the remote control by an operator may be called a “manual operation mode” in the following description. In other words, the mobile robot 10 may travel in the operation site while switching between the autonomously travel and the travel under the manual control of the operator. The mobile robot 10 may execute a presettask such as inspection, maintenance, transportation, or light work while travelling the operation site. In this disclosure, the mobile robot 10 is a robot in a broad sense and may be any robot that can perform both of travel under the remote control by an operator and autonomous travel. Examples of the mobile robot 10 include an automobile that can travel while switching between automatic driving and manual driving according to an operation by an operation at a remote location. Examples of the mobile robot 10 further include flying objects such as a drone, a multicopter, and an unmanned flying object.

[0022] The operation site where the mobile robot 10 is installed is, for example, the monitored area is an area (also referred to as a target site, or simply a site) in which the mobile robot 10 is installed. Examples of the monitored area include an outdoor area such as a business place, a factory, a chemical plant, a construction site, a substation, a farm, a field, a cultivated land, or a disaster site; and an indoor area such as an office, a school, a factory, a warehouse, a commercial facility, a hospital, or a nursing facility. In other words, the operation site may be any location where there is a need for the mobile robot 10 to perform tasks that have been manually performed by humans.

[0023] The route management server 50 is a server computer that stores and manages information on a route on which the mobile robot 10 autonomously travels. The route management server 50 stores and manages a captured image transmitted from the mobile robot 10, and transmits the captured image to the display apparatus 60 used by the operator.

[0024] The route management server 50 may be implemented by a single computer or multiple computers to which divided functions or means are feely allocated. All or a part of the functions of the route management server 50 may be implemented by a server computer residing in a cloud environment or a server computer residing in an on-premises environment. The route management server 50 is an example of an “information processing apparatus.”

[0025] The display apparatus 60 may be implemented by a computer such as a laptop personal computer (PC), operated by an operator residing at a control site different from the operation site, to perform predetermined operations to the mobile robot 10.The operator performs operations, for example, to control the mobile robot 10 to move or execute a predetermined task via an operation screen displayed on the display apparatus 60 at the control site such as an office. For example, the operator remotely controls the mobile robot 10 while viewing an image of the operation site displayed at the display apparatus 60.

[0026] Although one mobile robot 10 and one display apparatus 60 are connected to each other via the communication network 100 in FIG. 2, the display apparatus 60 may be connectable to multiple mobile robots 10 located at one operation site or connectable to multiple mobile robots 10 located at different operation sites. Further, although the display apparatus 60 islocated at the remote control site different from the operation site in which the mobile robot 10 is installed in FIG. 2, the display apparatus 60 may be located at the operation site in which the mobile robot 10 is installed. Furthermore, the display apparatus 60 is not limited to a laptop PC. For example, the display apparatus 60 may be a desktop PC, a tablet communication terminal, a smartphone, or a wearable communication terminal.

[0027] Configuration of Mobile RobotReferring to FIGS. 3A and 3B, a specific configuration of the mobile robot 10 is described below. FIG. 3A is a schematic diagram illustrating a first configuration of the mobile robot. FIG. 3B illustrates a second configuration of the mobile robot as a variation.

[0028] The mobile robot 10 illustrated in FIG. 3 A includes a housing 11, a bumper 12, the camera 111, a capture position adjuster 14, a support 15, a movement mechanism 16 including movement mechanisms 16a and 16b, a self-position recognition device 17, and range sensors 112 and 113. The housing 11 contains a controller. The controller is placed in the body of the mobile robot 10 and controls processing or operation of the mobile robot 10. The bumper 12 is disposed on the front side of the housing 11 in the travel direction of the mobile robot 10 and serves as a buffer in the event of a collision of the mobile robot 10 with something. The bumper 12 may be provided not only on the front side but also on a lateral side or the rear side of the housing 11.

[0029] The camera 111 is a pan-tilt- zoom (PTZ) camera. A PTZ camera is a camera that allows remote control of panning (pan angle), tilting (tilt angle), and zooming (zoom ratio). The camera 111 captures images of subjects (objects) such as a person, an object, or a landscape at the operation site where the mobile robot 10 is installed, to obtain captured images.The camera 111 may be a digital camera (general-purpose image-capturing device), such as a digital single-lens reflex camera or a compact digital camera, which acquires a planar image (detailed image). The data relating to the captured image obtained by the camera 111 (i.e., captured image data) is transmitted to the display apparatus 60 via a communication session established by a server computer such as the route management server 50.

[0030] The capture position adjuster 14 is a movable device that adjusts the image capturing direction (orientation) of the camera 111. The capture position adjuster 14 drives and rotates to adjust the image capturing direction of the camera 111 and to adjust the amount of zoom (magnification) of imaging by the camera 111. The capture position adjuster 14 may be integral with the camera 111. In other words, the camera 111 may have the function of adjusting the capture position.

[0031] The captured image obtained by the camera 111 may be a video image or a still image, or both of the video image and the still image. The captured image obtained by the camera 111may include audio data together with image data. Further, the camera 111 may be a wide- angle image-capturing device that acquires a spherical (360-degree) panoramic image.

[0032] A wide-angle image-capturing device is, for example, a spherical image-capturing device that captures two hemispherical images of a subject and generates a spherical (panoramic) image from the two hemispherical images. The wide-angle image-capturing device may be, for example, a wide-angle camera or a stereo camera that acquires a wide-angle image having an angle of view equal to or larger than a predetermined value. In other words, the wide-angle image-capturing device is an image-capturing means that acquires an image (a spherical image or a wide-angle image) captured using a lens having a focal length shorter than a predetermined value.

[0033] The mobile robot 10 may include multiple cameras 111. In such a case, the mobile robot 10 may include, as the cameras 111, both a wide-angle image -capturing device that captures a wide-angle image of a subject and a general-purpose image-capturing device that captures a part of the image captured by the wide-angle image-capturing device to acquire a detailed image (planar image).

[0034] The camera 111 may include a thermal image-capturing device that images far-infrared rays (infrared light) or a special camera such as an infrared camera that images near-infrared rays (infrared light). The camera 111 being a thermal image-capturing device acquires a captured image (thermography) by detecting far-infrared rays emitted from the subject, and the subject is recognizable from the captured image. The camera 111 being an infrared camera acquires a captured image (infrared image) in which the subject is imaged without being affected by disturbance light in the visible light wavelength band, and the subject is recognizable from the captured image.

[0035] The support 15 is for securing (fixing) the camera 111 and the capture position adjuster 14 on the mobile robot 10 (housing 11). The support 15 may be a pole attached to the housing 11 or a mount attached to the housing 11.

[0036] The movement mechanism 16 (16a and 16b) is a unit that causes the mobile robot 10 to move and includes wheels, travel motors, travel encoders, steering motors, and steering encoders. The mobile robot 10 is controlled by a known technique. The mobile robot 10 receives a travel instruction from, for example, an operator operating the display apparatus 60, and the movement mechanism 16 causes the mobile robot 10 to move or travel based on the received travel instruction. The movement mechanism 16 may be of a bipedal type or a single-wheel type. The shape of the mobile robot 10 is not limited to vehicle-like shapes such as that illustrated in FIGS. 3A and 3B. For example, the mobile robot 10 may be shaped like a crawler (caterpillar) robot, a bipedal walking humanoid robot, or a robot that mimics acreature or a particular character.

[0037] The self-position recognition device 17 includes cameras 17f, 17b, 171, and 17r to capture images of subject in the front (f), back (b), left (1), and right (r) directions of the mobile robot 10, respectively. The data relating to the captured image obtained by the cameras 17f, 17b, 171, and 17r (i.e., captured image data) is transmitted to the display apparatus 60 via a communication session established by a server computer such as the route management server 50.

[0038] The self-position recognition device 17 is not limited to a device employing the cameras 17f to 17r. For example, the self-position recognition device 17 may include a global positioning system (GPS) sensor and simultaneous localization and mapping (SLAM) device using light detection and ranging (LIDAR). Alternatively, the self -position recognition device 17 may include a magnetic induction device to be used in an environment in which, for example, a magnetic tape is laid on a traveling route.

[0039] As described above, since the cameras 17f to 17r are provided as the self -position recognition device 17 on four sides of the front, back, left, and right sides of the mobile robot 10, not only images captured in the front and back direction but also images captured in the left and right direction are transmitted to the display apparatus 60. Since the captured images in the left and right directions are transmitted, the operator viewing the display apparatus 60 can easily find the widening of the road width, which increases the recognition accuracy of an intersection.

[0040] The range sensors 112 and 113 are detection sensors to detect obstacles in the surroundings while the mobile robot 10 travels. Each of the range sensors 112 and 113 may be an image sensor such as a stereo camera or a camera with an area sensor in which photoelectric conversion elements are arranged in a plane. Alternatively, each of the range sensors 112 and 113 may be a distance measuring sensor such as a time of flight (TOF) sensor, a LIDAR sensor, or a radar sensor.

[0041] A variation of the configuration of the mobile apparatus is described below with reference to FIG. 3B. In a mobile robot 10a illustrated in FIG. 3B, the camera 17a is disposed on an upper portion of the support 15. The camera 17a is a wide-angle image-capturing device such as an omnidirectional camera that can capture an image of the surroundings of the mobile robot 10. In this case as well, not only the images captured in the front and back directions but also the images captured in the left and right directions are transmitted to the display apparatus 60. Accordingly, the operator viewing the display apparatus 60 can easily find the widening of the road width. Thus, the recognition accuracy of an intersection increases.

[0042] The mobile robot 10 may include various sensors to detect information on the surroundings ofthe mobile robot 10 in addition to the above components. Examples of various sensors include a barometer, a thermometer, a photometer, a motion sensor (human presence sensor), a gas sensor, an odor sensor, and an illuminometer. The mobile robot 10 may further include a movable arm that performs an additional operation other than traveling.

[0043] Hardware ConfigurationReferring to FIGS. 4 and 5, a description is given of a hardware configuration of the mobile robot 10 according to the present embodiment. FIG. 4 is a block diagram illustrating the hardware configuration of the mobile robot 10 according to the present embodiment.

[0044] As illustrated in FIG. 4, the mobile robot 10 includes a central processing unit (CPU) 101, a memory 102, an auxiliary memory 103, the camera 111, the range sensor 112 for horizontal detection, the range sensor 113 for oblique detection, a navigation satellite system 114, an inertial measurement unit (IMU) 115, a battery 121, a start switch 142, an emergency stop switch 143, motor drivers 122a and 122b (left and right), travel motors 132a and 132b (left and right), brake drivers 123a and 123b (left and right), brake motors 133a and 133b (left and right), and a power switch 141.

[0045] The CPU 101 controls the entire operation of the mobile robot 10. The memory 102 is a temporary storage area for the CPU 101 to execute programs such as a travel program.

[0046] The camera 111 includes a spherical camera, a stereo camera, and an infrared camera. The range sensor 112 for horizontal direction detection and the range sensor 113 for oblique direction detection are, for example, LiDAR sensors.

[0047] The navigation satellite system 114 receives radio waves from a satellite and measures the position of the mobile robot 10 on the earth based on the received result. The navigation satellite system 114 is also referred to as a GNSS.

[0048] The IMU 115 is an inertial measurement sensor and estimates the position and posture (translational motion and rotational motion in orthogonal triaxial directions) of the mobile robot 10 by an accelerometer and an angular speed sensor. The IMU 115 is an example of an internal sensor.

[0049] The battery 121 is a power source for the mobile robot 10 to travel. The start switch 142 is a switch for starting the mobile robot 10. The emergency stop switch 143 is a switch for stopping the mobile robot 10 as desired.

[0050] The motor drivers 122a and 122b are drivers for the travel motors 132a and 132b, respectively. The brake drivers 123a and 123b are drivers for the brake motors 133a and133b, respectively. The power switch 141 is a switch to turn on or off the power of the mobile robot 10.

[0051] The travel program to be executed on the mobile robot 10 in the present embodiment may be recorded, in a file format installable or executable by a computer, on a computer-readable recording medium, such as a compact disc-read only memory (CD-ROM), a flexible disk (FD), a compact disc-recordable (CD-R), or a digital versatile disk (DVD).

[0052] Alternatively, the travel program executed on the mobile robot 10 according to the present embodiment may be stored in a computer connected to a network such as the Internet. The travel program may be allowed to be downloaded by the mobile robot 10 via a network. Alternatively, the travel program executed on the mobile robot 10 according to the present embodiment can be provided or distributed via a network such as the Internet. The travel program executed on the mobile robot 10 according to the present embodiment may be prestored in, for example, a ROM.

[0053] Display ApparatusFIG. 5 is a block diagram illustrating a hardware configuration of an example of the route management server 50 and the display apparatus 60 according to the present embodiment. As illustrated in FIG. 5B, each of the route management server 50 and the display apparatus 60 is implemented by a computer 500 and includes a CPU 501, a ROM 502, a RAM 503, a hard disk (HD) 504, a hard disk drive (HDD) controller 505, a display 506, an external device connection interface (FF) 508, a network FF 509, a bus line 510, a keyboard 511, a pointing device 512, a digital versatile disc rewritable (DVD-RW) drive 514, and a medium FF 516.

[0054] The CPU 501 controls the entire operation of the computer 500. The ROM 502 stores programs, such as an initial program loader (IPL), for driving the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various kinds of data such as a program. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various kinds of information such as a cursor, a menu, a window, characters, or an image. The external device connection FF 508 is an interface for connecting to various external devices. Examples of the external device include, but are not limited to, a universal serial bus (USB) memory and a printer. The network FF 509 is an interface for performing data communication via a network. The bus line 510 may be an address bus or a data bus, which electrically connects various elements such as the CPU 501 illustrated in FIG. 5.

[0055] The keyboard 511 is a kind of input device including multiple keys used for inputting items such as characters, numerical values, or various instructions. The pointing device 512 is a kind of input device used to, for example, select various instructions, execute variousinstructions, select a target for processing, and move a cursor. The DVD-RW drive 514 reads and writes various data from and to a DVD-RW 513 serving as a removable recording medium according to the present embodiment. The DVD-RW drive 514 is not limited to the drive for DVD-RWs and may be, for example, a drive for a digital versatile disc-recordable (DVD-R). The medium I / F 516 controls the reading or writing (storing) of data from or to a recording medium 515 such as a flash memory.

[0056] Functional ConfigurationReferring to FIG. 6 to FIG. 9, a functional configuration of the communication system 1 according to the present embodiment is described below. FIG. 6 is a block diagram illustrating the functional configuration of the communication system 1 according to the present embodiment. FIG. 6 illustrates some of the apparatuses and the communication terminal (the display apparatus 60) illustrated in FIG. 2 relating to the processing or operation to be described below.

[0057] Functional Configuration of Mobile Robot (Controller)The mobile robot 10 includes a communication unit 31, a determination unit 32, an imagecapture control unit 33, a state detection unit 34, a location-information acquisition unit 35, a destination-candidate acquisition unit 36, a route-information generation unit 37, a routeinformation management unit 38, a destination setting unit 39, a travel control unit 40, an image recognition unit 41, a mode setting unit 42, an autonomous travel unit 43, a manual operation processing unit 44, a task execution unit 45, a map-information management unit 46, a first display processing unit 47, a learning unit 48, and a storing-reading unit 49. These are units of functions or means implemented or caused to function by one or more of the hardware elements illustrated in FIG. 4 operating in accordance with the instructions of the CPU 101 according to the travel program loaded to the memory 102. The mobile robot 10 further includes a storage unit 3000 implemented by the memory 102 illustrated in FIG. 4.

[0058] The communication unit 31 has the function of transmitting and receiving various types of data or information to and from other apparatuses or communication terminals via the communication network 100.

[0059] The determination unit 32 is implemented by processing of the CPU 101 and executes various determinations. The image-capture control unit 33 is implemented by processing of the CPU 101 for, for example, the camera 111 and controls imaging processing of the camera 111.

[0060] Regarding the imaging process, the image-capture control unit 33 includes an image processing unit 33a, an image-capture condition correction unit 33b, a correction-information management unit 33c, and a size determination unit 33d. The image processing unit 33a performs template matching on the lx image captured in the executing of a task using thetemplate image, and outputs the coordinates on the lx image that matches most the template image. The image-capture condition correction unit 33b corrects the image-capture conditions (pan and tilt) based on the coordinates of the inspected object in the lx image identified by the image processing unit 33 a. The correction-information management unit 33c manages (stores and reads information in and from) a correction-information management table. The size determination unit 33d extracts feature points in the lx image and determines the size of the template image so that the number of feature points is equal to or greater than a threshold.

[0061] The state detection unit 34 is implemented by the processing of the CPU 101 for, for example, the camera 111 and the range sensors 112 and 113, and detects the state of the mobile robot 10 or the surroundings of the mobile robot 10 using various sensors. For example, the state detection unit 34 measures the distance to an object (obstacle) present around the mobile robot 10 and outputs the measured distance as distance data. Further, the state detection unit 34 may use the distance data to acquire data indicating the location of the mobile robot 10 based on a match with an environment map by applying SLAM. SLAM is a technology that allows simultaneous processing of self-localization and environment map generation. The state detection unit 34 further detects the direction to which the mobile robot 10 is heading or faces using, for example, the IMU 115.

[0062] The location-information acquisition unit 35 is implemented by the processing of the CPU 101 for the navigation satellite system 114 and acquires location information indicating the current location of the mobile robot 10 using the navigation satellite system 114. For example, the location-information acquisition unit 35 acquires coordinate information indicating the latitude and the longitude of the current location of the mobile robot 10 using the navigation satellite system 114.

[0063] The destination-candidate acquisition unit 36 is implemented by, for example, the processing of the CPU 101 and acquires an image of a candidate of the destination to which the mobile robot 10 travels (may be referred to as a “destination candidate image” in the following description). Specifically, the destination-candidate acquisition unit 36 acquires an image captured under the control of the image-capture control unit 33 as a destination candidate image. For example, the captured image is an image of a partial area of the site where the mobile robot 10 is installed.

[0064] The route-information generation unit 37 is implemented by, for example, the processing of the CPU 101 and generates route information indicating a travel route of the mobile robot 10. The route-information generation unit 37 generates route information indicating a route from the current location to the final destination (that is, travel destination) based on the location of the destination candidate selected by the operator of the mobile robot 10. Example methodsof generating the route information include a method of connecting the waypoints 8 from the current location to the final destination with a straight line, and a method of minimizing the travel time by avoiding an obstacle using a captured image or information of the obstacle obtained by the state detection unit 34.

[0065] The route-information management unit 38 is implemented by, for example, the processing of the CPU 101 and stores the route information generated by the route -information generation unit 37 for management in a route-information management database (DB) 3002.

[0066] The destination setting unit 39 is implemented by, for example, the processing of the CPU 101 and sets a travel destination of the mobile robot 10. For example, based on the current location of the mobile robot 10 acquired by the location-information acquisition unit 35 and the route information generated by the route-information generation unit 37, the destination setting unit 39 sets one of destination candidates selected by the operator of the mobile robot 10, as the travel destination to which the mobile robot 10 heads next.

[0067] The travel control unit 40 is implemented by, for example, the processing of the CPU 101 for the motor drivers 122a and 122b and controls the travel of the mobile robot 10 by driving the travel motors 132a and 132b. The travel control unit 40 controls the mobile robot 10 to travel, for example, according to a drive instruction from the autonomous travel unit 43 or the manual operation processing unit 44.

[0068] The image recognition unit 41 is implemented by, for example, the processing of the CPU 101 and performs image recognition on a captured image acquired by the image-capture control unit 33. For example, the image recognition unit 41 performs image recognition to determine whether a specific subject is captured in the acquired captured image. The specific subject is, for example, an obstacle on the travel route or around the travel route of the mobile robot 10, an intersection such as a crossroad or an L-shaped road, or a sign or a signal at the site.

[0069] The mode setting unit 42 is implemented by, for example, the processing of the CPU 101 and sets an operation mode of the travel of the mobile robot 10. The mode setting unit 42 sets either an autonomous travel mode in which the mobile robot 10 autonomously travels or a manual operation mode in which the mobile robot 10 travels under the control operations manually made by the operator.

[0070] The autonomous travel unit 43 is implemented by, for example, the processing of the CPU 101 and controls the autonomous travel of the mobile robot 10. For example, the autonomous travel unit 43 outputs an instruction for driving the mobile robot 10 to the travel control unit 40, such that the mobile robot 10 travels on the travel route indicated by the route informationgenerated by the route-information generation unit 37.

[0071] The manual operation processing unit 44 is implemented by, for example, the processing of the CPU 101 and controls the processing instructed by the manual operation for the mobile robot 10. For example, the manual operation processing unit 44 outputs an instruction for driving the mobile robot 10 to the travel control unit 40 according to a manual operation command transmitted from the display apparatus 60.

[0072] The task execution unit 45 is implemented by, for example, the processing of the CPU 101 and causes the mobile robot 10 to execute a preset task when the mobile robot 10 reaches the destination or in response to a request from the operator. Examples of the preset task executed by the task execution unit 45 include processing of capturing images for inspection of a facility at the site and performing light work using a movable arm.

[0073] The map-information management unit 46 is implemented by, for example, the processing of the CPU 101 and manages map information representing an environment map of the operation site where the mobile robot 10 is installed, using a map-information management DB 3003.

[0074] The first display processing unit 47 is implemented by, for example, the processing of the CPU 101 and generates a display image (screen image) to be displayed on the display apparatus 60. For example, the first display processing unit 47 performs processing on the captured image acquired by the image-capture control unit 33 to generate a display image to be displayed on the display apparatus 60.

[0075] The storing-reading unit 49 is implemented by, for example, the processing of the CPU 101 and stores various data or information in the storage unit 3000 or reads various data or information from the storage unit 3000.

[0076] Destination-Candidate Management TableFIG. 7 is a conceptual diagram illustrating a destination-candidate management table according to the present embodiment. The destination-candidate management table is a table for managing one or more candidates for the destination of the mobile robot 10. The storage unit 3000 stores a destination-candidate management DB 3001 in the form of the destinationcandidate management table as illustrated in FIG. 7.

[0077] The destination-candidate management table stores destination candidate data for each site identifier (ID) for identifying the site where the mobile robot 10 is installed. In the destination candidate data, a candidate ID for identifying a destination candidate, location information indicating the location of the destination candidate, captured image data obtainedby capturing an image of an inspected object at the destination candidate, and an X value (parameter) are associated with one another. The location information is coordinate information including the latitude and longitude that indicate the location of the destination candidate at the site, and the posture of the mobile robot 10 at the time of the image capturing. The attitude may be a clockwise or counterclockwise rotation angle relative to a reference direction. In a case where the mobile robot 10 is a flying object such as a drone, the location information includes information such as the speed, posture, or altitude of the flying object in addition to the coordinate information indicating the latitude and longitude. The captured image data is an electronic file of image data captured by the mobile robot 10 at the location of the destination candidate at the site. The captured image data includes a high- magnification image of an inspected object captured at high magnification; pan, tilt, and zoom (P, T, Z) at the capturing of the high-magnification image, a lx image of the facility including the inspected object captured at equal magnification; and pan, tilt, and zoom (P, T, Z) at capturing the lx image. In the table, any information for accessing the captured image, such as a storage location, may be used.The destination candidate is a candidate for a relay point (referred to as a destination) on a route along which the mobile robot 10 travels. In the present embodiment, a destination which the mobile robot 10 actually visits to execute a task is referred to as an inspection point. The candidate of the travel destination of the mobile robot 10 includes not only a candidate of the destination of the mobile robot 10 but also a candidate of the place to be excluded from the travel route of the mobile robot 10.

[0078] The X value is a parameter for determining the range (size) of the template image cut out from the lx image in the destination-candidate management DB 3001 when the task is executed. A detailed description thereof will be given later.

[0079] Route-Information Management TableFIG. 8 is a conceptual diagram illustrating a route-information management table according to the present embodiment. The route-information management table is a table for managing route information indicating a travel route of the mobile robot 10. The storage unit 3000 stores a route-information management DB 3002 in the form of the route-information management table as illustrated in FIG. 8.

[0080] The route-information management table stores, for each site ID identifying the site where the mobile robot 10 is installed, a route ID for identifying a travel route of the mobile robot 10 and route information indicating the travel route of the mobile robot 10 in association with one another. The route information indicates the travel route of the mobile robot 10 for reaching next destinations one by one in order. The route information is generated by the route-information generation unit 37, for example, when the mobile robot 10 starts traveling.

[0081] Map-Information Management TableFIG. 9 is a conceptual diagram illustrating a map-information management table according to the present embodiment. The map information management table is a table for managing map information, which is information on an environment map of the operation site where the mobile robot 10 is installed. The storage unit 3000 stores a map-information management DB 3003 in the form of the map-information management table as illustrated in FIG. 9.

[0082] The map information management table stores map information in which the site ID and the site name for identifying the operation site where the mobile robot 10 is installed, and the storage location of the environment map of the operation site are associated with one another. The storage location is, for example, destination information for accessing the storage area of the mobile robot 10 or an external server in which the environment map is stored, and is represented by a uniform resource locator (URL) or a uniform resource identifier (URI).

[0083] Correction-Information Management TableFIG. 10 is a conceptual diagram illustrating an example of the correction-information management table. The correction-information management table indicates the coordinates on the lx image captured in the execution of a task, matching the center of the inspected object in order to keep the inspected object within the angle of view. The storage unit 3000 stores a correction-information management DB 3004 in the form of the correctioninformation management table as illustrated in FIG. 10. The correction-information management table includes items of a correction ID, an inspection result ID, a correction value, and an inspection point ID. The correction ID is identification information that is numbered not to overlap another correction ID when the image of the inspected object is corrected. The inspection result ID is identification information that is numbered not overlap with another inspection result ID for a series of inspections performed by the mobile robot 10 while traveling along the travel route. The correction value is coordinates (pixel value) at which the center of the inspected object is captured in the lx image of the facility captured in the execution of a task. The correction value is determined by an operation of the operator. The inspection point ID is identification information of a destination (that is, an inspection point) at which the mobile robot 10 has captured an image of the inspected object in the execution of the task. If the inspection point ID is known, the position and posture of the mobile robot 10 at the inspection point are known (FIG. 7). The candidate ID in the destination-candidate management table of FIG. 7 is used as the inspection point ID.

[0084] Image Data Management DBFIG. 11 is a conceptual diagram illustrating an example of an image data management table. The image data management table is a table for managing image data captured by the mobile robot 10 at a target point (inspection point) on the travel route when a task is executed. The storage unit 3000 stores an image data management DB 3005 in the form of the image datamanagement table as illustrated in FIG. 11. The image data management table associates the inspection point ID, the image-capture time, the lx image of the facility, and the image of the inspected object with the inspection result ID. One travel route generally includes multiple inspection points (destinations). In FIG. 11, the image-capture time, the lx image of the facility, and the image of the inspected object are stored for each inspection point. The image of the inspected object is captured at a high magnification.

[0085] Since the situation at the image capturing affects the accuracy of template matching for the lx image, situation information indicating the situation at the image capturing other than the image image-capture time is preferably stored. The situation at the image capturing is, for example, weather, season, or ambient brightness. In other words, the situation information includes information indicating the weather under which the image was captured, information indicating the season in which the image was captured, or information indicating the ambient brightness. Further, the conditions (pan, tilt, and zoom ratio) under which the image of the inspected object is captured and the lx image of the facility may be stored.

[0086] Functional Configuration of Display ApparatusReferring back to FIG. 6, a description is given below of a functional configuration of the display apparatus 60. The display apparatus 60 includes a communication unit 51, an operation reception unit 52, a display control unit 53, a determination unit 54, a manualoperation command generation unit 55, an autonomous-travel request generation unit 56, a second display processing unit 57, and a storing-reading unit 59. These units are functions implemented by or means caused to function by one or more of the elements illustrated in FIG. 5 operating according to the instructions from the CPU 501 executing a display apparatus control program loaded to the RAM 503. The display apparatus 60 further includes a storage unit 5000 implemented by the ROM 502, HD 504, or a recording medium 515 illustrated in FIG. 5.

[0087] The communication unit 51 is implemented by, for example, the processing of the CPU 501 in relation to the network I / F 509 and transmits or receives various types of data or information to or from another device or communication terminal.

[0088] The operation reception unit 52 is implemented by, for example, the processing of the CPU 501 in relation to the keyboard 511 or the pointing device 512 and receives various selections or inputs from the operator. The display control unit 53 is implemented by, for example, the processing of the CPU 501 and controls the display 506 to display various screens. The determination unit 54 is implemented by the processing of the CPU 501 and performs various determinations.

[0089] The manual-operation command generation unit 55 is implemented by, for example, theprocessing of the CPU 501 and generates a manual operation command (instruction) for moving the mobile robot 10 by a manual operation in response to an input operation performed by the operator.

[0090] The autonomous -travel request generation unit 56 is implemented by, for example, the processing of the CPU 501 and generates an autonomous travel request to cause the mobile robot 10 to autonomously travel. The autonomous-travel request generation unit 56 transmits an autonomous travel request to the mobile robot 10, for example, based on information on the destination candidate selected by the operator.

[0091] The second display processing unit 57 is implemented by, for example, the processing of the CPU 501 and generates a display image to be displayed on a display such as the display 506. The second display processing unit 57 performs processing, for example, on the captured image acquired by the mobile robot 10 to generate a display image to be displayed on the display apparatus 60. The function of at least one of the first display processing unit 47 included in the mobile robot 10 and the second display processing unit 57 included in the display apparatus 60 suffices for the communication system 1.

[0092] The storing-reading unit 59 is implemented by, for example, the processing of the CPU 501 and stores various data or information in the storage unit 5000 or reads various data or information from the storage unit 5000.

[0093] Processes or OperationDestination-Candidate Registration ProcessFirst, referring to FIGS. 12 to 15, a description is given below of a process of registering a destination candidate of the mobile robot 10 according to the present embodiment. FIG. 12 is a sequence diagram illustrating the process of registering a destination candidate according to the present embodiment.

[0094] The display apparatus 60 starts controlling the mobile robot 10 in response to a predetermined input operation by the operator (S 11). The communication unit 51 transmits an operation start request to the mobile robot 10 (S 12). Accordingly, the communication unit 31 of the mobile robot 10 receives the operation start request transmitted from the display apparatus 60.

[0095] Subsequently, the image-capture control unit 33 starts image capturing by the camera 111 (S 13). The image-capture control unit 33 acquires image data captured by the camera 111. In the present embodiment, moving images are acquired by the image capturing in S 13. The communication unit 31 transmits the data of captured images acquired in S13 to the display apparatus 60 (S14). Thus, the communication unit 51 of the display apparatus 60 receives the captured image data transmitted from the mobile robot 10.

[0096] Subsequently, the display control unit 53 of the display apparatus 60 displays an operation screen 200 including the captured image data received in S14 on the display such as the display 506 (S15).

[0097] FIG. 13 is a diagram illustrating an example of the operation screen. The operation screen 200 illustrated in FIG. 13 is a display screen for the operator to remotely control the mobile robot 10.

[0098] The operation screen 200 includes a site display area 210 and a site display area 230. In the site display area 210, captured image data (a planar image) captured by the camera 111 and received in S14 is displayed. In the site display area 230, captured image data (special image) captured by the camera 111 and received in S 14 is displayed. The special image displayed in the site display area 230 is an omnidirectional image of the site captured by the camera 111 as described above. Examples of the omnidirectional image include a spherical image, a wide- angle view image, or a hemispherical image. Alternatively, the camera 111 may combine images captured while the camera 111 rotates, so as to obtain an omnidirectional image as the special image. In the site display area 230, further, a mobile robot display image 235 indicating the presence of the mobile robot 10 is superimposed on the special image. In the operation screen 200, further, coordinate information represented by latitude (Lat) and longitude (Lon) is displayed as the location information indicating the current location of the mobile robot 10. In a case where the mobile robot 10 is a flying object such as a drone, the operation screen 200 may display other information such as the speed and the posture (position) or altitude of the flying object, in addition to the coordinate information represented by the latitude and longitude. The operation screen 200 may display the captured images obtained by the camera 111 as live streaming images that are distributed in real-time through a computer network such as the Internet.

[0099] The operation screen 200 further includes an operation icon 250 for allowing the operator to remotely control the mobile robot 10. The operation icon 250 includes multiple direction instruction buttons each of which is pressed to request movement of the mobile robot 10 in a certain horizontal direction (e.g., forward, backward, right rotation, or left rotation). The operator can remotely control the mobile robot 10 by selecting the operation icon 250 while viewing the planar image displayed in the site display area 210 and the special image such as the spherical image displayed in the site display area 230.In the present embodiment, the travel of the mobile robot 10 is remotely controlled by receiving the selection on the operation icon 250 displayed on the operation screen 200. Alternatively, the travel of the mobile robot 10 may be controlled by a dedicated controller, such as a keyboard or a game pad including a joystick.

[0100] The operation screen 200 further includes a destination-candidate registration button 270, which is pressed to register a destination candidate using the planar image displayed in the site display area 210 or the special image displayed in the site display area 230. In the following description, a destination candidate is registered in response to the pressing of the destination-candidate registration button 270 by the operator viewing the special image displayed in the site display area 230. Similar processing is performed also in a case where a destination candidate is registered in response to the pressing of the destination-candidate registration button 270 by the operator viewing the planar image displayed in the site display area 210. The operation screen 200 further includes a destination setting button 290, which is pressed to set a destination of the mobile robot 10.

[0101] In S16, the display apparatus 60 performs a manual operation processing for the mobile robot 10 using the operation screen 200 displayed in S15. Details of the processing of S16 will be described later.

[0102] When the operator presses the destination-candidate registration button 270 on the operation screen 200, the operation reception unit 52 receives a request to register, as a destination candidate, an area included in the special image displayed in the site display area 230 (S17). Then, the communication unit 51 transmits a destination-candidate registration request to the mobile robot 10 (S 18). The destination candidate registration request is, for example, a request to capture an image of the area included in the special image displayed in the site display area 230. Accordingly, the mobile robot 10 receives the destination-candidate registration request transmitted from the display apparatus 60.

[0103] Subsequently, the location-information acquisition unit 35 of the mobile robot 10 acquires location information indicating the current location of the mobile robot 10 using the navigation satellite system 114 (S19). Specifically, the location-information acquisition unit35 acquires coordinate information including the latitude and longitude of the current location of the mobile robot 10. In a case where the mobile robot 10 is a flying object such as a drone, the location-information acquisition unit 35 acquires, as the location information, information such as the speed and the posture or altitude of the flying object in addition to the coordinate information including the latitude and longitude. The destination-candidate acquisition unit36 acquires a captured image captured by the camera 111 at the current location of the mobile robot 10 as a destination candidate image (S20). In the present embodiment, as the destination candidate image, a still image is acquired by the destination-candidate acquisition unit 36.

[0104] Then, the storing-reading unit 49 stores the destination candidate data including the location information acquired in S19 and the captured image acquired in S20 in the destinationcandidate management DB 3001 (destination-candidate management table) illustrated in FIG.7 (S21). The storing-reading unit 49 registers, as the destination candidate data, the location information and the captured image each associated with a candidate ID for identifying the destination candidate. The candidate ID may be assigned automatically by the storingreading unit 49.

[0105] As described above, the communication system 1 displays the captured image captured by the mobile robot 10 at the display apparatus 60 operated by the operator who remotely controls the mobile robot 10. This allows the operator to remotely control the mobile robot 10 while visually checking the surroundings of the mobile robot 10 in real time. Further, the communication system 1 registers in advance a captured image obtained by capturing a specific area of the site, as a candidate of the destination for the autonomous travel of the mobile robot 10, in response to an input operation of the operator made in the manual operation for the mobile robot 10.

[0106] Manual Operation Process Subsequently, referring to FIGS. 14 and 15, a description is given in detail of the manual operation process for the mobile robot 10 in S 16 according to the embodiment. The operator performs manual operations to remotely control the mobile robot 10 to travel along a travel route. This operation is referred to as teaching travel of the travel route. The mobile robot 10 registers destination candidates in the teaching travel. FIG. 14 is a sequence diagram of the manual operation process for the mobile robot 10.

[0107] First, the operation reception unit 52 of the display apparatus 60 receives selection on the operation icon 250 on the operation screen 200 displayed in SI 5, according to an input operation of the operator (S31).The manual-operation command generation unit 55 generates a manual-operation command according to the selection on the operation icon 250 selected in S31 (S32). In the example of FIG. 13, since the operator selects the direction instruction button marked by an up arrow (indicating “forward”) on the operation icon 250, the manual-operation command generation unit 55 generates a command for causing the mobile robot 10 to move forward. Then, the communication unit 51 transmits the manual-operation command generated in S32 to the mobile robot 10 (S33). The mobile robot 10 receives the manual-operation command transmitted from the display apparatus 60. Regarding the input operation on the operation icon 250 by the operator, in a case where the operator selects the direction instruction button marked by a down arrow indicating “backward” while the mobile robot 10 is moving forward, the displayed captured image may be switched to a rear side image of the mobile robot 10, and the mobile robot 10 may move backward from that point in time. The display apparatus 60 may transmit the manual-operation command to the mobile robot 10, for example, via a managed cloud platform such as AWS IOT CORE.

[0108] Subsequently, the mode setting unit 42 sets the mobile robot 10 to operate in the manual operation mode (S34). Then, the manual operation processing unit 44 outputs an instruction to drive to the travel control unit 40 based on the manual-operation command received in S33. The travel control unit 40 controls the mobile robot 10 to travel in response to the instruction to drive from the manual operation processing unit 44 (S35).

[0109] Further, the learning unit 48 performs imitation learning (machine learning) of the travel routes traveled by the processing of the manual operation processing unit 44, according to the manual operation (S36). The learning unit 48 performs imitation learning of the travel routes, for example, based on the captured images acquired in the traveling in the manual operation mode by the manual operation processing unit 44 and the detection data obtained by the state detection unit 34.

[0110] Then, the mobile robot 10 executes registration of the destination candidate while traveling (S37). FIG. 15 is a flowchart of a process of registering a destination candidate according to the travel state of the mobile apparatus.

[0111] First, the determination unit 32 determines whether a preset task has been executed by the task execution unit 45 (S51). Specifically, the task execution unit 45 causes the mobile robot 10 to execute a preset task according to, for example, a task execution request from the operator or a schedule set in advance. Then, the determination unit 32 determines whether the preset task is executable under the control of the task execution unit 45. Examples of the preset task include capturing images for inspection of equipment at the site and performing light work using the movable arm.

[0112] The mobile robot 10 performs inspection work of an inspected object such as a meter or a valve, for example, when entering an inspection area while traveling the site. At this time, the mobile robot 10 stops traveling to capture an image of the inspected object. Such operation may be used to trigger the registration of the destination candidate. In the present embodiment, the lx image of the facility including the inspected object and the high- magnification image of the inspected object are captured in the execution of the task.

[0113] When determining that the preset task has been executed by the task execution unit 45 (YES in S51), the determination unit 32 proceeds the process to S56. By contrast, when determining that the preset task is not executed by the task execution unit 45 (NO in S51), the determination unit 32 proceeds the process to S52.

[0114] Next, the determination unit 32 determines whether the mobile robot 10 has stopped (S52). Specifically, when it is detected that driving control of the movement mechanism 16 by the travel control unit 40 is stopped, the determination unit 32 determines that the mobile robot 10has stopped. When the determination unit 32 determines that the mobile robot 10 has stopped (YES in S52), the determination unit 32 proceeds the process to S56. By contrast, when the determination unit 32 determines that the mobile robot 10 has not stopped (i.e., the mobile robot 10 is moving) (NO in S52), the determination unit 32 proceeds to S53.

[0115] The determination unit 32 then determines whether an intersection is detected near the mobile robot 10 (S53). Specifically, the image recognition unit 41 performs image recognition on a captured image acquired by the camera 111. Then, the determination unit 32 determines whether an intersection has been detected in the captured image as a result of the processing of the image recognition unit 41. The mobile robot 10 can acquire an image as viewed from different directions on the travel route at a time regardless of the travel direction, by acquiring a special image captured omnidirectionally at a distinctive location such as an intersection. The intersection is an example of a specific subject detected by the image recognition unit 41. The specific subject is not limited to an intersection but may be, for example, an obstacle on or around the travel route of the mobile robot 10, or a sign or a signal at the operation site. The specific subject differs depending on the type of the mobile robot 10, such that the specific subject differs between the case where the mobile robot 10 travels on a road surface and the case where the mobile robot 10 flies like a drone. The mobile robot 10 is preliminarily set with information on the specific subject to be detected.

[0116] When determining that an intersection has been detected near the mobile robot 10 (YES in S53), the determination unit 32 proceeds the process to S56. By contrast, when determining that an intersection has not been detected near the mobile robot 10 (NO in S52), the determination unit 32 proceeds the process to S54.

[0117] Subsequently, the determination unit 32 determines whether the current location of the mobile robot 10 is close to the destination candidate registered in the destination-candidate management DB 3001 (S54). Specifically, the determination unit 32 refers to the location information of the destination candidates stored in the destination-candidate management DB 3001 and determines whether there is a destination candidate close to the current location of the mobile robot 10 acquired by the location-information acquisition unit 35. The determination of whether the current location is close to the destination candidate is made based on, for example, the difference in value between the location information of the destination candidate preliminarily set and the location information indicating the current location of the mobile robot 10. Alternatively, the determination of whether the current location is close to the destination candidate may be made by performing image recognition on a captured image of the destination candidate (destination candidate image) and a captured image at the current location of the mobile robot 10.

[0118] When determining that the current location of the mobile robot 10 is close to the destinationcandidate registered in the destination-candidate management DB 3001 (YES in S54), the determination unit 32 proceeds the process to S56. By contrast, when determining that the current location of the mobile robot 10 is not close to the destination candidate registered in the destination-candidate management DB 3001 (NO in S54), the determination unit 32 proceeds the process to S55.

[0119] Subsequently, the determination unit 32 determines whether the direction instructed by the manual-operation command transmitted from the display apparatus 60 has been changed (S55). Specifically, the determination unit 32 determines whether the travel direction instructed by the manual-operation command has been changed, based on the manualoperation command received in S33. For example, when the manual-operation command instructs a characteristic operation, the determination unit 32 determines that the travel direction has been changed. Examples of the characteristic operation include steering by a certain amount or more, accelerating, and braking, which are performed, for example, when the mobile robot 10 travels through an intersection or a curve.

[0120] When it is determined that the travel direction instructed by the manual-operation command transmitted from the display apparatus 60 has been changed (YES in S55), the determination unit 32 proceeds the process to S56. By contrast, when the determination unit 32 determines that the travel direction instructed by the manual-operation command transmitted from the display apparatus 60 is not changed (NO in S55), the determination unit 32 ends the process.

[0121] In S56, the location-information acquisition unit 35 acquires the location information indicating the current location of the mobile robot 10 using, for example, the navigation satellite system 114. In addition, the destination-candidate acquisition unit 36 acquires a captured image captured by the camera 111 at the current location of the mobile robot 10 as a destination candidate image indicating a destination candidate of the mobile robot 10 (S57).

[0122] Then, the storing-reading unit 49 stores the destination candidate data including the location information acquired in S 19 and the captured image acquired in S20 in the destinationcandidate management DB 3001 (destination-candidate management table) illustrated in FIG. 7 (S58). The storing-reading unit 49 registers, as the destination candidate data, the location information and the captured image each associated with a candidate ID for identifying the destination candidate. The candidate ID may be assigned automatically by the storingreading unit 49.

[0123] As described above, the mobile robot 10 automatically captures an image of the surroundings of the mobile robot 10 using the camera 111 mounted on the mobile robot 10, based on a predetermined determination criterion in accordance with the travel state of the mobile robot 10, and registers in advance the captured image indicating a candidate of the travel destinationof the mobile robot 10. The determination criterion by the determination unit 32 is not limited to those described above referring to S51 to S55 but is appropriately set in accordance with details of the manual operation of the mobile robot 10 and the surroundings of the mobile robot 10.The determination criterion by the determination unit 32 may be, for example, a state change of the mobile robot 10 to be recognized by the operator when the operator selects the destination of the mobile robot 10, the condition of the operation site, or an environmental change at the operation site.

[0124] Destination SettingSubsequently, referring to FIGS. 16 to 19, a description is given of setting the destination of the mobile robot 10 using the destination candidates registered through the above-described process. FIG. 16 is a sequence diagram illustrating a process of setting the destination of the mobile robot 10 according to the present embodiment.

[0125] First, the operation reception unit 52 of the display apparatus 60 receives selection of the destination setting button 290 by an input operation of the operator on the operation screen 200 (S71).

[0126] Subsequently, the communication unit 51 transmits to the mobile robot 10 a request to acquire data indicating a destination candidate, whish is referred to as a destination candidate acquisition request (S72). The destination candidate acquisition request includes the site ID for identifying the site where the mobile robot 10 is installed. The communication unit 31 of the mobile robot 10 receives the destination candidate acquisition request transmitted from the display apparatus 60.

[0127] Subsequently, the storing-reading unit 49 searches the destination-candidate management DB 3001 (destination-candidate management table) illustrated in FIG. 7 using the site ID indicated by the destination candidate acquisition request received in S72, as a search key, to retrieve the destination candidate data associated with the same site ID as the received site ID (S73).

[0128] Then, the communication unit 31 transmits the destination candidate data retrieved in S73 to the display apparatus 60 that is the source of the request (S74). The destination candidate data retrieved in S73 includes, for each of multiple destination candidates, the candidate ID, the captured image, and the location information. Thus, the communication unit 51 of the display apparatus 60 receives and acquires the destination candidate data transmitted from the mobile robot 10.

[0129] Subsequently, the display control unit 53 of the display apparatus 60 displays a selectionscreen 400 including the destination candidate data received in S74 on the display such as the display 506 (S75). Specifically, the second display processing unit 57 generates the selection screen 400 based on the received destination candidate data. Then, the display control unit 53 displays the selection screen 400 generated by the second display processing unit 57. Alternatively, the first display processing unit 47 of the mobile robot 10 may generate the selection screen 400. In this case, the first display processing unit 47 generates the selection screen 400 based on the destination candidate data retrieved in S73. In S74, the communication unit 31 transmits screen data of the selection screen 400 including the destination candidate data generated by the first display processing unit 47 to the display apparatus 60.

[0130] FIGS. 17 and 18 illustrate examples of the selection screen. The selection screens 400 illustrated in FIGS. 17 and 18 are each a display screen that allows the operator to select the destination to which the mobile robot 10 travels.

[0131] The selection screen 400 includes an image display area 410, which displays multiple captured images 415a to 417f (may be collectively referred to as the captured images 415) selectably by the operator. The captured images 415a to 417f are the destination candidate images included in the destination candidate data received in S74. The selection screen 400 further includes an OK button 430 to be pressed to complete the selection and a cancel button 435 to be pressed to cancel the selection.The operator selects one or more captured images 415 displayed in the image display area 410 using an input means such as the pointing device 512.

[0132] When the operator selects some of the captured images 415 displayed in the image display area 410, the operation reception unit 52 of the display apparatus 60 receives the selection of destination candidate image, which indicates an area to be set as the destination (S76). In the selection screen 400 illustrated in FIG. 18, for example, the captured images 415a, 415b, and 415c are selected by the operator. When some of the captured images 415 displayed in the image display area 410 are selected, the selected captured images are displayed in a selection result display area 420 on the selection screen 400 as illustrated in FIG. 18. The selection result display area 420 displays the selected captured images in the order selected by the operator (No.l to No.3). The operator selects, as the travel destination of the mobile robot 10, for example, a captured image including an area where the mobile robot 10 is to execute a task, or a captured image including an area of a branch, such as an intersection, on the travel route at the site in a case where the mobile robot 10 travels to the location where the task is executed.

[0133] As described above, the display apparatus 60 selects an area indicated by the captured image 415 as the travel destination of the mobile robot 10 using the captured image 415 (destinationcandidate image) indicating a travel destination candidate of the mobile robot 10, which has been captured in advance by the mobile robot 10. This increases the operability in selecting the travel destination of the mobile robot 10, for example, as compared to the method of allowing the operator to input a character string indicating location information or a keyword.

[0134] Subsequently, when the selection of the destination candidate image is received in S76 and the operator presses the OK button 430, the autonomous-travel request generation unit 56 of the display apparatus 60 generates autonomous-travel request information (S77). The autonomous-travel request information includes one or more candidate IDs respectively associated with one or more captured images 415 having been selected in S76 and information indicating the order in which the selection of the captured images is received in S76.

[0135] Subsequently, the communication unit 51 transmits the autonomous-travel request information generated in S77 to the mobile robot 10 (S78). The communication unit 31 of the mobile robot 10 receives the autonomous-travel request information transmitted from the display apparatus 60.

[0136] Then, the mobile robot 10 starts autonomous travel based on the autonomous-travel request information received in S78 (S79).

[0137] Autonomous TravelSubsequently, referring to FIG. 19, a description is given in detail of the autonomous travel of the mobile robot 10, performed in S79. FIG. 19 is a flowchart illustrating an autonomous travel process performed by the mobile robot 10 according to the present embodiment.

[0138] First, when the autonomous -travel request information is received in S78, the mode setting unit 42 of the mobile robot 10 sets the mobile robot 10 to operate in the autonomous travel mode (S91).

[0139] Subsequently, the route-information generation unit 37 generates route information indicating an autonomous travel route of the mobile robot 10 based on the autonomous-travel request information received in S78 (S92). Specifically, based on the candidate IDs and the order of selecting the captured images 415 specified by the candidate IDs, which are indicated by the received autonomous-travel request information, the route-information generation unit 37 generates a travel route such that the mobile robot 10 autonomously travels the areas indicated by the captured images 415, in the order in which the captured images 415 are selected by the operator. The travel route generated by the route-information generation unit 37 is not limited to the route in the order of selecting the captured images 415. Alternatively, the routeinformation generation unit 37 may generate route information based on locations where thecaptured images 415 are captured, which are stored in the destination-candidate management DB 3001, such that the distance or travel time of the travel route connecting the locations indicated by the selected captured images 415 becomes shorter.

[0140] Then, the route-information management unit 38 stores the route information generated in S92 in the route-information management DB 3002 (route-information management table) illustrated in FIG. 8 in association with the site ID received in S72 (S93).

[0141] Subsequently, the location-information acquisition unit 35 acquires location information indicating the current location of the mobile robot 10 (S94).

[0142] The destination setting unit 39 sets the travel destination of the mobile robot 10 based on the current location of the mobile robot 10 acquired by the location-information acquisition unit 35 and the route information generated in S92 (S95). Specifically, the destination setting unit39 sets, as the travel destination, for example, the location of a destination to which the mobile robot 10 should go next, from among multiple destinations each specified by the candidate ID in the generated route information. For example, in a case where autonomous travel just starts, the destination setting unit 39 sets, as the travel destination, the location of the destination specified by the first candidate ID in the route information. Then, the destination setting unit 39 generates a travel route from the acquired current location of the mobile robot 10 to the travel destination that is thus set. Example methods for the destination setting unit 39 to generate the travel route include a method of connecting the current location and the destination with a straight line, and a method of minimizing the travel time by avoiding an obstacle using a captured image or information of the obstacle obtained by the state detection unit 34.

[0143] The method for the destination setting unit 39 to generate the travel route is not limited to generating the travel route using the location information of travel destination candidates registered in the destination-candidate management DB 3001. Alternatively, the following method may be adopted. Based on the result of image recognition by the first display processing unit 47 on the captured image captured by the camera 111, the destination setting unit 39 may identify a location in the captured image 415 set as the travel destination.

[0144] Then, the travel control unit 40 controls the mobile robot 10 to travel to the set travel destination through the travel route generated in S92. In this case, the travel control unit 40 controls the mobile robot 10 to autonomously travel according to a drive command from the autonomous travel unit 43 (step S96).

[0145] When the mobile robot 10 has not reached the destination (NO in S97), the travel control unit40 repeats the process from S94 and continues the autonomous travel until the mobile robot10 reaches the destination.

[0146] When the mobile robot 10 has reached the destination (inspection point) on the travel route (YES in step S97) under the control of the travel control unit 40, the task execution unit 45 executes the task (S98). In the present embodiment, the task execution unit 45 causes the mobile robot 10 to capture an image of the inspected object.

[0147] After the task is completed, the mobile robot 10 repeats the process from S94 and continues the autonomous travel if the mobile robot 10 has not yet reached the final destination (NO in S99).

[0148] When the mobile robot 10 has reached the final destination (YES in S99), the travel control unit 40 ends the travel. The travel control unit 40 may control the mobile robot 10 to travel to the start point.

[0149] In this manner, the mobile robot 10 autonomously travels to the location indicated in the captured image 415 selected by the operator as the destination of the mobile robot 10. In the autonomous travel mode, the mobile robot 10 autonomously travels according to the generated route information or using the learning data learned in the manual operation mode.

[0150] Storing High-Magnification Image and lx Image of Inspected ObjectA detailed description is given below of image capturing of an inspected object.

[0151] FIG. 20 is a flowchart of a process in which the mobile robot 10 captures a high- magnification image of an inspected object and a lx image (second image data) of the facility in registration of a destination candidate under the control of an operator. FIG. 20 illustrates the process performed in steps S16 and S17 of FIG. 12.

[0152] The operator operates the display apparatus 60 to move the mobile robot 10 to the destination candidate. Specifically, the mobile robot 10 travels according to the drive command based on the manual-operation command transmitted from the display apparatus 60. Then, the operator checks the captured image data displayed by the display control unit 53. The display apparatus 60 receives from the operator the adjustments of pan, tilt, and zoom settings of the camera 111 to capture an image of the inspected object having a size equal to or larger than a threshold size (S101). The communication unit 51 of the display apparatus 60 transmits the adjusted pan, tilt, and zoom ratio to the mobile robot 10.

[0153] The determination unit 54 of the display apparatus 60 determines whether an inspection point registration button (e.g., the destination-candidate registration button 270 in FIG. 13) has been pressed (S102). When the determination in step S102 is Yes, the determination unit 54determines whether a registration button has been pressed (S 103). The inspection point registration button is a button for storing the location information and the captured image data in the destination-candidate management DB 3001, and the registration button is a confirmation button. Either one of the inspection point registration button and the registration button suffices.

[0154] If the determination in step S103 is Yes, the communication unit 51 transmits a registration request for the high-magnification image of the inspected object and the lx image of the facility to the mobile robot 10. The communication unit 31 of the mobile robot 10 receives the request. The image-capture control unit 33 captures the high-magnification image at the pan, tilt, and zoom settings received in step S101 and stores the images of the inspected object in the destination-candidate management DB 3001 (S104).

[0155] The image-capture control unit 33 stores the location information and the image-capture conditions (pan, tilt, and zoom settings) under which the high-magnification image is acquired in the destination-candidate management DB 3001 (S105).

[0156] In step S106, the image-capture control unit 33 adjusts the zoom ratio to lx zoom while keeping the same orientation (pan and tilt) of the camera 111. The image-capture control unit 33 captures a lx image with lx zoom, and stores the pan, tilt, and zoom in the destinationcandidate management DB 3001 (SI 07). The term “equal magnification or lx” of lx image is used as an expression for capturing a wide range. For example, the magnification may be 2-fold magnification. The zoom ratio of the lx image is set as appropriate.

[0157] In the case of the lx image (second image data) captured in the registration of the destination candidate in this way, probably the inspected object is captured at the center since the operator intervenes. However, the image quality of the lx image may be degraded due to various reasons. For example, the zoom is not set at lx, a person or an object such as a fallen leave is captured, or there is influence of, for example, the setting sun. When the image quality of the lx image is degraded, the image quality of the template image is also degraded.

[0158] FIGS. 21A and 21B illustrate a high-magnification image and a lx image stored in the registration of a destination candidate, respectively. The high-magnification image of FIG. 21A is captured at a zoom ratio larger than 1 so that the inspected object 150 is captured in a large size. The zoom ratio is 7.6 times, but this is merely an example. The inspected object 150 is an analog meter, but this is merely an example. The lx image of FIG. 2 IB has a zoom ratio of lx, and thus the entire facility including the inspected object 150 is captured.

[0159] Quality Check of lx Image Captured in Registration of Destination CandidateAs described later, the template image is cut out from around the center of the lx imagecaptured in registration of the destination candidate. If the image quality around the center of the lx image is not good, there is a high possibility that the image processing unit 33a fails to detect the inspected object by the template matching. Accordingly, the size determination unit 33d preferably determine the size of the template image to make the template image suitable for the template matching.

[0160] FIG. 22 is a flowchart of a process in which the size determination unit 33d determines the qualities around the center of the lx image.

[0161] First, the size determination unit 33d extracts feature points from the entire lx image stored in step S107 of FIG. 20 (S201). There are known methods of extracting the feature points such as the scale -invariant feature transform (SHFT), the speeded up robust features (SURF), and the histograms of oriented gradients (HOG). Any other method capable of extracting the feature points may be used.

[0162] In step S202, the size determination unit 33d counts the number of feature points in the having the size calculated by multiplying the initial cropping range by X (the parameter for size determination). The cropping range refers to the area cut out as a template image from the lx image of the destination-candidate management DB 3001 in the execution of a task. The initial cropping range is set to the minimum for including the entire inspected object. For example, it is assumed that the resolution of the camera 111 is 960x1280, and an image of the inspected object is captured at the maximum magnification of lOx in the execution of the task. Since the inspected object needs to be captured in the range of 1 / 10 of the resolution, the initial cropping range is as follows.Height: 960 10 = 96Width: 1280 ■? 10 = 128The template image is the center area of the prestored lx image stored in the destinationcandidate management DB 3001, and the size of the template image is calculated by multiplying the initial cropping range by X. The initial value of the X value is 1, but may be 2 or greater.

[0163] In step S203, the size determination unit 33d determines whether the number of feature points in the cropping range is equal to or greater than the threshold. When the determination in step S203 is No, the cropping range is narrow, and thus the size determination unit 33d increases the multiplying value X by one (S204).

[0164] When the determination in step S203 is Yes, the size determination unit 33d stores the value of the multiplying value X in the destination-candidate management DB 3001 in association with the inspection point ID (S205).

[0165] When the number of feature points is small, it is difficult for the image processing unit 33a to determine the coordinates that match the inspected object at the time of template matching. In other words, the deviation from the inspected object captured in the lx image increases.In contrast, in the present embodiment, the size determination unit 33d ensures that the number of feature points is equal to or greater than the threshold. Accordingly, it is easy to determine the coordinates that match the inspected object in the lx image in the destinationcandidate management DB 3001.

[0166] Image Capturing of Inspected Object in Execution of TaskThe image capturing of the inspected object in task execution will be described. FIG. 23 is a flowchart of a process in which the mobile robot 10 captures an image of an inspected object in autonomous travel.FIG. 23 illustrates the process performed in step S98 of FIG. 19.

[0167] First, when the task of image capturing is executed, the image-capture control unit 33 acquires the current position and posture of the mobile robot 10 (S 111). The location is detected by the location-information acquisition unit 35, and the posture is detected by the state detection unit 34 using, for example, SLAM.

[0168] The image-capture condition correction unit 33b compares the position and posture stored in the destination-candidate management DB 3001 with the current position and posture, and calculates the change amounts in the pan and tilt (SI 12). The method of calculating the change amounts will be described in detail later.

[0169] The image-capture control unit 33 corrects the pan and tilt of the camera 111 based on the change amounts. The image-capture control unit 33 having the zoom ratio of lx captures a lx image (S 113). The zoom ratio of lx is an example. The zoom ratio is set similar to that of the lx image in the destination-candidate management DB 3001.

[0170] Then, the image processing unit 33a performs template matching on the lx image captured in step S 113 (SI 14). The matching process will be described in detail later with reference to FIG. 25. The template matching is a process in which the image processing unit 33a repeats the summing of square errors of pixel values while shifting the template image from the upper left end on the lx image to the right by one pixel. When the template image reaches the right end of the lx image, the image processing unit 33a returns the template image to the left end, shifts the template image downward by one pixel, and repeats the summing of square errors of pixel values while shifting the template image to the right by one pixel. The image processing unit 33a repeats this horizontal scanning until the template image reaches the lower end of the lx image. The image processing unit 33a identifies the coordinates on the lx image having the smallest of all the sums of square errors.

[0171] The image-capture condition correction unit 33b adjusts the pan and tilt so that the optical axis coincides with (matches most) the coordinates identified in step SI 14 (SI 15). Click centering is known as such a function. Click centering is a function of panning and tilting of a camera having panning and tilting function so that the clicked position is located at the center when a mouse is left-clicked on the monitor of the camera. In the present embodiment, the coordinates identified in step SI 14 are used instead of the operator’s clicking. Click centering will be described below.It is assumed that the resolution of the image is “1280 x 960”, and the angles of view at the minimum magnification are “44 degrees in the vertical direction and 58 degrees in the horizontal direction”. When the clicked coordinates are x and y, the rotation angle in the pan direction is expressed by Expression 1, and the rotation angle in the tilt direction is expressed by Expression 2 (the unit is radian).

[0172] Expression 1 tan / x - 640 \

[0173] Expression 2

[0174] Subsequently, the image-capture control unit 33 acquires the zoom ratio at the time of capturing the high-magnification image stored in the destination-candidate management DB 3001, and changes the zoom ratio (S 116). The image-capture control unit 33 controls the camera 111 to capture an image of the inspected object with the pan, tilt, and zoom ratio of steps SI 15 and SI 16 (SI 17).When the template matching is successful, the optical axis is directed to the inspected object. Accordingly, the inspected object in an increased size falls in the angle of view even when the magnification is increased.

[0175] Calculation of Change Amount in Pan and Tilt based on Position and PostureFIG. 24 is a diagram illustrating the calculation of the change amounts in the pan and tilt in step S 112 in FIG. 23 in the execution of a task. When the mobile robot 10 is at the position and posture illustrated in FIG. 24 at the time of storing the destination candidate and at the execution of the task, the pan and tilt are calculated as follows. It is assumed that a distance D between the center of the mobile robot 10 and the camera 111 and a focus value F of the camera that approximates the distance to an inspected object S are known. Further, in Expression 3, the value (1.0 or -1.0) is 1.0 when the outer product of the vectors AP2 and P2S is negative, and is -1.0 when the outer product is positive.

[0176] Expression 3In FIG. 24, the reference signs represents the following items.PanS: Inspected objectO: Position of the mobile robot 10 in the destination-candidate management DB 3001 Pi: Position of the camera at the time of destination candidate registration ai: Posture at destination candidate registrationPi: Pan at the time of destination candidate registrationA: Position of the mobile robot 10 (center of axle) in the execution of the taskP2: Position of the camera in the execution of the task012: Posture in the execution of the task 2: Pan in the execution of the taskOA: Distance between the mobile robot 10 at the time of destination candidate registration and the mobile robot 10 in the execution of the task tiltl: Tilt at the time of destination candidate registration F: Focal length of the camera 111 Template MatchingFIG. 25 is a flowchart illustrating a process of template matching in step S 114 in detail. FIGS. 26A to 26E each illustrate an image for explaining the process of FIG. 25.

[0177] The image processing unit 33a acquires the lx image and the zoom ratio thereof from the destination-candidate management DB 3001 (S 121). The zoom ratio is assumed to be 1. FIG. 26A illustrates a lx image acquired from the destination-candidate management DB 3001.

[0178] The image processing unit 33a cuts out a template image from the lx image captured in registration of the destination candidate (S122). FIG. 26B illustrates a template image cut out from the lx image. The cutting out does not mean deletion from the original lx image butmeans processing of copying the template image. The details will be described with reference to FIG. 27. In the case of the lx image of the facility (destination candidate) captured at the registration of the destination candidate, probability the inspected object is captured at the center since a human being intervenes. This template image may also be stored in the image data management DB 3005.

[0179] The image processing unit 33a executes template matching on the current lx image (current first image data) captured at the destination in the execution of the task using the template image (S123). FIG. 26C illustrates the current lx image captured in the execution of the task.

[0180] The image processing unit 33a calculates the coordinates having the maximum matching rate (S124). FIG. 26D illustrates the template image overlapping the lx image at the position where the matching rate is maximum. FIG. 26E illustrates an image of the inspected object captured with the zoom ratio of the high-magnification image stored in the destinationcandidate management DB 3001 in the execution of the task.

[0181] FIG. 27 is a diagram illustrating the process of step S122. It one example, the resolution (height x width) of the lx image is set to “960 * 1280” pixels, and the zoom ratio of the high- magnification image stored in the destination-candidate management DB 3001 is lOx. In this case, a height H and a width W of the template image are calculated as follows.H = (960 / 10) x X valueW = (1280 / 10) x X valueSuppose the X value is 2, the height H is 192, and the width W is 256. Even if the pan and tilt are adjusted by the template matching, the inspected object is not necessarily captured at the center of the lx image depending on the quality of the lx image in the destination-candidate management DB 3001 or the lx image captured in the execution of the task. In the present embodiment, the image processing unit 33a can extract a wider range as the template image by using the X value as the parameter. In addition, although the template image also includes the background of the inspected object, this processing increases the accuracy of determination of the coordinates of the inspected object by template matching. For example, a plant is often equipped with multiple similar- shaped meters adjacent to one other. Accordingly, if only the inspected object is cut out as the template image, the image of another meter not the inspected object may match the template image.

[0182] In FIG. 27, the image processing unit 33a cuts out the template image from the lx image stored in the destination-candidate management DB 3001 when the task is executed. Alternatively, the high-magnification image stored in the destination-candidate management DB 3001 may be used as the template image. In this case, the image processing unit 33a reduces the high-magnification image stored in the destination-candidate management DB 3001 at the zoom ratio and sets the reduced high-magnification image as the template image.To be more specific, the image processing unit 33a reduces the high-magnification image at the ratio of the magnification of the lx image captured in the execution of the task relative to the high-magnification image so as to obtain the template image. Note that the zoom ratio of the lx image is not necessarily lx.

[0183] However, even when multiple meters are adjacent to one other, only one meter may be imaged in the high-magnification image so that the inspected object is clearly recognized. When the template image is such a high-magnification image, as described above, it is possible that the coordinates of a nearby meter is undesirably determined as having a maximum matching rate by template matching. Accordingly, the template image cut out from the lx image stored in the destination-candidate management DB 3001 is more preferable depending on the inspected object captured by the mobile robot 10.

[0184] Correction of Image of Inspected ObjectIt is possible that the inspected object may be absent, or a part of the inspected object is missing in the image captured in the execution of the task due to, for example, a difference in weather conditions. In such a case, it is preferable that the operator registers correction information for correcting the cropping range of the template image.

[0185] FIG. 28 a diagram illustrating an inspection result display screen 300 displayed on the display apparatus 60 by the operator. When the autonomous travel is finished, the operator can connect the display apparatus 60 to the mobile robot 10 and display the inspection result display screen 300. The display apparatus 60 may be constantly connected to the mobile robot 10. The display apparatus 60 acquires from the mobile robot 10 the lx image and the high-magnification image stored in the image data management DB 3005 by the mobile robot 10.

[0186] The inspection result display screen 300 displays a template image 301 and images 302 and 303 of the inspected object for each destination (inspection point) on the travel route. In FIG. 28, the destinations having the destination candidate IDs P001, P002, and P003 are illustrated. The template image 301 may be a high-magnification image. The images 302 and 303 of the inspected object are displayed for each date and time of inspection. A correction button 304 is displayed for each of the images 302 and 303 of the inspected object. The operator presses the correction button 304 to transition to a correction screen 310.

[0187] FIG. 29 illustrates the correction screen 310 displayed by the display apparatus 60. The correction screen 310 displays a template image 311 and a lx image 313 captured in the execution of a task. A rectangular frame 312 is overlaid on the lx image 313. The operator presses the up, down, left, and right buttons on a direction icon 314 to move the rectangular frame 312, thereby adjusting the position of the rectangular frame 312 so that the inspectedobject 315 is displayed preferably at the center of the rectangular frame 312. The coordinates of the rectangular frame 312 are the correction value and examples of designated coordinates designated by the operator. Since the operator has adjusted the rectangular frame 312, the inspected object 315 is captured at the coordinates represented by the correction values of the lx image 313. When the operator presses a correction value registration button 316, the communication unit 51 of the display apparatus 60 transmits the inspection result ID, the inspection point ID, and the correction value to the mobile robot 10. These are stored in the correction-information management DB 3004 and are used for cutting out a template image at the next information of the same inspection point.

[0188] FIG. 30 is a flowchart of a process performed by the display apparatus 60 when the operator corrects the image of the inspected object. It is assumed that the display control unit 53 of the display apparatus 60 displays the inspection result display screen 300.

[0189] The operation reception unit 52 determines whether the correction button 304 on the inspection result display screen 300 is pressed (S131).

[0190] When the correction button 304 is pressed (Yes in S 131), the display control unit 53 displays the template image used for template matching and the lx image acquired in the execution of the task (S132).

[0191] The operation reception unit 52 determines whether the operation of moving the rectangular frame 312 vertically or laterally is received (S133). When such an operation is received, the display control unit 53 displays the rectangular frame 312 at the position shifted in the direction of operation of the rectangular frame 312 (S134).

[0192] When the operation of moving the rectangular frame 312 vertically or laterally is not received (No in S133), the operation reception unit 52 determines whether the correction value registration button 316 is pressed (S135). When the button is not pressed (No in S135), the process returns to step S133.

[0193] When the button is pressed (Yes in S135), the communication unit 51 transmits the inspection result ID, the inspection point ID, and the correction value to the mobile robot 10 (S136).These pieces of information are received by the communication unit 31 of the mobile robot 10 and stored by the correction-information management unit 33c in the correction-information management DB 3004 via the storing -reading unit 49.

[0194] Cutting out of Template Image using Correction InformationThe image processing unit 33a acquires a more appropriate template image from the lx image in the destination-candidate management DB 3001 by using correction information. FIG. 31is a flowchart of a process in which the image processing unit 33a cuts out a template image using correction information. The process of FIG. 31 is executed in step S122 of FIG. 25.

[0195] The image processing unit 33a acquires correction information from the correctioninformation management DB 3004 using the inspection point ID at which the mobile robot 10 currently stays as a key (S141). Multiple pieces of correction information may be acquired.

[0196] In step S 142, the image processing unit 33a acquires, from the image data management DB 3005, the image-capture time of the image data associated with the inspection result ID and the inspection point ID same as those of the acquired correction information. Accordingly, the correction information is associated with the image-capture time.

[0197] The image processing unit 33a identifies the correction information having the image-capture time closest to the current time (S 143).

[0198] The image processing unit 33a cuts out a template image from the lx image captured in the execution of the task based on the identified correction information (S144). The image processing unit 33a can cut out the template image including the inspected object by cutting out the template image using the correction information of the image-capture time close to the current time. In this way, even when the inspected object is not captured at the center of the lx image stored in the destination-candidate management DB 3001, the template image including the image of the inspected object is extracted.

[0199] The image processing unit 33a may cut out the template image from the lx image stored in the destination-candidate management DB 3001. Alternatively, instead of cutting out the template image with the image processing unit 33a, the image-capture control unit 33 may direct the optical axis to the coordinates indicated by the correction information in the lx image in the execution of the task and capture the lx image with the zoom ratio stored in the destination-candidate management DB 3001.

[0200] In addition, in step S142, the image processing unit 33a preferably identifies the correction information under the condition close to the current condition in consideration of the season and the weather in addition to the time. In this case, the season and the weather are associated with the captured image data in the image data management DB 3005. The current season is apparent from calendar information, and the weather is acquired by the mobile robot 10 from, for example, a weather information server.

[0201] The image processing unit 33a may identify the correction information of a similar lx image captured in the past execution of the task, which is similar to the lx image captured in the current execution of the task.

[0202] FIG. 32 is a flowchart of another process in which the image processing unit 33a cuts out a template image using correction information. Step S141 may be the same as that of FIG. 31.

[0203] In step S 142-2, the image processing unit 33a acquires a lx image associated with the inspection result ID and the inspection point ID same as those of acquired correction information.

[0204] In step S 143-2, the image processing unit 33a calculates the inner product (cosine similarity) of the lx image acquired in the execution of the task and the lx image acquired in S 142-2 and identifies the correction information associated with the lx image having the inner product per pixel closest to 1. In other words, the image processing unit 33a identifies the lx image having the highest similarity with the lx image acquired in the execution of the task.

[0205] In this way, the image processing unit 33a cuts out the template image from the lx image captured in the current execution of the task using the correction information used in capturing of the similar lx image to the lx image captured in the current execution of the task.

[0206] As described above, the mobile robot 10 according to the present embodiment stops at a preset position in the execution of inspection and searches for an inspected object by template matching from a wide-angle image captured in the execution of a task. Accordingly, the mobile robot 10 determines the optimal image capturing direction and acquires an enlarged image of the inspected object.

[0207] Second EmbodimentIn the present embodiment, controlling the mobile robot 10 by the route management server 50 will be described.FIG. 33 is a functional block diagram of the communication system 1 according to the present embodiment. As illustrated in FIG. 33, the mobile robot 10 includes the communication unit 31, the determination unit 32, the image-capture control unit 33, the state detection unit 34, the location-information acquisition unit 35, the travel control unit 40, and the autonomous travel unit 43. The route management server 50 includes a communication unit 71, the image processing unit 33a, the image-capture condition correction unit 33b, the correctioninformation management unit 33c, the size determination unit 33d, the destination-candidate acquisition unit 36, the route-information generation unit 37, the route-information management unit 38, the destination setting unit 39, the image recognition unit 41, the mode setting unit 42, the manual operation processing unit 44, the task execution unit 45, the mapinformation management unit 46, the first display processing unit 47, the learning unit 48, and a storing-reading unit 49. The route management server 50 includes the destination-candidatemanagement DB 3001, the route-information management DB 3002, the map-information management DB 3003, the correction-information management DB 3004, and the image data management DB 3005. Each function may be similar to that of the first embodiment.

[0208] In FIG. 33, the route management server 50 is separate from the mobile robot 10, but the function of the route management server 50 may be incorporated into the controller of the mobile robot 10. The controller may be detachable.

[0209] Storing High-Magnification Image and lx Image of Inspected Object FIG. 34 is a flowchart of a process in which the mobile robot 10 captures a high-magnification image of an inspected object and a lx image of the facility (destination candidate) in registration of the destination candidate under the control of an operator. FIG. 34 illustrates the process performed in steps S16 and S17 of FIG. 12.

[0210] S301: When the operator controls the mobile robot 10 to travel to a destination candidate, the operator checks the captured image data and adjusts the pan, tilt, and zoom ratio of the camera 111 so that the inspected object is captured. The communication unit 51 of the display apparatus 60 transmits the pan, tilt, and zoom ratio to the route management server 50.

[0211] S302: The communication unit 71 of the route management server 50 transmits the received pan, tilt, and zoom ratio to the mobile robot 10.

[0212] S3O3: The image-capture control unit 33 of the mobile robot 10 controls the camera 111 to capture an image with the received pan, tilt, and zoom ratio. The communication unit 31 transmits the captured image data to the route management server 50.

[0213] S304: The communication unit 71 of the route management server 50 transmits the received captured image data to the display apparatus 60.

[0214] S305: When the operator determines that the inspected object in the captured image is equal to or larger in size than the threshold size, the operator operates the registration button. The communication unit 51 of the display apparatus 60 transmits the information indicating the pressing of the button to the route management server 50.

[0215] S306: The communication unit 71 of the route management server 50 receives the information indicating the pressing of the registration button. The storing-reading unit 49 stores the high-magnification image captured with the pan, tilt, and zoom ratio of step S3O3 in the destination-candidate management DB 3001.

[0216] S307: The communication unit 71 of the route management server 50 acquires the pan, tilt,zoom ratio, location information, and posture from the mobile robot 10.

[0217] S3O8: The storing-reading unit 49 stores the pan, tilt, zoom ratio, location information, and posture in the route management server 50.

[0218] S309: The communication unit 71 of the route management server 50 requests the mobile robot 10 to set the zoom ratio to 1.

[0219] S310: The communication unit 31 of the mobile robot 10 receives the request to set the zoom ratio to lx, and the image-capture control unit 33 sets the zoom ratio to lx.

[0220] S311: The image-capture control unit 33 captures a lx image while keeping the pan and tilt unchanged.

[0221] S312: The communication unit 31 of the mobile robot 10 transmits the lx image, the pan, tilt, and zoom ratio to the route management server 50. The storing-reading unit 49 of the route management server 50 stores the lx image, the pan, the tilt, and the zoom ratio in the destination-candidate management DB 3001.

[0222] In this way, the route management server 50 communicates with the mobile robot 10 and stores the high-magnification image and the lx image.

[0223] Image Capturing of Inspected Object in Execution of TaskFIG. 35 is a sequence diagram illustrating a process in which the mobile robot 10 captures an image of an inspected object in autonomous travel. FIG. 35 illustrates the process performed in step S98 of FIG. 19.

[0224] S321: The communication unit 31 of the mobile robot 10 receives the execution instruction of the task from the route management server 50. In the instruction, the image-capture control unit 33 of the mobile robot 10 transmits the current location and posture of the mobile robot 10 to the route management server 50. The location is detected by the location-information acquisition unit 35, and the posture is detected by the state detection unit 34 using, for example, SLAM.

[0225] S322: The communication unit 71 of the route management server 50 receives the location information (position) and the posture. The image-capture condition correction unit 33b compares the position and posture stored in the destination-candidate management DB 3001 with the current position and posture, and calculates the change amounts in the pan and tilt.

[0226] S323: The communication unit 71 of the route management server 50 transmits the changeamounts to the mobile robot 10.

[0227] S324: The communication unit 31 of the mobile robot 10 receives the change amounts. The image-capture control unit 33 controls the camera 111 to capture a lx image with the pan and tilt corrected based on the change amounts.

[0228] S325: The communication unit 31 of the mobile robot 10 transmits the lx image to the route management server 50.

[0229] S326: When the communication unit 71 of the route management server 50 receives the lx image, the image processing unit 33a performs template matching on the l image received in step S325. The image-capture condition correction unit 33b determines the pan and tilt directions in which the optical axis coincides with the most matching coordinates.

[0230] S327: The communication unit 71 of the route management server 50 transmits to the mobile robot 10 a request to capture a high-magnification image of the inspected object and a lx image of the facility. The request is accompanied by the pan and tilt in the most matching direction and the zoom ratio in the high-magnification image of the destination-candidate management DB 3001.

[0231] S328: The communication unit 31 of the mobile robot 10 receives these pieces of information. The image-capture control unit 33 sets the camera 111 to have the received pan and tilt.

[0232] S329: The image-capture control unit 33 sets the camera 111 to have the received zoom ratio.

[0233] S33O: The image-capture control unit 33 captures an image of the inspected object to generate an inspected object image.

[0234] S331: The communication unit 31 of the mobile robot 10 transmits the image of the inspected object to the route management server 50.The storing-reading unit 49 of the route management server 50 stores the image of the inspected object, the lx image of the facility, the inspection result ID, the inspection point ID, and the image image-capture time in the image data management DB.

[0235] The mobile robot 10 of the present embodiment provides to effects similar to those of the first embodiment. Further, the mobile robot 10 communicates with the route management server 50; and the route management server 50 performs template matching, corrects the imagecapture conditions, and stores the image of the inspected object captured by the mobile robot 10.

[0236] Third EmbodimentA description is given of a second embodiment in which the display apparatus 60 is a head mounted display (HMD). The HMD is a kind of display device that is worn on the head so as to cover both eyes with two displays in a goggle-like shape. The HMD displays parallax images to the left and right eyes that lead to stereoscopic vision. Providing a stereoscopic view is not a requisite.In addition to HMDs that simply provide stereoscopically view of videos, there are various types of HMDs such as an HMD called virtual reality (VR) headset (goggles), augmented reality (AR) headset, or mixed reality (MR) headset. A VR headset displays an artificial virtual world. An AR headset reads the real world stereoscopically and displays virtual information superimposed on the real world. An MR headset is an extension of the AR headset and displays an object which is not actually present in the real world in a three- dimensional manner. In the present embodiment, since the operator browses with the HMD the image data captured by the mobile robot 10, a simple HMD or AR headset would be used.

[0237] FIG. 36 is a block diagram illustrating an example of the hardware configuration of an HMD used as the display apparatus 60. As illustrated in FIG. 36, the display apparatus 60 which is an HMD has a configuration of a computer. The display apparatus 60 includes a CPU 201, a ROM 202, a RAM 203, an external device connection I / F 205, a display 207, an operation device 208, a medium I / F 209, a bus line 211, a speaker 212, an electronic compass 218, a gyro sensor 219, and an accelerometer 220.

[0238] The CPU 201 controls the entire operation of the HMD. The ROM 202 stores a program such as an initial program loader (IPL) to boot the CPU 201. The RAM 203 is used as a work area for the CPU 201.

[0239] The external device connection PF 205 is an interface for connecting various external devices. The external device in this case is, for example, a communication management server 3 or an earphone 6 having a microphone.

[0240] The display 207 is a display apparatus such as a liquid crystal display or an organic EL display that displays various images.

[0241] The operation device 208 is an input means operated by an operator to, for example, select or execute various instructions, select a target for processing, or move a cursor being displayed. Examples of the input means include various operation buttons, a power switch, a physical button, and a line-of-sight operation circuit that operates in response to detection of the line of sight of the operator.

[0242] The medium I / F 209 controls the reading or writing (storing) of data from or to a recordingmedium 209m such as a flash memory. Examples of the recording medium 209m include a DVD and a BLU-RAY DISC.

[0243] The speaker 212 is a circuit that converts an electric signal into physical vibration to generate sound such as music or voice. The electronic compass 218 calculates the direction of the HMD from the Earth's magnetism and outputs direction information. The gyro sensor 219 detects a change in angle (roll, pitch, and yaw) of the HMD as the HMD moves.

[0244] The accelerometer 220 is a sensor that detects acceleration in triaxial directions. Examples of the bus line 211 include an address bus and a data bus, which electrically connect the components including the CPU 201 to one another.

[0245] The display apparatus 60 according to the present embodiment can detect the posture of the display apparatus 60 by the signal of the gyro sensor 219 and thus can identify the direction in which the operator is viewing. The camera 111 can capture a spherical image, a wide-angle image, or a hemispherical image. Accordingly, the display apparatus 60 can display the direction in which the operator is viewing on the display 207 of the HMD. This makes it easier for the operator to view various directions compared with operating the area displayed in the site display area 210 with the pointing device 512 such as a mouse.

[0246] The position of the mobile robot 10 when the operator is viewing in a certain direction with the HMD is identified by the state detection unit 34. Accordingly, the display apparatus 60 can display annotation of an object such as a building present in the direction of view of the operator from the position of the mobile robot 10 in a manner superimposed on the image data transmitted from the mobile robot 10. In this case, for example, the system administrator sets an annotation of the building using the environment map generated by the mobile robot 10.

[0247] FIG. 37 illustrates an example of image data displayed by the HMD used as the display apparatus 60. In the image data of FIG. 37, a current position 601 of the mobile robot 10 and a name 602 “laboratory building” of the building in the image data are indicated. The mobile robot 10 receives the direction in which the operator is viewing from the display apparatus 60, and identifies on the environment map a building present in the direction in which the operator is viewing from the position of the mobile robot 10. The image-capture control unit 33 superimposes information such as the current position and the name of the building on the captured image data, and the communication unit 31 transmits the image data to the display apparatus 60. As the annotation, in addition to the name 602 of the building, the contents of accidents that occurred in the past, and events to be observed with caution may be displayed. Accordingly, the operator can easily grasp which direction the operator is viewing and what the operator is viewing.The mobile robot 10 of the present embodiment provides effects similar to those of the first embodiment or the second embodiment. Further, the use of an HMD as the display apparatus 60 is advantageous in that the operator can easily view the image data captured by the mobile robot 10.

[0249] Applied CasesAlthough the example embodiments of the present invention are described above, the abovedescribed embodiments are illustrative and do not limit the scope of the present invention. Thus, numerous additional modifications and variations are possible in light of the above teachings. For example, elements and / or features of different illustrative embodiments may be combined with each other and / or substituted for each other within the scope of the present invention. Any one of the above-described operations may be performed in various other ways, for example, in an order different from the one described above.

[0250] For example, since the direction information (pan and tilt) in the image capturing is stored in the destination-candidate management DB 3001, the image processing unit 33a may focus on the direction when searching for the inspected object in the lx image of the facility captured in the execution of the task by template matching. The image processing unit 33a may search for the inspected object with reference to the zoom ratio.

[0251] In the present embodiment, the image processing unit 33a detects the coordinates of an inspected object from the lx image by performing template matching, but the coordinates may be detected by image recognition using machine learning. For example, a person in charge prepares a neural network such as a convolutional neural network in which the input is a lx image and the output is annotation information (rectangular coordinates representing the position of the inspected object), and uses a model having learned the correspondence between the input and the output. In this case, the model can identify the coordinates of the inspected object from the lx image captured in the execution of a task without using a template image.

[0252] In the description of the present embodiment, the inspected object is captured at the center of the lx image stored in the destination-candidate management DB 3001. However, the inspected object does not need to be at the center of the lx image as long as the coordinates of the inspected object are known.

[0253] In the present embodiment, the description has been given of the case where the pan and tilt are adjusted so that the optical axis coincides with the coordinates of the lx image detected by the template matching in step S 115 of FIG. 23, but a slight deviation may be allowed. Even if there is a slight deviation, it is sufficient that the inspected object is in the angle of view when the image-capture control unit 33 performs image capturing at the zoom ratio stored in thedestination-candidate management DB 3001.

[0254] At the inspection point, the image-capture control unit 33 may capture the inspected object at a predetermined zoom ratio (for example, 5x to lOx) instead of the zoom ratio stored in the destination-candidate management DB 3001. Alternatively, the image-capture control unit 33 may adjust the zoom ratio stored in the destination-candidate management DB 3001 by the ratio of the distances to the inspected object at the time of storing the zoom ratio and the distances at the time of executing the task. In this way, the image-capture control unit 33 can image the inspected object in the similar size to the high-magnification image of the destination-candidate management DB 3001.

[0255] In addition, a lx image in the execution of the task may be additionally stored in the destination-candidate management DB 3001. The first display processing unit 47 of the mobile robot 10 calculates the similarity between the already stored lx image and the lx image newly captured in the execution of the task, and adds a lx image that is not similar to any stored lx image (having the similarity equal to or less than a certain value). In this way, the mobile robot 10 can store various lx images of the same inspection point in the destination-candidate management DB 3001. When executing the task, a template image is cut out from each of the lx images narrowed down by the time and the conditions such as weather, template matching is performed on each of the template images, and a median value of the results is identified as the matched coordinates.

[0256] In the configuration illustrated in, for example, FIG. 6, processing units are divided into blocks in accordance with the main functions in order to facilitate the understanding the processing by the mobile robot 10 and the display apparatus 60. No limitation is intended by how the processing units are divided by the unit of process or by the name. The processing performed by the mobile robot 10 and the display apparatus 60 may be divided into a greater number of processing units depending on the content of the processing. Further, a single processing unit can be divided into multiple processing units.

[0257] Each of the functions of the described embodiments may be implemented by one or more processing circuits or circuitry. The “processing circuit or circuitry” in the present specification includes a programmed processor to execute each function by software, such as a processor implemented by an electronic circuit, and devices, such as an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), and conventional circuit modules arranged to perform the recited functions.

[0258] Embodiments of the present disclosure can provide significant improvements in computer capability and functionality. These improvements allow operators to take advantage of computers that provide more efficient and robust interaction with tables that is a way to storeand present information on information processing apparatuses. In addition, the embodiments of the present disclosure can provide a better operator experience through the use of a more efficient, powerful, and robust user interface. Such a user interface provides a better interaction between a human and a machine.

[0259] The present invention can be implemented in any convenient form, for example using dedicated hardware, or a mixture of dedicated hardware and software. The present invention may be implemented as computer software implemented by one or more networked processing apparatuses. The processing apparatuses include any suitably programmed apparatuses such as a general purpose computer, a personal digital assistant, a Wireless Application Protocol (WAP) or third-generation (3G)-compliant mobile telephone, and so on. Since the present invention can be implemented as software, each and every aspect of the present invention thus encompasses computer software implementable on a programmable device. The computer software can be provided to the programmable device using any conventional carrier medium (carrier means). The carrier medium includes a transient carrier medium such as an electrical, optical, microwave, acoustic or radio frequency signal carrying the computer code. An example of such a transient medium is a Transmission Control Protocol / Intemet Protocol (TCP / IP) signal carrying computer code over an IP network, such as the Internet. The carrier medium may also include a storage medium for storing processor readable code such as a floppy disk, a hard disk, a compact disc read-only memory (CD- ROM), a magnetic tape device, or a solid state memory device.

[0260] The functionality of the elements disclosed herein may be implemented using circuitry or processing circuitry which includes general purpose processors, special purpose processors, integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or combinations thereof which are configured or programmed, using one or more programs stored in one or more memories, to perform the disclosed functionality. Processors are considered processing circuitry or circuitry as they include transistors and other circuitry therein. In the disclosure, the circuitry, units, or means are hardware that carry out or are programmed to perform the recited functionality. The hardware may be any hardware disclosed herein which is programmed or configured to carry out the recited functionality.There is a memory that stores a computer program which includes computer instructions. These computer instructions provide the logic and routines that enable the hardware (e.g., processing circuitry or circuitry) to perform the method disclosed herein. This computer program can be implemented in known formats as a computer-readable storage medium, a computer program product, a memory device, a record medium such as a CD-ROM or DVD, and / or the memory of an FPGA or ASIC.

[0261] This patent application is based on and claims priority to Japanese Patent Application Nos. 2023-104800, filed on June 27, 2023, and 2024-011340, filed on January 29, 2024, in the Japan Patent Office, the entire disclosure of which is hereby incorporated by reference herein. [Reference Signs List]

[0262] 1 Communication system10 Mobile robot50 Route management server60 Display apparatus

Claims

[CLAIMS]

1. A mobile apparatus configured to capture an image of an object at a preset position in autonomous travel, the mobile apparatus comprising: an image processing unit configured to perform image processing on first image data to output coordinates of the object in the first image data including an image of the object and an image of surrounding of the object captured at the preset position in autonomous travel; an image-capture condition correction unit configured to correct a condition of image capturing of the object based on the coordinates of the object in the first image data output by the image processing unit; and an image-capture control unit configured to capture an image of the object under the condition corrected by the image-capture condition correction unit.

2. The mobile apparatus according to claim 1, wherein the image processing unit is configured to: obtain second image data before performing the image processing, the second image data being different from the first image data and being data including an image of the object and an image of surrounding of the object; extract, as a template image, a range including the object from the second image data; and perform template matching on the first image data with the template image in the image processing.

3. The mobile apparatus according to claim 2, further comprising a size determination unit configured to detect feature points from the second image data and determine a size of the template image in which a number of the feature points is equal to or greater than a threshold.

4. The mobile apparatus according to claim 1, wherein the image-capture control unit further includes a correction-information management unit configured to store in advance designated coordinates in association with the preset position in a storage unit, the designated coordinates being coordinates of the object designated by an operator for the first image data, wherein the image processing unit is configured to: acquire a template image from the first image data based on the designated coordinates of the object; and perform template matching on the first image data with the template image in the image processing.

5. The mobile apparatus according to claim 4, wherein the correction-information management unit is configured to store, in the storage unit, the first image data in association with the preset position, the coordinates of the objectin the first image data, and a capture time of the first image data, wherein the storage unit stores multiple past first image data captured in past, and wherein the image processing unit is configured to: acquire one of the multiple past first image data associated with a capture time closest to a time of current autonomous travel; and acquire the template image from the acquired past first image data based on coordinates of the object associated with the acquired past first image data.

6. The mobile apparatus according to claim 4, wherein the correction-information management unit is configured to store, in the storage unit, the first image data in association with the preset position, the coordinates of the object in the first image data, and weather under which the first image data is obtained, and the storage unit stores multiple past first image data captured in past, and wherein the image processing unit is configured to: acquire one of the multiple past first image data associated with the weather similar to weather of current autonomous travel; and acquire the template image from the acquired past first image data based on coordinates of the object associated with the acquired past first image data.

7. The mobile apparatus according to claim 4, wherein the correction-information management unit is configured to store, in the storage unit, the first image data in association with the preset position and the coordinates of the object in the first image data, and the storage unit stores multiple past first image data captured in past, and wherein the image processing unit is configured to: identify one of the multiple past first image data having a highest similarity to the first image data captured in current autonomous travel; and acquire the template image from the identified past first image data based on the coordinates of the object associated with the identified past first image data.

8. The mobile apparatus according to any one of claims 1 to 7, wherein the image-capture control unit is configured to store, in the storage unit, pan, tilt, and zoom settings set by an operator before the autonomous travel, the pan, tilt, and zoom settings for capturing at the preset position an image of the object having a size equal to or larger than a threshold size, wherein, in autonomous travel, the image-capture condition correction unit is configured to correct pan and tilt settings of an image-capturing means to include, in an angle of view of the image-capturing means, the coordinates of the object in the first image data output by the image processing unit, and wherein the image-capture control unit is configured to capture the image of the object withthe corrected pan and tilt settings and the stored zoom setting.

9. The mobile apparatus according to claim 2, wherein the second image data and a high-magnification image of the object having a magnification higher than a magnification of the second image data are stored in a storage unit in advance, and wherein the template image extracted by the image processing unit is an image obtained by reducing the high-magnification image.

10. The mobile apparatus according to claim 9, wherein the high-magnification image is an image of the object captured with pan, tilt, and zoom settings that are set by an operator to capture at the preset position an image of the object having a size equal to or larger than a threshold size, wherein the second image data is data captured with the pan and tilt settings set by the operator and a zoom ratio of equal magnification, and wherein the image processing unit is configured to reduce the high-magnification image at a ratio of a magnification of the second image data relative to the high-magnification image to obtain the template image.

11. An information processing apparatus configured to communicate via a network with a mobile apparatus that captures an image of an object at a preset position in autonomous travel, the information processing apparatus comprising: a communication unit configured to receive first image data including an image of the object and an image of surrounding of the object captured at the preset position in autonomous travel; an image processing unit configured to perform image processing on the first image data to output coordinates of the object in the first image data; and an image-capture condition correction unit configured to correct a condition of image capturing of the object based on the coordinates of the object in the first image data output by the image processing unit, wherein the communication unit is configured to transmit, to the mobile apparatus, a request to capture the object under a corrected condition corrected by the image-capture condition correction unit.

12. A method for capturing an image of an object at a preset position with a mobile apparatus performing autonomous travel, the method comprising: performing image processing on first image data to output coordinates of the object in the first image data including an image of the object and an image of surrounding of the object captured at the preset position in autonomous travel; correcting a condition for image capturing of the object based on the coordinates of the objectin the output first image data; and capturing an image of the object under the corrected condition.

13. A recording medium storing a plurality of program codes which, when executed by one or more processors, causes the processors to perform a method for capturing an image of an object at a preset position with a mobile apparatus performing autonomous travel, the method comprising: performing image processing on first image data to output coordinates of the object in the first image data, the first image data including an image of the object and an image of surrounding of the object captured at the preset position in autonomous travel; correcting a condition for image capturing of the object based on the coordinates of the object in the output first image data; and capturing an image of the object under the corrected condition.

Images