Robotic system and method of robotic operation

Abstract

A robot system has a self-propelled robot, an operation unit, a display, a surrounding camera configured to the self-propelled robot and to capture a surrounding situation of the self-propelled robot, and a processing circuit configured to generate a self-propelled robot simulation image and to generate a composite image including a surrounding situation image captured by the surrounding camera and the generated self-propelled robot simulation image.

Description

Technical Field

[0001] This disclosure relates to robot systems and robot operation methods. Background Technology

[0002] Previously, it was known to take pictures of the area around an autonomously moving robot, and to move the robot while viewing the pictures (see, for example, Patent Document 1).

[0003] Patent Document 1: Japanese Patent Application Publication No. 2013-031897

[0004] Among autonomous walking robots (hereinafter referred to as self-propelled robots), some are equipped with robotic arms. When such self-propelled robots are made to walk, the robotic arms are prone to interfering with surrounding objects. The aforementioned prior art makes no mention of this problem. Summary of the Invention

[0005] This disclosure was made to solve the aforementioned problems, and its purpose is to provide a robot system and robot operation method that can prevent interference between a self-propelled robot with a robotic arm and surrounding objects.

[0006] To achieve the above objectives, one aspect of this disclosure relates to a robot system comprising: a self-propelled robot including a robotic arm having one or more joints; an operation unit for operating the self-propelled robot based on operations performed by an operator; a display for visual recognition by the operator; an ambient camera mounted on the self-propelled robot to capture images of the environment surrounding the robot; and a processing circuit configured to generate a simulated image of the self-propelled robot that continuously simulates the posture of the self-propelled robot, including the posture of the robotic arm, and to generate a composite image comprising the ambient image captured by the ambient camera and the generated simulated image of the self-propelled robot, and display it on the display. Here, the phrase "continuously simulates" is used to explicitly describe a continuous animation of the simulated image of the self-propelled robot generated by the simulated image generation unit, and that the simulated image of the self-propelled robot is a momentary image of that animation; it has no other particular meaning.

[0007] In addition, other aspects of this disclosure include robot operation methods such as: operating a self-propelled robot equipped with a robotic arm; generating a simulated image of the self-propelled robot that continuously simulates the posture of the self-propelled robot, including the posture of the robotic arm; setting up a surrounding camera to capture the surrounding conditions of the self-propelled robot; generating a composite image comprising the surrounding condition image captured by the surrounding camera and the simulated image of the self-propelled robot; and displaying the composite image.

[0008] This disclosure provides a robot system and robot operation method that can prevent interference between a self-propelled robot with a robotic arm and its surrounding objects. Attached Figure Description

[0009] Figure 1 This is a schematic diagram illustrating an example of the structure of a robot system according to an embodiment of the present disclosure.

[0010] Figure 2 It means Figure 1 A top view of an example of the structure of an operating unit.

[0011] Figure 3 It is a schematic representation Figure 1 A diagram showing the shooting range of the surrounding cameras.

[0012] Figure 4 It means Figure 3 The functional modules of the control system structure of the robot system.

[0013] Figure 5 It is an overhead viewpoint diagram that shows a composite image of the surrounding environment and a simulated image of the self-propelled robot as an image observed from the perspective of the self-propelled robot.

[0014] Figure 6 This is a top-view diagram that shows a composite image of the surrounding environment and a simulated image of the self-propelled robot as an image obtained from a viewpoint above.

[0015] Figure 7 It is a first-person viewpoint image that combines images of the surrounding environment with simulated images of the self-propelled robot, presented as an image observed from the self-propelled robot.

[0016] Figure 8 It is a diagram showing a composite image of a self-propelled robot's predetermined movement path superimposed with an image of the surrounding conditions.

[0017] Figure 9This is a composite image showing the pose changes of the robotic arm of a self-propelled robot, superimposed with an image of the self-propelled robot and an image of the surrounding conditions.

[0018] Figure 10A This is a screenshot showing an animation of the pose changes of the robotic arm of a self-propelled robot.

[0019] Figure 10B This is a screenshot showing an animation of the pose changes of the robotic arm of a self-propelled robot.

[0020] Figure 10C This is a screenshot showing an animation of the pose changes of the robotic arm of a self-propelled robot.

[0021] Figure 10D This is a screenshot showing an animation of the pose changes of the robotic arm of a self-propelled robot. Detailed Implementation

[0022] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Furthermore, in all the following drawings, the same or equivalent elements will be labeled with the same reference numerals, and repeated descriptions will be omitted. Additionally, since the following drawings are for illustrating the present disclosure, there may be omissions of elements unrelated to the present disclosure, inaccurate dimensions due to exaggeration, simplifications, and inconsistencies in the forms of corresponding elements in multiple drawings. Furthermore, the present disclosure is not limited to the following embodiments.

[0023] (Implementation Method)

[0024] Figure 1 This is a schematic diagram illustrating an example of the structure of a robot system 100 according to an embodiment of the present disclosure.

[0025] [Hardware Structure]

[0026] Reference Figure 1 The robot system 100 according to the embodiment includes: a self-propelled robot 1, which has robot arms 121A and 121B; and an operation unit 2, which includes an operation section 21 for operating the self-propelled robot 1. Figure 2 21A, 21B); Analog image generation unit 115 ( Figure 4 It generates a simulated image 160 of the self-walking robot 1, which simulates the posture of the self-walking robot 1 at all times, including the posture of the robot arms 121A and 121B (see reference). Figures 5-7 A surrounding camera 17, which is installed on the self-propelled robot 1, captures images of the surrounding environment of the self-propelled robot 1; a composite image generation unit 116 (see reference 116). Figure 4It generates an image 50 containing the surrounding conditions captured by the surrounding camera 17 (see reference). Figures 5-7 Composite images 501, 601, and 701, including the simulated image 160 of the self-propelled robot generated by the simulated image generation unit 115 (see reference). Figures 5-7 ); and the display unit 23 of the aforementioned operation unit 2 (see reference 2) Figure 2 The display shows composite images 501, 601, and 701 generated by the composite image generation unit 116. The structure will be described in detail below.

[0027] The robot system 100 of this embodiment includes a self-propelled robot 1 and an operation unit (control console) 2. The self-propelled robot 1 includes a walking part 11 capable of autonomous walking and an arm 13 provided on the walking part 11.

[0028] The self-propelled robot 1 and the operating unit 2 are connected, for example, via a data communication network 3. The self-propelled robot 1 and the operating unit 2 can also be directly connected via wired or wireless means.

[0029] The above-mentioned elements in the robot system 100 will be described in detail below.

[0030] <Applications of Robot System 100>

[0031] The purpose of the robot system 100 is not particularly limited. Hereinafter, we will illustrate the case of the self-walking robot 1 performing care in a private residence.

[0032] <Data Communication Network 3>

[0033] Data communication network 3 can be any network capable of data communication. Examples of data communication network 3 include the Internet, LAN (Local Area Network), and WAN (Wide Area Network).

[0034] <Self-propelled Robot 1>

[0035] Reference Figure 1 The self-walking robot 1 can be basically configured to include a walking part 11 capable of autonomous walking and an arm (robot arm) 13 disposed on the walking part 11.

[0036] Here, the self-propelled robot 1 has a walking part 11, a lifting part 12, and an arm 13.

[0037] The traveling unit 11 is, for example, a trolley (hereinafter referred to as trolley 11). The trolley 11 has wheels 11a at its base, consisting of front wheels and rear wheels. One of the front wheels and the rear wheels is a steering wheel, and at least one of the front wheels and the rear wheels is a drive wheel. In addition, a lifting unit 12 is provided at the front of the trolley, and a shelf 11b for placing items is provided at the rear of the trolley 11.

[0038] The trolley 11 is also equipped with a battery and a motor. The trolley 11 moves autonomously by using the battery as a power source to drive the wheels 11a. In addition, the lifting unit 12, the arm 13, and the robot-side display unit 14, robot-side microphone 15, and robot-side playback unit 16 (described later) operate using the battery as a power source.

[0039] The lifting unit 12 includes a base 122 and a lifting shaft 123 that moves up and down relative to the base 122. The lifting shaft 123 extends, for example, in the vertical direction.

[0040] At the upper part of the lifting shaft 123, the base ends of the first robot arm 121A and the second robot arm 121B are rotatably disposed around the central axis of the lifting shaft 123. The second robot arm 121B is disposed above the first robot arm 121A. The first robot arm 121A and the second robot arm 121B can interchange their respective rotational positions, without left-right distinction.

[0041] The first robotic arm 121A and the second robotic arm 121B are each composed of multi-joint robotic arms, and each has a manipulator 124A and a manipulator 124B at its front end.

[0042] The robotic arms 124A and 124B are not particularly limited, but are formed in a shape that can grasp an object.

[0043] A surrounding camera 17 is provided in front of the lifting shaft 123. In addition, the surrounding camera 17 is located on the right side of the trolley 11 (shown with reference to reference numeral 17) and at the rear (in... Figure 1 (not shown in the middle) and the left side (in Figure 1 (Not shown in the image). These four ambient cameras are positioned at the same height as each other. The four ambient cameras 17 are devices used by the operator P to confirm the surrounding conditions (environment) of the self-propelled robot 1. The ambient cameras 17 will be described in detail later.

[0044] A hand-end camera 18 is provided at the end of the second robotic arm 121B. The hand-end camera 18 is a device used by the operator P to confirm the object to be held by the pair of robotic arms 124A and 124B.

[0045] A robot-side display 14 is mounted on the upper end of the lifting shaft 123 via a support member 125. The robot-side display 14 is, for example, a liquid crystal display.

[0046] A robot-side microphone 15, a robot-side playback unit 16, and a main camera 19 are provided at appropriate locations on the robot-side display unit 14.

[0047] The robot-side display unit 14, robot-side microphone 15, robot-side playback unit 16, and main camera 19 are a device assembly for dialogue between the self-propelled robot 1 and a human (hereinafter referred to as the interlocutor). The robot-side display unit 14 displays information (image information, text information, etc.) to be transmitted to the interlocutor. The robot-side microphone 15 acquires the interlocutor's voice. The robot-side playback unit 16, for example, is composed of a speaker, and plays the sound information to be transmitted to the interlocutor. The main camera 19 captures the interlocutor's image.

[0048] The trolley 11 also includes a computing circuit module Cm1 and a robot-side communication unit 113. The computing circuit module Cm1 includes a processor Pr1 and a memory Me1. As described later, the computing circuit module Cm1 constitutes the robot control unit (controller) 112, the analog image generation unit 115, the composite image generation unit 116, and the interference warning unit 117 (see reference). Figure 4 The analog image generation unit 115, the composite image generation unit 116, and the interference warning unit 117 may also be composed of a portion or all of the arithmetic circuit module Cm2 described later.

[0049] <Operation Unit 2>

[0050] Figure 2 It means Figure 1 This is a top view of an example of the structure of the operating unit 2. The operating unit 2 is not particularly limited as long as it can operate the self-propelled robot 1. Figure 2 As shown, the operation unit 2 can be formed by integrating the left and right operation parts 21A and 21B, or it can be composed of multiple operation parts formed separately. The operation parts are not particularly limited as long as they can be operated by the operator. Examples of operation parts (operation tools, operation components) include indicator keys, joysticks, handles, touch panels, etc.

[0051] Additionally, in operation unit 2, such as Figure 2 As shown, the operation units 21A and 21B, the operation-side display unit 23, the operation-side microphone 25, and the operation-side playback unit 26 can be integrated, or the operation units 21A and 21B, the operation-side display unit 23, the operation-side microphone 25, and the operation-side playback unit 26 can be formed separately.

[0052] Reference Figure 2 The operating unit 2 has a main body 20. The main body 20 is formed as a flat rectangular box.

[0053] A left-hand operation unit 21A and a right-hand operation unit 21B are respectively provided at the left and right ends of the main body 20. The left-hand operation unit 21A and the right-hand operation unit 21B constitute the operation unit 21. A set of operation keys 29 is provided on the left-hand operation unit 21A and the right-hand operation unit 21B. This set of operation keys 29 is configured in the same way as the known operation key set of a game console. Therefore, the description of this set of operation keys 29 is omitted. If the operator P operates this set of operation keys 29 appropriately with both hands, the walking part, the lifting part, and the arm 13 of the self-propelled robot 1 will move according to the operation. That is, the operation unit 21 is configured to output key operation signals for operating the walking part, the lifting part, and the arm 13 of the self-propelled robot 1.

[0054] An operation-side display unit 23, which is visually recognizable by the operator P, is provided at the center of the upper surface of the main body 20. The operation-side display unit 23 is, for example, a touchscreen. However, the operation-side display unit 23 only needs to display images and does not necessarily need to be a touchscreen. For example, the operation-side display unit 23 can be a liquid crystal display separately from the operation unit 2, or it can be a head-mounted display. The operation-side display unit 23 displays information (image information, text information, etc.) required by the operator P to operate the self-propelled robot 1. For example, the main image captured by the main camera 19 and the hand-end image captured by the hand-end camera 18 are appropriately displayed on the operation-side display unit 23. Additionally, the composite images 501, 601, and 701 (described later) are displayed on the operation-side display unit 23. Figures 5-7 ).

[0055] An operation-side microphone 25 and an operation-side playback unit 26 are provided at appropriate locations on the upper surface of the main body 20. The operation-side microphone 25 acquires the voice of the person speaking to it. The operation-side playback unit 26 is, for example, a speaker, and plays the voice of the person speaking to it acquired by the robot-side microphone 15. The operation-side playback unit 26 also includes a headset earpiece 26a. A sound output terminal is provided at appropriate locations on the main body 20. If a connection cable 30 for the headset earpiece 26a is connected to the sound output terminal, the output of the operation-side playback unit 26 switches from a speaker to the headset earpiece 26a, and the voice of the person speaking to it acquired by the robot-side microphone 15 is played from the headset earpiece 26a.

[0056] The main body 20 houses an arithmetic circuit module Cm2 and an operation-side communication unit 28. The arithmetic circuit module Cm2 includes a processor Pr2 and a memory Me2. As described later, the arithmetic circuit module Cm2 constitutes the operation control unit 27 (see reference). Figure 4 ).

[0057] <Surrounding Cameras 17>

[0058] Figure 3It is a schematic representation Figure 1 A diagram showing the shooting range of camera 17 around the area.

[0059] Reference Figure 3 Four peripheral cameras 17 are respectively positioned at the front, right side, rear, and rear of the self-propelled robot 1. These four peripheral cameras 17 are arranged symmetrically with respect to the predetermined central axis C of the self-propelled robot 1 in both front-to-back and left-to-right configurations when viewed from above (when viewed from above). Furthermore, these four peripheral cameras 17 are positioned at the same height relative to each other along the midway point of the self-propelled robot 1 in the height direction.

[0060] Each of the surrounding cameras 17 is a wide-angle camera, specifically one with a 180-degree field of view. Therefore, the shooting ranges 151A to 151D of the four surrounding cameras 17 overlap with each other at the two lateral ends of each surrounding camera 17.

[0061] Here, the surrounding camera 17 consists of a 3D camera (three-dimensional camera). A 3D camera is a camera that can acquire not only two-dimensional information in the horizontal and vertical directions (X and Y) but also depth (Z) information. Examples of 3D cameras include, for example, cameras that use multiple cameras to achieve stereoscopic effect using parallax, cameras that use time-of-flight (ToF) technology, and cameras that use patterned light with structured illumination. These cameras are well known, so detailed descriptions are omitted.

[0062] Here, by combining the images captured by the four surrounding cameras 17 and performing image processing, an image obtained from an overhead viewpoint (hereinafter referred to as the overhead viewpoint image, see reference) is obtained. Figure 5 The image obtained by observing the surroundings from an upward viewpoint (hereinafter referred to as the top viewpoint image, see reference). Figure 6 ), and images obtained from the surroundings observed by the self-propelled robot 1 (hereinafter referred to as first-person viewpoint images, see reference ). Figure 7 These three types of images. Since the images captured by the surrounding camera 17 contain depth information, such image processing is possible.

[0063] As will be described later, these images are combined with simulated images of the self-propelled robot to create composite images.

[0064] [Structure of the Control System]

[0065] Figure 4 It means Figure 1 The functional modules of the control system structure of the robot system 100.

[0066] The structure of the control system of the robot system 100 will be described below as "basic structure", "structure related to composite image" and "structure related to interference warning".

[0067] <Basic Structure>

[0068] {Structure of Operation Unit 2}

[0069] Reference Figure 4 The operation unit 2 includes an operation unit 21, an operation-side display unit 23, an operation-side microphone 25, an operation-side playback unit 26, an operation control unit 27, and an operation-side communication unit 28.

[0070] The operation unit 21 outputs the key operation signal corresponding to the operation of the operator P on a set of operation keys 29 to the operation control unit 27.

[0071] The operation-side display unit 23 displays an image based on the image display signal input from the operation control unit 27. Additionally, the operation-side display unit 23 outputs composite image specification information, predetermined movement path information, and arm animation information, which will be described in detail later. Furthermore, the operation-side display unit 23 outputs display image switching information.

[0072] The operator microphone 25 acquires the voice of the operator P and outputs it as the operator's voice signal to the operation control unit 27.

[0073] The operation-side playback unit (interference warning reporting unit) 26 plays the speaker's voice and the interference warning voice signal respectively, based on the speaker's voice signal and the interference warning voice signal input from the operation control unit 27. The operation-side playback unit 26 corresponds to the interference warning reporting device.

[0074] The operation control unit 27 generates an operation signal corresponding to the key operation signal input from the operation unit 21 and outputs it to the operation side communication unit 28. This operation signal is generated, for example, based on pre-set allocation information for "the movement of the walking part, the movement of the lifting part, and the movement of the arm of the self-propelled robot" for "the combination of key operation signals of a set of operation keys 29".

[0075] Additionally, the operation control unit 27 outputs the operator's voice signal input from the operation-side microphone 25 to the operation-side communication unit 28. Furthermore, the operation control unit 27 outputs composite image specification information, predetermined movement path information, and arm animation information input from the operation-side display unit 23 to the operation-side communication unit 28.

[0076] On the other hand, the operation control unit 27 appropriately generates display signals for the composite image, the hand-end image, and the main image based on the composite image signal, the hand-end image signal, and the main image signal input from the operation-side communication unit 28, and outputs them to the operation-side display unit 23. At this time, the operation control unit 27 switches the display signals for the composite image, the hand-end image, and the main image according to the display switching information input from the operation-side display unit 23.

[0077] In addition, the operation control unit 27 outputs an interference warning image signal to the operation side display unit 23 based on the interference warning signal input from the operation side communication unit 28, and generates an interference warning sound signal based on the interference warning signal and outputs it to the operation side microphone 25.

[0078] In addition, the operation control unit 27 outputs the interlocutor's voice signal input from the operation side communication unit 28 to the operation side playback unit 26.

[0079] The operator-side communication unit 28 is composed of a communicator capable of data communication. The operator-side communication unit 28 converts the operation signals, operator voice signals, composite image specification information, predetermined movement path information, and arm animation information input from the operation control unit 27 into communication data (data packets) and sends them to the robot-side communication unit 113.

[0080] In addition, the operation-side communication unit 28 receives communication data from the robot-side communication unit 113, including composite image signal, hand-end image signal, main image signal, interference warning signal, and interlocutor voice signal, and restores them to composite image signal, hand-end image signal, main image signal, interference warning signal, and interlocutor voice signal respectively, and outputs them to the operation control unit 27.

[0081] Here, these communications are conducted via data communication network 3.

[0082] Here, the operation control unit 27 is composed of an arithmetic circuit module Cm2 having a processor Pr2 and a memory Me2. The operation control unit 27 is a functional module that executes a control program stored in the memory Me2 by the processor Pr2 in the arithmetic circuit module Cm2. Specifically, the arithmetic circuit module Cm2 is composed of, for example, a microcontroller, MPU, FPGA (Field Programmable Gate Array), PLC (Programmable Logic Controller), etc. They can be composed of a single arithmetic circuit module for centralized control, or they can be composed of multiple arithmetic circuit modules for distributed control.

[0083] {Structure of one side of the self-propelled robot}

[0084] The self-propelled robot 1 includes a walking unit 11, a lifting unit 12, an arm 13, a robot-side display unit 14, a robot-side microphone 15, a robot-side playback unit 16, a surrounding camera 17, a hand-mounted camera 18, a main camera 19, a robot control unit 112, a robot-side communication unit 113, an analog image generation unit 115, a composite image generation unit 116, and an interference warning unit 117.

[0085] The robot-side communication unit 113 is composed of a communicator capable of data communication. The robot-side communication unit 113 receives communication data such as operation signals, operator voice signals, composite image specification information, predetermined movement path information, and arm animation information from the operator-side communication unit 28, and restores them to operation signals, operator voice signals, composite image specification information, predetermined movement path information, and arm animation information, and outputs them to the robot control unit 112.

[0086] In addition, the robot-side communication unit 113 converts the composite image signal, hand-end image signal, main image signal, interference warning signal, and interlocutor voice signal input from the robot control unit 112 into communication data (data packets) and sends them to the operation-side communication unit 28.

[0087] The robot control unit 112 outputs the operation signals input from the robot-side communication unit 113 to the walking unit 11, the lifting unit 12, and the arm 13.

[0088] In addition, the robot control unit 112 outputs the composite image specification information, predetermined movement path information and arm animation information input from the robot-side communication unit 113 to the composite image generation unit 116.

[0089] In addition, the robot control unit 112 appropriately generates an image display signal and outputs it to the robot-side display unit 14.

[0090] Additionally, the robot control unit 112 outputs the operator's voice signal input from the robot-side communication unit 113 to the robot-side playback unit 16. Furthermore, in this case, the robot control unit 112 may, for example, cause the robot-side display unit 14 to display an image (e.g., an illustration) of a person wearing a uniform corresponding to the specified work site, and convert the operator's voice signal into a voice suitable for that person (e.g., a soft voice corresponding to the gender of the employee).

[0091] In addition, the robot control unit 112 outputs the composite image signal input from the composite image generation unit 116, the hand-end image signal input from the hand-end camera 18, and the main image signal input from the main camera 19 to the robot-side communication unit 113.

[0092] The walking unit 11, the lifting unit 12, and the arm 13 operate according to the operation signals input from the robot control unit 112.

[0093] The robot-side display unit 14 displays an image based on the image display signal input from the robot control unit 112.

[0094] The robot's side microphone 15 acquires the voice of the person speaking to it (e.g., a customer) and outputs it as the person speaking to it to the robot control unit 112.

[0095] The robot-side playback unit 16 plays sound based on the operator's voice signal input from the robot control unit 112. The robot-side playback unit 16 is, for example, a speaker.

[0096] The surrounding camera 17 captures images of the conditions (environment) around the walking robot 1 and outputs them as surrounding condition images to the composite image generation unit 116 and the interference warning unit 117.

[0097] The hand-end camera 18 captures the environment of the hand-end of the second robot arm 121B and outputs it as a hand-end image to the robot control unit 112. Examples of the environment of the hand-end of the second robot arm 121B include objects to be held by the robotic arm 124B.

[0098] The main camera 19 captures a field of view equivalent to that of a standing person and outputs it as the main image to the robot control unit 112. When the self-propelled robot 1 is facing a person with whom it is speaking, the image of the person with whom it is speaking is included in this main image.

[0099] Here, the robot control unit 112, analog image generation unit 115, composite image generation unit 116, and interference warning unit 117 are constituted by an arithmetic circuit module Cm1 having a processor Pr1 and a memory Me1. The processor Pr1 is an example of a processing circuit. The analog image generation unit 115, composite image generation unit 116, and interference warning unit 117 can also be referred to as an analog image generation circuit, a composite image generation circuit, and an interference warning circuit, respectively. The robot control unit 112, analog image generation unit 115, composite image generation unit 116, and interference warning unit 117 are functional modules implemented by the processor Pr1 in the arithmetic circuit module Cm1 executing the control program stored in the memory Me1. Specifically, the arithmetic circuit module Cm1 is composed of, for example, a microcontroller, MPU, FPGA (Field Programmable Gate Array), PLC (Programmable Logic Controller), etc. They can be composed of a single arithmetic circuit module for centralized control, or they can be composed of multiple arithmetic circuit modules for distributed control.

[0100] Here, the functions of the elements disclosed in this specification can be executed using circuitry or processing circuitry, including general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), existing circuits, and / or combinations thereof configured or programmed to perform the disclosed functions. A processor, because it contains transistors and other circuitry, can be considered a processing circuit or circuit. In this disclosure, a “unit” or “section” refers to hardware that performs or is programmed to perform the listed functions. The hardware can be the hardware disclosed in this specification, or it can be other known hardware programmed or configured to perform the listed functions. In the case of a processor where the hardware is considered a type of circuit, a “unit” or “section” is a combination of hardware and software used in the configuration of the hardware and / or processor.

[0101] <Structures related to composite images>

[0102] The following sections will explain the structures related to composite images in turn, according to each of the constituent elements.

[0103] {Analog Image Generation Unit 115}

[0104] Reference Figure 1 and Figure 4 The joints of the first and second robotic arms 121A and 121B of the self-propelled robot 1 are powered by motors MA (see reference). Figure 4 The motor MA is driven to change its posture. Each joint is equipped with a rotation angle detection unit EA (see reference 1) to detect the rotation angle of the motor MA. Figure 4 The rotation angle detection unit EA is, for example, composed of an encoder. Therefore, the posture of the first and second robot arms 121A and 121B can be obtained in real time using the rotation angle of the motor MA of each joint.

[0105] The simulation image generation unit 115 generates arm images that simulate the posture of the first and second robot arms 121A and 121B at all times based on the rotation angles output from the rotation angle detection unit EA of each joint of the first and second robot arms 121A and 121B.

[0106] The lifting section 12 of the self-propelled robot 1 is equipped with a motor ML that raises and lowers the lifting shaft 123 (see reference). Figure 4 Rotation angle detection unit EL (refer to) detects the rotation angle of a digit. Figure 4The rotation angle detection unit EL is, for example, composed of an encoder. Therefore, the posture of the lifting unit 12 can be obtained in real time using the rotation angle of the motor ML. The simulation image generation unit 115 generates a lifting unit image that simulates the posture of the lifting unit 12 at all times based on the rotation angle output from the rotation angle detection unit EL.

[0107] Furthermore, the simulation image generation unit 115 synthesizes the aforementioned arm image and the aforementioned lifting unit image to generate a self-propelled robot simulation image 160 (see reference) that continuously simulates the posture of the self-propelled robot 1, including the postures of the first and second robot arms 121A and 121B. Figures 5 to 7 Furthermore, the simulated image 160 of the self-propelled robot is output to the composite image generation unit 116. In generating the simulated image 160 of the self-propelled robot, for example, CAD data of the self-propelled robot 1 can be used. The simulated image 160 of the self-propelled robot can also be simplified to the extent that it does not significantly compromise the clarity of the posture of the self-propelled robot 1.

[0108] Specifically, the simulation image generation unit 115 generates any one of three self-walking robot simulation images based on the composite image specification information input from the composite image generation unit 116: a self-walking robot simulation image 160 obtained from a viewpoint overlooking the self-walking robot 1, a self-walking robot simulation image 160 obtained from an observation of the self-walking robot 1 from above, and a self-walking robot simulation image 160 composed of the arm simulation part 160a described later, which is arranged in the periphery of the surrounding situation image 50 (here, the left and right ends of the upper end) as observed from the self-walking robot 1.

[0109] {Composite image generation unit 116}

[0110] The composite image generation unit 116 performs image processing by combining the images captured from the four surrounding cameras 17 as described above, and generates three types of images: a top-view image, an upper-view image, and a first-person view image. Furthermore, these images are combined with simulated images of the self-propelled robot input from the simulation image generation unit 115 to create a composite image.

[0111] In this case, since the self-propelled robot simulation image contains three-dimensional information, it can be combined with the overhead view image, the top view image, and the first-person view image to correctly convert the self-propelled robot simulation image into images of these three viewpoints.

[0112] Figure 5 It is an overhead view diagram showing a composite image 501 of the surrounding situation image 50 and the self-propelled robot simulation image 160 as an image obtained from the viewpoint of overlooking the self-propelled robot. Figure 6It is an upper viewpoint diagram showing a composite image 601 of the surrounding situation image 50 and the self-walking robot simulation image 160 as an image obtained from observing the self-walking robot from an upper viewpoint. Figure 7 This is a first-person viewpoint image 701, which is a composite image 50 of the surrounding situation and a simulated image 160 of the self-propelled robot, shown as an image observed from the self-propelled robot. Figures 5-7 For example, it shows a self-walking robot 1 moving around in a person's home for care purposes.

[0113] Reference Figure 5 The composite image 501, viewed from above, is positioned in front of the surrounding situation image 50, which is viewed from above the self-propelled robot 1, and includes a simulated image 160 of the self-propelled robot 1, also viewed from above the self-propelled robot 1. The surrounding situation image 50 is captured by a wide-angle surrounding camera 17 and is therefore distorted.

[0114] Reference Figure 6 The composite image 601 of the upper viewpoint is positioned in front of the surrounding situation image 50 obtained from viewing the self-walking robot 1 from above, and includes a simulated image 160 of the self-walking robot 1 obtained from viewing the self-walking robot 1 from above.

[0115] Reference Figure 7 The first-person viewpoint composite image 701 includes an arm simulation portion 160a simulating a portion of the robot arms 121A and 121B of the self-propelled robot 1 in the peripheral portion (here, the left and right ends of the upper part) of the surrounding situation image 50 observed from the self-propelled robot 1, serving as a self-propelled robot simulation image 160. Specifically, the end portions 50a of the robot arms 121A and 121B are reflected at the left and right ends of the upper part of the surrounding situation image 50. The arm simulation portion 160a is shown connected to the end portions 50a of the robot arms 121A and 121B reflected in the surrounding situation image 50.

[0116] Furthermore, since the surrounding camera 17 is positioned below and in front of the robot arms 121A and 121B, the portions of the robot arms 121A and 121B, except for their distal ends, are not reflected in the surrounding situation image. Therefore, as described above, the arm simulation portion 160a of the self-propelled robot simulation image 160 is positioned at the left and right ends of the upper part of the surrounding situation image 50, connected to the distal ends 50a of the robot arms 121A and 121B reflected in the surrounding situation image 50. In this case, if it is desired to show the simulated portion of the base end of the robot arm in the self-propelled robot simulation image 160 (located behind the surrounding camera 17), it would be equivalent to showing it in the center of the surrounding situation image 50, thus obscuring the crucial central portion of the surrounding situation image 50. Therefore, for the simulated portion of the robot arm in the self-propelled robot simulation image 160, the portion corresponding to the base end of the robot arm is not shown, but is shown separately at the left and right ends of the upper part of the surrounding situation image 50, thereby allowing the central portion of the surrounding situation image 50 to be shown. In addition, the simulated portion of the robot arm in the self-walking robot simulation image 160 can be schematically (simplified), for example, the portion corresponding to the base end of the robot arm can be positioned above or below the surrounding image 50 to generate the self-walking robot simulation image 160.

[0117] The composite image generation unit 116 generates the three composite images 501, 601, and 701 through the above synthesis. Specifically, if composite image specification information is input from the robot control unit 112, the composite image generation unit 116 outputs the composite image specification information to the analog image generation unit 115, generates the specified composite image among the three composite images 501, 601, and 701, and outputs it to the robot control unit 112.

[0118] <Structure related to the predetermined movement path 802 of the self-propelled robot 1>

[0119] Figure 8 It is a diagram showing a composite image of the self-propelled robot 1's predetermined movement path 802 and the surrounding situation image 50 superimposed.

[0120] Reference Figure 8 In the composite image 801, the predetermined movement path 802 of the self-propelled robot 1 is shown overlapping with the surrounding situation image 50. The predetermined movement path 802 is shown as extending from the self-propelled robot simulation image 160 to the target location.

[0121] If the composite image generation unit 116 receives predetermined movement path information from the robot control unit 112, it displays the predetermined movement path 802 of the self-propelled robot 1 overlaid with the surrounding situation image 50. In this case, the composite image generation unit 116 generates the predetermined movement path 802, for example, based on the target position of the self-propelled robot 1 indicated by the predetermined movement path information and the current position of the self-propelled robot 1. The current position of the self-propelled robot 1 is obtained, for example, based on the rotation angle of the motor driving the walking unit of the self-propelled robot 1.

[0122] Alternatively, the composite image generation unit 116 can also generate a predetermined movement path 802 based on the operation signal received by the robot control unit 112. In this case, the movement (walking) target value (command value) of the self-propelled robot 1 in the operation signal is the target position of the self-propelled robot 1. Furthermore, the predetermined movement path information does not include the movement target position of the self-propelled robot 1. Figure 8 The predetermined movement path 802 is shown in the composite image from an overhead viewpoint, but similarly, the predetermined movement path 802 can also be shown in the composite image from an upper viewpoint or a first-person viewpoint.

[0123] <Structures related to arm animation>

[0124] Figure 9 It is a composite image 901 showing an arm animation 803 illustrating the pose changes of the robotic arms 121A and 121B of the self-walking robot 1, superimposed with a self-walking robot simulation image 160 and a surrounding situation image 50. Figures 10A to 10D These are images from a frame of arm animation 803 showing the posture changes of the robotic arm 121 of the self-propelled robot 1. Figures 10A to 10D In the animation, robot arms 121A and 121B are shown in a simplified form. The illustration of the U-shaped cables is also omitted. The robot arms in arm animation 803 can be shown faithfully as the actual robot arms 121A and 121B, or they can be shown in a more simplified form.

[0125] Reference Figure 9 If the composite image generation unit 116 receives arm animation information from the robot control unit 112, it displays the arm animation 803 overlaid with the self-propelled robot simulation image 160 and the surrounding situation image 50. Alternatively, the arm animation 803 may be displayed overlaid only with the self-propelled robot simulation image 160 or only with the surrounding situation image 50. Figures 10A to 10D As shown, the animation 803 shows the changes in the robot arms 121A and 121B.

[0126] In this case, the composite image generation unit 116 generates arm animation 803, for example, based on the target position (pose) of the robot arms 121A and 121B represented by the arm animation information and the current position (pose) of the robot arms 121A and 121B. The current position of the self-propelled robot 1 is obtained based on the rotation angle output from the rotation angle detection unit EA of each joint of the first and second robot arms 121A and 121B.

[0127] Alternatively, the composite image generation unit 116 can also generate a predetermined movement path 802 based on the operation signals received by the robot control unit 112. In this case, the position command values ​​of the robot arms 121A and 121B in the operation signals are the target positions of the robot arms 121A and 121B. Furthermore, the arm animation information does not include the target positions of the robot arms 121A and 121B. Figure 9 Arm animation 803 is shown in a composite image from an upper viewpoint, but similarly, arm animation 803 can also be shown in a composite image from a top-down viewpoint or a first-person viewpoint.

[0128] <Structures related to interference warnings>

[0129] The interference warning unit 117 generates an interference warning signal based on the surrounding situation image input from the surrounding camera 17 and the posture of the self-propelled robot 1, and outputs it to the robot control unit 112.

[0130] The surrounding environment image contains three-dimensional information. The interference warning unit 117 first extracts the three-dimensional outlines (hereinafter referred to as objects) of objects existing in the lateral and directional directions of the self-propelled robot 1 from the surrounding environment image through image processing. Next, the interference warning unit 117 uses the depth information of the surrounding environment image to obtain the distance between the extracted object and the self-propelled robot 1. Then, the interference warning unit 117 determines, for example, whether the self-propelled robot 1 interferes with the object based on the distance and orientation of the extracted object. If the interference warning unit 117 determines that the self-propelled robot 1 is interfering with the object, it outputs an interference warning signal to the robot control unit 112.

[0131] The interference warning signal is then sent to the operation control unit 27 via the robot control unit 112, the robot-side communication unit 113, and the operation-side communication unit 28. Based on the interference warning signal, the operation control unit 27 causes the operation-side display unit 23 to display the interference warning and the operation-side playback unit 26 to play the interference warning sound.

[0132] [action]

[0133] Next, the operation (robot operation method) of the robot system 100 configured as described above will be explained.

[0134] Reference Figure 1 and Figure 2 Operator P operates the operation unit 21 of the operation unit 2, causing the self-walking robot 1 to move within the individual's residence to perform caregiving tasks. During this movement, the self-walking robot 1 performs the necessary caregiving tasks. Operator P primarily performs these tasks while viewing the main image and hand-eye image displayed on the operation-side display unit 23 of the operation unit 2. Operator P can switch between the main image, hand-eye image, and composite image by touching the operation-side display unit 23. Furthermore, as needed, operator P can communicate with caregivers or their associates using the operation-side microphone 25 and operation-side playback unit 26 of the operation unit 2, and the robot-side display unit 14, robot-side microphone 15, and robot-side playback unit 16 of the self-walking robot 1.

[0135] Furthermore, when the operator P moves the self-propelled robot 1, he touches the operation-side display unit 23, causing the operation-side display unit 23 to display the desired composite images 501, 601, and 701. Regarding the composite images 501, 601, and 701, as the self-propelled robot 1 moves, the surrounding situation image 50 changes constantly, and if the posture of the arm 13 and the lifting unit 12 changes due to the operation, the self-propelled robot simulation image 160 changes constantly. In this case, especially since the posture of the arm in the self-propelled robot simulation image 160 changes constantly, the operator P can make the self-propelled robot 1 move in a way that does not interfere with the surrounding objects.

[0136] Furthermore, in this case, if operator P touches the operation-side display unit 23 and inputs predetermined movement path information including the target position of the self-propelled robot 1, a composite image 801 including the predetermined movement path 802 of the self-propelled robot 1 is displayed on the operation-side display unit 23. Operator P can reliably make the self-propelled robot 1 walk while referring to the predetermined movement path 802.

[0137] If operator P touches the operation-side display unit 23 and inputs arm animation information including the target positions of the robot arms 121A and 121B of the self-propelled robot 1, a composite image 901 including arm animation 803 is displayed on the operation-side display unit 23. Operator P can reliably operate the robot arms 121A and 121B while referring to the arm animation 803 to perform tasks appropriately.

[0138] Furthermore, if the self-propelled robot 1 interferes with surrounding objects while walking, an interference warning display is shown on the operation-side display unit 23, and an interference warning sound is played from the operation-side playback unit 26. Based on the interference warning display and sound, the operator P infers the possibility of interference and operates the operation unit 2 to cause the self-propelled robot 1 to perform the necessary interference avoidance actions.

[0139] (Other implementation methods)

[0140] In the above embodiment, the simulated image generation unit 115 may also be configured to generate a simulated image 160 of the self-walking robot that omits the posture changes of the lifting unit 12.

[0141] As described above, according to the embodiments of this disclosure, interference between the self-propelled robot 1 equipped with robotic arms 121A and 121B and surrounding objects can be avoided.

[0142] In addition, the robot arms 121A and 121B are equipped with rotation angle detection units EA that detect the rotation angle of the motors MA that drive each joint. The simulation image generation unit 116 is configured to generate a self-walking robot simulation image 160 based at least on the rotation angle detected by the rotation angle detection units EA corresponding to each joint of the robot arms 121A and 121B.

[0143] Therefore, since the self-propelled robot simulation image 160 is generated based on the rotation angles detected by the rotation angle detection unit EA corresponding to each joint of the robot arms 121A and 121B, the postures of the robot arms 121A and 121B in the self-propelled robot simulation image 160 become the correct postures in real time. As a result, interference between the self-propelled robot 1 equipped with robot arms 121A and 121B and surrounding objects can be avoided more reliably.

[0144] Furthermore, the robot system 100 is configured such that, when the composite image generation unit 116 generates a composite image 701 from a first-person perspective observed by the self-propelled robot 1, the simulation image generation unit 115 generates a self-propelled robot simulation image 160 in which an arm simulation portion 160a simulating at least a portion of the robot arms 121A and 121B in the self-propelled robot 1 that is not reflected in the surrounding situation image 50 is connected to the aforementioned portion 50a of the robot arm that is reflected in the surrounding situation image, and the composite image generation unit 116 generates a composite image 50 from a first-person perspective in which the arm simulation portion 160a of the self-propelled robot simulation image 160 is connected to the portion 50a of the robot arm that is reflected in the surrounding situation image 50.

[0145] Therefore, even in the first-person viewpoint composite image 701 of the surrounding situation image 50, which does not show the entirety of the robotic arms 121A, 121B of the self-walking robot 1 due to the configuration of the surrounding camera 17, it is possible to appropriately generate a self-walking robot simulation image 160, which includes an arm simulation portion 160a that simulates at least a portion of the parts of the robotic arms 121A, 121B in the self-walking robot 1 that are not shown in the surrounding situation image 50.

[0146] In addition, the composite image generation unit 116 is configured to generate a composite image 801 that shows the predetermined movement path 802 of the self-propelled robot 1 overlapping with the surrounding situation image 50.

[0147] Therefore, the operator P can make the self-walking robot walk reliably while watching the predetermined movement path 802 of the self-walking robot 1.

[0148] Furthermore, the composite image generation unit 116 is configured to generate a composite image 601 that overlaps at least one of the following: an arm animation 803 showing the posture changes of the robot arms 121A and 121B of the self-propelled robot 1, and an image 601 showing the surrounding situation image 50 and a simulated image 160 of the self-propelled robot. Therefore, the operator P can reliably operate the robot arms 121A and 121B to perform tasks while viewing the animation 803.

[0149] In addition, the robot system 100 also has an interference warning unit 117, which determines whether the robot arms 121A and 121B interfere with objects around the self-propelled robot 1 based on the surrounding situation image captured by the surrounding camera 17 and the posture of the self-propelled robot 1, and outputs an interference warning signal if interference is determined to have occurred.

[0150] Therefore, interference warning signals can be used to avoid interference between the robot arms 121A and 121B and objects around the self-propelled robot 1.

[0151] Furthermore, the display unit 23 is configured to display an image indicating an interference warning based on the interference warning signal output from the interference warning unit 116.

[0152] Therefore, the operator P can view the display on the display unit 23 to understand the possibility of interference between the robot arms 121A and 121B and the objects around the self-propelled robot 1.

[0153] The robot system 100 also includes an interference warning reporting unit 26, which is separate from the display 23 and reports interference warnings based on the interference warning signals output from the interference warning unit 116.

[0154] Therefore, the operator P can understand the possibility of interference between the robot arms 121A and 121B and objects around the self-propelled robot 1 through the report from the interference warning report unit 26.

[0155] Based on the above description, many improvements and other implementations will be apparent to those skilled in the art. Therefore, the above description should be interpreted as illustrative only.

[0156] The functions of the elements disclosed in this specification can be executed using circuitry or processing circuitry, including general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), existing circuits, and / or combinations thereof configured or programmed to perform the disclosed functions. A processor, because it contains transistors and other circuitry, can be considered a processing circuit or a circuit. In this disclosure, a circuit, unit, or means is hardware that performs or is programmed to perform the listed functions. The hardware can be the hardware disclosed in this specification, or it can be other known hardware programmed or configured to perform the listed functions. In the case where the hardware is considered a processor, a type of circuit, the circuit, means, or unit is a combination of hardware and software used in the configuration of the hardware and / or processor.

[0157] One aspect of the robot system disclosed herein includes: a self-propelled robot including a robotic arm having one or more joints; an operating unit that is operated by an operator to operate the self-propelled robot; a display that is visually recognized by the operator; an ambient camera mounted on the self-propelled robot and capturing images of the environment surrounding the self-propelled robot; and a processing circuit configured to generate a simulated image of the self-propelled robot that continuously simulates the posture of the self-propelled robot, including the posture of the robotic arm, and to generate a composite image that includes the ambient image captured by the ambient camera and the generated simulated image of the self-propelled robot and displays it on the display.

[0158] According to this structure, since the display shows a simulated image of the self-propelled robot, which simulates the posture of the self-propelled robot including the posture of the robot arm, together with images of the surrounding environment captured by surrounding cameras, the operator can view the display to operate the control unit to avoid interference between the self-propelled robot with the robot arm and surrounding objects.

[0159] Alternatively, the robotic arm may include: one or more motors that drive one or more joints respectively; and one or more rotation angle detection units that detect the rotation angle of one or more motors respectively, wherein the processing circuit is configured to generate a simulated image of the self-walking robot based at least on the rotation angle detected by the one or more rotation angle detection units.

[0160] In the robot system described above, the processing circuit may also be configured to generate the self-propelled robot simulation image by connecting an arm simulation portion that simulates at least a portion of the robot arm in the self-propelled robot that is not reflected in the surrounding situation image to a portion of the robot arm that is reflected in the surrounding situation image, when generating a composite image from a first-person perspective observed by the self-propelled robot, and to generate the composite image from the first-person perspective by connecting the arm simulation portion of the generated self-propelled robot simulation image to a portion of the robot arm that is reflected in the surrounding situation image.

[0161] In the robot system described above, the processing circuit can also be configured to generate a composite image showing the predetermined movement path of the self-propelled robot superimposed on the surrounding situation image.

[0162] In the robot system described above, the processing circuit may also be configured to generate a composite image that overlays an arm animation showing the posture changes of the robot arm of the self-propelled robot with an image of the surrounding environment or a simulated image of the self-propelled robot.

[0163] In the robot system described above, the processing circuit can also determine whether the robot arm interferes with objects around the self-propelled robot based on the surrounding environment image captured by the surrounding camera and the posture of the self-propelled robot. If interference is determined to have occurred, an interference warning signal is output.

[0164] In the robot system described above, the display can also be configured to display an image indicating an interference warning based on the output interference warning signal.

[0165] The robot system described above may also include an interference warning reporter, which is configured separately from the display and reports interference warnings based on the output interference warning signal.

[0166] One aspect of the robot operation method disclosed herein includes: operating a self-propelled robot equipped with a robotic arm; generating a simulated image of the self-propelled robot that continuously simulates the posture of the self-propelled robot, including the posture of the robotic arm; setting a surrounding camera around the self-propelled robot to capture the surrounding conditions; generating a composite image including the surrounding condition image captured by the surrounding camera and the simulated image of the self-propelled robot; and displaying the composite image.

[0167] This structure can prevent self-propelled robots with robotic arms from interfering with surrounding objects.

Claims

1. A robot system, wherein, have: Self-propelled robots, including robotic arms with more than one joint; The operation unit, subject to operations performed by the operator, is used to operate the self-propelled robot. The display is visually identified by the operator. Surrounding cameras, mounted on the self-propelled robot, capture images of the environment surrounding the self-propelled robot; and Processing circuit, The processing circuit is configured as follows: Generate simulated images of the self-walking robot, including the posture of the robotic arm, at all times. It generates a composite image that includes images of the surrounding environment captured by the surrounding cameras and a simulated image of the self-propelled robot, and displays it on the display. The composite images include an overhead view composite image obtained from a viewpoint overlooking the self-propelled robot, an upper view composite image obtained from a viewpoint above the self-propelled robot, and a first-person view composite image observed from the self-propelled robot. The designated composite image from the overhead viewpoint composite image, the top viewpoint composite image, and the first-person viewpoint composite image will be displayed on the display.

2. The robot system according to claim 1, wherein, The robotic arm has the following features: One or more motors, each driving one or more joints; and One or more rotation angle detection units are used to detect the rotation angle of the one or more motors. The processing circuit is configured to generate a simulated image of the self-walking robot based on at least one rotation angle detected by the rotation angle detection unit.

3. The robot system according to claim 1 or 2, wherein, The processing circuit is configured as follows: In the case of generating the first-person viewpoint composite image The self-propelled robot simulation image is generated by connecting a simulated arm portion (at least a portion of the robot arm in the self-propelled robot that is not reflected in the surrounding situation image) to a portion of the robot arm that is reflected in the surrounding situation image. A composite image from the first-person viewpoint is generated by connecting the simulated arm portion of the generated self-walking robot simulation image with a portion of the robot arm already reflected in the surrounding situation image.

4. The robot system according to claim 1 or 2, wherein, The processing circuit is configured to generate a composite image showing the predetermined movement path of the self-propelled robot superimposed on the surrounding situation image.

5. The robot system according to claim 1 or 2, wherein, The processing circuit is configured to generate a composite image that overlays an arm animation showing the posture changes of the robot arm of the self-propelled robot with an image of the surrounding conditions or a simulated image of the self-propelled robot.

6. The robot system according to claim 1 or 2, wherein, The processing circuit determines whether the robot arm interferes with objects around the self-propelled robot based on images of the surrounding environment captured by the surrounding camera and the posture of the self-propelled robot. If interference is determined to have occurred, an interference warning signal is output.

7. The robot system according to claim 6, wherein, The display is configured to show an image representing an interference warning based on the output interference warning signal.

8. The robot system according to claim 6, wherein, It also includes an interference warning reporter, which is configured separately from the display and reports interference warnings based on the output interference warning signal.

9. A robot operation method, in, include: Operating a self-propelled robot equipped with a robotic arm; Generate simulated images of the self-walking robot that simulate the posture of the self-walking robot, including the posture of the robot arm, at all times; A surrounding camera is installed on the self-propelled robot to capture the situation around the self-propelled robot; Generate a composite image comprising images of the surrounding conditions captured by the surrounding cameras and simulated images of the self-propelled robot; and Display the composite image. The composite images include an overhead view composite image obtained from a viewpoint overlooking the self-propelled robot, an upper view composite image obtained from a viewpoint above the self-propelled robot, and a first-person view composite image observed from the self-propelled robot. The designated composite image from the overhead viewpoint composite image, the top viewpoint composite image, and the first-person viewpoint composite image will be displayed on the monitor.

Images