Robot, image generation system, image generation device, robot control device, and image generation method

Abstract

The present invention realizes a camera trajectory close to an inspection operation of a maintenance person even when a camera of a robot arm is subjected to an operation constraint. To this end, a robot (100) is provided with: an in-robot transceiver (106) that receives line-of-sight information of a maintenance person and information of a plurality of maintenance points in appearance inspection work; a camera trajectory generator (115) that generates a camera trajectory of a camera (101) from the line-of-sight information, the information of the maintenance points, and specification conditions of the camera; a robot operation constraint generator (116) that generates an operation constraint of the robot from environment information for robot operation and a mechanical constraint of the robot; a camera avoidance trajectory generator (118) that generates an avoidance trajectory of the camera for avoiding the operation constraint of the robot; an overall operation generator (119) that generates robot operation information of the robot using the camera trajectory, the operation constraint of the robot, and the avoidance trajectory of the camera; and a controller (113) that controls the robot using the robot operation information generated by the overall operation generator.

Description

Robot, Image Generation System, Image Generation Device, Robot Control Device, and Image Generation Method

[0001] The present invention relates to a robot, an image generation system, an image generation device, a robot control device, and an image generation method for photographing an image of an object.

[0002] In recent years, with the decline of the working population, the need for automation of tasks that have been conventionally performed by humans has increased, and work substitution by robots has been progressing. As an example of work substitution by robots, there is an appearance inspection work. Appearance inspection work is work for checking an object from various angles and confirming whether there are any abnormalities such as scratches or dirt on the appearance. For the automation of appearance inspection work, image data from various positions and angles are required.

[0003] For example, Patent Document 1 discloses that "an appearance inspection system for performing an appearance inspection of an inspection object includes a display device, a robot for moving an imaging device, and while the robot is moving the imaging device, the imaging device images each of a plurality of inspection parts of the inspection object, and based on each image obtained from the imaging device, an inspection unit for inspecting whether there are defects in each of the plurality of inspection parts, and a display control unit for displaying on the display device an inspection result matrix representing the inspection results of the inspection unit for each of the plurality of inspection parts for each inspection object."

[0004] Japanese Unexamined Patent Application Publication No. 2020-003300

[0005] When automating, for example, the appearance inspection performed by a maintenance worker in a narrow space under a railway vehicle using a camera provided on a robot arm, the robot arm may be restricted in its operation by the environment and may not be able to execute the camera trajectory for collecting image data as performed by the maintenance worker. Therefore, the technology of Patent Document 1 that does not consider the operation restrictions of the robot arm cannot be applied.

[0006] An object of the present invention is to provide a robot, an image generation system, an image generation device, a robot control device, and an image generation method capable of appropriately photographing an object even in an environment where operation restrictions occur when a robot equipped with a camera photographs the object.

[0007] To solve the aforementioned problems, the present invention provides a robot equipped with a camera, comprising: an in-robot transceiver that receives eye-line information of a maintenance worker and information of multiple maintenance points during visual inspection work; a camera trajectory generator that generates a camera trajectory of the camera from the eye-line information, the maintenance point information and the specifications of the camera; a robot motion constraint generator that generates motion constraints of the robot from environmental information in which the robot operates and the mechanical constraints of the robot; a camera avoidance trajectory generator that generates an avoidance trajectory of the camera that avoids the motion constraints of the robot; an overall motion generator that generates robot motion information of the robot using the camera trajectory, the motion constraints of the robot and the avoidance trajectory of the camera; and a controller that controls the robot using the robot motion information generated by the overall motion generator.

[0008] According to the present invention, it is possible to provide a robot, an image generation system, an image generation device, a robot control device, and an image generation method that enable a robot equipped with a camera to appropriately photograph an object even in environments where movement constraints occur when taking pictures, by setting an appropriate avoidance trajectory.

[0009] This is a side view of the robot of the embodiment. This is a perspective view of the railway vehicle under-vehicle environment, which is the inspection target environment of the embodiment. This is an external view of the input device. This is a functional block diagram explaining the functions of the robot. This is a functional block diagram of the visual inspection work measurement device and the input device. This is a hardware configuration diagram of the computer. This is a diagram showing the inspection work of a maintenance worker in the railway vehicle under-vehicle environment. This is a view of the inspection target facing directly from the axle side of the wheel. This is a diagram explaining the camera trajectory. This is a diagram showing the state in which the camera photographs the first inspection target O1. This is a diagram showing the state in which the camera photographs the second inspection target O2. This is a diagram showing the state in which the camera moves to a position to photograph the intermediate point O2' in the middle of the inspection path P2. This is a diagram showing the state in which the robot changes its posture so that the camera goes under the axle. This is a diagram showing the state in which the robot changes its posture so that the camera goes under the axle. This is a diagram showing the state in which the robot moves the camera to a position to photograph the intermediate point O2'. This is a diagram showing the state in which the robot moves the camera to a position to photograph the third inspection target O3. This is a diagram showing the state in which the camera photographs the first inspection target O1. This figure shows the state in which the camera photographs the second inspection target O2. This figure shows the state in which the robot changes its posture so that the camera goes under the axle. This figure shows the state in which the robot changes its posture so that the camera goes under the axle. This figure shows the state in which the robot positions the camera to photograph the second inspection target O2. This figure illustrates the camera trajectory and shooting operation when an intermediate point O2' is provided. This figure illustrates the camera trajectory and shooting operation when it is not possible to avoid camera operation constraints by adding a trajectory (when an intermediate point cannot be provided). This is a flowchart illustrating the image generation operation in the image generation system of the embodiment. This figure illustrates other configurations of the image generation system of the embodiment. This figure illustrates a system configuration in which a camera trajectory is generated outside the robot, the generated camera trajectory is notified to the robot, and an inspection image is acquired.

[0010] The image generation system of this embodiment is an image generation system for inspection images, comprising a visual inspection work measurement device 600 and a robot 100, and is configured so that the robot 100 acquires an image of the inspection target that the maintenance worker is looking at during a visual inspection. The visual inspection work measurement device 600 obtains time-series data of the maintenance worker's gaze information, the inspection route and the inspection target, and the position of the eyes during a visual inspection performed by the maintenance worker. The robot 100 then generates a camera trajectory, which is the time change in the position and direction of the camera 101, from the trajectory of the maintenance worker's gaze point and the direction of their gaze obtained by the visual inspection work measurement device 600, and generates operation information for the robot 100 that satisfies this camera trajectory in the inspection environment. The robot 100 operates based on the generated operation information and acquires an image of the inspection target that the maintenance worker is looking at by photographing the inspection target with the camera 101.

[0011] Furthermore, the visual inspection work measurement device 600 may determine the gaze point trajectory, which is the time change in the position of the gaze point on the inspection object that the maintenance worker is looking at, and the direction of the maintenance worker's line of sight for each gaze point. The robot 100 may then generate a camera trajectory, which is the time change in the position and direction of the camera 101, from the gaze point trajectory and line of sight of the maintenance worker determined by the visual inspection work measurement device 600, and generate operation information for the robot 100 that satisfies this camera trajectory in the inspection environment. Embodiments of the present invention will be described in detail below with reference to the drawings.

[0012] Figure 1 is a side view of the robot 100 according to the embodiment. The robot 100 according to the embodiment is a quadruped walking robot and is equipped with a four-legged movement mechanism 103 at the bottom of the main body 104 that can move in all directions on a plane. The robot 100 is also equipped with a serial link mechanism arm 102 with a camera 101 at its tip at the top of the main body 104. Furthermore, the main body 104 of the robot 100 has a transceiver 106, and inside the main body 104 is a controller (not shown) that controls the robot using sensors to detect the robot's position and orientation, and force sensors at each joint of the legs.

[0013] The robot in the embodiment shown in Figure 1 is a quadruped robot, but it is not limited to this, as long as the camera arm has redundancy in its degrees of freedom and can assume multiple postures with a fixed camera position. Furthermore, the locomotion method is not limited to the quadruped locomotion mechanism 103, but may also be a crawler-type locomotion mechanism.

[0014] Figure 2 is a perspective view of the railway vehicle under-vehicle environment 700, which is the inspection environment of the embodiment. In the inspection environment of the embodiment, the first inspection target structure 1, the second inspection target structure 2, and the third inspection target structure 3 are located behind the wheels 4 that straddle the rails 5a and 5b. A maintenance worker 800 (not shown) performs a visual inspection of the first inspection target structure 1, the second inspection target structure 2, and the third inspection target structure 3 with the wheels 4 in between.

[0015] Figure 3 is an external view of the input device 200 (output device) that inputs operation commands to the robot 100 (Figure 1) of the embodiment and checks the status of the robot 100. The input device 200 also inputs operation commands to the visual inspection work measurement device 600, which will be described later, and checks the status of the visual inspection work measurement device 600.

[0016] The input device 200 is an information terminal (computer) equipped with a display unit 201, an input unit 202, and a transceiver 204. The input device 200 transmits operation commands entered by the user from the input unit 202 to the robot 100 and the visual inspection work measurement device 600 via the transceiver 204, receives status information from the robot 100 and the visual inspection work measurement device 600, and displays it on the display unit 201 for the user to confirm.

[0017] Next, a robot according to an embodiment, an image generation device that generates inspection images using the robot, and an image generation system will be described.

[0018] Figure 4 is a functional block diagram illustrating the functions of the robot 100. The robot 100 functions as an image generation device that generates inspection images, and together with the input device 200 and the visual inspection work measurement device 600, it constitutes an image generation system.

[0019] As shown in Figure 4, the robot 100 consists of a camera 101, a position / attitude sensor 107, a force sensor 108, an environmental measurement sensor 109, a robot controller 110, an arm 102, a movement mechanism 103, and a transceiver 106.

[0020] Camera 101 is an imaging device attached to the tip of the arm 102 (Figure 1) of the robot 100, which captures image data for visual inspection. Position and orientation sensor 107 is a sensor that measures the position and orientation of the robot 100. For example, it is an acceleration sensor or a gyroscope. Force sensor 108 is a torque sensor that measures the torque of each joint of the robot, and measures the load applied to the robot 100.

[0021] The environmental measurement sensor 109 is a point cloud sensor capable of measuring the three-dimensional shape of the environment surrounding the robot 100. The arm 102 is a serial link mechanism with a camera 101 at its tip.

[0022] The mobile mechanism 103 is a four-legged mobile mechanism capable of moving in all directions on a plane. The transceiver 106 is a wireless communication unit connected to the input device 200, which receives operation instructions for the robot 100 from the input device 200 and transmits status information of the robot 100 to the input device 200.

[0023] The robot controller 110 consists of a sensor signal processor 111, a motion generator 112, a motion executer 113 (controller), and a motion result saver 114 (image group generator).

[0024] The sensor signal processor 111 converts electrical signals from the camera 101, position / attitude sensor 107, force sensor 108, and environmental measurement sensor 109 into physical parameters and notifies the motion generator 112 and motion result saver 114.

[0025] The motion execution unit 113 drives and controls the arm 102 and the movement mechanism 103 based on a time-series motion pattern (robot motion information) of each joint of the arm 102 and the movement mechanism 103, which is generated so that the position of the camera 101 follows the camera trajectory generated by the motion generator 112, which will be described later.

[0026] The motion generator 112 consists of a camera trajectory generator 115, a robot motion dynamic generator 116 (robot motion constraint generator), a relay point generator 117, a camera motion constraint avoidance trajectory generator 118 (camera avoidance trajectory generator), and an overall motion generator 119.

[0027] The camera trajectory generator 115 calculates a camera trajectory that captures an image similar to the image the maintenance worker 800 saw while performing the visual inspection, based on the gaze information, inspection target and inspection route, and time-series data of the eye position during the visual inspection work performed by the maintenance worker 800, stored in the visual inspection work data saver 203 of the input device 200, as well as the specifications of the camera 101 such as resolution and field of view.

[0028] The robot motion dynamic generator 116 identifies obstacles and other objects from the working environment information of the robot 100 measured by the environmental measurement sensor 109, and calculates the area in which the camera 101 can move due to the robot's motion, based on the robot's mechanical constraints. In other words, the robot motion dynamic generator 116 determines the motion constraints of the camera 101.

[0029] Details of the relay point generator 117 and the camera operation constraint avoidance trajectory generator 118 will be described later.

[0030] The overall motion generator 119 generates time-series patterns (robot motion information) for each joint of the arm 102 and the movement mechanism 103 based on the camera trajectory, so that the position of the camera 101 follows the camera trajectory generated by the camera trajectory generator 115.

[0031] The operation result saver 114 consists of an avoidance operation image remover 120 and an inspection image generator 121, and generates an image of the object to be inspected from the image captured by the camera 101 of the robot 100. As will be described in detail later, the image of the object to be inspected is generated by removing images from the image captured by the camera 101 when an avoidance operation was performed due to operational constraints of the camera 101 in the inspection target environment.

[0032] Next, using the functional block diagram in Figure 5, we will explain the functions of the input device 200 and the visual inspection work measurement device 600, which constitute the image generation system together with the robot 100 described in Figure 4.

[0033] As shown in Figure 5, the visual inspection work measurement device 600 consists of a gaze measurement sensor 601, an inspection target measurement sensor 602, a position / orientation sensor 603, and a visual inspection work controller 610.

[0034] The gaze measurement sensor 601 is a sensor that measures the gaze of the maintenance worker 800 while he is working, for example, a sensor that measures the eye movements of the maintenance worker 800. The inspection target measurement sensor 602 is a sensor that measures the inspection target during the visual inspection work, for example, a camera that takes an image of the structure in the direction the maintenance worker 800 is facing, or a sensor that can measure the three-dimensional shape of the structure. The position and orientation sensor 603 is a sensor that measures changes in the position and orientation of the visual inspection work measurement device 600, and is an acceleration sensor or a gyroscope sensor.

[0035] The visual inspection work controller 610 consists of a sensor signal processor 611, a visual inspection work extractor 612, a measurement result saver 613, and a transceiver 604. The sensor signal processor 611 converts electrical signals from the gaze measurement sensor 601, the inspection target measurement sensor 602, and the position / attitude sensor 603 into physical parameters.

[0036] The visual inspection work extractor 612 consists of a gaze detector 614, an inspection target extractor 615, and an eye position calculator 616. Based on the measurements of the gaze measurement sensor 601, the inspection target measurement sensor 602, and the position / orientation sensor 603, the visual inspection work extractor 612 obtains time-series data of the gaze information of the maintenance worker 800 during the visual inspection work, the inspection target and inspection path, and the position of the eyes. The visual inspection work extractor 612 may also obtain the gaze point trajectory, which is the time change in the position of the gaze point on the inspection target that the maintenance worker 800 is looking at, and the direction of the maintenance worker's gaze for each gaze point. In this specification, gaze information and eye position are sometimes collectively referred to as gaze information.

[0037] The measurement result storage unit 613 stores time-series data of the maintenance worker 800's gaze information, inspection target and inspection route, and eye position during the visual inspection work, which are obtained by the visual inspection work extractor 612. The transceiver 604 is a communication unit that receives work commands for the visual inspection work measurement device 600 from the input device 200 (described later) and transmits the time-series data of the maintenance worker 800's gaze information, inspection target and inspection route, and eye position during the visual inspection work, which are stored in the measurement result storage unit 613, to the input device 200.

[0038] The input device 200 consists of a display unit 201, an input unit 202, a visual inspection work data storage unit 203, and a transceiver 204. It inputs operation commands to the visual inspection work measurement device 600 and the robot 100, and also checks their status. The functions of the display unit 201, the input unit 202, and the transceiver 204 are explained in Figure 3, and are omitted here.

[0039] The visual inspection work data storage device 203 stores time-series data of the maintenance worker 800's gaze information, inspection target and inspection route, and eye position received from the visual inspection work measurement device 600, and transmits it to the robot 100.

[0040] Next, we will describe the specific hardware configuration that realizes the robot controller 110, the input device 200, and the visual inspection work controller 601.

[0041] Specifically, the robot controller 110, input device 200, and visual inspection work controller 601 of the embodiment are realized by the computer 900 shown in the hardware configuration diagram of Figure 6. The computer 900 has a CPU (Central Processing Unit) 901, ROM (Read Only Memory) 902, RAM 903, HDD (Hard Disk Drive) 904, input / output I / F (Interface) 905, communication I / F 906, and media I / F 907. The HDD 904 may be an SSD (Solid State Drive).

[0042] The CPU 901 operates based on a program stored in the ROM 902 or the HDD 904 and controls each part of the computer 900. The ROM 902 stores a boot program executed by the CPU 901 when the computer 900 is started up, a program related to the hardware of the computer 900, and the like.

[0043] The CPU 901 controls an input device 910 such as a mouse or a keyboard and an output device 911 such as a display or a printer via the input / output I / F 905. The CPU 901 acquires data from the input device 910 via the input / output I / F 905 and outputs the generated data to the output device 911. Note that, together with the CPU 901, a GPU (Graphics Processing Unit) or the like may be used as a processor.

[0044] The HDD 904 stores a program executed by the CPU 901 and data used by the program and the like. The communication I / F 906 receives data from another device via a communication network (for example, NW (Network) 920) and outputs it to the CPU 901, and also transmits data generated by the CPU 901 to another device via the communication network.

[0045] The media I / F 907 reads a program or data stored in the recording medium 912 and outputs it to the CPU 901 via the RAM 903. The CPU 901 loads a program related to the target processing from the recording medium 912 onto the RAM 903 via the media I / F 907 and executes the loaded program. The recording medium 912 is an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as a MO (Magneto Optical disk), a magnetic recording medium, a semiconductor memory, or the like.

[0046] For example, when the computer 900 or the like functions as the robot controller 110 of the embodiment, the CPU 901 of the computer 900 realizes the functions of each processing unit of the sensor signal processor 111, the motion generator 112, and the motion executor 113 by executing the program loaded onto the RAM 903.

[0047] The CPU 901 reads and executes a program related to the target process from the recording medium 912. Additionally, the CPU 901 may read a program related to the target process via a communication network (NW920) from another device.

[0048] Also, when the CPU 901 of the computer 900 executes a program, the functions of the sensor signal processor 611 and the appearance inspection work extractor 612 of the appearance inspection work controller 601 are realized.

[0049] Hereinafter, the operation of the image generation system according to the embodiment will be described in detail. First, the procedure of the appearance inspection work will be described with reference to FIGS. 7A, 7B, and 7C.

[0050] FIG. 7A is a diagram showing the state of the inspection work of the maintenance worker 800 in the under-railway vehicle environment 700 of the inspection target environment shown in FIG. 2. The appearance inspection work measurement device 600 obtains the gaze point trajectory, which is the time change of the position of the gaze point on the inspection target that the maintenance worker 800 is gazing at, and the line-of-sight direction of the maintenance worker for each gaze point.

[0051] Specifically, the maintenance worker 800 performs an appearance inspection of the first inspection target structure 1, the second inspection target structure 2, and the third inspection target structure 3 from the front side of the wheel 4. At this time, after the maintenance worker 800 inspects while gazing at the first inspection target O1 of the first inspection target structure 1, the line of sight is moved to the second inspection target O2 of the second inspection target structure 2 for inspection, and further, the line of sight is moved to the third inspection target O3 of the third inspection target structure 3 for inspection. In this specification, the first inspection target O1, the second inspection target O2, and the third inspection target O3 may be collectively referred to as maintenance points.

[0052] The appearance inspection work measurement device 600 of the image generation system according to the embodiment measures the line-of-sight movement of the maintenance worker 800 from the first inspection target O1 to the second inspection target O2 and the line-of-sight movement from the second inspection target O2 to the third inspection target O3, obtains the line-of-sight information, the inspection target, the inspection route, and the position of the eyes of the maintenance worker 800, and records them as time-series data. The first inspection target O1, the second inspection target O2, and the third inspection target O3 are included in the inspection target.

[0053] More specifically, the visual inspection work extractor 612 (see Figure 5) of the visual inspection work measurement device 600 calculates the gaze line of the maintenance worker 800 during the visual inspection using the gaze detector 614, calculates the visual inspection target and inspection path using the inspection target extractor 615, calculates the position of the maintenance worker 800's eyes during the visual inspection using the eye position calculator 616, and inputs the measured results as time-series data into the measurement result saver 613. In other words, the visual inspection work extractor 612 obtains gaze information, inspection target and inspection path, and time-series data of the eye position.

[0054] Furthermore, the inspection target extractor 615 calculates the gaze vector for the inspection target that the maintenance worker 800 is fixated on within the inspection target environment, based on the gaze detected by the gaze detector 614 and the eye position calculated by the eye position calculator 616, and extracts the maintenance worker 800's gaze position from the three-dimensional shape of the inspection target measured by the inspection target measurement sensor 602. By analyzing the time-series data, areas with particularly high frequency and duration of fixation are designated as inspection targets.

[0055] Figure 7B shows the inspection target viewed from directly opposite the axle side of the wheel 4. The inspection target extractor 615 extracts each inspection target based on time-series data of the gaze position of the maintenance worker 800. This makes it possible to calculate the inspection path P1 based on the movement of the gaze from the first inspection target O1 to the second inspection target, and the inspection path P2 based on the movement of the gaze from the second inspection target O2 to the third inspection target.

[0056] In the image generation system of this embodiment, the camera trajectory generator 115 calculates movement trajectories (camera trajectories P3, P4) from the calculated inspection paths P1 and P2, as shown in Figure 7C, in which the difference between the shooting direction of the camera 101 mounted on the robot arm and the field of view direction of the maintenance worker 800 is within a predetermined value. By controlling the robot 100 to satisfy the calculated camera trajectories of the camera 101 and photographing the inspection target environment, the maintenance worker 800 can capture a visual image of the inspection target that has been visually inspected.

[0057] However, due to constraints on the robot's (camera's) movement relative to obstacles, it may not be possible to achieve the camera trajectory obtained from measurements taken by the maintenance worker 800 of the visual inspection work measurement device 600. Here, an example of how to avoid camera movement constraints and photograph the inspection target with the robot 100's camera 101 is explained with reference to Figures 8A to 8G and 9A to 9E.

[0058] Figure 8A shows the state in which the robot 100 is taking a picture of the first inspection target O1. The robot 100 moves the camera 101 so that it fills the camera trajectory P3 (Figure 7C). Then, as shown in Figure 8B, the robot 100 positions the camera 101 to take a picture of the second inspection target O2.

[0059] Next, as shown in Figure 8C, if the camera trajectory P4 interferes with the operating constraint range of the camera 101 due to the robot's movement constraints, the robot 100 moves the camera 101 to a position where it photographs an intermediate point O2' along the inspection path P2 corresponding to the camera trajectory P4 (Figure 7C).

[0060] Here, the case where the camera trajectory interferes with the camera movement constraint range means that the camera trajectory overlaps with the range in which the robot 100 cannot move the camera 101 continuously in one direction according to the camera trajectory.

[0061] More specifically, the relay point O2' is a point within the range where there are no operational constraints on the camera 101 when moving along the inspection path P2 from the second inspection target O2 to the third inspection target O3, and also within the range where there are no operational constraints on the camera 101 when moving along the inspection path P2 from the third inspection target O3 to the second inspection target O2. In other words, the relay point O2' can be used to capture images from different robot 100 postures.

[0062] After photographing the relay point O2', the robot 100 changes its posture so that the camera 101 is positioned under the axle of the wheel 4, as shown in Figures 8D and 8E. Then, as shown in Figure 8F, the robot 100 moves the camera 101 to a position to photograph the relay point O2' and resumes photography. In other words, the robot 100 moves from the relay point O2' according to a trajectory that avoids camera movement constraints and returns to the relay point O2'.

[0063] Subsequently, as shown in Figure 8G, the robot 100 moves the camera 101 to a position where it can photograph the third inspection target O3, so as to fill the camera trajectory P4 (Figure 7C).

[0064] As described above, although the camera 101 moves along the camera trajectory P4 in accordance with a trajectory that avoids camera movement constraints, it appears to be moving along the camera trajectory P4, so the image generation system can acquire a continuous series of images of the inspection target.

[0065] When the camera trajectory interferes with the operational constraint range of camera 101, and when performing a visual inspection, it may be sufficient to have specific inspection images of the first inspection target O1, the second inspection target O2, and the third inspection target O3. The operation of robot 100 in this case is explained with reference to Figures 9A to 9E. This operation of robot 100 is also applied when it is not possible to avoid the operational constraints of camera 101 by adding the camera operation constraint avoidance trajectory described above (when relay points cannot be provided).

[0066] Figure 9A shows the same state as Figure 8A, where the robot 100 is taking a picture of the first inspection target O1. The robot 100 moves the camera 101 so that it fills the camera trajectory P3 (Figure 7C). Then, as shown in Figure 8B, the robot 100 positions the camera 101 to take a picture of the second inspection target O2.

[0067] Subsequently, as shown in Figures 9C and 9D, the robot 100 changes its posture so that the camera 101 is positioned to slide under the axle of the wheel 4. Then, as shown in Figure 9E, the robot 100 positions the camera 101 to photograph the second inspection target O2. In other words, the robot 100 moves the camera 101 according to an alternative camera trajectory that avoids the constraints on the camera 101's movement from the position of photographing the second inspection target O2 to the position of photographing the third inspection target O3, without moving the camera 101 to satisfy the camera trajectory P4 (Figure 7C). That is, no inspection image is taken for the camera trajectory P4.

[0068] Next, the shooting operation of the camera 101 that photographs the object to be inspected will be explained with reference to Figures 10A and 10B.

[0069] Figure 10A shows the camera trajectory when the relay point O2' described in Figures 8A to 8G is provided. In this case, the camera trajectory consists of the camera trajectory P3 shown as a solid line, camera trajectory P4', camera trajectory P4'', and the camera movement constraint avoidance trajectories (P8, P9) shown as dotted lines. Camera trajectories P4 and P4'' are trajectories obtained by dividing camera trajectory P4 at the relay point O2', and are almost identical trajectories.

[0070] Therefore, if a relay point O2' is provided, the images captured by camera 101 along camera trajectories P3, P4', and P4'' are approximately equal to the images of the inspection target that the maintenance worker 800 visually observes during the visual inspection. When camera 101 is moving along avoidance trajectories (P8, P9), shooting is temporarily suspended (no shooting is performed). Alternatively, camera 101 may be configured to take pictures while moving, and the images captured along the avoidance trajectories (P8, P9) of camera movement constraints may not be displayed during display.

[0071] From the perspective of the camera 101's movement, the camera 101 moves from camera trajectory P4' to avoidance trajectory P8 via relay point O2', and then returns to camera trajectory P4'' from avoidance trajectory P9 via relay point O2'. In other words, the point where the camera 101 moves from camera trajectory P4' to avoidance trajectory P8 and the point where it returns from avoidance trajectory P9 to camera trajectory P4'' are both relay point O2' at the same location.

[0072] Figure 10B shows the camera trajectory when the camera operation constraint cannot be avoided by adding the avoidance trajectory described in Figures 9A to 9E (when relay points cannot be provided). In this case, the camera trajectory consists of camera trajectory P3 and alternative camera trajectories (P5, P6, P7) that avoid the camera operation constraint. Camera 101 continues to take pictures while moving along this camera trajectory.

[0073] The alternative camera trajectories (P5, P6, P7) that avoid camera operation constraints deviate significantly from camera trajectory P4. Therefore, since the images of the inspection target that maintenance worker 800 visually observes during the visual inspection differ, a warning will be displayed for the images captured by the alternative camera trajectories (P5, P6, P7) indicating that they do not represent the inspection target.

[0074] Next, the image generation operation in the image generation system of the embodiment will be explained using the flowchart in Figure 11. The flowchart in Figure 11 is activated when the user inputs a start command to the input unit 202 of the input device 200.

[0075] In step S1, the visual inspection work measurement device 600 (Figure 5) uses the visual inspection work extractor 612 to obtain time-series data of the maintenance worker 800's gaze information, the inspection target and inspection route, and the position of the eyes during the visual inspection work, based on the measured values ​​of the gaze measurement sensor 601, the inspection target measurement sensor 602, and the position / orientation sensor 603. In this case, it is desirable that the position information be in absolute coordinate system.

[0076] In step S2, the camera trajectory generator 115 (Figure 4) of the motion generator 112 of the robot controller 110 calculates the camera trajectory for capturing the visual images that the maintenance worker 800 saw while performing the visual inspection, based on the gaze information of the maintenance worker 800 during the visual inspection work, the inspection target and inspection route, and the time-series data of the eye position obtained in step S1, as well as the specifications of the camera 101 such as resolution and field of view.

[0077] In step S3, the robot motion dynamic generator 116 (Figure 4) of the robot 100 generates camera motion constraints based on environmental information and the robot's mechanism. Specifically, the robot motion dynamic generator 116 identifies obstacles and other objects from the working environment information of the robot 100 measured by the environmental measurement sensor 109, and calculates the area in which the camera 101 can move based on the robot's mechanical constraints. In other words, the robot motion dynamic generator 116 determines (generates) motion constraints for the camera 101.

[0078] In step S4, the robot controller 110 determines whether the camera trajectory calculated in step S2 interferes with the camera motion constraints generated in step S3. If there is no interference (No in S4), the process proceeds to step S5; if there is interference (Yes in S4), the process proceeds to step S21.

[0079] In step S5, the overall motion generator 119 generates robot motion information that reproduces the camera trajectory calculated in step S2.

[0080] In step S6, the robot is operated based on the robot motion information, and images are captured by the camera to generate a set of images. Specifically, based on the robot motion information generated in step S5, the motion execution unit 113 drives and controls the arm 102 and the movement mechanism 103, and the images captured by the camera 101 at that time are saved in the motion result saver 114. Then, the motion result saver 114 generates a set of inspection images from the saved images using the inspection image generator 121. For example, the captured images are aggregated to generate an inspection video.

[0081] In step S7, the generated image set is displayed. Specifically, the robot controller 110 transmits the inspection image set generated by the inspection image generator 121 (Figure 4) to the input device 200 via the transceiver 106 and transceiver 204, displays the inspection image set on the display unit 201, and ends the process.

[0082] In step S21, the motion generator 112 determines whether it is possible to avoid the camera motion constraint by adding a trajectory to the camera trajectory calculated in step S2. If it is possible to avoid the constraint (Yes in S21), the process proceeds to step S22; if it is not possible to avoid the constraint (No in S21), the process proceeds to step S31.

[0083] In step S22, a trajectory that avoids the camera motion constraints described in Figures 8A to 8G is generated. The method for generating the trajectory that avoids the camera motion constraints is described in detail below.

[0084] To generate a trajectory that avoids camera movement constraints, the relay point generator 117 (Figure 4) first generates relay points. For example, if the camera trajectory P4 generated by the camera trajectory generator 115 shown in Figure 7(c) interferes with the camera movement constraints, the relay point generator 117 generates a relay point O2' in the middle of the camera trajectory P4.

[0085] In detail, the relay point O2' is generated in the overlapping area of ​​the first and second operating ranges, with respect to the first operating range that does not interfere with the camera operation constraint when the camera is moved along the camera trajectory P4 from the second inspection target O2 to the third inspection target O3, and the second operating range that does not interfere with the camera operation constraint when the camera is moved along the camera trajectory P4 from the third inspection target O3 to the second inspection target O2.

[0086] Next, the camera motion constraint avoidance trajectory generator 118 (Figure 4) generates a trajectory for the generated relay point O2' that moves from the second inspection target O2 to the relay point O2', then moves away from the relay point O2', and returns to the relay point O2' in a position that allows movement from the relay point O2' to the third inspection target O3, and this is used as the camera motion constraint avoidance trajectory.

[0087] In step S23, the overall motion generator 119 generates robot motion information that reproduces the camera trajectory, including the trajectory for avoiding camera motion constraints. Specifically, in step S2, the overall motion generator 119 generates time-series patterns for each joint of the arm 102 and the moving mechanism 103 based on the camera trajectory generated by the camera trajectory generator 115 and the trajectory for avoiding camera motion constraints generated by the camera motion constraint avoidance trajectory generator 118 in step S22, and uses this as robot motion information.

[0088] In step S24, the robot is operated based on the robot motion information, and a set of images is generated by deleting the images taken during the camera motion avoidance trajectory from the set of images taken by the camera, and then the process proceeds to step S7.

[0089] More specifically, based on the robot motion information generated in step S23, the motion execution unit 113 drives and controls the arm 102 and the movement mechanism 103, and the images captured by the camera 101 at that time are saved in the motion result saver 114. Then, the motion result saver 114 removes the images from the saved images using the avoidance motion image remover 120 to remove the images of the camera motion constraint avoidance trajectory, and the inspection image generator 121 generates a group of inspection images. For example, the captured images are aggregated to generate an inspection video.

[0090] In step S31, an alternative camera trajectory is generated that avoids the camera operation constraints described in Figures 9A to 9E.

[0091] In step S32, robot motion information is generated that reproduces the camera trajectory, including the alternative camera trajectory, and the process proceeds to step S6.

[0092] Next, other configurations of the image generation system of the embodiment will be described with reference to Figure 12. Figure 12 is a diagram showing the overall network configuration 500 of the image generation system. As shown in Figure 12, the robot 100, the input device 200, and the visual inspection work measurement device 600 are connected via the network 400 and can communicate various types of data. A storage and external processing device 300 may also be provided on the same network 400.

[0093] By providing a storage and external processing unit 300, it becomes possible to store data saved by the robot 100 outside of the robot. Similarly, data saved by the visual inspection work measurement device 600 can also be stored externally. In particular, by storing visual inspection data from maintenance personnel externally, it can be used as training data when building the robot 100's functions based on machine learning. Furthermore, even if processing is difficult for the robot controller 110 mounted on the robot 100, it is expected that the data can be processed by performing calculations on the externally provided storage and external processing unit 300.

[0094] In the above example, the generation of the camera trajectory for acquiring inspection images is performed by a motion generator 112 mounted on the robot 100. However, the camera trajectory may also be generated outside the robot 100, and the generated camera trajectory may be notified to the robot 100 to acquire inspection images.

[0095] For example, as shown in Figure 13, an image generation device 510 having an action generator 112 and an action result saver 114 receives time-series data of the gaze information of the maintenance worker 800 during the visual inspection work, the inspection target and inspection route, and the position of the eyes from the visual inspection work measurement device 600, generates a camera trajectory, and notifies the robot 100 of the robot motion information. The robot 100 then acquires the image captured by the camera 101 and notifies the input device 200 of the inspection image for display.

[0096] Furthermore, in Figure 13, the configuration of the robot 100, image generation device 510, visual inspection work measurement device 600, and input device 200 can also be called an image generation system.

[0097] Alternatively, the robot control device 520 (dashed line frame) may be composed of the motion generator 112, the transceiver 511 (transceiver inside the robot control device, transceiver inside the image generation device), and the visual inspection work measurement device 600. The robot control device 520 generates robot motion information for the robot 100 equipped with a camera 101 based on the gaze information of the maintenance worker performing the visual inspection, the inspection target and inspection route, and time-series data of the position of the eyes, and notifies the robot 100. As a result, the robot 100 can capture images of the maintenance worker's view with the camera 101.

[0098] Furthermore, the present invention is not limited to the embodiments described above, and various modifications are included. The embodiments described above are described in detail for the purpose of clearly illustrating the present invention, and are not necessarily limited to those having all the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment.

[0099] 100 Robot 101 Camera 102 Arm 103 Movement mechanism 106 Transceiver (Transceiver inside robot) 110 Robot controller 112 Motion generator 113 Motion executer (Controller) 114 Motion result saver (Image group generator) 115 Camera trajectory generator 116 Robot motion dynamic generator (Robot motion constraint generator) 117 Relay point generator 118 Camera motion constraint avoidance trajectory generator (Camera avoidance trajectory generator) 119 Overall motion generator 120 Avoidance motion image remover 121 Inspection image generator 200 Input device (Output device) 201 Display unit 202 Input unit 203 Visual inspection work data saver 510 Image generation device 511 Transceiver (Transceiver inside robot control device, Transceiver inside image generation device) 520 Robot control device 600 Visual inspection work measurement device 601 Eye-tracking sensor 602 Inspection target measurement sensor 603 Position / orientation sensor 604 Transmitter / receiver 610 Visual inspection work controller 612 Visual inspection work extractor 613 Measurement result saver 614 Eye-tracking detector 615 Inspection target extractor 616 Eye position calculator O1 First inspection target (maintenance point) O2 Second inspection target (maintenance point) O3 Third inspection target (maintenance point)

Claims

1. A robot equipped with a camera, comprising: an internal transceiver that receives eye-line information of a maintenance worker and information of multiple maintenance points during visual inspection work; a camera trajectory generator that generates a camera trajectory of the camera from the eye-line information, the maintenance point information and the camera's specifications; a robot motion constraint generator that generates motion constraints of the robot from environmental information in which the robot operates and the robot's mechanical constraints; a camera avoidance trajectory generator that generates an avoidance trajectory of the camera that avoids the robot's motion constraints; an overall motion generator that generates robot motion information of the robot using the camera trajectory, the robot's motion constraints and the camera avoidance trajectory; and a controller that controls the robot using the robot motion information generated by the overall motion generator.

2. The robot according to claim 1, wherein the overall motion generator adds the avoidance trajectory to the camera trajectory, and the point at which the camera moves from the camera trajectory to the avoidance trajectory and the point at which it returns from the avoidance trajectory to the camera trajectory are substantially the same position.

3. A robot according to claim 1, comprising: an image group generator that generates an image group using images taken by the camera when the robot operates based on the robot operation information, and deletes images taken in the avoidance trajectory from the image group; and an output device that outputs the image group generated by the image group generator.

4. The robot according to claim 1, wherein the camera temporarily stops taking pictures in the avoidance trajectory.

5. A robot according to claim 3 or 4, characterized in that the image when moving to the avoidance trajectory generated by the camera avoidance trajectory generator and the image when returning to the camera trajectory are continuous.

6. The robot according to claim 1, characterized in that the eye-line information includes the position of the maintenance worker's eyes and their line of sight.

7. An image generation system comprising a robot equipped with a camera and an image generation device that generates a group of images using images captured by the camera, wherein the image generation device comprises: a camera trajectory generator that generates a camera trajectory of the camera from acquired eye-line information of a maintenance worker during visual inspection work, information of multiple maintenance points, and specifications of the camera; a robot motion constraint generator that generates motion constraints of the robot from environmental information in which the robot operates and mechanical constraints of the robot; a camera avoidance trajectory generator that generates an avoidance trajectory of the camera that avoids the motion constraints of the robot; an overall motion generator that generates robot motion information of the robot using the camera trajectory, the motion constraints of the robot, and the avoidance trajectory of the camera; an internal transceiver in the image generation device that transmits the robot motion information to the robot and receives images captured by the camera; an inspection image generator that generates a group of images using images taken when the robot operates based on the robot motion information received by the internal transceiver in the image generation device; and an output device that outputs the group of images generated by the inspection image generator, wherein the robot is An image generation system comprising: a robot-internal transceiver that receives the robot motion information and transmits images captured by the camera; and a controller that controls the robot based on the robot motion information.

8. An image generation device that generates an image group using images captured by a camera provided on a robot according to claim 1, comprising: an internal transceiver for receiving images captured by the camera; an image group generator for generating an image group using images received by the internal transceiver; and an output device for outputting the image group generated by the image group generator.

9. A robot control device for a robot equipped with a camera, comprising: a transceiver within the robot control device that receives eye-line information of a maintenance worker and multiple maintenance points during visual inspection work; a camera trajectory generator that generates a camera trajectory of the camera from the eye-line information, the maintenance point information and the camera's specifications; a robot motion constraint generator that generates motion constraints of the robot from environmental information in which the robot operates and the robot's mechanical constraints; a camera avoidance trajectory generator that generates an avoidance trajectory of the camera that avoids the robot's motion constraints; an overall motion generator that generates robot motion information of the robot using the camera trajectory, the robot's motion constraints and the camera avoidance trajectory; and a controller that controls the robot using the robot motion information generated by the overall motion generator.

10. An image generation method for an image generation apparatus that controls a robot equipped with a camera to generate a group of images, comprising: a step of detecting the line of sight of a maintenance worker and a plurality of maintenance points in an external inspection operation; a step of generating a camera trajectory of the camera from the line of sight information, the maintenance point information and the camera's specifications; a step of determining the robot's motion constraints from the environment information in which the robot operates and the robot's mechanical constraints; a step of generating an avoidance trajectory for the camera that avoids the robot's motion constraints; a step of generating an avoidance trajectory for the camera that avoids the motion constraints using the camera trajectory and the robot's motion constraints; a step of generating robot motion information using the avoidance trajectory, the camera trajectory and the robot's motion constraints; and a step of generating a group of images using images taken by a camera of a robot operating based on the robot motion information.

Images