Robot, image generation system, image generation device, robot control device, and image generation method

Abstract

Even when the robot arm's camera is subject to movement constraints, it achieves a camera trajectory that closely resembles the inspection movements of a maintenance worker. [Solution] A robot 100 equipped with a camera 101 is provided with the following: a robot-internal transceiver 106 that receives eye-line information of a maintenance worker and information of multiple maintenance points during visual inspection work; a camera trajectory generator 115 that generates a camera trajectory from the eye-line information, maintenance point information and camera specifications; a robot motion constraint generator 116 that generates motion constraints for the robot from environmental information in which the robot operates and mechanical constraints of the robot; a camera avoidance trajectory generator 118 that generates an avoidance trajectory for the camera that avoids the robot motion constraints; an overall motion generator 119 that generates robot motion information for the robot using the camera trajectory, robot motion constraints and camera avoidance trajectory; and a controller 113 that controls the robot using the robot motion information generated by the overall motion generator.

Description

Technical Field

[0001] The present invention relates to a robot for photographing an image of an object, an image generation system, an image generation device, a robot control device, and an image generation method.

Background Art

[0002] In recent years, with the decline of the working population, the need for automation of tasks that have been performed manually by humans has increased, and work substitution by robots has been progressing. As an example of work substitution by robots, there is an appearance inspection work. Appearance inspection work is work for checking an object from various angles and confirming whether there are any abnormalities such as scratches or dirt on the appearance. For the automation of appearance inspection work, image data from various positions and angles are required.

[0003] For example, Patent Document 1 discloses that "an appearance inspection system for performing an appearance inspection of an inspection object includes a display device, a robot for moving an imaging device, and while the robot is moving the imaging device, the imaging device images each of a plurality of inspection parts of the inspection object, and based on each image obtained from the imaging device, an inspection unit for inspecting the presence or absence of defects for each of the plurality of inspection parts, and a display control unit for displaying, on the display device, an inspection result matrix representing the inspection results by the inspection unit for each of the plurality of inspection parts for each inspection object."

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When automating visual inspections performed by maintenance personnel in confined spaces, such as under railway vehicles, using a camera mounted on a robotic arm, the robotic arm may be subject to operational constraints from the environment, preventing it from executing the same image data acquisition camera trajectory as the maintenance personnel. For this reason, the technology described in Patent Document 1, which does not take into account operational constraints of the robotic arm, cannot be applied.

[0006] The object of the present invention is to provide a robot, an image generation system, an image generation device, a robot control device, and an image generation method that enable a robot equipped with a camera to appropriately photograph an object even in environments where operational constraints occur. [Means for solving the problem]

[0007] To solve the aforementioned problems, the present invention provides a robot equipped with a camera, comprising: an in-robot transceiver that receives eye-line information of a maintenance worker and information of multiple maintenance points during visual inspection work; a camera trajectory generator that generates a camera trajectory of the camera from the eye-line information, the maintenance point information and the specifications of the camera; a robot motion constraint generator that generates motion constraints of the robot from environmental information in which the robot operates and the mechanical constraints of the robot; a camera avoidance trajectory generator that generates an avoidance trajectory of the camera that avoids the motion constraints of the robot; an overall motion generator that generates robot motion information of the robot using the camera trajectory, the motion constraints of the robot and the avoidance trajectory of the camera; and a controller that controls the robot using the robot motion information generated by the overall motion generator. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a robot, an image generation system, an image generation device, a robot control device, and an image generation method that enable a robot equipped with a camera to appropriately photograph an object even in environments where movement constraints occur when taking pictures, by setting an appropriate avoidance trajectory. [Brief explanation of the drawing]

[0009] [Figure 1] This is a side view of the robot according to the embodiment. [Figure 2] This is a perspective view of the environment under a railway vehicle, which is the environment to be inspected in the embodiment. [Figure 3] This is an external view of the input device. [Figure 4] This is a functional block diagram explaining the robot's functions. [Figure 5] This is a functional block diagram of the visual inspection work measurement device and input device. [Figure 6] This is a hardware configuration diagram of a computer. [Figure 7A] This diagram shows the inspection work performed by maintenance personnel in the environment beneath railway vehicles. [Figure 7B] This is a view of the object being inspected, seen directly from the axle side of the wheel. [Figure 7C] This is a diagram illustrating the camera's trajectory. [Figure 8A] This diagram shows the camera capturing the first test target, O1. [Figure 8B] This diagram shows the camera capturing the second target of inspection, O2. [Figure 8C] This diagram shows the camera moving to a position where it can photograph the intermediate point O2' along the inspection route P2. [Figure 8D] This diagram shows the robot changing its posture so that the camera is positioned under the axle. [Figure 8E] This diagram shows the robot changing its posture so that the camera is positioned under the axle. [Figure 8F] This diagram shows the robot moving its camera to a position where it can photograph relay point O2'. [Figure 8G] This diagram shows the robot moving its camera to a position where it can photograph the third inspection target, O3. [Figure 9A] This diagram shows the camera capturing the first test target, O1. [Figure 9B] This diagram shows the camera capturing the second target of inspection, O2. [Figure 9C] This is a diagram showing the state in which the robot's posture is changed so that the camera dives under the axle. [Figure 9D] This is a diagram showing the state in which the robot's posture is changed so that the camera dives under the axle. [Figure 9E] This is a diagram showing the state in which the camera of the robot reaches the position for photographing the second inspection target O2. [Figure 10A] This is a diagram for explaining the camera trajectory and the photographing operation when the relay point O2' is provided. [Figure 10B] This is a diagram for explaining the camera trajectory and the photographing operation when adding a trajectory cannot avoid camera operation constraints (when a relay point cannot be provided). [Figure 11] This is a flowchart for explaining the image generation operation in the image generation system of the embodiment. [Figure 12] This is a diagram for explaining other configurations of the image generation system of the embodiment. [Figure 13] This is a diagram for explaining the system configuration in which a camera trajectory is generated outside the robot, and the generated camera trajectory is notified to the robot to obtain inspection images.

Mode for Carrying Out the Invention

[0010] The image generation system of this embodiment is an image generation system for inspection images, comprising a visual inspection work measurement device 600 and a robot 100, and is configured so that the robot 100 acquires an image of the inspection target that the maintenance worker is looking at during a visual inspection. The visual inspection work measurement device 600 obtains time-series data of the maintenance worker's gaze information, the inspection path and the inspection target, and the position of the eyes during a visual inspection performed by the maintenance worker. The robot 100 then generates a camera trajectory, which is the time change in the position and direction of the camera 101, from the trajectory of the maintenance worker's gaze point and the direction of their gaze obtained by the visual inspection work measurement device 600, and generates operation information of the robot 100 that satisfies this camera trajectory in the inspection environment. The robot 100 operates based on the generated operation information and acquires an image of the inspection target that the maintenance worker is looking at by photographing the inspection target with the camera 101.

[0011] Alternatively, the visual inspection work measurement device 600 may determine the gaze point trajectory, which is the time change in the position of the gaze point on the inspection object that the maintenance worker is looking at, and the direction of the maintenance worker's line of sight for each gaze point. The robot 100 may then generate a camera trajectory, which is the time change in the position and direction of the camera 101, from the gaze point trajectory and line of sight of the maintenance worker determined by the visual inspection work measurement device 600, and generate operation information for the robot 100 that satisfies this camera trajectory in the inspection environment. Embodiments of the present invention will be described in detail below with reference to the drawings.

[0012] Figure 1 is a side view of the robot 100 according to the embodiment. The robot 100 of this embodiment is a quadruped walking robot, and is equipped with a four-legged locomotion mechanism 103 at the bottom of its main body 104 that can move in all directions on a plane. The robot 100 is also equipped with a serial link mechanism arm 102 at the top of its main body 104, which has a camera 101 at its tip. Furthermore, the main body 104 of the robot 100 has a transceiver 106, and inside the main body 104 is a controller (not shown) that controls the robot using sensors to detect the robot's position and orientation, and force sensors at each joint of the legs.

[0013] The robot in the embodiment shown in Figure 1 is a quadruped robot, but it is not limited to this, as long as the camera arm has redundancy in its degrees of freedom and can assume multiple postures with a fixed camera position. Furthermore, the locomotion method is not limited to the quadruped locomotion mechanism 103, but may also be a crawler-type locomotion mechanism.

[0014] Figure 2 is a perspective view of the railway vehicle under-vehicle environment 700, which is the inspection environment in the embodiment. In this embodiment, the inspection environment is configured such that the first inspection target structure 1, the second inspection target structure 2, and the third inspection target structure 3 are positioned behind the wheel 4 that straddles rails 5a and 5b. A maintenance worker 800 (not shown) performs a visual inspection of the first inspection target structure 1, the second inspection target structure 2, and the third inspection target structure 3, with the wheel 4 in between.

[0015] Figure 3 is an external view of the input device 200 (output device) which inputs operation commands to the robot 100 (Figure 1) of the embodiment and checks the status of the robot 100. The input device 200 also inputs operation commands to the visual inspection work measurement device 600, which will be described later, and checks the status of the visual inspection work measurement device 600.

[0016] The input device 200 is an information terminal (computer) equipped with a display unit 201, an input unit 202, and a transceiver 204. The input device 200 transmits operation commands entered by the user from the input unit 202 to the robot 100 and the visual inspection work measurement device 600 via the transceiver 204, and receives status information from the robot 100 and the visual inspection work measurement device 600, which is then displayed on the display unit 201 for the user to confirm.

[0017] Next, a robot according to an embodiment, an image generation device that generates inspection images using the robot, and an image generation system will be described.

[0018] Figure 4 is a functional block diagram illustrating the functions of robot 100. The robot 100 functions as an image generation device that generates inspection images, and together with the input device 200 and the visual inspection work measurement device 600, it constitutes an image generation system.

[0019] As shown in Figure 4, the robot 100 consists of a camera 101, a position / attitude sensor 107, a force sensor 108, an environmental measurement sensor 109, a robot controller 110, an arm 102, a movement mechanism 103, and a transceiver 106.

[0020] Camera 101 is an imaging device attached to the tip of the arm 102 (Figure 1) of the robot 100, and is used to capture image data for visual inspection. The position and orientation sensor 107 is a sensor that measures the position and orientation of the robot 100. For example, it could be an accelerometer or a gyroscope. The force sensor 108 is a torque sensor that measures the torque of each joint of the robot and measures the load applied to the robot 100.

[0021] The environmental measurement sensor 109 is a point cloud sensor capable of measuring the three-dimensional shape of the environment surrounding the robot 100. Arm 102 is a serial link mechanism with a camera 101 at its tip.

[0022] The mobile mechanism 103 is a four-legged mobile mechanism capable of moving in all directions on a plane. The transceiver 106 is a wireless communication unit connected to the input device 200. It receives operation instructions for the robot 100 from the input device 200 and transmits status information of the robot 100 to the input device 200.

[0023] The robot controller 110 consists of a sensor signal processor 111, a motion generator 112, a motion executer 113 (controller), and a motion result saver 114 (image group generator).

[0024] The sensor signal processor 111 converts electrical signals from the camera 101, position / attitude sensor 107, force sensor 108, and environmental measurement sensor 109 into physical parameters and notifies the motion generator 112 and motion result saver 114.

[0025] The motion executer 113 drives and controls the arm 102 and the movement mechanism 103 based on a time-series motion pattern (robot motion information) of each joint of the arm 102 and the movement mechanism 103, which is generated so that the position of the camera 101 follows the camera trajectory generated by the motion generator 112, which will be described later.

[0026] The motion generator 112 consists of a camera trajectory generator 115, a robot motion dynamic generator 116 (robot motion constraint generator), a relay point generator 117, a camera motion constraint avoidance trajectory generator 118 (camera avoidance trajectory generator), and an overall motion generator 119.

[0027] The camera trajectory generator 115 calculates a camera trajectory that captures an image similar to the image the maintenance worker 800 saw while performing the visual inspection, based on the gaze information, inspection target and inspection route, and time-series data of the eye position during the visual inspection work performed by the maintenance worker 800, stored in the visual inspection work data saver 203 of the input device 200, as well as the specifications of the camera 101, such as its resolution and field of view.

[0028] The robot motion dynamic generator 116 identifies obstacles and other objects from the working environment information of the robot 100 measured by the environmental measurement sensor 109, and calculates the area in which the camera 101 can move due to the robot's motion, based on the robot's mechanical constraints. In other words, the robot motion dynamic generator 116 determines the motion constraints of the camera 101.

[0029] Details of the relay point generator 117 and the camera operation constraint avoidance trajectory generator 118 will be described later.

[0030] The overall motion generator 119 generates time-series patterns (robot motion information) for each joint of the arm 102 and the movement mechanism 103 based on the camera trajectory, so that the position of the camera 101 follows the camera trajectory generated by the camera trajectory generator 115.

[0031] The operation result saver 114 consists of an avoidance operation image remover 120 and an inspection image generator 121, and generates an image of the object to be inspected from the image captured by the camera 101 of the robot 100. As will be described in detail later, the image of the object to be inspected is generated by removing images from the image captured by the camera 101 when an avoidance operation was performed due to operational constraints of the camera 101 in the inspection environment.

[0032] Next, using the functional block diagram in Figure 5, we will explain the functions of the input device 200 and the visual inspection work measurement device 600, which constitute the image generation system along with the robot 100 described in Figure 4.

[0033] As shown in Figure 5, the visual inspection work measurement device 600 consists of a gaze measurement sensor 601, an inspection target measurement sensor 602, a position / orientation sensor 603, and a visual inspection work controller 610.

[0034] The eye-tracking sensor 601 is a sensor that measures the gaze of the maintenance worker 800 while he is working, and is, for example, a sensor that measures the eye movements of the maintenance worker 800. The inspection target measurement sensor 602 is a sensor that measures the inspection target during the visual inspection work. For example, it is a camera that takes an image of the structure in the direction that the maintenance worker 800 is facing, or a sensor that can measure the three-dimensional shape of the structure. The position and orientation sensor 603 is a sensor that measures changes in the position and orientation of the visual inspection work measurement device 600, and is an acceleration sensor or gyroscope sensor.

[0035] The visual inspection work controller 610 consists of a sensor signal processor 611, a visual inspection work extractor 612, a measurement result saver 613, and a transceiver 604. The sensor signal processor 611 converts electrical signals from the gaze measurement sensor 601, the object being inspected measurement sensor 602, and the position / attitude sensor 603 into physical parameters.

[0036] The visual inspection work extractor 612 consists of a gaze detector 614, an inspection target extractor 615, and an eye position calculator 616. Based on the measurements of the gaze measurement sensor 601, the inspection target measurement sensor 602, and the position / orientation sensor 603, the visual inspection work extractor 612 obtains time-series data of the gaze information of the maintenance worker 800 during the visual inspection work, the inspection target and inspection path, and the position of the eyes. The visual inspection work extractor 612 may also obtain the gaze point trajectory, which is the time change in the position of the gaze point on the inspection target that the maintenance worker 800 is looking at, and the direction of the maintenance worker's gaze for each gaze point. In this specification, gaze information and eye position are sometimes collectively referred to as gaze information.

[0037] The measurement result storage unit 613 stores time-series data of the maintenance worker 800's gaze information, inspection target and inspection route, and eye position during the visual inspection work, which are obtained by the visual inspection work extractor 612. The transceiver 604 is a communication unit that receives work commands for the visual inspection work measurement device 600 from the input device 200 (described later) and transmits to the input device 200 time-series data of the maintenance worker 800's line of sight during the visual inspection work, the inspection target and inspection route, and the position of the eyes, which are stored in the measurement result saver 613.

[0038] The input device 200 consists of a display unit 201, an input unit 202, a visual inspection work data storage unit 203, and a transceiver 204. It inputs operation commands to the visual inspection work measurement device 600 and the robot 100, and also checks their status. The functions of the display unit 201, the input unit 202, and the transceiver 204 are as explained in Figure 3, and will not be explained here.

[0039] The visual inspection work data storage device 203 stores time-series data of the maintenance worker 800's gaze information, inspection target and inspection route, and eye position received from the visual inspection work measurement device 600, and transmits it to the robot 100.

[0040] Next, we will describe the specific hardware configuration that realizes the robot controller 110, the input device 200, and the visual inspection work controller 601.

[0041] Specifically, the robot controller 110, input device 200, and visual inspection work controller 601 of the embodiment are realized by the computer 900 shown in the hardware configuration diagram of Figure 6. The computer 900 has a CPU (Central Processing Unit) 901, ROM (Read Only Memory) 902, RAM 903, HDD (Hard Disk Drive) 904, input / output I / F (Interface) 905, communication I / F 906, and media I / F 907. The HDD 904 may be an SSD (Solid State Drive).

[0042] The CPU 901 operates based on programs stored in the ROM 902 or HDD 904, controlling various parts of the computer 900. The ROM 902 stores boot programs executed by the CPU 901 when the computer 900 starts up, as well as programs related to the computer 900's hardware.

[0043] The CPU 901 controls input devices 910, such as a mouse or keyboard, and output devices 911, such as a display or printer, via the input / output interface 905. The CPU 901 acquires data from the input devices 910 and outputs the generated data to the output devices 911 via the input / output interface 905. A GPU (Graphics Processing Unit) or similar component may also be used as a processor in addition to the CPU 901.

[0044] HDD904 stores programs executed by CPU901 and data used by those programs. Communication I / F906 receives data from other devices via a communication network (e.g., NW(Network)920) and outputs it to CPU901, and also transmits data generated by CPU901 to other devices via the communication network.

[0045] The media interface 907 reads a program or data stored in the recording medium 912 and outputs it to the CPU 901 via the RAM 903. The CPU 901 loads the program related to the desired processing from the recording medium 912 onto the RAM 903 via the media interface 907 and executes the loaded program. The recording medium 912 can be an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto Optical Disk), a magnetic recording medium, or a semiconductor memory.

[0046] For example, if a computer 900 functions as the robot controller 110 of the embodiment, the CPU 901 of the computer 900 executes a program loaded onto the RAM 903 to realize the functions of the sensor signal processor 111, the motion generator 112, and the motion executer 113.

[0047] The CPU 901 reads the program related to the target process from the recording medium 912 and executes it. In addition, the CPU 901 may read the program related to the target process from another device via a communication network (NW920).

[0048] Furthermore, the CPU 901 of the computer 900 executes a program to realize the functions of the sensor signal processor 611 and the visual inspection work extractor 612 of the visual inspection work controller 601.

[0049] The operation of the image generation system of this embodiment will be described in detail below. First, the procedure for visual inspection will be explained using Figures 7A, 7B, and 7C.

[0050] Figure 7A shows the inspection work performed by a maintenance worker 800 in the railway vehicle under-vehicle environment 700, which is the inspection target environment shown in Figure 2. The visual inspection work measurement device 600 determines the gaze point trajectory, which is the change in the position of the gaze point on the inspection target that the maintenance worker 800 is looking at over time, and the direction of the maintenance worker's line of sight for each gaze point.

[0051] In detail, the maintenance worker 800 visually inspects the first inspection target structure 1, the second inspection target structure 2, and the third inspection target structure 3 from the front side of the wheel 4. At this time, the maintenance worker 800 first observes and inspects the first inspection target O1 of the first inspection target structure 1, then shifts his gaze to inspect the second inspection target O2 of the second inspection target structure 2, and then shifts his gaze to inspect the third inspection target O3 of the third inspection target structure 3. In this specification, the first inspection target O1, the second inspection target O2, and the third inspection target O3 are sometimes collectively referred to as maintenance points.

[0052] The visual inspection work measurement device 600 of the image generation system in this embodiment measures the gaze movement of the maintenance worker 800 from the first inspection target O1 to the second inspection target O2 and from the second inspection target O2 to the third inspection target O3, and determines the gaze information of the maintenance worker 800, the inspection targets and inspection path, and the position of the eyes, and records them as time-series data. The first inspection target O1, the second inspection target O2, and the third inspection target O3 are included in the inspection targets.

[0053] More specifically, the visual inspection work extractor 612 (see Figure 5) of the visual inspection work measurement device 600 calculates the gaze line of the maintenance worker 800 during the visual inspection using the gaze detector 614, calculates the visual inspection target and inspection path using the inspection target extractor 615, calculates the position of the maintenance worker 800's eyes during the visual inspection using the eye position calculator 616, and inputs the measured results as time-series data into the measurement result saver 613. In other words, the visual inspection work extractor 612 obtains gaze information, inspection target and inspection path, and time-series data of the eye position.

[0054] Furthermore, the inspection target extractor 615 calculates the gaze vector for the inspection target that the maintenance worker 800 is fixated on within the inspection target environment, based on the gaze detected by the gaze detector 614 and the eye position calculated by the eye position calculator 616. It also extracts the fixation position of the maintenance worker 800 from the three-dimensional shape of the inspection target measured by the inspection target measurement sensor 602. By analyzing the time-series data, areas with particularly high fixation frequency and duration are designated as inspection targets.

[0055] Figure 7B shows the inspection target viewed directly from the axle side of wheel 4. The inspection target extractor 615 extracts each inspection target based on time-series data of the gaze position of the maintenance worker 800. This makes it possible to calculate the inspection path P1 based on the movement of the gaze from the first inspection target O1 to the second inspection target, and the inspection path P2 based on the movement of the gaze from the second inspection target O2 to the third inspection target.

[0056] In the image generation system of this embodiment, the camera trajectory generator 115 calculates, from the calculated inspection paths P1 and P2, movement trajectories (camera trajectories P3, P4) such that the difference between the shooting direction of the camera 101 mounted on the robot arm and the field of view direction of the maintenance worker 800 is within a predetermined value, as shown in Figure 7C. By controlling the robot 100 to satisfy the calculated camera trajectories of the camera 101 and photographing the inspection target environment, the maintenance worker 800 can capture a visual image of the inspection target that has been visually inspected.

[0057] However, due to constraints on the robot's (camera's) movement relative to obstacles, it may not be possible to achieve the camera trajectory obtained from measurements taken by the maintenance worker 800 of the visual inspection work measurement device 600. Here, an example of how to avoid camera movement constraints and photograph the inspection target with the robot 100's camera 101 is explained with reference to Figures 8A to 8G and 9A to 9E.

[0058] Figure 8A shows the state in which the robot 100 is taking a picture of the first inspection target O1. The robot 100 moves the camera 101 so that it fills the camera trajectory P3 (Figure 7C). Then, as shown in Figure 8B, the robot 100 positions the camera 101 to take a picture of the second inspection target O2.

[0059] Next, as shown in Figure 8C, if the camera trajectory P4 interferes with the range of motion constraints of the camera 101 due to the robot's motion constraints, the robot 100 moves the camera 101 to a position where it photographs an intermediate point O2' along the inspection path P2 corresponding to the camera trajectory P4 (Figure 7C).

[0060] Here, the case where the camera trajectory interferes with the camera movement constraint range means that the camera trajectory overlaps with the range in which the robot 100 cannot move the camera 101 continuously in one direction according to the camera trajectory.

[0061] More specifically, relay point O2' is a point within the range where there are no operational constraints on camera 101 when moving along inspection path P2 from the second inspection target O2 to the third inspection target O3, and also within the range where there are no operational constraints on camera 101 when moving along inspection path P2 from the third inspection target O3 to the second inspection target O2. In other words, relay point O2' can be used to capture images from different robot 100 postures.

[0062] After photographing the relay point O2', the robot 100 changes its posture so that the camera 101 is positioned under the axle of the wheel 4, as shown in Figures 8D and 8E. Then, as shown in Figure 8F, the robot 100 moves the camera 101 to a position to photograph the relay point O2' and resumes photography. In other words, the robot 100 moves from the relay point O2' following a trajectory that avoids camera movement constraints and returns to the relay point O2'.

[0063] Subsequently, as shown in Figure 8G, the robot 100 moves the camera 101 to a position that photographs the third inspection target O3 so as to fill the camera trajectory P4 (Figure 7C).

[0064] As described above, although camera 101 moves along the camera trajectory P4 in accordance with a trajectory that avoids camera movement constraints, it appears to be moving along camera trajectory P4, so the image generation system can acquire continuous images of the inspection target.

[0065] When the camera trajectory interferes with the operational constraint range of camera 101, and when performing a visual inspection, it may be sufficient to have specific inspection images of the first inspection target O1, the second inspection target O2, and the third inspection target O3. The operation of robot 100 in this case is explained in Figures 9A to 9E. This operation of robot 100 is also applied when it is not possible to avoid the operational constraints of camera 101 by adding the previously described camera operation constraint avoidance trajectory (when relay points cannot be provided).

[0066] Figure 9A shows the same state as Figure 8A, where the robot 100 is taking a picture of the first inspection target O1. The robot 100 moves the camera 101 so that it fills the camera trajectory P3 (Figure 7C). Then, as shown in Figure 8B, the robot 100 positions the camera 101 to take a picture of the second inspection target O2.

[0067] Subsequently, as shown in Figures 9C and 9D, the robot 100 changes its posture so that the camera 101 is positioned to slide under the axle of the wheel 4. Then, as shown in Figure 9E, the robot 100 positions the camera 101 to photograph the second inspection target O2. In other words, the robot 100 moves the camera 101 according to an alternative camera trajectory that avoids the constraints on the camera 101's movement from the position of photographing the second inspection target O2 to the position of photographing the third inspection target O3, without moving the camera 101 to satisfy the camera trajectory P4 (Figure 7C). That is, no inspection image is taken for camera trajectory P4.

[0068] Next, the shooting operation of the camera 101 that photographs the object to be inspected will be explained with reference to Figures 10A and 10B.

[0069] Figure 10A shows the camera trajectory when the relay point O2' described in Figures 8A to 8G is provided. In this case, the camera trajectory consists of the camera trajectory P3 shown as a solid line, camera trajectory P4', camera trajectory P4'', and the camera movement constraint avoidance trajectories (P8, P9) shown as dotted lines. Camera trajectories P4 and P4'' are the trajectories obtained by dividing camera trajectory P4 at the relay point O2', and are almost identical trajectories.

[0070] Therefore, if a relay point O2' is provided, the images captured by camera 101 along camera trajectories P3, P4', and P4'' are approximately equal to the images of the inspection target that the maintenance worker 800 visually observes during the visual inspection. When camera 101 is moving along the avoidance trajectories (P8, P9), it temporarily stops capturing images (does not capture images). Alternatively, camera 101 may be configured to capture images while moving, and the images captured along the avoidance trajectories (P8, P9) of the camera movement constraints may not be displayed during display.

[0071] From the perspective of the camera 101's movement, camera 101 moves from camera trajectory P4' to avoidance trajectory P8 via relay point O2', and then returns to camera trajectory P4'' from avoidance trajectory P9 via relay point O2'. In other words, the point where camera 101 moves from camera trajectory P4' to avoidance trajectory P8 and the point where it returns from avoidance trajectory P9 to camera trajectory P4'' are both relay point O2' at the same location.

[0072] Figure 10B shows the camera trajectory when the camera movement constraint cannot be avoided by adding the avoidance trajectories described in Figures 9A to 9E (when relay points cannot be provided). In this case, the camera trajectory consists of camera trajectory P3 and alternative camera trajectories (P5, P6, P7) that avoid the camera movement constraint. Camera 101 continues to take pictures while moving along this camera trajectory.

[0073] The alternative camera trajectories (P5, P6, P7) that avoid camera operation constraints deviate significantly from camera trajectory P4. Therefore, since the images of the inspection target that maintenance worker 800 visually observes during the visual inspection differ, a warning will be displayed for the images captured by the alternative camera trajectories (P5, P6, P7) indicating that they do not represent the inspection target.

[0074] Next, the image generation operation in the image generation system of the embodiment will be explained using the flowchart in Figure 11. The flowchart in Figure 11 is activated when the user inputs a start command to the input unit 202 of the input device 200.

[0075] In step S1, the visual inspection work measurement device 600 (Figure 5) uses the visual inspection work extractor 612 to obtain time-series data of the maintenance worker 800's gaze information, the inspection target and inspection path, and the position of the eyes during the visual inspection work, based on the measured values ​​of the gaze measurement sensor 601, the inspection target measurement sensor 602, and the position / orientation sensor 603. In this case, it is desirable that the position information be in absolute coordinate system.

[0076] In step S2, the camera trajectory generator 115 (Figure 4) of the motion generator 112 of the robot controller 110 calculates the camera trajectory for capturing the visual images that the maintenance worker 800 saw while performing the visual inspection, based on the gaze information of the maintenance worker 800 during the visual inspection work, the inspection target and inspection path, and the time-series data of the eye position obtained in step S1, as well as the specifications of the camera 101 such as resolution and field of view.

[0077] In step S3, the robot motion dynamic generator 116 (Figure 4) of the robot 100 generates camera motion constraints based on environmental information and the robot's mechanism. Specifically, the robot motion dynamic generator 116 identifies obstacles and other objects from the working environment information of the robot 100 measured by the environmental measurement sensor 109, and calculates the area in which the camera 101 can move based on the robot's mechanical constraints. In other words, the robot motion dynamic generator 116 determines (generates) motion constraints for the camera 101.

[0078] In step S4, the robot controller 110 determines whether the camera trajectory calculated in step S2 interferes with the camera motion constraints generated in step S3. If there is no interference (No in S4), proceed to step S5; if there is interference (Yes in S4), proceed to step S21.

[0079] In step S5, the overall motion generator 119 generates robot motion information that reproduces the camera trajectory calculated in step S2.

[0080] In step S6, the robot is operated based on the robot motion information, and images are captured by the camera to generate a set of images. Specifically, based on the robot motion information generated in step S5, the motion execution unit 113 drives and controls the arm 102 and the movement mechanism 103, and the images captured by the camera 101 at that time are saved in the motion result saver 114. Then, the motion result saver 114 generates a set of inspection images from the saved images using the inspection image generator 121. For example, the captured images are aggregated to generate an inspection video.

[0081] In step S7, the generated image set is displayed. Specifically, the robot controller 110 transmits the inspection image set generated by the inspection image generator 121 (Figure 4) to the input device 200 via the transceiver 106 and transceiver 204, displays the inspection image set on the display unit 201, and terminates the process.

[0082] In step S21, the motion generator 112 determines whether it is possible to avoid the camera motion constraint by adding a trajectory to the camera trajectory calculated in step S2. If it is possible to avoid it (Yes in S21), proceed to step S22; if it is not possible to avoid it (No in S21), proceed to step S31.

[0083] In step S22, a trajectory that avoids the camera motion constraints described in Figures 8A to 8G is generated. The method for generating the trajectory that avoids the camera motion constraints is described in detail below.

[0084] To generate a trajectory that avoids camera motion constraints, the relay point generator 117 (Figure 4) first generates relay points. For example, if the camera trajectory P4 generated by the camera trajectory generator 115 shown in Figure 7(c) interferes with the camera motion constraints, the relay point generator 117 generates a relay point O2' in the middle of the camera trajectory P4.

[0085] In detail, the relay point O2' is generated in the overlapping area of ​​the first and second operating ranges, with respect to the first operating range that does not interfere with the camera operation constraint when the camera is moved along the camera trajectory P4 from the second inspection target O2 to the third inspection target O3, and the second operating range that does not interfere with the camera operation constraint when the camera is moved along the camera trajectory P4 from the third inspection target O3 to the second inspection target O2.

[0086] Next, the camera motion constraint avoidance trajectory generator 118 (Figure 4) generates a trajectory for the generated relay point O2' that moves from the second inspection target O2 to the relay point O2', then moves away from the relay point O2', and returns to the relay point O2' in a position that allows movement from the relay point O2' to the third inspection target O3. This trajectory is used as the camera motion constraint avoidance trajectory.

[0087] In step S23, the overall motion generator 119 generates robot motion information that reproduces the camera trajectory, including the trajectory for avoiding camera motion constraints. Specifically, in step S2, the overall motion generator 119 generates time-series patterns for each joint of the arm 102 and the moving mechanism 103 based on the camera trajectory generated by the camera trajectory generator 115 and the trajectory for avoiding camera motion constraints generated by the camera motion constraint avoidance trajectory generator 118 in step S22, and uses this as robot motion information.

[0088] In step S24, the robot is operated based on the robot motion information, and a set of images is generated by deleting the images taken during the camera motion avoidance trajectory from the set of images taken by the camera, and then the process proceeds to step S7.

[0089] More specifically, based on the robot motion information generated in step S23, the motion executer 113 drives and controls the arm 102 and the movement mechanism 103, and the images captured by the camera 101 at that time are saved in the motion result saver 114. Then, the motion result saver 114 removes the images from the saved images using the avoidance motion image remover 120 to remove the images of the avoidance trajectory due to the camera motion constraint, and the inspection image generator 121 generates a group of inspection images. For example, the captured images are aggregated to generate an inspection video.

[0090] In step S31, an alternative camera trajectory is generated that avoids the camera operation constraints described in Figures 9A to 9E.

[0091] In step S32, robot motion information is generated that reproduces the camera trajectory, including the alternative camera trajectory, and the process proceeds to step S6.

[0092] Next, Figure 12 illustrates the other configurations of the image generation system according to the embodiment. Figure 12 shows the network configuration 500 of the entire image generation system. As shown in Figure 12, the robot 100, the input device 200, and the visual inspection work measurement device 600 are connected via the network 400 and can communicate various types of data. A storage and external processing device 300 may also be provided on the same network 400.

[0093] By providing a storage and external processing unit 300, data saved by the robot 100 can be stored externally. Similarly, data saved by the visual inspection work measurement device 600 can also be stored externally. In particular, by storing visual inspection data from maintenance personnel externally, it can be used as training data when building the robot 100's functions based on machine learning. Furthermore, even if processing is difficult for the robot controller 110 mounted on the robot 100, it is expected that the data can be processed by performing calculations on the externally provided storage and external processing unit 300.

[0094] In the above example, the generation of the camera trajectory for acquiring inspection images is performed by a motion generator 112 mounted on the robot 100. However, the camera trajectory may also be generated outside the robot 100, and the generated camera trajectory may be notified to the robot 100 to acquire inspection images.

[0095] For example, as shown in Figure 13, an image generation device 510 having a motion generator 112 and a motion result saver 114 receives time-series data of the gaze information of the maintenance worker 800 during the visual inspection work, the inspection target and inspection route, and the position of the eyes from the visual inspection work measurement device 600, generates a camera trajectory, and notifies the robot 100 of the robot motion information. The robot 100 then acquires the image captured by the camera 101 and notifies the input device 200 of the inspection image for display.

[0096] Furthermore, in Figure 13, the configuration of the robot 100, image generation device 510, visual inspection work measurement device 600, and input device 200 can also be considered an image generation system.

[0097] Alternatively, the robot control device 520 (indicated by the dashed line frame) may be composed of the motion generator 112, the transceiver 511 (transceiver inside the robot control device, transceiver inside the image generation device), and the visual inspection work measurement device 600. The robot control device 520 generates robot motion information for the robot 100 equipped with the camera 101 based on the gaze information of the maintenance worker performing the visual inspection, the inspection target and inspection route, and time-series data of the position of the eyes, and notifies the robot 100. As a result, the robot 100 can capture images of the maintenance worker's view with the camera 101.

[0098] Furthermore, the present invention is not limited to the embodiments described above, and various modifications are included. The embodiments described above are described in detail for the purpose of clearly illustrating the present invention, and are not necessarily limited to those having all the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. [Explanation of Symbols]

[0099] 100 robots 101 Camera 102 Arm 103 Moving mechanism 106 Transmitter / Receiver (Transmitter / Receiver inside the robot) 110 Robot Controller 112 Motion generator 113. Executor (Controller) 114 Operation result saver (image group generator) 115 Camera trajectory generator 116. Robot Motion Dynamic Generator (Robot Motion Constraint Generator) 117 Relay point generator 118 Camera Movement Constraint Avoidance Trajectory Generator (Camera Avoidance Trajectory Generator) 119 Overall motion generator 120 Avoidance Action Image Remover 121 Inspection Image Generator 200 Input devices (output devices) 201 Display section 202 Input Section 203 Visual Inspection Work Data Storage Device 510 Image Generator 511 Transmitters and receivers (transmitters and receivers within robot control devices, transceivers and receivers within image generation devices) 520 Robot Control Device 600 Visual Inspection Work Measurement Device 601 Eye-tracking sensor 602 Measurement sensors to be inspected 603 Position and orientation sensor 604 Transmitter / Receiver 610 Visual Inspection Work Controller 612 Visual Inspection Work Extractor 613 Measurement Result Storage Device 614 Eye-tracking detector 615 Extractor for testing 616 Eye position calculator O1 First inspection target (maintenance point) O2 Second inspection target (maintenance point) O3 Third inspection target (maintenance point)

Claims

1. A robot equipped with a camera, A robot-based transceiver that receives eye-level information from a maintenance worker and information from multiple maintenance points during visual inspection work, A camera trajectory generator that generates the camera trajectory of the camera from the aforementioned viewpoint information, the maintenance point information, and the specifications of the camera, A robot motion constraint generator that generates motion constraints for the robot from environmental information in which the robot operates and the mechanical constraints of the robot, A camera avoidance trajectory generator that generates an avoidance trajectory for the camera that avoids the motion constraints of the robot, A total motion generator that generates robot motion information for the robot using the camera trajectory, the robot's motion constraints, and the camera's avoidance trajectory, The system includes a controller that controls the robot using the robot motion information generated by the overall motion generator. A robot characterized by the following features.

2. In the robot according to claim 1, The overall motion generator is, The avoidance trajectory is added to the aforementioned camera trajectory, The point at which the camera moves from the camera trajectory to the avoidance trajectory and the point at which it returns from the avoidance trajectory to the camera trajectory are approximately the same location. A robot characterized by the following features.

3. In the robot according to claim 1, An image group generator that generates an image group using images captured by the camera when the robot operates based on the robot motion information, and deletes images captured in the avoidance trajectory from the image group, The system comprises an output device that outputs the image group generated by the image group generator. A robot characterized by the following features.

4. In the robot according to claim 1, The camera will temporarily stop taking pictures during the avoidance trajectory. A robot characterized by the following features.

5. In the robot according to claim 3 or 4, The image taken when transitioning to the avoidance trajectory generated by the camera avoidance trajectory generator and the image taken when returning to the camera trajectory are continuous. A robot characterized by the following features.

6. In the robot according to claim 1, The aforementioned eye-line information includes the position and gaze information of the maintenance worker. A robot characterized by the following features.

7. An image generation system comprising a robot equipped with a camera and an image generation device that generates a group of images using images captured by the camera, The image generation device is A camera trajectory generator that generates the camera trajectory of the camera from the maintenance worker's eye-view information and information on multiple maintenance points acquired during the visual inspection work, and the specifications of the camera, A robot motion constraint generator that generates motion constraints for the robot from environmental information in which the robot operates and the mechanical constraints of the robot, A camera avoidance trajectory generator that generates an avoidance trajectory for the camera that avoids the motion constraints of the robot, A total motion generator that generates robot motion information for the robot using the camera trajectory, the robot's motion constraints, and the camera's avoidance trajectory, An image generation device includes a transceiver that transmits the robot motion information to the robot and receives the image captured by the camera, An inspection image generator that generates a set of images using images taken when the robot operates, based on the robot motion information received by the transceiver inside the image generation device, The system comprises an output device that outputs the image group generated by the inspection image generator, The aforementioned robot, A robot-internal transceiver that receives the robot motion information and transmits the image captured by the camera, The system includes a controller that controls the robot based on the robot motion information. An image generation system characterized by the following features.

8. An image generation device that generates a group of images using images captured by a camera installed in the robot described in claim 1, An image generation device includes a transceiver that receives the image captured by the aforementioned camera, An image group generator that generates an image group using images received by the transceiver within the image generation device, The system comprises an output device that outputs the image group generated by the image group generator. An image generation device characterized by the following features.

9. A robot control device for a robot equipped with a camera, A transceiver inside a robot control unit that receives eye-level information from a maintenance worker and multiple maintenance points during visual inspection work, A camera trajectory generator that generates the camera trajectory of the camera from the aforementioned viewpoint information, the maintenance point information, and the specifications of the camera, A robot motion constraint generator that generates motion constraints for the robot from environmental information in which the robot operates and the mechanical constraints of the robot, A camera avoidance trajectory generator that generates an avoidance trajectory for the camera that avoids the motion constraints of the robot, A total motion generator that generates robot motion information for the robot using the camera trajectory, the robot's motion constraints, and the camera's avoidance trajectory, The system includes a controller that controls the robot using the robot motion information generated by the overall motion generator. A robot control device characterized by the following.

10. An image generation method for an image generation device that generates a series of images by controlling a robot equipped with a camera, Steps to detect the eye-level information of maintenance personnel and multiple maintenance points during visual inspection work, A step of generating the camera trajectory of the camera from the aforementioned viewpoint information, the maintenance point information, and the specifications of the camera, The steps include determining the robot's operational constraints from environmental information in which the robot operates and the robot's mechanical constraints, The steps include generating an avoidance trajectory for the camera that avoids the movement constraints of the robot, Using the aforementioned camera trajectory and the robot's motion constraints, the camera's avoidance trajectory is generated to avoid the motion constraints. A step of generating robot motion information using the avoidance trajectory, the camera trajectory, and the robot motion constraints, The steps include generating a set of images using images taken by a camera of a robot that operates based on the robot motion information, An image generation method that includes [a specific feature / method].

Images