ROBOT CONTROL SYSTEM, ROBOT CONTROL METHOD, AND PROGRAM

Abstract

To make it possible to automatically cut out an image captured with a camera during the work of the robot and a region within the image and to present the cut-out one to an operator or a monitor.SOLUTION: A robot control system comprises: a robot 10; a control device 20 that transmits a control command to the robot 10; a camera 3 that captures and obtains a captured image of at least a work environment of the robot 10; an image region selection device 30 that selects at least one region of the captured image which is important for the work of the robot 10, on the basis of the captured image, and sensor information acquired from a sensor that is placed on the robot 10 and / or in the environment and that detects the state of the robot 10 and / or the environment; and an image display device 40 that displays the region of the captured image, which has been selected by the image region selection device 30.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a robot control system, a robot control method, and a program having a function of monitoring the working status of a robot. [Background technology]

[0002] In order to improve production efficiency and reduce labor costs, there are increasing efforts to use robots to replace tasks previously performed by humans, such as assembling industrial products, welding, and transporting. In these robot systems, cameras are installed in appropriate positions, and tasks are often carried out by humans (operators) operating the robot based on images from the cameras, or by controlling the robot automatically.

[0003] In this case, the appropriate position, angle of view, and number of cameras depend on the work content. For example, if the work area including the work object and the robot is large, a wide-angle camera must be used, and the camera image will be large. Therefore, the operator or monitor needs to check the camera image by zooming in or moving the line of sight. The operator is the person who operates the robot when the robot is operated manually. The monitor is the person who monitors the work status of the robot, whether the robot is operated manually or automatically. In the case of manual operation, the operator and the monitor may be the same person.

[0004] In addition, in tasks such as assembling parts, including tightening screws, both an overview of the work environment for situation recognition and motion planning, and a local perspective to prevent occlusion for fine positioning, are necessary. Therefore, operators and supervisors need to check images by switching between multiple camera images.

[0005] As described above, in a robot system using camera images, the operator or observer needs to move his or her line of sight or switch viewpoints, which places a burden on the operator or observer. Therefore, Patent Document 1 describes a system that automatically switches the appropriate camera image for remotely controlling the robot based on pre-stored camera image switching information. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2017 / 033359 Summary of the Invention [Problem to be solved by the invention]

[0007] However, although the technology described in Patent Document 1 enables switching between a plurality of camera images, it does not mention cropping or zooming of camera images.

[0008] In view of the above circumstances, there has been a demand for a method of switching between images captured by a camera or areas within an image while a robot is working and presenting them to an operator or observer. [Means for solving the problem]

[0009] In order to solve the above problems, a robot control system according to one embodiment of the present invention includes a robot, a control device that transmits control commands to the robot, a camera that captures at least an image of the robot's working environment to obtain a captured image, an image area selection device that selects at least a partial area of ​​the captured image that is important for the robot's work based on the captured image and sensor information obtained from sensors disposed on the robot and / or the environment that detect the state of the robot and / or the state of the environment, and an image display device that displays the area of ​​the captured image selected by the image area selection device. Effect of the Invention

[0010] According to at least one aspect of the present invention, an image or an area within an image that is important for the robot's work captured by a camera is selected and displayed on an image display device. This makes it possible to automatically crop or zoom the image, reducing the burden on the operator or observer of switching between images captured by the camera and moving their line of sight. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic diagram showing a configuration example of a robot control system to which the present invention is applied; [Diagram 2] 1 is a block diagram showing an example of a hardware configuration of an image region selection device provided in a robot control system. FIG. [Diagram 3] 1 is a block diagram showing an example of the internal configuration of an image region selection device provided in a robot control system. FIG. [Figure 4] 5 is a flowchart showing an example of the operation of the image region selection device and the image display device in the first embodiment of the present invention when there is a single camera image. [Diagram 5] FIG. 2 is an explanatory diagram showing an example of a heat map for an input camera image and an image display of an image display device in the first embodiment of the present invention. [Figure 6] 10 is a flowchart showing an example of the operation of an image region selection device and an image display device in a case where there are a plurality of camera images in a second embodiment of the present invention. [Figure 7] FIG. 11 is an explanatory diagram showing an example of an input camera image and an obtained heat map in the second embodiment of the present invention. [Figure 8] FIG. 11 is an explanatory diagram showing an example of an image display of an image display device in a second embodiment of the present invention. [Figure 9] FIG. 11 is an explanatory diagram showing another example of an image display of the image display device in the second embodiment of the present invention. [Figure 10] 13 is a flowchart showing an example of the operation of the image region selection device and the image display device when automatically generating an operation command for a robot in the third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, examples of embodiments of the present invention will be described with reference to the accompanying drawings. In this specification and the accompanying drawings, components having substantially the same functions or configurations are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0013] <First embodiment> First, a robot control system according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 5. FIG.

[0014] Fig. 1 is a schematic diagram showing an example of the configuration of a robot control system to which the present invention is applied. In the robot control system 100 shown in Fig. 1, a robot 10 is a device capable of handling objects and performing predetermined tasks such as assembling and transporting parts. Here, the configuration of the robot 10 is not important, and it may be a robot arm alone or may include a moving device such as crawlers or wheels.

[0015] The robot controller 20 is a device that outputs control commands to the robot 10 to control the operation of the robot 10 based on operation commands such as joint angles and forces (torque) of the robot 10 input to the robot controller 20. The control commands are signals that indicate, for example, current values ​​and voltage values ​​for actuators (motors, etc.) provided in the joints and end effector 11 of the robot 10. When the robot 10 receives a control command from the robot controller 20, a built-in drive circuit supplies a drive signal to the corresponding actuator.

[0016] The robot control device 20 may be configured to automatically control the operation of the robot 10 based on the camera image captured by the camera 3. A configuration in which the robot control device 20 automatically controls the operation of the robot 10 will be described in the third embodiment.

[0017] The camera 3 is an imaging device for capturing images of the working environment and surrounding environment of the robot 10. There may be one camera 3 or multiple cameras as shown in FIG. 1. In FIG. 1, three cameras, namely, cameras 3a, 3b, and 3c, are installed as the camera 3. The camera 3a is attached to the robot arm, and the cameras 3b and 3c are installed around the robot 10 (for example, a workroom or a wall of a building). Here, the working environment corresponds to the movable area of ​​the robot 10, that is, the range in which the robot 10 can move (work area). The surrounding environment corresponds to the surrounding area outside the movable area of ​​the robot 10. The camera 3 is assumed to be a network camera without a posture changing function and a hardware zoom function.

[0018] In this embodiment, a case where a single camera image is used will be described, and in the second embodiment, a case where a plurality of camera images are used will be described.

[0019] The image area selection device 30 is a device that selects an image or an area within the image that is important for the work of the robot 10 based on the camera image captured by the camera 3 and information obtained from sensors arranged in the robot 10 and / or the environment (hereinafter referred to as "sensor information"). Here, the type of sensor information of the robot 10 and the environment does not matter. For example, the sensor information may be the current value of a motor provided in the joints of the robot 10, or the output signal of a tactile sensor or an inertial sensor externally attached to the robot 10. Furthermore, the sensor information may be the output signal of a temperature sensor that measures the work environment, or a gaze measurement sensor (not shown) attached to the operator of the robot 10. In this way, each sensor detects the state of the robot 10 and / or the state of the environment, and outputs a detection signal according to the detection content.

[0020] The image display device 40 is a device that selectively displays the image or an area within the image selected by the image area selection device 30. The robot operation device 50 is a device that includes buttons, joysticks, etc. that correspond to the movements of the robot 10, receives input from the operator, and transmits operation commands corresponding to the contents of the input to the robot control device 20.

[0021] [Hardware configuration of each device] Next, the hardware configuration of the image region selection device 30 included in the robot control system 100 will be described with reference to Fig. 2. Here, an example of the hardware configuration of the computer included in the image region selection device 30 will be described.

[0022] 2 is a block diagram showing an example of the hardware configuration of a computer included in the image region selection device 30. The illustrated computer 60 is an example of hardware constituting a computer used in the robot control device 20 and the image region selection device 30. The computer 60 can be, for example, a personal computer.

[0023] The computer 60 includes a CPU (Central Processing Unit) 61, a ROM (Read Only Memory) 62, and a RAM (Random Access Memory) 63, each of which is connected to a bus 64. The computer 60 further includes a non-volatile storage 66, an input / output interface 67, and a network interface 68.

[0024] The CPU 61 reads out from the ROM 62 the program code of the software that realizes the functions of the image region selection device 30 according to this embodiment, and loads the program into the RAM 63 and executes it. Variables, parameters, etc. generated during the arithmetic processing of the CPU 61 are temporarily written into the RAM 63. The variables, parameters, etc. written into the RAM 63 are read out by the CPU 61 as appropriate. Although the CPU 61 is used as the arithmetic processing device, other processors such as an MPU (Micro Processing Unit) may also be used.

[0025] The non-volatile storage 66 is an example of a recording medium, and is capable of storing data used by a program, data obtained by executing a program, and the like. For example, the non-volatile storage 66 stores learning data and learning models, which will be described later. In addition, the non-volatile storage 66 may store an OS (Operating System) and a program executed by the CPU 61. As the non-volatile storage 66, a semiconductor memory, a HDD (Hard Disk Drive), an SSD (Solid State Drive), a disk device using magnetism or light, or the like is used.

[0026] The input / output interface 67 is an interface for communicating signals and data with each sensor and each actuator included in the robot control system 100. The input / output interface 67 may also function as an A / D (Analog / digital) converter and / or a D / A converter (not shown) for processing input or output signals. The sensor information in this specification includes information obtained not only from each sensor but also from each actuator.

[0027] The network interface 68 may be, for example, a network interface card (NIC) or a modem. The network interface 68 is configured to be capable of transmitting and receiving various data to and from an external device via a communication network such as a LAN or the Internet to which a terminal is connected, or a dedicated line.

[0028] [Internal structure of the image region selection device] 3 is a block diagram showing an example of the internal configuration of image region selection device 30. Image region selection device 30 includes a learning data storage unit 31, a learning unit 32, a learning model storage unit 33, and an inference unit .

[0029] The learning data storage unit 31 stores learning data used for learning the learning model 33a. The learning data includes at least camera images of the working environment and surrounding environment of the robot 10, which are acquired in time series while the robot 10 is working, and sensor information of the robot 10 and the environment.

[0030] The learning unit 32 uses the learning data stored in the learning data storage unit 31 to make the learning model 33a learn predicted values ​​of sensor information of the robot 10 and the environment, and a heat map (important areas and importance of a camera image). For example, the learning unit 32 performs learning of the learning model 33a by machine learning to adjust model parameters of the learning model 33a. As an example, the learning model 33a can be configured using a neural network. The learning method of the learning model 33a is not limited to machine learning by deep learning using a neural network, and may be another learning method.

[0031] The learning model storage unit 33 stores the learning model 33a and its model parameters. For example, when a neural network is used for the learning model 33a, the model parameters are weights such as the degree of connection between neurons constituting the neural network and the firing threshold of the neurons. The learning model storage unit 33 is realized by a non-volatile storage 66, for example. The learning model 33a stored in the learning model storage unit 33 is a trained model (inference program) reflecting the results of learning (model parameters).

[0032] The inference unit 34 uses the learning model 33a stored in the learning model storage unit 33 to make inferences (predictions) about the input camera image and the sensor information of the robot 10 and the environment, and outputs the predicted values. The inference unit 34 also includes a heat map generation unit 331 that generates a heat map, and uses the learning model 33a to infer a heat map (important area, importance) for the input camera image. Details of the heat map and the heat map generation unit 331 will be described later. The result of the inference by the inference unit 34 is output to the robot control device 20 and the image display device 40. The inference unit 34 processes (post-processes) the camera image based on the inference result, and outputs data required for displaying the image on the image display device 40. Examples of processing the camera image include cutting out a specific area and enlarging it.

[0033] Next, an example of a method in which the image area selection device 30 selects an area in a camera image that is important for the robot 10 to perform a task (hereinafter referred to as an "important area") in the robot control system 100 will be described. The method of selecting an important area in a camera image is divided into a teaching phase and an operation phase. For example, a learning mode and an operation mode are prepared in the robot control system 100, and the robot transitions to the teaching phase or the operation phase by selecting either the learning mode or the operation mode displayed on a menu screen (not shown) with the robot operation device 50.

[0034] [Robot control system teaching phase] In the teaching phase, first, the image region selection function of the image region selection device 30 is disabled, and then a task is performed using the robot 10. The image region selection device 30 stores the camera images acquired during the task and the sensor information of the robot 10 and the environment as learning data in the learning data storage unit 31. Learning is performed in the learning unit 32 using the learning data at this time as teacher data. Note that the operator may control the robot 10 using the robot operation device 50, or the robot 10 may automatically reproduce a previously planned operation of the robot 10.

[0035] Next, in the learning section 32 of the image region selection device 30, when the robot 10 performs a certain task, model parameters of a learning model 33a that predicts the camera image and the sensor information of the robot 10 and the environment from the next time (e.g., time t+1) onwards, based on the camera image at the current time (e.g., time t) and the sensor information of the robot 10 and the environment, are learned and stored in the learning model storage section 33. Learning of the learning model 33a is performed for each type of task. Furthermore, the accuracy of the learning is improved by performing learning multiple times for one type of task.

[0036] As described above, the learning model 33a has a heat map generating unit 331 therein that generates a heat map indicating an area in the camera image, particularly an image area (important area) required for the prediction. The learning model 33a is configured to optimize the heat map so that the heat map value is large or small in the image area required for prediction, so that the prediction error for each data is reduced during the learning process. Each data is the camera image, the robot 10, and sensor information of the environment. The learning unit 32 is configured to set an area where the value of the heat map generated by the heat map generating unit 331 of the learning model 33a is equal to or greater than a set threshold as an important area. The important area in this embodiment is assumed to be at least a part of the camera image, but may be the entire image.

[0037] The learning model 33a compares the camera image and the sensor information of the robot 10 and the environment at time t+1 with the camera image and the sensor information of the robot 10 and the environment corresponding to time t+1 stored in the learning model storage unit 33, and calculates the error (corresponding to the prediction error) of each data. In this embodiment, the value of this heat map is regarded as the "importance" of the area in the camera image. Here, the area of ​​the image is not limited to an area having a certain area. The area of ​​the image may be a point, and the value of the heat map may be a single value at that point.

[0038] An image or an area within the image that is important for the work of the robot 10 is searched for according to a predetermined area search algorithm. The shape of the area is assumed to be circular, but it may be elliptical or rectangular. As an example, the following area search algorithm is considered. First, the learning model 33a detects a work object in a camera image, and sets a position (x coordinate, y coordinate) that serves as a reference for search within the image based on the type of work and the work object, etc. The learning model 33a forms a provisional area with multiple pixels contained within a certain radius centered on the reference position, and calculates a prediction error using an image of the provisional area. Next, the learning model 33a changes the range of the provisional area (for example, makes the range smaller) and calculates a prediction error.

[0039] The learning model 33a then compares the prediction error when the previous provisional region is used with the prediction error when the current provisional region is used, and determines whether the prediction error is smaller than the previous one. If the prediction error is smaller, the learning model 33a further reduces the range of the provisional region, calculates and compares the prediction error, and repeats these processes. If the prediction error gradually decreases and then starts to increase, the provisional region having the range and reference position immediately before the prediction error started to increase is considered to be an important region for the work of the robot 10.

[0040] Furthermore, by calculating and comparing prediction errors by changing not only the range of the provisional region but also the reference position in the image of the provisional region, it is possible to identify a region that can be estimated to be important for the work of the robot 10. In the process of setting the range of such a region and calculating and comparing prediction errors, the heat map generating unit 331 of the learning model 33a generates a heat map by assigning a heat map value to each pixel in the image or to a pixel representing the region according to the prediction error. As described above, a region consisting of pixels whose heat map value is equal to or greater than a set threshold value is identified as an important region. Depending on the type of work (for example, work that requires a bird's-eye view), the range of the provisional region is enlarged while calculating and comparing prediction errors, and a region that is important for the work of the robot 10 is identified.

[0041] [Operation phase of robot control system] Next, the operation phase of the robot control system 100 will be described with reference to FIG. 4 is a flowchart showing an example of the operation of the inference unit 34 of the image region selection device 30 and the image display device 40 when there is a single camera image. First, the robot control device 20 receives a motion command for the robot 10 input by the operator from the robot operation device 50, and generates control commands for each actuator for controlling the robot 10 based on the motion command. The motion command is, for example, the joint angles of the robot 10, and the attitude (position) and force (torque) of the end effector 11. The robot 10 receives the control command from the robot control device 20 and starts its motion (task) (S1).

[0042] Next, the operator or observer checks the camera image displayed on the screen of the image display device 40 to determine whether the work of the robot 10 has been completed (S2). For example, an icon of a work completion button 74 (see FIG. 5C described later) is displayed in the camera image displayed on the image display device 40. When the operator or observer determines that the work has been completed, he or she clicks on the icon of the work completion button 74 using the robot operation device 50. When the inference unit 34 of the image area selection device 30 detects that the work completion button 74 has been operated, it determines that the work has been completed. Note that a mechanical work completion button may be provided on the robot operation device 50.

[0043] When the operator or observer determines that the work of the robot 10 is completed (YES in S2), the operator or observer operates the work completion button 74 and then ends the work of the robot 10 (S7).

[0044] On the other hand, if the operator or supervisor determines that the robot 10 has not completed its work (NO in S2), the inference unit 34 acquires camera images and sensor information of the robot 10 and the environment, inputs them into the learning model 33a (S3), and obtains an output (inference result) for the input.

[0045] Next, the inference unit 34 acquires an important area of ​​the camera image as an output in response to the input of the camera image and the sensor information of the robot 10 and the environment to the learning model 33a in step S3 (S4). The important area of ​​the camera image corresponds to an area with a high value (importance) of the heat map. The inference unit 34 also acquires a predicted value of the camera image and the sensor information of the robot 10 and the environment as an output in response to the input of each data to the learning model 33a in step S3 (S5). Note that this predicted value is used when the robot 10 automatically performs a task in the third embodiment.

[0046] Next, the image region selection device 30 transmits information (e.g., position, range) of the important region of the camera image inferred by the inference unit 34 to the image display device 40. As a result, the image region selection device 30 displays an image including the important region in the camera image obtained by the inference unit 34 in step S4 on the image display device 40 (S6). As a result, the image display device 40 displays an enlarged important region in the camera image on the screen. After the processes of steps S5 and S6, the process returns to the determination process of step S2. If it is not determined that the work is completed, the camera image and the sensor information of the robot 10 and the environment are input to the learning model 33a. In this way, the processes of steps S2 to S6 are repeated until it is determined that the work is completed, that is, from the start of the work to the end of the work. FIG. 5 shows an example of an image display of the image display device 40.

[0047] [Image display example] Fig. 5 shows an example of a heat map and image display for an input camera image in the first embodiment. Fig. 5A shows an input camera image, Fig. 5B shows a heat map, and Fig. 5C shows an example of an image display on the image display device 40. In the example shown in Fig. 5, as an example, an area with a large heat map value, i.e., an area with high importance, is displayed dark, and an area with a small heat map value, i.e., an area with low importance, is displayed light.

[0048] When the heat map 72 shown in Fig. 5B is obtained for the input camera image 71 shown in Fig. 5A, the image display device 40 displays an image of an area (specific area Ai) where the value of the heat map 72 is large, as shown in Fig. 5C. Fig. 5C shows an example of an image 73 in which an area of ​​the input camera image 71 corresponding to the specific area Ai of the heat map 72 is enlarged and displayed. Specifically, in the image 73, a local area including the work object 12 and the end effector 11 of the image 71, which is the input camera image, is enlarged.

[0049] As described above, the robot control system (robot control system 100) according to the first embodiment is configured to include a robot (robot 10) that performs work, a control device (robot control device 20) that transmits control commands to the robot based on operation commands, a camera (camera 3) that captures at least the robot's work environment to obtain a captured image, an image area selection device (image area selection device 30) that selects at least a partial area of ​​the captured image that is important for the robot's work based on the captured image and sensor information obtained from a sensor that detects the state of the robot and / or the state of the robot and / or the environment placed in the environment, and an image display device (image display device 40) that displays the area of ​​the captured image selected by the image area selection device.

[0050] The robot control system 100 according to the first embodiment configured as described above sequentially acquires important areas of the image captured by the camera 3, and updates the display of the image or the area in the image on the image display device 40, thereby presenting the image or the area in the image that is important for the work to the operator or the observer according to the work situation. That is, the robot control system 100 automatically cuts out the image or the area in the image that is important for the work of the robot 10 captured by the camera 3 by the image area selection device 30, and displays it enlarged on the image display device 40. This allows the robot control system 100 to reduce the load on the operator or the observer due to switching of the image captured by the camera 3 or the area in the image and the movement of the line of sight. For example, it is possible to automatically switch and display on the image display device 40 an image from a viewpoint that overlooks the work environment because the work area is wide, and an image from a viewpoint that requires information on fine details.

[0051] In the robot control system (robot control system 100) according to the first embodiment, the image region selection device (image region selection device 30) is configured to include a learning model (learning model 33a) that is trained to predict captured images and sensor information from the next time onward from the captured image captured by the camera at the current time and sensor information, and a heat map generation unit (heat map generation unit 331) that generates a heat map indicating areas in the captured image within the learning model. The learning model optimizes the heat map so that values ​​are larger or smaller in areas necessary for prediction so that prediction errors are reduced during the learning process, and sets areas where the values ​​are equal to or greater than a set threshold as important areas (specific areas Ai).

[0052] Moreover, the robot control system (robot control system 100) according to the first embodiment includes a model memory unit that stores a learning model (learning model 33a) for selecting an important area (specific area Ai) in a captured image, a learning unit that learns the learning model, a learning data memory unit that includes at least captured images and sensor information obtained while the robot is working and stores time-series learning data used for learning the learning model, and an inference unit that infers important areas in the captured image using the learning model stored in the model memory unit.

[0053] Note that there may be multiple important regions for one camera image. For example, if there are multiple important regions in a camera image, the camera image may be divided and images may be displayed for each important region. In this case, if there are many important regions in a camera image, it is advisable to obtain the importance of each important region in step S4 and display images of important regions with higher importance first.

[0054] <Second embodiment> Next, as a second embodiment of the present invention, a robot control system in the case where there are multiple camera images will be described with reference to Fig. 6 to Fig. 9. The basic configuration of the robot control system according to the second embodiment is the same as the robot control system 100 shown in Fig. 1.

[0055] Fig. 6 is a flowchart showing an example of the operation of the inference unit 34 of the image region selection device 30 and the image display device 40 when there are multiple camera images in the second embodiment. In the second embodiment, the difference from the first embodiment is that there are multiple camera images acquired by the camera 3, and the parts where the flowchart in Fig. 6 differs from Fig. 4 shown in the first embodiment are steps S14, and S16 to S17. The processes of steps S11 to S13, S15, and S18 are the same as the processes of steps S1 to S3, S5, and S7 in Fig. 4, so a detailed description will not be given.

[0056] In FIG. 6, if the operator or observer determines that the robot 10 has not completed its work (NO in S12), the inference unit 34 of the image area selection device 30 acquires multiple camera images and sensor information of the robot 10 and the environment, inputs them to the learning model 33a (S13), and obtains an output (inference result) for the input.

[0057] Next, the inference unit 34 acquires important areas and importance (heat map values) for each of the multiple camera images as output in response to the input of the multiple camera images, and the sensor information of the robot 10 and the environment to the learning model 33a in step S13 (S14). The inference unit 34 also acquires predicted values ​​of the multiple camera images, and the sensor information of the robot 10 and the environment as output in response to the input of each data to the learning model 33a in step S13 (S15).

[0058] Next, the inference unit 34 compares the importance obtained in step S14 between the multiple camera images and selects a camera image or a desired area to display (S16). In this example, two or more camera images are selected in descending order of importance.

[0059] Next, the image region selection device 30 transmits the camera image or the important region selected by the inference unit 34 to the image display device 40. As a result, the image region selection device 30 displays the camera image or the important region obtained by the inference unit 34 in step S14 on the image display device 40 (S17).

[0060] After the processes of steps S15 and S17, the process returns to the determination process of step S12, and if the work is not completed (NO in S12), the processes of steps S13 to S17 are repeated. Fig. 7 shows an example of an image displayed on the image display device 40.

[0061] Fig. 7 shows examples of input camera images and obtained heat maps in the second embodiment. For example, Fig. 7A shows an input camera image (image 81) of camera 3a, Fig. 7B shows an input camera image (image 82) of a camera that can overlook the entire robot 10 such as camera 3c, and Fig. 7C shows an input camera image (image 83) of camera 3b. Images 81 to 83 in Figs. 7A to 7C show specific regions Ai1 to Ai3 in which the heat map values ​​are relatively higher than the other pixels, respectively.

[0062] [Example of image display] Fig. 8 shows an example of an image display of the image display device 40 in the second embodiment. In Fig. 8, an image 91 corresponds to the input camera image 81, an image 92 corresponds to the input camera image 82, and an image 93 corresponds to the input camera image 83.

[0063] As shown in FIG. 7A to FIG. 7C, when heat maps including specific regions Ai1 to Ai3 are obtained for the multiple input camera images 81 to 83 in FIG. 8, the image display device 40 preferentially displays camera images of areas with high importance having large heat map values. FIG. 8 shows an example in which camera images to be displayed are arranged in order (predetermined positions) according to the importance of the camera images. Here, on the display screen 90, images are displayed large according to the importance of the camera images, and other small images are displayed in a reduced size. In this way, the arrangement of the multiple images acquired by the camera 3 is determined according to the importance, so that it becomes clear which image the operator or observer should check with priority. Furthermore, since the area (size) of the images differs depending on the priority, it becomes clearer which image the operator or observer should check with priority.

[0064] For example, in Figures 7A to 7C, the heat map values ​​in the specific regions Ai1 to Ai3 are Ai1>Ai2>Ai1 in order. Therefore, in the display screen 90 shown in Figure 8, the image 91 with the largest heat map value is displayed largest in the left and center areas, and the image 92 with the second largest heat map value and the image 93 with the third largest heat map value are displayed vertically in the right area, reduced by the same reduction ratio. It is needless to say that the arrangement of the image with the largest heat map value and the other images is not limited to the example of the arrangement in Figure 8.

[0065] Here, the method of displaying images in image display device 40 (image size, shape, arrangement, etc.) does not matter, and the area of ​​the image to be displayed may be changed according to the value (importance) of the heat map as shown in FIG.

[0066] [Other examples of image display] Fig. 9 shows another example of image display of the image display device 40 in the second embodiment. The example shown in Fig. 9 is an example in which the area of ​​the displayed camera image is changed according to the importance of the camera image.

[0067] 9, images 91A to 93A corresponding to the images 91 to 93 shown in FIG. 8 are displayed on a display screen 90A. The relative positional relationship of the images 91A to 93A and the ratio of the long side to the short side (aspect ratio) of each image are the same as those of the images 91 to 93. However, the area (size) relationship of the images 91A to 93A is different from that of the images 91 to 93. That is, in FIG. 9, the areas of the images 92A and 93A are different. In this way, the images may be displayed larger in order of the heat map value (higher importance). In this way, the areas of the multiple images acquired by the camera 3 are determined according to the importance, so that the operator or the observer can clearly see which image should be checked with priority.

[0068] Note that display screen 90A has blank spaces below image 91A and to the right of image 93A. In order to effectively utilize the display area of ​​display screen 90A, the sizes, shapes, and arrangements of images 91A to 93A can be changed as appropriate within the range of a rule that images are displayed larger in descending order of importance.

[0069] The robot control system 100 according to the second embodiment configured as described above, when there are multiple images captured by the camera 3, sequentially acquires the important area and importance of the images that are important for the work of the robot 10, and updates the display of the images on the image display device 40 based on the importance of each image, thereby presenting the images that are important for the work to the operator or observer according to the work situation. That is, the robot control system 100 uses the image area selection device 30 to compare the importance of multiple images captured by the camera 3, automatically selects an image based on the importance, and displays it on the image display device 40. As a result, the robot control system 100 according to this embodiment can reduce the load on the operator or observer due to switching between multiple images captured by the multiple cameras 3 and moving the line of sight. For example, multiple images such as an image from a viewpoint overlooking the work environment and an image from a local viewpoint that prevents occlusion for fine positioning can be automatically switched and displayed on the image display device 40.

[0070] Note that the configuration of cropping and enlarging the image area of ​​the camera image in the first embodiment may be applied to the robot control system 100 according to the second embodiment, which switches between a plurality of camera images. By configuring in this way, the effects of the first embodiment can be obtained in addition to the effects of the second embodiment.

[0071] <Third embodiment> Next, as a third embodiment of the present invention, a robot control system in which an operation command for the robot 10 is automatically generated in parallel with the display of an image on the image display device 40, and the robot 10 operates autonomously, will be described with reference to Fig. 10. The basic configuration of the robot control system according to the third embodiment is the same as that of the robot control system 100 shown in Fig. 1.

[0072] FIG. 10 is a flowchart showing an example of the operation of the inference unit 34 of the image region selection device 30 and the image display device 40 when an operation command for the robot 10 is automatically generated in the third embodiment. In the third embodiment, the difference from the first embodiment is that an operation command for the robot 10 is automatically generated, and the flowchart in FIG. 10 differs from FIG. 4 shown in the first embodiment in that step S27 has been added. The processes in steps S21 to S26 and S28 are the same as the processes in steps S1 to S7 in FIG. 4, and therefore will not be described in detail. In this embodiment, the inference unit 34 of the image region selection device 30 and the image display device 40 are automatically generated in the third embodiment. important The area (and importance) and the predicted value are used to generate a motion command for the robot 10.

[0073] The image area selection device 30 controls the robot 10 by transmitting a command regarding the operation of the robot 10 to the robot control device 20 based on the important area (importance) of the camera image, the camera image, and the predicted values ​​of the sensor information of the robot 10 and the environment outputted from the learning model 33a by the inference unit 34. At this time, the robot 10 may be operated by an operator by accepting input from the robot operation device 50, or the robot 10 may be operated completely automatically without accepting input from the robot operation device 50. The process of step S27 is as follows.

[0074] After completing the process of step S25, the inference unit 34 inputs a predicted value relating to the operation of the robot 10 out of the predicted values ​​of the camera image, robot 10 and sensor information acquired in step S25 to the robot control device 20 (S27). The robot control device 20 outputs a control command to the robot 10 using the predicted value input from the image region selection device 30 as an operation command, and controls the operation (task) of the robot 10. Here, the predicted value relating to the operation of the robot 10 corresponds to the operation command of the robot 10, and is, for example, the joint angles of the robot 10, and the attitude (position) and force (torque) of the end effector 11.

[0075] After the processes of steps S26 and S27, the process returns to the determination process of step S22, and if the work is not completed (NO in S22), the processes of steps S23 to S27 are repeated.

[0076] The robot control system 100 according to the third embodiment configured as described above presents an image or an area within the image that is important for the work to an operator or a supervisor according to the work situation by updating the display of the image or an area within the image on the image display device 40 while the robot 10 automatically performs the work. As a result, the robot control system 100 according to this embodiment, like the first embodiment, can reduce the burden on the operator or supervisor caused by switching the image captured by the camera 3 or an area within the image and moving the line of sight, and can also cause the robot 10 to automatically perform the work.

[0077] The configuration of switching between multiple camera images in the second embodiment may be applied to the robot control system 100 according to the third embodiment that automatically generates an operation command for the robot 10. With this configuration, the effects of the second embodiment can be obtained in addition to the effects of the third embodiment. That is, the robot control system 100 according to this embodiment can reduce the burden on an operator or observer caused by switching between multiple images captured by multiple cameras 3 and moving their line of sight, and can also cause the robot 10 to automatically perform a task.

[0078] In the third embodiment, since the robot 10 automatically generates operation commands, it is possible to delete the robot operation device 50 of the robot control system 100. In the third embodiment, since an operator does not operate the robot operation device 50, it may be unnecessary to check the work status using the image display device 40 in order to input operation commands. Therefore, the image display device 40 of the robot control system 100 may be deleted. However, the image display device 40 may be left in the robot control system 100 so that an observer can check the work status due to the automatic operation of the robot 10.

[0079] <Modification> The present invention is not limited to the above-mentioned first to third embodiments, and of course, various other application examples and modifications are possible without departing from the gist of the present invention described in the claims. For example, the above-mentioned embodiments are detailed and specific descriptions of the configurations of the robot control system and the image area selection device in order to easily explain the present invention, and are not necessarily limited to those including all of the components described. In addition, it is possible to replace a part of the configuration of one embodiment with a component of another embodiment. It is also possible to add a component of another embodiment to the configuration of one embodiment. It is also possible to add, replace, or delete other components to, a part of the configuration of each embodiment.

[0080] In addition, the image area selection device 30 and each camera 3, or the robot control device 20 and each camera 3 may be configured to be able to communicate with each other, and each camera 3 may be provided with a posture change function and a hardware zoom function. For example, when a person operates the robot 10 in the teaching phase, commands such as camera posture and zoom are output from the robot operation device 50 to the camera 3 via the image area selection device 30 or the robot control device 20. This allows the operator to change the posture and zoom the camera 3 in order to display an image that makes it easy to operate the robot 10. The camera image at this time is used as learning data. In addition, in the third embodiment, when automatically generating an operation command for the robot 10, it is not limited to a configuration in which the image obtained by the camera 3 is post-processed (cutting out and enlarging the area) and displayed, but it is also possible to adopt a configuration in which the image area selection device 30 directly gives commands such as zoom to the camera 3 according to the situation of the robot 10 to acquire an image.

[0081] In addition, the above-mentioned configurations, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. As the hardware, a broad processor device such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) may be used.

[0082] Furthermore, each component of the image region selection device 30 according to each of the above-described embodiments may be implemented in the robot control device 20. Furthermore, the processing performed by a certain processing unit of the image region selection device 30 may be realized by a single piece of hardware, or may be realized by distributed processing using multiple pieces of hardware.

[0083] In the flowcharts shown in FIGS. 4, 6, and 10, a plurality of processes may be executed in parallel or the order of the processes may be changed, as long as this does not affect the results of the processes. [Explanation of symbols]

[0084] 10...robot, 20...robot control device, 3, 3a, 3b, 3c...camera, 30...image area selection device, 40...image display device, 50...robot operation device, 31...learning data storage unit, 32...learning unit, 33...learning model storage unit, 33a...learning model, 331...heat map generation unit, 34...inference unit, 60...computer, 61...CPU, 71...input camera image, 72...heat map, 73...image, 74...task completion button, 81, 82, 83...image (input camera image and heat map), 90, 90A...display screen, 91-93, 91A-93A...image, 100...robot control system

Claims

1. A robot that performs the work, A control device that transmits a control command to the robot based on the operation command; A camera for capturing an image of at least a working environment of the robot; an image area selection device that selects at least a partial area of ​​the captured image that is important for the operation of the robot based on the captured image and sensor information acquired from a sensor that detects a state of the robot and / or a state of the environment disposed in the robot and / or the environment; an image display device that displays the area of ​​the captured image selected by the image area selection device, The image region selection device comprises: A learning model that is trained to predict the captured image and the sensor information at a next time and thereafter based on the captured image captured by the camera at a current time and the sensor information; a heat map generating unit that generates a heat map indicating an area in the captured image within the learning model, The learning model is optimized so that the values ​​in the regions necessary for prediction are increased or decreased so that prediction errors are reduced during the learning process. Regions where the values ​​of the heat map are equal to or greater than a set threshold are considered to be important regions. Robot control system.

2. The image region selection device comprises: a model storage unit that stores the learning model for selecting the important area in the captured image; A learning unit that learns the learning model; The captured image and the sensor information are obtained during the operation of the robot, a learning data storage unit that stores time-series learning data used for learning the learning model; an inference unit that infers the important area in the captured image by using the learning model stored in the model storage unit; The robot control system of claim 1 .

3. the inference unit compares the importance of the captured image output by the inference unit of the image region selection device among a plurality of the captured images based on the value of the heat map for each of the captured images; The arrangement of at least the captured images to be displayed on the image display device is changed according to the importance. The robot control system of claim 2 .

4. the inference unit compares the importance of the captured image output by the inference unit of the image region selection device among a plurality of the captured images based on the value of the heat map for each of the captured images; The area of ​​the captured image to be displayed on the image display device is changed according to the importance. The robot control system of claim 2 .

5. The image display device displays on a screen an image obtained by cutting out the area of ​​the captured image selected by the image area selection device. The robot control system of claim 1 .

6. a robot operation device that transmits the operation command to the control device in response to an operation by an operator. The robot control system of claim 1 .

7. A robot that performs the work, A control device that transmits a control command to the robot based on the operation command; A camera for capturing an image of at least a working environment of the robot; an image area selection device that selects at least a partial area of ​​the captured image that is important for a task of the robot based on the captured image and sensor information acquired from a sensor that detects a state of the robot and / or a state of the environment disposed in the robot and / or the environment, The image region selection device comprises: A learning model that is trained to predict the captured image and the sensor information at a next time and thereafter based on the captured image captured by the camera at a current time and the sensor information; a heat map generating unit that generates a heat map indicating an area in the captured image within the learning model, The learning model is optimized so that the value becomes larger or smaller in the region necessary for prediction so that the prediction error is reduced during the learning process. The region where the value of the heat map is equal to or greater than a set threshold is considered to be an important region, The image region selection device generates a prediction value regarding the operation of the robot based on the captured image and the predicted value of the sensor information by the learning model, and outputs the prediction value to the control device as the operation command. Robot control system.

8. A control device that transmits a control command to the robot based on the operation command; A camera for capturing an image of at least a working environment of the robot; an image area selection device that selects at least a partial area of ​​the captured image that is important for the operation of the robot based on the captured image and sensor information acquired from a sensor that detects a state of the robot and / or a state of the environment disposed in the robot and / or the environment; an image display device that displays the area of ​​the captured image selected by the image area selection device, The image region selection device comprises: A learning model that is trained to predict the captured image and the sensor information at a next time and thereafter based on the captured image captured by the camera at a current time and the sensor information; a heat map generating unit that generates a heat map indicating an area in the captured image within the learning model, The learning model is optimized so that the values ​​in the regions necessary for prediction are increased or decreased so that prediction errors are reduced during the learning process. Regions where the values ​​of the heat map are equal to or greater than a set threshold are considered to be important regions. Robot control system.

9. A control device that transmits a control command to the robot based on the operation command; A camera for capturing an image of at least a working environment of the robot; an image area selection device that selects at least a partial area of ​​the captured image that is important for a task of the robot based on the captured image and sensor information acquired from a sensor that detects a state of the robot and / or a state of the environment disposed in the robot and / or the environment, The image region selection device comprises: A learning model that is trained to predict the captured image and the sensor information at a next time and thereafter based on the captured image captured by the camera at a current time and the sensor information; a heat map generating unit that generates a heat map indicating an area in the captured image within the learning model, The learning model is optimized so that the value becomes larger or smaller in the region necessary for prediction so that the prediction error is reduced during the learning process. The region where the value of the heat map is equal to or greater than a set threshold is considered to be an important region, The image region selection device generates a prediction value regarding the operation of the robot based on the captured image and the predicted value of the sensor information by the learning model, and outputs the prediction value to the control device as the operation command. Robot control system.

10. A robot control method using a robot control system including a control device that transmits a control command to a robot based on an operation command, and an image area selection device, comprising: The image region selection device, A process of selecting at least a partial area of ​​the captured image that is important for the work of the robot based on a captured image acquired by a camera that captures at least an image of a working environment of the robot and sensor information acquired from a sensor that detects a state of the robot and / or a state of the environment disposed on the robot and / or the environment; Executing a process of outputting the selected area of ​​the captured image to an image display device; The image region selection device includes a learning model that is trained to predict the captured image and the sensor information at the next time and thereafter from the captured image captured by the camera at the current time and the sensor information, and a heat map generation unit that generates a heat map indicating regions in the captured image within the learning model, and the learning model optimizes the heat map so that values ​​are large or small in regions necessary for prediction so that prediction errors are reduced in the learning process. A region where the value of the heat map is equal to or larger than a set threshold value is selected as an important region. A method for controlling a robot.

11. A program for causing a computer to execute an image area selection device of a robot control system including a control device that transmits a control command to a robot based on an operation command and an image area selection device, the program comprising: a step of selecting at least a partial area of ​​the captured image that is important for the work of the robot based on a captured image acquired by a camera that captures at least an image of a working environment of the robot and sensor information acquired from a sensor disposed on the robot and / or the environment that detects a state of the robot and / or a state of the environment; outputting the selected area of ​​the captured image to an image display device; The image region selection device includes a learning model that is trained to predict the captured image and the sensor information at the next time and thereafter from the captured image captured by the camera at the current time and the sensor information, and a heat map generation unit that generates a heat map indicating regions in the captured image within the learning model, and the learning model optimizes the heat map so that values ​​are large or small in regions necessary for prediction so that prediction errors are reduced in the learning process. A region where the value of the heat map is equal to or larger than a set threshold value is selected as an important region. program.

Images