Control system and control method

Abstract

To provide a control system for a robot device capable of reducing work about teaching of operation of the robot device by appropriately dealing with a work environment in which mutual positional relation among an environment sensor, a robot and a work object changes.SOLUTION: A control system 100 includes a data conversion adjustment part 865 for determining and outputting conversion contents showing that input / output data of an operation pattern generation part 864 becomes the same data when the operation pattern generation part 864 performs learning on the basis of an output of an environment sensor 12. Further, the control system 100 includes an environment information conversion part 866 for converting environment information and an input of a state of a robot device 1 on the basis of the conversion contents determined by the data conversion adjustment part 865, and outputting the converted input to the operation pattern generation part 864, and a control signal conversion part 867 for converting output contents of the operation pattern generation part 864 on the basis of the conversion contents determined by the data conversion adjustment part 865, and outputting the converted output contents to the robot device 1.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a control system for a robotic device that performs a task. [Background technology]

[0002] Robot devices, such as industrial robots, mainly perform repeated operations only at memorized positions, so in order to have the robot perform a task, programming is required to specify positions that correspond to the task content. For this reason, there are cases where it is not possible to flexibly respond to position shifts of the work object or changes in the task content.

[0003] It is known to control a robot device to perform a predetermined task using machine learning techniques including neural networks. In a plurality of task situations where the positions of the work objects are different, a learning model is constructed by machine learning the correspondence between information from an environmental sensor such as a camera and an operation command for the robot device, and the output from the environmental sensor is input to the learning model to generate an operation command to control the robot device, thereby making it possible to make the robot device perform a task even in task situations where the positions of the work objects are different.

[0004] A control system for a robot device using a machine learning learner is described in Patent Document 1. The control system described in Patent Document 1 has an image prediction unit that has been trained by machine learning to predict an image captured when the robot performs an appropriate operation based on an image acquired by a robot camera. This control system generates an operation command so that the current image approaches the predicted image output from the image prediction unit.

[0005] According to the technology described in Patent Document 1, movements are generated using an image prediction unit that has been trained using machine learning for images captured by a robot camera, making it possible to flexibly respond to positional deviations of the work object and generate movements accordingly. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2019-107704 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, a robot device may perform the same task in different positions or under different conditions. In such cases, the same task must be performed in an environment where the positional relationship between the environmental sensor, the robot, and the work object, such as the orientation, has changed.

[0008] The control system described in Patent Document 1 uses a learning model that learns the appropriate robot behavior for camera images through machine learning, but the learning model can only output appropriate inference results in an environment where the relative positions of the image sensor, robot, and work object at the time of machine learning are valid. If the relative positions of the image sensor, robot, and work object, such as their orientation, are different from those at the time of machine learning in the environment, the model cannot output the appropriate robot behavior for the image, and the robot cannot operate appropriately.

[0009] Therefore, it is necessary to conduct machine learning again to adapt the robot device to a different environment, which can be a cumbersome task.

[0010] The present invention aims to provide a control system and control method for a robot device that can appropriately respond to a work environment in which the relative positions of the environmental sensor, robot, and work object change in a control system that uses a learning model developed by machine learning, and that can reduce the work involved in teaching the robot device how to operate. [Means for solving the problem]

[0011] In order to achieve the above object, the present invention is configured as follows.

[0012] a control system including a robot device having a robot arm configured with one or more joints and an end effector; an environmental sensor that acquires environmental information related to a working environment of the robot device including the robot device and a work target handled by the robot device; and a motion pattern generation unit that learns a correspondence relationship between a control pattern of the robot device and the environmental information and a state of the robot device, and outputs a control signal for the robot device corresponding to an input of the environmental information and the state of the robot device, the learning control device further including a data conversion adjustment unit that determines and outputs conversion content that turns input / output data of the motion pattern generation unit into data similar to that learned by the motion pattern generation unit, based on an output of the environmental sensor; an environmental information conversion unit that converts input of the environmental information and the state of the robot device based on the conversion content determined by the data conversion adjustment unit, and outputs the converted data to the motion pattern generation unit; and a control signal conversion unit that converts the output content of the motion pattern generation unit based on the conversion content determined by the data conversion adjustment unit, and outputs the converted data to the robot device. The data conversion adjustment unit estimates the reference coordinates of the work object and the reference coordinates of the robot device in the coordinate system of the environmental sensor, and adjusts the conversion contents of the environmental information conversion unit and the control signal conversion unit based on the difference in position and orientation of the coordinate systems of the work object and the robot device relative to the coordinate system of the environmental sensor.

[0013] a control method for a control system including a robot device having a robot arm and an end effector, an environmental sensor that acquires environmental information related to a work environment of the robot device including the robot device and a work object handled by the robot device, and a learning-type control device having a motion pattern generation unit that learns a correspondence between the environmental information and a control pattern of the robot device for a state of the robot device and outputs a control signal for the robot device corresponding to an input of the environmental information and the state of the robot device, the control method comprising the steps of: acquiring the work environment using the environmental sensor and comparing it with pre-stored work environment reference coordinates of the robot device and the work object to estimate relative coordinates of the acquired work environment; setting a robot state transformation amount based on the estimated relative coordinates and the work environment reference coordinates such that images of the robot device and the work object at the estimated relative coordinates have a positional relationship similar to that of the images of the robot device and the work object at the work environment reference coordinates; and converting a control signal for the robot device based on the robot transformation amount. Effect of the Invention

[0014] According to the present invention, it is possible to provide a control system and a control method for a robot device that uses a learning model developed by machine learning, and that can appropriately respond to a work environment in which the relative positions of the environmental sensor, the robot, and the work object change, thereby reducing the work involved in teaching the robot device to operate. [Brief description of the drawings]

[0015] [Figure 1] 1 is a schematic configuration diagram of a robot device and a control system to which the present invention is applied; [Diagram 2] FIG. 2 is a schematic diagram showing a hardware configuration for executing software in the control system of the present invention. [Diagram 3] FIG. 2 is an external view showing an example of work performed by a robot device. [Figure 4] FIG. 1 is a schematic diagram showing an example of a software configuration when machine learning a learning model in a control system according to a first embodiment of the present invention. [Diagram 5] FIG. 2 is a schematic diagram showing an example of a software configuration when control is executed using a machine-learned learning model in the control system according to the first embodiment of the present invention. [Figure 6] 1 is a flowchart showing a method for performing control using a machine-learned learning model in a control system according to a first embodiment of the present invention. [Figure 7A] FIG. 2 is a schematic diagram showing adjustment contents of data conversion in the control system according to the first embodiment of the present invention. [Figure 7B] FIG. 2 is a schematic diagram showing adjustment contents of data conversion in the control system according to the first embodiment of the present invention. [Figure 7C] FIG. 2 is a schematic diagram showing adjustment contents of data conversion in the control system according to the first embodiment of the present invention. [Figure 8] FIG. 11 is a schematic diagram showing the configuration of a control robot device and a system according to a second embodiment of the present invention. [Figure 9] FIG. 11 is a schematic diagram showing an example of a software configuration when control is executed using a machine-learned learning model in a control system according to a second embodiment of the present invention. [Figure 10A] FIG. 11 is a schematic diagram showing adjustment contents of data conversion in a control system according to a second embodiment of the present invention. [Figure 10B] FIG. 11 is a schematic diagram showing adjustment contents of data conversion in a control system according to a second embodiment of the present invention. [Figure 11] FIG. 11 is a schematic diagram showing an example of a software configuration when control is executed using a machine-learned learning model in a control system according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. EXAMPLES

[0017] Example 1 The configurations of a robot apparatus 1 and a control system 100 for the robot apparatus 1 in the first embodiment will be described with reference to FIGS.

[0018] Fig. 1 is a schematic diagram showing an example of the configuration of a control system 100 for a robot device 1 to which the present invention is applied. In Fig. 1, the control system 100 includes the robot device 1, a camera 12 which is an environmental sensor, and a learning-type control device 8. The robot device 1 has a mobile cart 9, a robot arm 10 attached to the mobile cart 9, and an end effector 11 attached to the tip of the robot arm 10, and performs a task on a work object 2 on a work table 3.

[0019] Each device of the robot device 1 is connected to a robot control device 7 and operates in response to control commands from the robot control device 7 (motor current of the robot arm 10, motor current of the end effector 11, etc.), and transmits the state of the robot device 1 (voltage of an angle sensor attached to a joint of the robot arm 10, etc.) to the robot control device 7.

[0020] The robot controller 7 is connected to the learning controller 8 via the network 6, and converts the state of the robot device 1 obtained from the robot device 1 (joint angles of the robot arm 10, hand position of the end effector 11, etc.) and transmits it to the learning controller 8. The robot controller 7 is configured to calculate a control command for the robot device 1 based on the operation command output by the learning controller 8 (target hand position of the end effector 11, etc.) and the state of the robot device 1 input from the robot device 1.

[0021] In addition, a camera 12 that measures the state of the work object 2 and the robot device 1 on the workbench 3 is connected to the learning control device 8 via a network 6, and transmits the captured images of the work object 2 and the robot device 1 to the learning control device 8.

[0022] 2 is a schematic diagram showing an example of a hardware configuration for executing software in a control system of a robot device 1 according to a first embodiment of the present invention. Note that an interface is indicated as "I / F" in FIG. 2.

[0023] In FIG. 2, the robot control device 7 of the robot device 1 is a computer in which a control device 71, a communication interface 72, a control interface 73, and a storage device 74 are electrically connected to one another.

[0024] The control device 71 includes a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), and the like, and is configured to execute information processing based on programs and various data.

[0025] The storage device 74 is an auxiliary storage device such as a hard disk drive, and stores a control program 741 executed by the control device 71 .

[0026] The control interface 73 is an interface that is connected to the robot arm 10, the end factor 11, and the movable carriage 9 of the robot apparatus 1, and is used to transmit and receive control commands to the robot arm 10, the end factor 11, and the movable carriage 9 of the robot apparatus 1, and data relating to the status of the robot arm 10, the end factor 11, and the movable carriage 9 of the robot apparatus 1, and is configured appropriately according to the equipment that constitutes the robot apparatus 1.

[0027] The communication interface 72 is an interface that connects to the learning control device 8 and is used to send and receive operational commands to the robot device 1 and data regarding the state of the robot device 1, and communicates with the communication interface 82 of the learning control device 8 via the network 6.

[0028] When the robot control device 7 is started by turning on the power or the like, it loads and executes a control program 741 stored in the storage device 74 in the control device 71. The control program 741 generates a control command for the robot device 1 based on an operation command for the learning control device 8 input from the communication interface 72 and a state of the robot arm 10, etc. of the robot device 1 input from the control interface 73, and outputs the control command from the control interface 73 to the robot device 1. In addition, the robot control device 7 outputs the state of the robot arm 10, etc. of the robot device 1 input from the control interface 73 to the learning control device 8 from the communication interface 72.

[0029] The learning control device 8 is a computer in which a control device 81, a communication interface 82, an input device 83, and a storage device 84 are electrically connected to one another. The control device 81 includes a CPU, a RAM, a ROM, etc., and is configured to execute information processing based on programs and various data. The communication interface 82 is an interface that connects to the robot control device 7 and transmits and receives operation commands for the robot device 1 and the state of the robot device 1 to the robot control device 7, and communicates with the communication interface 72 of the robot control device 7 via the network 6.

[0030] In addition, the communication interface 82 communicates with the camera 12 via the network 6. The input device 83 is a device that takes in input from a user, such as a mouse or a keyboard, and controls the execution of the program of the learning control device 8. The storage device 84 is an auxiliary storage device, such as a hard disk drive, and stores a learning program 851, a process execution program 861, an operation command program 862, learning data 852, and model parameter data 863, which are executed by the control device 81.

[0031] When the learning type control device 8 is started by turning on the power, etc., the learning program 851 , the process execution program 861 , and the operation command program 862 stored in the storage device 84 are loaded into the control device 81 .

[0032] When a user instructs the learning program 851 to execute learning via the input device 83, the learning program 851 generates learning data 852 using the state of the robot device 1 input from the communication interface 82, stores the data in the storage device 84, and performs machine learning on a learning model, which will be described later, using the learning data 852 to generate model parameter data 863.

[0033] When a user instructs the robot apparatus 1 to start work via the input device 83, the process execution program 861 determines the timing to execute the operation command program 862 based on the state of the robot apparatus 1 obtained from the communication interface 82, and outputs an instruction to execute the operation program 862.

[0034] When the process execution program 861 instructs the operation command program 862 to execute an operation, the operation command program 862 reads the model parameter data 862, generates an operation command for the robot device 1 using the state of the robot device 1 obtained from the communication interface 82, the image from the camera 12, and a learning model described later, and outputs the operation command from the communication interface 82 to the robot control device 7.

[0035] In addition, the learning control device 8 and the robot control device 7 may be configured with the same hardware, and the control program 741, learning program 851, process execution program 861, and control command program 862 may be configured to be executed on the same hardware. When the learning control device 8 is configured by hardware, the learning program 851 serves as a learning section, the process execution program 861 serves as a process execution section, and the operation command program 862 serves as an operation command section.

[0036] When the motion command program 862 is configured by hardware, the motion pattern generating section 864, the data conversion adjusting section 865, the environmental information converting section 866, and the control signal converting section 867 are also configured by hardware.

[0037] The control interface 73 and the communication interface 72 may be of the same configuration, and the robot control device 7 may be configured to control the robot device 1 via the network 6. The robot control device 7 and the learning control device 8 do not have to be connected via the network 6, and the communication interfaces 72, 82 may be directly connected. A configuration may be adopted in which a plurality of operation command programs 862 are prepared according to the work content, and a configuration may be adopted in which the process execution program 861 determines the progress of the work based on the state of the robot device 1 and the images captured by the camera 12, and selects a program to be executed from the plurality of operation command programs.

[0038] Furthermore, the learning program 851, the process execution program 861, and the control command program 862 do not have to be stored in the same learning control device 8, and different learning control devices 8 may be configured to communicate the learning data 852 and the model parameter data 863 via the network 6.

[0039] Next, in the control system 100 of the robot apparatus 1 according to the first embodiment of the present invention, the learning control device 8 is configured to learn an action in response to environmental information and output the action as a control command to the robot apparatus 1. A method for estimating a positional relationship between a work object and the robot and converting and outputting the control command will be described with reference to Figs. 3 to 7C.

[0040] FIG. 3 is an external view showing an example of a task executed by the robot device 1. As shown in FIG.

[0041] An example of the operation of the robot device 1 will be described with reference to FIG.

[0042] In manufacturing sites where a wide variety of products are produced in small quantities, some tasks (such as assembly work) may be performed in multiple work locations.

[0043] For example, as shown in Fig. 3, a plurality of robot devices 1 may assemble work objects 2 arranged on a plurality of work tables 3 or work objects 2 transported by an automatic guided vehicle 5. In such on-site work, the work content may be changed depending on the fluctuation in demand. For example, a robot device 1 performing assembly work at a certain work table 3 may move to a different work table 3 and perform work depending on a change in the production plan.

[0044] At this time, the robot device 1 may perform the work from a different direction relative to the work table 3 in order to shorten the moving distance or to avoid interference with other robot devices 1. Furthermore, depending on the change in the production plan, the delivery location of the work object 2 transported by the automatic guided vehicle 5 may be changed, and the positional relationship between the robot device 1 and the work object 2 during the assembly work may be changed.

[0045] In order for the robot device 1 to perform work appropriately at the above-mentioned work site, it is necessary to flexibly respond to positional deviations of the work object 2 that occur due to deviations in the stopping position of the robot device 1 or deviations in the part supply position of the automatic guided vehicle 5, and to respond to changes in the positional relationship between the robot device 1 and the work object 2, such as the position and orientation.

[0046] Fig. 4 is a schematic diagram showing the software configuration of the learning program 851 executed by the learning control device 8. A method in which the learning control device 8 learns a method for generating an operation command for the robot device 1 will be described with reference to Fig. 4. Note that the learning program 851 is configured to include a data collection unit 853, a data collection instruction unit 854, and a learning processing unit 855 as software modules when deployed in the control device 81 and executed.

[0047] The data collection unit 853 includes a data collection instruction unit 854 that instructs the collection of learning data 852 based on an instruction from a user via the input device 83, and acquires, via the communication interface 82, the camera image of the camera 12 while the robot device 1 is operating, the robot hand position which is the hand position of the end effector 11, and a robot operation command which is the target hand position of the end effector 11, and stores them as learning data 852. For example, a plurality of hand positions of the robot device 1 are designated in advance in the control program 741 of the robot control device 7, and the robot control device 7 is programmed to move to the designated hand positions at a predetermined time, and data is acquired for each predetermined sampling while the robot device 1 is performing a task using the program, and the time-series data is stored as one piece of learning data.

[0048] The data collection unit 853 then collects a plurality of pieces of learning data while a plurality of tasks are being performed under different task conditions, such as by slightly changing the position of the task target 2.

[0049] The learning processing unit 855 performs machine learning on the learning model 856 using the learning data 852 collected by the data collecting unit 853. The learning processing unit 855 includes a learning model 856, which is a neural network that receives the camera image and the robot hand position of the learning data 852 as input and outputs an estimated result of a robot operation command, and a parameter update amount calculation unit 857 that compares the operation command of the learning data 852 with the estimated operation command of the learning model 856 and calculates the update amount of the model parameters of the learning model 856 (weights such as the degree of connection between neurons of the neural network and the firing threshold of neurons) so that the contents of the learning data 852 and the output of the learning model 856 match.

[0050] Then, the learning processing unit 855 continues to update the model parameters of the learning model 856 according to the update amount calculated by the parameter update amount calculation unit 857 until a predetermined condition is reached, and when the predetermined condition is reached, it saves the model parameters of the learning model 856 at that time as model parameter data 863.

[0051] Note that the neural network of learning model 856 shown in FIG. 4 is illustrated as a simple Multi-Layer Perceptron (MLP), but a different network may be used, for example, a configuration using a Convolutional Neural Network (CNN) or a Recurrent Neural Network (RNN), or further a configuration combining these networks.

[0052] Fig. 5 is a schematic diagram showing the software configuration of the action command program 862 executed by the learning control device 8. Fig. 6 is a flowchart showing a method in which the action pattern generation unit 864, the data conversion adjustment unit 855, the environmental information conversion unit 866, and the control signal conversion unit 867 of the action command program 862 output action commands to the robot device 1 and generate actions for the robot device 1.

[0053] 7A, 7B and 7C are schematic diagrams showing how the data conversion adjustment unit 865 sets the data conversion contents, and how the environmental information conversion unit 866 and the control signal conversion unit 867 convert the environmental information and the robot operation command.

[0054] Fig. 7A shows the positional relationship between the robot device 1, work objects 2a and 2b, and the camera 12 when the model parameter data 863 is learned. Fig. 7B and Fig. 7C show the positional relationship between the robot device 1, work objects 2a and 2b, and the camera 12 that is different from when the model parameter data 863 is learned.

[0055] A method in which the learning control device 8 estimates the positional relationship between the work objects 2a and 2b and the robot device 1, and converts and outputs a control command will be described with reference to Figs. 5 to 7C. All steps in Fig. 6 are executed by the learning control device 8 at every predetermined sampling. When the operation command program 862 receives an operation execution instruction from the process execution program 861, the operation command program 861 stored in the storage device 84 is deployed and executed in the control device 81, thereby providing an operation pattern generation unit 864, a data conversion adjustment unit 865, an environmental information conversion unit 866, and a control signal conversion unit 867 as software modules. The operation in the positional relationship shown in Figs. 7A, 7B, and 7C shows an operation of gripping and inserting the work object 2b into a recess of the work object 2a as an example for explanation.

[0056] The operation command program 862 is composed of a data conversion adjustment unit 865 that sets changes to environmental information related to the camera image and the robot hand position, and changes to robot operation commands based on the camera image from the camera 12 input via the communications interface 82, an environmental information conversion unit 866 that receives the camera image and the robot hand position as input and changes the contents related to the position and orientation of the two pieces of input information, a movement pattern generation unit 864 that has a learning model 856 reflecting the model parameter data 863, receives the camera image and the robot hand position converted by the environmental information conversion unit 866 as input, and outputs a robot operation command, and a control signal conversion unit 867 that changes the contents related to the position and orientation of the robot operation command of the movement pattern generation unit 864. When an operation execution instruction is input from the process execution program 861 and the program is executed, the robot operation command output from the control signal conversion unit 867 is output to the robot control device 7 via the communications interface 82.

[0057] When the operation command program 862 is started by the process execution program 861, the data conversion adjustment unit 865 acquires a camera image in step S100, and then estimates the relative coordinates (work object coordinates) of the work objects 2a, 2b with respect to the reference coordinates of the camera 12 in step S101.

[0058] As shown in Figs. 7A, 7B and 7C, the work object coordinates are defined in advance as the x-axis pointing in the direction of the recess of the work object 2a, and the y-axis pointing from there in the direction in which the work object 2b is placed.

[0059] The reference state is defined as the state when the model parameter data 863 shown in FIG. 7A is learned, in which the orientations of the camera reference coordinates and the work object coordinates are the same and the origin of the coordinates of the work object 2a is on the x-axis of the camera coordinates.

[0060] The work object coordinates in the state shown in Fig. 7B are estimated to be relative coordinates obtained by moving the work object coordinates in the positive x direction and negative y direction of the camera reference coordinates with respect to the reference state, and then rotating them by 90 degrees. The work object coordinates in the state shown in Fig. 7C are estimated to be relative coordinates inverted around the x axis of the work object coordinates with respect to the reference state, because the position of work object 2b relative to work object 2a is different from the position shown in Fig. 7A.

[0061] The estimation of the relative coordinates of the work objects 2a, 2b may be configured to recognize the positions and shapes of the work objects 2a, 2b using template matching, which finds points in an image that match a template image, and estimate the work object coordinates based on the recognition results, or an object detection method based on R-CNN (Regions with Convolutional Neural Networks) may be used to recognize the positions and shapes of the work objects 2a, 2b and estimate the work object coordinates.

[0062] In step S102, the relative coordinates (robot coordinates) of the robot device 1 with respect to the coordinates of the work object are estimated. The coordinates of the robot device 1 use a unique coordinate definition (coordinates centered on the connection part between the robot arm 10 and the movable carriage 9) used by the robot control device 7 for controlling the robot device 1.

[0063] Then, as shown in FIG. 7A, the state in which the robot coordinates have moved in the positive direction of the x-axis and the negative direction of the y-axis of the work object coordinates and then rotated 90 degrees is defined as the reference state.

[0064] The robot coordinates in the state shown in FIG. 7B are estimated to be relative coordinates obtained by moving in the positive direction of the x-axis and the positive direction of the y-axis of the work object coordinate system with respect to the reference state, and then rotating by −90 degrees.

[0065] 7C, the robot coordinates are estimated as relative coordinates inverted around the y-axis of the robot coordinates after moving in the positive direction of the x-axis and the positive direction of the y-axis of the work object coordinates. Note that the robot coordinates may be calculated by calculating global coordinates for the entire work site using the self-position estimation result based on the amount of movement of the mobile carriage 9 of the robot device 1, and calculating from the difference between the coordinates of the work object in the global coordinates.

[0066] In step S103, the amount of image conversion in the environment information conversion unit 866 is set.

[0067] In the state shown in Fig. 7B, in order for the captured images of the work objects 2a and 2b to be similar to the reference state shown in Fig. 7A, the images are moved in the negative direction of the x-axis and the positive direction of the y-axis of the camera reference coordinates, and then rotated by -90 degrees to obtain an image similar to the reference state. For this reason, the transformation amount is set to a movement in the negative direction of the x-axis and the positive direction of the y-axis of the camera reference coordinates, and a rotation of -90 degrees.

[0068] In the state shown in Figure 7C, in order for the imaging results of the work objects 2a and 2b to be similar to the reference state shown in Figure 7A, if the image is inverted around the x-axis of the camera reference coordinates, the image will be similar to the reference state, so the transformation amount is set to inversion around the x-axis of the camera reference coordinates.

[0069] In step S104, the amount of conversion of the robot state in the environment information conversion unit 866 is set.

[0070] In the state shown in Figure 7B, in order for the relative position between the work object coordinates and the robot coordinates to be the same as the reference state shown in Figure 7A, the robot coordinates must be rotated 90 degrees on the robot coordinates and then moved in the negative direction on the x-axis and the positive direction on the y-axis to achieve a positional relationship similar to that of the reference state; therefore, the transformation amount is set to a rotation of 90 degrees on the robot coordinates and movement in the negative direction on the x-axis and the positive direction on the y-axis.

[0071] In the state shown in Figure 7C, in order for the relative position of the coordinates of the work object and the robot coordinates to be the same as the reference state shown in Figure 7A, the origin of the robot coordinates must be inverted around the y axis in the robot coordinate system and moved in the negative direction of the x axis and in the positive direction of the y axis in the robot coordinate system, which will result in a positional relationship similar to that of the reference state. Therefore, the transformation amount is set to inversion of the robot coordinate system around the y axis and movement in the negative direction of the x axis and in the positive direction of the y axis in the robot coordinate system.

[0072] In step S105, a change amount of the control command is set in the control signal change unit 867. The change amount of the control command is set so as to result in a transformation opposite to the transformation of the robot state set in step S104.

[0073] Next, in step S106, a motion command is output to the robot apparatus 1 to move to an initial position of the motion. The initial position of the motion is the starting point of the motion when the model parameter data 863 is learned, and is set in advance as a positive position on the x-axis of the work object coordinate system, as shown in Fig. 7A.

[0074] Then, as shown in Figures 7B and 7C, an initial position in the work object coordinates estimated in step S101 is set, and a position converted from the set initial position using the conversion amount set in step S105 is set as a target position, and an operation command is output to the robot device 1 so that the robot device 1 moves to the target position.

[0075] The operation command in step S106 is an operation command for a general industrial robot, and may be a Point to Point (PtP) operation command that moves the end effector 11 to a specified position and posture, or may be another operation command using a general trajectory planning method.

[0076] In step S107, the environment information conversion unit 866 acquires the camera image and the robot hand position, which are the environment information of the robot apparatus 1, and then in step S108, applies the conversion amount set by the data conversion adjustment unit 865 in steps S103 and S104 to the acquired data and outputs it. The environment information conversion unit 866 performs conversion to rotate or invert the output data of the camera 12, which is an environment sensor, based on the coordinate system of the camera 12.

[0077] The image transformation may be performed by image processing using affine transformation, and the robot hand position may be transformed by transforming coordinates using a simultaneous transformation matrix.

[0078] The motion pattern generation unit 864 inputs the environmental information changed by the environmental information change unit 866 in step S108 into the learning model 856 to which the model parameter data 863 has been applied, and estimates, generates, and outputs a motion command for the robot apparatus 1 in step S109.

[0079] In step S110, the control signal conversion unit 867 applies the conversion amount set by the data conversion adjustment unit 865 in step S105 to the operation command output from the operation pattern generation unit 864, and outputs it. The data conversion adjustment unit 865 estimates the reference coordinates of the work targets 2a, 2b and the reference coordinates of the robot apparatus 1 in the coordinate system of the camera 12, which is the environmental sensor, and adjusts the conversion contents of the environmental information conversion unit 866 and the control signal conversion unit 867 based on the difference in position and posture of the coordinate system of the work targets 2a, 2b and the robot apparatus 1 relative to the coordinate system of the camera 12, which is the environmental sensor.

[0080] In step S111, the operation command program 862 outputs the operation command output by the control signal conversion unit 867 to the robot control device 7 via the communication interface 82, and in the following step S112, it is determined whether the operation has been completed.

[0081] If the operation is not completed, the process returns to before step S107, and the operation command is repeatedly output.

[0082] Then, in step S112, if the operation is completed, the process ends. The environmental information conversion unit 866 converts the output data of the camera 12, which is an environmental sensor, by rotating or inverting it based on the coordinate system of the camera 12, and the control signal conversion unit 867 converts the control signal to the robot device 1 by rotating, translating, or inverting it based on the coordinate system of the robot device 1.

[0083] The completion of an action may be determined when a preset time has elapsed, or may be configured to determine that an action has ended when the action has moved to a specified position, or may be determined that an action has ended based on other conditions.

[0084] For the sake of simplicity, in Fig. 7A, Fig. 7B and Fig. 7C, conversions related to rotation and movement in two-dimensional plane coordinates are used, but conversions based on three-dimensional space coordinates may be used. In addition, the coordinate estimation and initial position setting of the work target object in steps S101 and S106 are set in advance, but they do not need to be limited to fixed settings. When the data collection unit 853 generates the learning data 852, the coordinate definitions and initial positions of the work targets 2a and 2b may be defined, and the coordinate definitions and initial position definitions may be included in the model parameter data 863. The data conversion adjustment unit 865 may be configured to read the model parameter data 863 including the coordinate definitions and initial position definitions, and to set the conversion amount using the defined coordinate definitions and initial position.

[0085] In this way, according to the first embodiment of the present invention, the positional relationship between the robot apparatus 1, the work objects 2a, 2b, and the camera 12, which is an image sensor, is determined, and the environmental information and control signals are converted so that the positional relationship is the same as when the learning model 856 was learned, thereby eliminating the need for re-learning to adapt to the environment and reducing the work involved in teaching the robot apparatus 1 to operate.

[0086] In other words, in a control system that uses a learning model developed by machine learning, it is possible to provide a control system and control method for a robot device that can appropriately respond to a work environment in which the relative positions of the environmental sensor, robot, and work object change, thereby reducing the work involved in teaching the robot device how to operate.

[0087] Furthermore, when the learning model 856 is to learn operations in significantly different work conditions, such as when the relative positions of the robot device 1, the work objects 2a, 2b, and the image sensor 12 change, the operation accuracy may decrease because the conditions are diverse and the model cannot be optimized appropriately. However, by applying Example 1 of the present invention, the learning model 856 does not need to respond to changes in the work situation, and the operation accuracy can be improved.

[0088] The control method of the control system according to the first embodiment will be described below.

[0089] A control method for a control system including a robot device 1 having a robot arm 10 and an end effector 11, a camera (environment sensor) 12 for acquiring environmental information on a work environment of the robot device 1 including the robot device 1 and work objects 2a, 2b handled by the robot device 1, and a learning-type control device 8 having a motion pattern generation unit 864 for learning a correspondence between the environmental information and a control pattern of the robot device 1 for a state of the robot device 1 and outputting a control signal for the robot device 1 corresponding to an input of the environmental information and the state of the robot device 1, the control method acquiring the work environment using the camera (environment sensor) 12 and estimating relative coordinates of the acquired work environment by comparing the estimated relative coordinates with prestored work environment reference coordinates of the robot device 1 and the work objects 2a, 2b, and based on the estimated relative coordinates and the work environment reference coordinates, setting a robot state transformation amount for making images of the robot device 1 and the work objects 2a, 2b in the estimated relative coordinates have a positional relationship similar to that of the images of the robot device 1 and the work objects 2a, 2b in the work environment reference coordinates, and transforming a control command for the robot device 1 based on the robot transformation amount.

[0090] Example 2 Next, as a control system 200 for a robot apparatus 1 according to a second embodiment of the present invention, an example in which a learning control device 8 estimates positional relationships between work targets 2a, 2b and a plurality of robot apparatuses 1a, 1b, converts and outputs a control command, and selects a robot apparatus 1a or 1b appropriate for a task will be described with reference to Figs. 8 to 10.

[0091] FIG. 8 is a schematic diagram showing a configuration example of a control system 200 of the robot devices 1a and 1b to which the second embodiment of the present invention is applied.

[0092] In FIG. 8, a control system 200 has a robot device 1a (equipped with a robot arm 10a and an end effector 11a) and a different robot device 1b (equipped with a robot arm 10b and an end effector 11ba), and the two robot devices 1a and 1b perform work on a work object 2 (2a, 2b) on a single work table 3.

[0093] The robot controllers 7 a and 7 b of the robot apparatus 1 a and the robot apparatus 1 b are connected to a learning controller 8 via a network 6 .

[0094] FIG. 9 is a schematic diagram showing the software configuration of an operation command program 862 executed by the learning control device 8 of the different robot apparatuses 1a and 1b to which the second embodiment of the present invention is applied.

[0095] 9, the operation command program 862 is configured to further include a control command transmission destination setting unit 868 that switches (selects) the output destination of an operation command output by a control signal conversion unit 867 to either the robot device 1a or the robot device 1b in response to an instruction from a data conversion adjustment unit 865. In other words, the data conversion adjustment unit 865 selects a robot device to be used for work from the multiple robot devices 1a, 1b based on the positional relationship between the work targets 2a, 2b and the multiple robot devices 1a, 1b.

[0096] 10A and 10B are schematic diagrams showing a method in which the data conversion adjustment unit 865 and the control command destination setting unit 868 of the different robot devices 1a and 1b to which the second embodiment of the present invention is applied switch the output destination of the operation command.

[0097] Like Fig. 7A, Fig. 10A shows the positional relationship between the robot device 1, work targets 2a and 2b, and the camera 12 when the model parameter data 863 is learned. Fig. 10B shows the positional relationship between the robot devices 1a and 1b, work targets 2a and 2b, and the camera 12. The method of generating the operations of the robot devices 1a and 1b is the same as the flowchart shown in Fig. 6, except for steps S102 and S111 described below.

[0098] In step S102 of FIG. 6, the data conversion adjustment unit 865 estimates the relative coordinates of the robot device 1a (robot a coordinates) and the relative coordinates of the robot device 1b (robot b coordinates) with respect to the work targets 2a and 2b, and further calculates the amount of change in each of the relative coordinates.

[0099] As shown in Fig. 10B, the robot a coordinates are in a state where there is no movement or rotation relative to the work object coordinates in the reference state shown in Fig. 10A, while the robot b coordinates are estimated to be relative coordinates that have moved in the positive x direction and positive y direction of the work object coordinates with respect to the reference state and then rotated -90 degrees, as in Fig. 7B. Thus, when performing work using the robot device 1a, it is sufficient to simply perform a process of rotating the image in the environment information conversion unit 866, but when performing work using the robot device 1b, it is necessary to rotate the image, input the robot state, and convert the operation command.

[0100] At this time, when converting the input of the robot state and the operation command, there is a possibility that the robot arm 10 will output an operation target that cannot be realized, and the task may not be completed, so it is desirable to keep the conversion amount of the input of the robot state and the operation command as small as possible.

[0101] The data conversion adjustment section 865 calculates the amount of change in each relative coordinate, and outputs the discrimination result of the robot device 1 a or 1 b having the smaller amount of change to the control command transmission destination setting section 868 .

[0102] In step S111 of FIG. 6, the control command destination setting unit 868 changes the destination of the communication interface 82 so that the control command is output to the robot apparatus 1a.

[0103] According to the second embodiment, in addition to being able to obtain the same effects as those of the first embodiment, in the case where two robot devices 1a and 1b perform work on a work target 2 (2a, 2b) on one workbench 3, the conversion amounts of environmental information and control signals for the multiple robot devices 1a, 1b are estimated and the robot devices 1a, 1b with the smallest conversion amounts are selected, thereby avoiding the output of an operation command that would put the robot arm 10 in an unachievable posture, and thus improving the success rate of the work.

[0104] Example 3 Next, a third embodiment of the present invention will be described.

[0105] Third Embodiment As a control system for a robot apparatus 1 according to a third embodiment of the present invention, an example in which an appropriate conversion amount is estimated from the conversion results of a plurality of environment information conversion units 866a, 866b,... and control signal conversion units 867a, 867b,... will be described with reference to FIG.

[0106] FIG. 11 is a schematic diagram showing the software configuration of an operation command program 862 executed by the learning control device 8 of a different robot device to which the third embodiment of the present invention is applied.

[0107] 11, the motion command program 862 includes a plurality of environment information conversion units 866a, 866b..., a plurality of control signal conversion units 867a, 867b..., and a plurality of motion pattern generation units 864a, 864b..., and is configured to input converted images from the plurality of environment information conversion units 866a, 866b..., and predicted images from the plurality of motion pattern generation units 864a, 864b..., to a data conversion adjustment unit 865. The converted images from the plurality of environment information conversion units 866a, 866b..., are output to the plurality of motion pattern generation units 864a, 864b..., and the predicted images from the plurality of motion pattern generation units 864a, 864b..., are output to the plurality of control signal conversion units 867a, 867b....

[0108] The system is further configured to include a control signal selection unit 869 that selects one control signal from the output results of the multiple control signal conversion units 867 based on the output result of the data conversion adjustment unit 865. The control signal selection unit 869 outputs the selected control signal to the communication interface 82 as a robot operation command.

[0109] The learning model 856 inside the motion pattern generating units 864a, 864b, . . . is configured to further output a predicted image that predicts (reproduces) the input camera image, together with the estimation result of the control signal.

[0110] At this time, the learning model 856 reproduces the input camera image with high accuracy if the input image is the same as the image at the time of learning, but if an image of an environment different from that at the time of learning is input, it attempts to reproduce the image at the time of learning and outputs an image different from the input camera image.

[0111] In other words, by calculating the difference between the input image and the predicted image, it is possible to determine how close the current environment is to the environment during learning.

[0112] The data conversion adjustment unit 865 calculates the difference between an image converted by multiple environmental information conversion units 866a, 866b, etc., and a combination (pair) of multiple predicted images corresponding to the converted images, determines the combination with the smallest difference, and outputs an instruction to change the selection of the control signal selection unit 869 so that the conversion content of the determined environmental information becomes the result.

[0113] For simplicity of explanation, the configuration is such that a combination with the smallest difference between a transformed image and a predicted image is selected from among a plurality of environment information conversion units 866a, 866b, etc., for which the conversion amounts have been set in advance. However, it is clear that the settings do not have to be predetermined, and a configuration in which calculations are repeated while changing the setting values ​​to search for setting values ​​that minimize the difference between a transformed image and a predicted image is also possible.

[0114] In this way, according to the third embodiment, it is possible to obtain the same effect as in the first embodiment. In addition, by configuring the conversion content to be selected using the prediction result of the learning model, it is possible to apply the conversion content that is more suitable for the learning model, thereby improving the operation accuracy of the robot device 1.

[0115] The present invention is not limited to the above-described embodiment, but includes various modifications.

[0116] For example, the robot arm 11 used in the explanation of the first to third embodiments is illustrated as a vertical articulated robot, but it may be a Cartesian coordinate robot, a horizontal articulated robot, a parallel link robot, etc. It is clear that a device other than the camera 12 may be used as a device (environmental sensor) for measuring the working status of the robot apparatus 1, and that it may be configured to measure the environmental conditions by measuring the shapes of the work target 2 and surrounding objects using, for example, a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) which is a range finder using a laser.

[0117] In the second embodiment, the number of robot devices is not limited to two, ie, 1a and 1b, but may be more than two.

[0118] In addition, the movement pattern generation unit 864 outputs the target hand position as a movement command based on the image captured by the camera 12 and the information on the hand position of the robot arm 10, but it may also output a target joint angle as a movement command based on the joint angle of the robot arm 10.

[0119] Also, it may be configured to operate at a fixed sampling period, and to output the relative movement amount of the hand position (the movement amount for one sampling) as a movement command.

[0120] Furthermore, it is clear that the learning data 852 and the model parameter data 862 may be configured to be shared with learning control devices 8 at other sites via the network 6. [Explanation of symbols]

[0121] 1, 1a, 1b... robot device, 2, 2a, 2b... work object, 3... work table, 6... network, 7... robot control device, 8... learning type control device, 9... mobile cart, 10, 10a, 10b... robot arm, 11, 11a, 11b... end effector, 12... camera, 71, 81... control device, 72, 82... communication interface, 73... control interface, 74, 84... storage device, 83... input device, 100, 200... control system system, 851... learning program, 852... learning data, 853... data collection unit, 854... data recording instruction unit, 855... learning processing unit, 856... learning model, 857... parameter update amount calculation unit, 861... process execution program, 862... operation command program, 863... model parameter data, 864... operation pattern generation unit, 865... data conversion adjustment unit, 866... ​​environmental information conversion unit, 867... control signal conversion unit, 869... control signal selection unit

Claims

1. A robotic device having a robot arm configured with one or more joints and an end effector; an environmental sensor for acquiring environmental information regarding a working environment of the robotic device, the environmental information including the robotic device and a work target object handled by the robotic device; a motion pattern generating unit that learns a correspondence relationship between the environmental information and a control pattern of the robot apparatus relative to a state of the robot apparatus, and outputs a control signal for the robot apparatus corresponding to an input of the environmental information and the state of the robot apparatus; and Equipped with The learning control device further comprises: a data conversion adjustment unit that determines and outputs conversion contents that make the input / output data of the motion pattern generation unit similar to the data learned by the motion pattern generation unit based on an output of the environmental sensor; an environmental information conversion unit that converts the input of the environmental information and the state of the robot apparatus based on the conversion content determined by the data conversion adjustment unit and outputs the converted information to the motion pattern generation unit; a control signal conversion unit that converts the output content of the motion pattern generation unit based on the conversion content determined by the data conversion adjustment unit and outputs the converted content to the robot device; Equipped with a control system characterized in that the data conversion adjustment unit estimates reference coordinates of the work object and reference coordinates of the robot device in the coordinate system of the environmental sensor, and adjusts the conversion contents of the environmental information conversion unit and the control signal conversion unit based on differences in position and orientation of the coordinate systems of the work object and the robot device relative to the coordinate system of the environmental sensor.

2. A robotic device having a robot arm composed of one or more joints and an end effector; an environmental sensor for acquiring environmental information regarding a working environment of the robotic device, the environmental information including the robotic device and a work target object handled by the robotic device; a motion pattern generating unit that learns a correspondence relationship between the environmental information and a control pattern of the robot apparatus relative to a state of the robot apparatus, and outputs a control signal for the robot apparatus corresponding to an input of the environmental information and the state of the robot apparatus; and Equipped with The learning control device further comprises: a data conversion adjustment unit that determines and outputs conversion contents that make the input / output data of the motion pattern generation unit similar to the data learned by the motion pattern generation unit based on an output of the environmental sensor; an environmental information conversion unit that converts the input of the environmental information and the state of the robot apparatus based on the conversion content determined by the data conversion adjustment unit and outputs the converted information to the motion pattern generation unit; a control signal conversion unit that converts the output content of the motion pattern generation unit based on the conversion content determined by the data conversion adjustment unit and outputs the converted content to the robot device; Equipped with the environmental information conversion unit converts the output data of the environmental sensor by rotating or inverting the output data with respect to a coordinate system of the environmental sensor; A control system according to claim 1, wherein the control signal conversion unit converts a control signal for the robot device by rotating, translating, or inverting the control signal with respect to a coordinate system of the robot device.

3. A robotic device having a robot arm composed of one or more joints and an end effector; an environmental sensor for acquiring environmental information regarding a working environment of the robotic device, the environmental information including the robotic device and a work target object handled by the robotic device; a motion pattern generating unit that learns a correspondence relationship between the environmental information and a control pattern of the robot apparatus relative to a state of the robot apparatus, and outputs a control signal for the robot apparatus corresponding to an input of the environmental information and the state of the robot apparatus; and Equipped with The learning control device further comprises: a data conversion adjustment unit that determines and outputs conversion contents that make the input / output data of the motion pattern generation unit similar to the data learned by the motion pattern generation unit based on an output of the environmental sensor; an environmental information conversion unit that converts the input of the environmental information and the state of the robot apparatus based on the conversion content determined by the data conversion adjustment unit and outputs the converted information to the motion pattern generation unit; a control signal conversion unit that converts the output content of the motion pattern generation unit based on the conversion content determined by the data conversion adjustment unit and outputs the converted content to the robot device; Equipped with Two or more of the robotic devices, the data conversion adjustment unit further comprises a control command destination setting unit that selects the robot device to be used for work from among the plurality of robot devices based on a positional relationship between the work target object and the plurality of robot devices.

4. A robotic device having a robot arm composed of one or more joints and an end effector; an environmental sensor for acquiring environmental information regarding a working environment of the robotic device, the environmental information including the robotic device and a work target object handled by the robotic device; a motion pattern generating unit that learns a correspondence relationship between the environmental information and a control pattern of the robot apparatus relative to a state of the robot apparatus, and outputs a control signal for the robot apparatus corresponding to an input of the environmental information and the state of the robot apparatus; and Equipped with The learning control device further comprises: a data conversion adjustment unit that determines and outputs conversion contents that make the input / output data of the motion pattern generation unit similar to the data learned by the motion pattern generation unit based on an output of the environmental sensor; an environmental information conversion unit that converts the input of the environmental information and the state of the robot apparatus based on the conversion content determined by the data conversion adjustment unit and outputs the converted information to the motion pattern generation unit; a control signal conversion unit that converts the output content of the motion pattern generation unit based on the conversion content determined by the data conversion adjustment unit and outputs the converted content to the robot device; Equipped with the motion pattern generation unit further outputs a control signal for the robot apparatus corresponding to the input of the environmental information and a state of the robot apparatus, and prediction information for the environmental information; The learning control device has a plurality of environment information conversion units and a control signal conversion unit each having a different conversion content, The control system is characterized in that the data conversion adjustment unit includes a control signal selection unit that compares the prediction information of the multiple environmental information of the operation pattern generation unit obtained via the multiple environmental information conversion units with the environmental information output by the environmental sensor, and selects the conversion content of the environmental information conversion unit that most closely matches.

5. A control method for a control system including a robot device having a robot arm and an end effector, an environmental sensor that acquires environmental information related to a work environment of the robot device including the robot device and a work target handled by the robot device, and a learning-type control device having a motion pattern generation unit that learns a correspondence relationship between a control pattern of the robot device and the environmental information and a state of the robot device, and outputs a control signal for the robot device corresponding to an input of the environmental information and the state of the robot device, acquiring the work environment using the environmental sensor, and comparing the acquired work environment with pre-stored work environment reference coordinates of the robot device and the work target object to estimate relative coordinates of the work environment; setting a robot state transformation amount based on the estimated relative coordinates and the work environment reference coordinates such that the images of the robot apparatus and the work object in the estimated relative coordinates have a similar positional relationship to the images of the robot apparatus and the work object in the work environment reference coordinates; A control method for a control system, comprising converting a control signal for the robot device based on the robot state conversion amount.

6. 6. The control method for a control system according to claim 5, A control method for a control system, comprising: estimating reference coordinates of the work object and reference coordinates of the robot device in a coordinate system of the environmental sensor; and adjusting the conversion content of the control signal of the robot device based on the difference in position and orientation of the coordinate system of the work object and the robot device relative to the coordinate system of the environmental sensor.

7. 6. The control method for a control system according to claim 5, converting the output data of the environmental sensor by rotating or inverting the output data with respect to a coordinate system of the environmental sensor; A control method for a control system, comprising converting a control signal for the robot device so as to rotate, translate, or invert the control signal with respect to a coordinate system of the robot device.

8. 6. The control method for a control system according to claim 5, Two or more of the robotic devices, a control method for a control system, comprising the steps of: selecting a robot device to be used for a task from among a plurality of robot devices based on a positional relationship between the work object and the plurality of robot devices;

9. 6. The control method for a control system according to claim 5, outputting a control signal for the robot apparatus corresponding to the environmental information and a state of the robot apparatus and prediction information for the environmental information; A control method for a control system, characterized by comparing predicted information of multiple pieces of environmental information with the environmental information output by the environmental sensor, and adjusting the conversion content so as to select the conversion content of the environmental information that most closely matches.

Images