Control device, control method, and program

Abstract

To enable a robot device to be controlled more properly.SOLUTION: A control device comprises: an acquisition unit that acquires measurement information obtained using a sensor for performing measurement related to an operation state of a robot device; and a control unit that controls operation of the robot device on the basis of the measurement information. The sensor includes a visual sensor and a force sensor. The control unit dynamically or stepwise changes a control ratio between visual control, which is the control based on visual measurement information obtained using the visual sensor, and force sense control, which is the control based on force sense measurement information obtained using the force sensor, in accordance with a status of work in which the robot device is used.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to a control device, a control method, and a program for controlling a robot device. [Background technology]

[0002] Conventionally, control devices for controlling robot devices have been widely used. In this method, a visual sensor and a force sensor are used to measure the operating state of the robot device, There are known techniques for controlling the operating state of a robot device to a target state (for example, (See Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7295344 Summary of the Invention [Problem to be solved by the invention]

[0004] Visual control is the control of a robot device using visual measurement information obtained using a visual sensor. Force control is the control of a robot device using force measurement information obtained using a force sensor. By utilizing the advantages of each of these, we aim to achieve both visual control and force control to control the movement of the robot device. It is desirable to control it appropriately.

[0005] The present disclosure provides a control device, a control method, and a control method for controlling a robot device more appropriately. and programs. [Means for solving the problem]

[0006] The control device according to the first aspect of the present disclosure includes: an acquisition unit that acquires measurement information obtained using a sensor for the measurement; a control unit that controls the operation of the robot device, and the sensors include a visual sensor and a force sensor. a visual sensor, and the control unit controls the The control is based on visual control, and the control is based on force measurement information obtained using the force sensor. The control ratio of a certain force sense control and a certain force sense control is dynamically adjusted depending on the situation of the work using the robot device. will be changed in stages.

[0007] A control method according to a second aspect of the present disclosure includes: and acquiring measurement information obtained using a sensor for the purpose of and controlling the operation of the robot device, wherein the sensors include a visual sensor and a force sensor. and performing the control based on visual measurement information obtained using the visual sensor. The control is a visual control, and the control is based on force measurement information obtained using the force sensor. The control ratio of the force sense control and the force sense control is dynamically adjusted according to the situation of the work using the robot device. This includes changing the system or changing it step by step.

[0008] A program according to a third aspect of the present disclosure includes a program for causing a control device to perform a program for controlling a robot device. acquiring measurement information obtained using a sensor for performing measurements; and controlling the operation of the robot device based on the visual information. and a force sensor, and the control is obtained using the visual sensor. Visual control is the control based on visual measurement information, and force measurement information obtained using the force sensor is The control ratio of the force feedback control, which is the control based on the constant information, and the control ratio of the force feedback control, which is the control based on the constant information, is set to This includes dynamically or gradually changing the settings depending on the situation. [Effects of the Invention]

[0009] According to one aspect of the present disclosure, there is provided a control device that enables more appropriate control of a robot device. It is possible to provide a device, a control method, and a program. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of a system configuration of a control system including a control device according to an embodiment. [Figure 2] 2 is a block diagram showing a functional block configuration of the control device according to the first embodiment. FIG. [Figure 3] FIG. 3 is a diagram for explaining an example of data collection by the control device according to the first embodiment. [Figure 4] FIG. 3 is a diagram for explaining an example of data collection by the control device according to the first embodiment. [Figure 5] FIG. 4 is a flow chart for explaining an example of data collection by the control device according to the first embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of a setting library stored in a control device according to the embodiment. [Figure 7] FIG. 10 is a block diagram showing a functional block configuration of a control device according to a second embodiment. [Figure 8] FIG. 10 is a flow chart for explaining an example of a control flow by a control device according to a second embodiment. [Figure 9] 10A and 10B are diagrams for explaining a specific example of control during the operation of "mounting components on a board" as control by the control device according to the second embodiment. [Figure 10] FIG. 10 is a block diagram showing a functional block configuration of a control device according to a third embodiment. [Figure 11]10A and 10B are diagrams for explaining a specific example of control during the operation of "mounting components on a board" as control by the control device according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] The embodiments will be described with reference to the drawings. The same or similar symbols are used for the parts.

[0012] (1) Overview of the embodiment The control device according to the embodiment includes a sensor for measuring the operating state of the robot device. an acquisition unit that acquires measurement information obtained using the and a control unit that controls the operation of the device, and the sensors include a visual sensor and a force sensor. The control unit performs the control based on visual measurement information obtained using the visual sensor. Visual control and force control, which is the control based on force measurement information obtained using the force sensor. The control ratio of the control and the control of the robot device is dynamically or stepwise adjusted according to the work situation using the robot device. Change.

[0013] In this way, the control ratio between visual control and force control can be adjusted dynamically or stepwise depending on the work situation. By changing to the visual control, the advantages of the visual control and the force control can be utilized. It is possible to appropriately control the operation of the robot device by achieving both the sense of touch and the force sense control.

[0014] "Robot device" refers to a device that can operate according to control commands output by a control device. Any robot device can be applied. The robot may be an industrial robot such as a computer, or may include a mobile object that can move automatically. Industrial robots include, for example, vertical articulated robots, SCARA robots, and parallel link robots. This includes robots, cartesian robots, and collaborative robots. For example, drones, vehicles configured to be able to drive themselves, automated guided vehicles, or mobile robots In the following embodiments, the robot device may be a combination of the robot and the industrial robot. The following mainly describes an example in which the device is a manipulator.

[0015] The "control command" is related to the control of the operation of the robot device, and is, for example, a target control amount. , operation amount, etc. "Outputting a control command" means to control a robot device based on the control command. It may be controlled directly, or if the robot device is equipped with a controller, it may be controlled by the controller. By outputting control commands to the controller, the controller controls the operation of the robot device. It may include the following.

[0016] "Operational state of the robot device" refers to the state of a part of the configuration of the robot device (for example, the end effector) Status regarding the operation of the robotic device and / or the object to be worked on by the robotic device In addition, "measurement of the operating state of the robot device" refers to the measurement of the structure of a part of the robot device. Measurement of conditions related to the operation of a component (e.g., an end effector) and / or the robotic device The measurement of the state of the object of the work used. "Object" refers to the state of the robotic device. A part of a robot device (for example, an engine) is an object that can be connected to the robot, such as a workpiece. The object (or effector) may also be considered an object.

[0017] A "task" is a job that a robotic device is to perform, and may include multiple steps. For example, parts transportation, part fitting, screw driving, processing, etc. The work is, for example, gripping a workpiece. The task may be a simple one such as releasing a work. It may be given by specifying a rate.

[0018] "Measurement information" is not limited to the sensor measurement data itself, but also includes characteristics calculated from the measurement data. The "measurement information" may be, for example, information that exists as the basic configuration of the robot. The distance may be measured or calculated using an encoder and / or a servo motor.

[0019] (2) First embodiment In the first embodiment, data to be used for control during actual work is collected before the work. For details of the control during actual work, see the second embodiment and This will be explained in the third embodiment.

[0020] (2.1) System Configuration FIG. 1 shows an example of the system configuration of a control system including a control device 100 according to a first embodiment. Here, the explanation will be focused on the hardware configuration of the control system.

[0021] In the illustrated example, the robotic device 200 is a manipulator. The manipulator 200 is a 6-axis vertical articulated industrial robot. The robot has a robot arm 21 and six joints 211 to 216. Each of the joints 211 to 216 is Each of the actuators has a built-in motor (not shown) and is configured to be rotatable around each axis.

[0022] The first joint portion 211 is connected to the base portion 221, and the tip side portion is rotated around the axis of the base. Instead of the base 221, a moving mechanism that can move automatically (self-propelled) may be provided. The second joint portion 212 is connected to the first joint portion 211, and the tip side portion thereof is slidable in the front-rear direction. The third joint portion 213 is connected to the second joint portion 212 via a link 222. The fourth joint portion 214 is connected to a link 223, and the tip portion of the fourth joint portion 214 is rotated in the up and down direction. The third joint portion 213 is connected to the link 223, and the tip end portion is rotated around the axis of the link 223. The fifth joint portion 215 is connected to the fourth joint portion 214 via a link 224. The sixth joint 216 is connected to the fifth joint 216 via a link 225. It is connected to the joint portion 215, and the tip portion rotates around the axis of the link 225. A gripper 226 is attached to the tip of the six-joint unit 216 together with a force sensor 320. The gripper 226 is an example of an end effector.

[0023] The visual sensor 310 is a sensor that measures images and detects the environment in which the robot device 200 operates. Observe each object (gripper 226, workpiece W1, workpiece W2) that exists in the boundary (work space) In the illustrated example, the visual sensor 310 is attached to the link 225. The visual sensor 310 is integrally provided with the robot device 200. The sensor 310 is fixed to the equipment in the work space, and the visual sensor 310 is connected to the robot device 200. The visual sensor 310 may be, for example, a digital camera or A camera such as a video camera may be used. , image data) is an example of visual measurement information.

[0024] The force sensor 320 is a sensor that measures force, and is connected to the robot device 200 (specifically, The force sensor is configured to measure the forces and moments acting on the gripper 226. The sensor 320 detects, for example, forces acting on the gripper 226 in the directions of the three axes of the X-axis, Y-axis, and Z-axis, and It may also be a six-axis force sensor that measures moments around the X-axis, Y-axis, and Z-axis. That is, the force sensor 320 detects the force of the assembly or gripper 22 supported by the robot device 200. The force and moment generated by contact between the force sensor 6 and an object can be measured. The measurement data of 20 is used to adjust the gripping force of the gripper 226 or to detect abnormal force applied to the gripper 226. It may also be used to detect whether or not a The term "force" is used to include the meaning of "moment."

[0025] The force sensor 320 may be, for example, a motor ( By measuring the current value of the actuator (not shown), it is possible to measure the force in the three axial directions of the X-axis, Y-axis, and Z-axis, and the force in the X-axis, Y-axis, and Z-axis directions. This may be realized by measuring the moments around the axis and the Z axis, or by measuring the front of the robot 200. The robot 200 uses pressure sensors on its surface and changes in the state of the jacket on its surface. The sensor may be a sensor that detects changes in air and / or liquid flow rate or capacitance. The detection device may be one that detects the change of the color.

[0026] Each of the joints 211 to 216 may have an encoder (not shown) built in. The encoder is an example of a sensor. The encoder detects the angle of each of the joints 211 to 216. The encoder measurement data is input to each joint 211. It may be used to control the angle of 216.

[0027] The control system may include a transport device 510 that transports the workpiece W2. The device 200 uses a gripper 226 (end effector) attached to the end of the arm. The end effector can be replaced with an external device depending on the application. Instead of the gripper 226, a welding gun, a tool, or the like may be attached. The conveying device 200 detects the workpiece W2 moving on the conveying device 510 using a visual sensor 310. The force sensor 320 can be used to perform tasks while tracking the robot. Then, the workpiece W1 as an assembly part grasped by the robot device 200 is transferred onto the conveying device 510. The work is to fit the workpiece W2 (assembly part, for example, a circuit board) that is flowing through.

[0028] The control device 100 includes a processor 101, a memory 102, and an external interface (I / F). The processor 101 includes a CPU (Central Processor F) 103. Furthermore, the processor 101 includes a microprocessor. processors, FPGAs (field-programmable gate arrays), and and at least one of DSP (digital signal processor) The memory 102 may include a RAM (Random Access Memory). ry), ROM (Read Only Memory), and auxiliary storage devices (e.g., hardware The processor is comprised of a hard disk drive, a solid-state drive, and a The control device 100 includes a plurality of computers. The control device 100 may be configured as a dedicated computer for the service to be provided. Not limited to the information processing device, but also general-purpose devices such as PCs (Personal Computers) It may be an information processing device such as a PLC (programmable logic controller). The controller may be a controller such as a controller.

[0029] The memory 102 stores a program executed by the processor 101. The server 101 executes a program stored in the memory 102. The functions of each functional block described later are realized together with the memory 1. 02 is a recognition label containing a trained model used for image recognition to recognize objects, for example. The library, the data collected by the pre-operation data collection, and the data collected based on the The recognition library and the setting library may also be stored. A library may be provided for each type of work that the robot device 200 can perform. The configuration information contained in the library is a trained model generated based on collected data. may include:

[0030] The external I / F 103 is, for example, a USB (Universal Serial Bus) It is an interface for connecting to an external device so that communication can be performed. The external I / F 103 is connected to an external device (including the robot device 200) by wire. The type and number of the external I / F 103 may be determined depending on the external device to be connected. In the illustrated example, the control device 100 is The robot device 200, the visual sensor 310, and the user interface 102 are connected to each other via the external I / F 103. The user I / F 400 is connected to the display interface (I / F) 400 as shown in FIG. In the illustrated example, the user I / F 400 is a control device 410 and an operation device 420. The user I / F 400 is provided separately from the control device 100. The display device 410 may be a liquid crystal display or an organic EL (Electro- The display device 410 may be a speaker, a luminescence display, or the like. The operation device 420 may be, for example, a keyboard, a mouse, or the like. The display device 410 and the operation device 42 are devices for performing operation inputs such as a touch panel. The display 0 may be integrally configured as a touch panel display. By using the display device 410 and the operation device 420, the status of the control device 100 can be confirmed. , and can operate the control device 100.

[0031] (2.2) Functional block configuration of the control device FIG. 2 is a block diagram showing the functional block configuration of the control device 100 according to the first embodiment. In the first embodiment, data to be used for control during work is collected before the work. This section mainly explains the functional block configuration related to the data collection. before the person (including the operator) starts using the control device 100 (and the robot device 200). For example, the control device 100 (and the robot device 200) may be debugged in advance before shipping. Data collection may be performed.

[0032] The control device 100 includes an action generating unit 110, an acquiring unit 120, a data collecting unit 130, and a data The system includes a data storage unit 140, a setting acquisition unit 150, and a library storage unit 160. In this embodiment, the control device 100 may include a control unit 170 or may include a control unit 170. It doesn't have to be.

[0033] The motion generation unit 110 changes the motion state of the robot device 200 from an arbitrary state to a target state. The motion generation unit 110 generates a control command for executing a forward motion. A reverse direction in which the operating state of the mobile device 200 is changed from a target state to an arbitrary state different from the target state. The motion generation unit 110 generates a control command for executing the motion. The driving unit 210 outputs the signal to the driving unit 210 of the robot device 200. The drive unit 210 includes a servo motor provided at each of the joints 211 to 216. The drive unit 210 may include a controller on the device 200 side. The motor is driven to operate the robot device 200.

[0034] Here, the goal state is the state that is realized when the purpose of the work (or process) is achieved, In the first embodiment, the robot device 200 is in an intermediate state of the end effector. The assembly part (work W1) being transported by the gripper (226) is transferred to the assembled part (work W2 ), so the target state is the state where the assembly part is attached to the subassembly part. Specifically, the target state is when the workpiece W1 is This is a state where it is fitted into the hole of W2, or an intermediate state where it is in contact with the hole at a position where it can be fitted into the hole.

[0035] The arbitrary state is a state different from the target state. For example, when workpiece W1 is inserted into a hole in workpiece W2, The arbitrary state is a state in which the object is located at a position away from the object. For example, the position of an arbitrary state may be set based on the position of the target state. It may be set by operation input (user input). Note that the term "position" does not mean "coordinates." It can also mean not only taste but also "attitude."

[0036] The acquisition unit 120 uses a sensor 300 to measure the operating state of the robot device 200. In the illustrated example, the sensor 300 includes a visual sensor 310 and a The sensor 300 includes a force sensor 320. The sensor 300 further includes another sensor 330 such as an encoder. The acquisition unit 120 may acquire the measurement data (i.e., image data) output by the visual sensor 310. The image recognition unit 121 may include an image recognition unit 121 that performs image recognition on the image data. The object (for example, the gripper 226, the workpiece W1, the workpiece W2) is subjected to image recognition such as feature extraction. The position of the object (for example, the coordinates of the feature) may be obtained by recognizing the object. The information is an example of visual measurement information.

[0037] The data collection unit 130 executes the reverse motion according to the control command output by the motion generation unit 110. During the execution, data including a set of the control command and the measurement information acquired by the acquisition unit 120 is repeatedly This allows the movement (forward movement) of the robot device 200 during actual work to be For example, the robot device 200 can be used to efficiently collect data that can be used for controlling the robot device 200. During the actual work, the control is based on the collected data and the measurement information obtained during the work. By generating a command, the robot can properly control the robot's movement from an arbitrary state to a target state. The data collection unit 130 can collect the control data output by the action generation unit 110. During the execution of the forward operation in accordance with the command, the control command and the measurement information acquired by the acquisition unit 120 The data may be collected repeatedly, including sets of

[0038] The motion generation unit 110 generates a motion state of the robot device 200 by generating a plurality of motion states different from the target state. Generates control commands to execute multiple patterns of reverse motion to change the state of the The data collection unit 130 may collect data for each of the multiple patterns of reverse motion. This allows for multiple patterns of data corresponding to multiple travel routes. Since the collected data can be used to control the operation of the robot device 200 during actual work, it is possible to provide versatility to the robot device 200. For example, if the workpiece W2 is not fixed or moves during work, Even when the workpiece W1 is moved, it becomes easy to attach the workpiece W1 to the workpiece W2.

[0039] The action generation unit 110 generates a control command that collects more data in an area closer to the target state. In this case, a control command may be generated that collects less data in an area farther from the target state. The closer the area is to the target state, the more precise and accurate control is required during actual work. By increasing the amount of data collected in the region close to the actual state, it is possible to achieve precise and accurate control. On the other hand, in areas far from the target state, detailed and accurate control is required. Therefore, the amount of data collected is smaller in areas farther from the target state. By generating control commands that reduce the amount of data collected, unnecessary data collection is suppressed, resulting in efficient data collection. This becomes possible.

[0040] The acquisition unit 120 sets the acquisition period ( For example, the sampling frequency can be shortened, and the amount of measurement information acquired increases as the area becomes farther from the target state. The acquisition period may be extended so that the number of times is increased. The amount of data collected increases in the area where the target state is reached, and decreases in the area where the target state is reached. It is possible.

[0041] The acquisition unit 120 acquires target measurement information obtained in the target state and target measurement information obtained during the execution of the reverse direction operation. The data collection unit 130 may acquire the difference between the current measurement information and the measured value as relative measurement information. During the execution of the reverse direction operation according to the control command, the control command and a set of relative measurement information are included. For example, the acquisition unit 120 may acquire data by using the visual sensor 3 in the target state. The visual measurement information (target position) of the object obtained by 10 and the The difference between the current measurement information (current position) of the object and the current measurement information (current position) of the object may be acquired as relative position information. In the environment of FIG. 1, the acquisition unit 120 sets the position of the hole in the workpiece W2 as the target position, and Each position on the movement path is taken as the current position, and the difference between the target position and each current position is taken as the relative position. This allows the control to be performed for each relative positional relationship between the target position and the current position of the object. Therefore, during actual work, the target position and the current position of the object are matched. It is possible to generate appropriate control commands based on the relative positional relationship of the This control can be applied even when the object moves during work.

[0042] The acquisition unit 120 acquires information about the robot device 200 in response to the output of the sensor 300 or a control command. Measurement information including velocity-related values ​​relating to the robotic device 200 may be acquired. The information may be at least one of the velocity, acceleration, and jerk of the object. 20 is a system for detecting an object from visual measurement information (position information of the object) obtained by the visual sensor 310. The acquisition unit 120 may derive the velocity, acceleration, and jerk of the robot. The speed, acceleration, and jerk of the object may be derived from the control command. The velocity, acceleration, and speed of the object are calculated from the measurement information obtained by the sensor 330 (for example, an encoder). The data collection unit 130 may derive the reverse direction motion and the jerk. During the process, data including a set of the control command and measurement information including a speed-related value is repeatedly The speed-related values ​​collected in this way may be used in control during actual work. It can be used to set speed-related target values. This will be explained in the third embodiment.

[0043] The data storage unit 140 stores the data collected by the data collection unit 130. Data Collection The data collected by the unit 130 includes a plurality of sets of measurement information (relative measurement information) and control commands. Each set may include measurement information (relative measurement information), speed-related values, and control commands. The data collected by the data collection unit 130 is such that the area closer to the target state has more data. A set may be included.

[0044] The setting acquisition unit 150 acquires the data collected by the data collection unit 130 (specifically, the data storage To perform a task using the robot device 200 based on the data stored in the robot device 200 The setting information for a certain task is acquired by For example, the setting acquisition unit 150 may acquire the data collected by the data collection unit 130. After the collection is completed, the library storage unit 160 acquires the setting library. The control unit 170 stores the setting library acquired by the setting unit 50. The library is used to control the robot device 200.

[0045] The setting library contains the operation parameters for each work process, the transition destinations and transition conditions for each process. It includes information about the transition conditions (normal value, timeout value, error value, etc.) and the operation parameters. The data may include correspondence information that associates the measurement information (relative measurement information) with the control command. It includes correspondence information that associates measurement information (relative measurement information), speed relation values, and control commands. The operational parameters may be calculated based on visual measurement information obtained by the visual sensor 310. The robot is controlled by visual control and by force measurement information obtained by the force sensor 320. The operation parameters may include a control ratio of the force feedback control to the robot control. The operating parameters may include information on the type of material (metal, resin, screw, connector, etc.). The setting acquisition unit 150 may also include information about features on the object (holes, edges, connectors, etc.). A configuration library based at least in part on operational inputs made via the operating device 420. may be obtained.

[0046] The setting acquisition unit 150 uses the data collected by the data collection unit 130 as learning data. Therefore, even if a trained model for deriving control commands from measurement information is obtained by machine learning, In this case, the control unit 170 may use a setting library including a trained model during actual work. The robot device 200 is controlled based on measurement information obtained during the work using the sensor. The trained model (learning model) uses machine learning to develop the inference ability to generate control commands. The type of machine learning is not particularly limited as long as it can be obtained more easily. Although not universally accepted, it is typically supervised or reinforcement learning. For example, neural networks such as deep neural networks (DNN) are used. The learning model may be generated based on a value function such as a state value function or an action value function. In this way, the setting acquisition unit 150 can obtain the optimal setting by machine learning. Determine the operation to infer the value, enable the inference, and generate the optimal value variable of the control command. Do it.

[0047] The setting acquisition unit 150 acquires a setting list including the control ratio between the visual control and the force-sense control for each process, for example. In this case, the control unit 170 may acquire the setting library during the actual work. Using a variable, the control ratio is changed dynamically or stepwise depending on the work situation. The details of such control will be explained in the second embodiment.

[0048] The setting acquisition unit 150 includes a speed-related value for the robot device 200 for each process, for example. In this case, the control unit 170 may acquire a setting library. A velocity-related value for the robotic device 200 may be derived from the information or control command. The control unit 170 calculates the speed relationship based on the setting library and measurement information obtained during the operation. The robot device 200 is repeatedly controlled to reduce the difference between the speed-related target value and the speed-related target value. The details of such control will be described in the third embodiment.

[0049] The motion generation unit 110 generates a reverse motion for each type of work using the robot device 200. The control command for executing the task may be generated. The task type is assumed to be the environment shown in Figure 1. and "mounting components on a circuit board." Other examples of work types include "packing food into boxes," "Screw tightening", "Pick and place", "AGV (Automated Guided Vehicle) The data collection unit 130 repeatedly collects data for each type of work. Data may be collected for each type of work by collecting data in return.

[0050] In this case, the setting acquisition unit 150 uses the data collected by the data collection unit 130 for each type of work. Based on the data, the control unit 170 can acquire the setting library for each type of work. ,Control for performing the work using a setting library corresponding to the type of work to be actually performed. For example, the control unit 170 may A list of setting libraries stored in the display device 410 is displayed, and the user can select an operation from the list. The robot device 200 is controlled using the setting library selected using the operator 420. The library storage unit 160 may store a recognition library in association with the setting library. If the system also memorizes the type of work to be performed, the system uses a recognition library corresponding to the type of work to be performed. Image recognition may be performed to perform the operation.

[0051] (2.3) An example of data collection 3 and 4 are diagrams illustrating an example of data collection by the control device 100 according to the first embodiment. This is a diagram for

[0052] As described above, the motion generation unit 110 converts the motion state of the robot device 200 from the target state to the corresponding state. Control for executing multiple patterns of reverse operation that change to multiple different arbitrary states The data collection unit 130 generates a command for each of the multiple patterns of reverse direction motion. As shown in FIG. 3, the action generation unit 110 repeatedly collects data in a region close to the target state. The control command is generated so that the amount of data collected increases in the area, and the amount of data collected decreases in the area farther from the target state. In Figure 3, the x-axis and y-axis are inclined relative to each other in the horizontal plane. The z-axis indicates the perpendicular direction. Specifically, the more data collected, the darker the color. The smaller the amount, the lighter the color.

[0053] In the example of FIG. 4, the motion generation unit 110 generates the motion when the workpiece W1 (component) is The target state is the hole in W2 (board), and a total of eight patterns from P1 to P8 are selected. The robot device 200 executes the reverse movement to the state P1. Although the movement paths are shown linearly, each movement path may not be linear. After pulling out the work W1 (component) upward from the hole in the work W2 (board), Alternatively, the path may be one that moves the workpiece W1 (component) diagonally upward. Among the patterns P1 to P8, P2, P4, and P8 move in the region R close to the target state. This means that the closer the location to the target state, the more data is collected. It becomes more.

[0054] (2.4) Specific examples of the operation of the control device FIG. 5 is a diagram illustrating an example of data collection by the control device 100 according to the first embodiment. FIG.

[0055] In step S101, a user (e.g., a worker) performs a The workpiece W1 held or supported by the vector is moved to a target state. 0 is one of multiple reverse operation patterns (i.e., any arbitrary state ) and / or the position information of the arbitrary state. The acquisition unit 120 then acquires the target state from the visual measurement information obtained using the visual sensor 310. The acquisition unit 120 acquires the position of the object as the target position using the force sensor 320. From the force measurement information obtained, the force (reaction force) applied to the object in the target state is calculated as the target force (target reaction force). Obtain as.

[0056] In step S102, the action generation unit 110 performs the action generation based on the information acquired in step S101. Based on this, a control command is output to the robot device 200 (drive unit 210), thereby To move an object by a predetermined distance.

[0057] In step S103, the acquisition unit 120 acquires the current measurement information. 0 acquires the current position of the object from the visual measurement information obtained using the visual sensor 310. The acquisition unit 120 calculates the difference between the current position of the object and the target position as a relative position (amount of position change). The acquisition unit 120 may acquire the force sensor information obtained by using the force sensor 320 as follows. The current force (reaction force) is acquired from the measurement information. The acquisition unit 120 calculates the current force (reaction force) and the target force. The difference between the force (reaction force) and the actual force (force change amount) may be acquired as the relative force (force change amount). , from the current visual measurement information or the control command of step S102, the velocity related value of the object (velocity , acceleration, and jerk) may be acquired.

[0058] In step S104, the data collecting unit 130 receives the control command of step S102 and The set of information (position change amount, force change amount, velocity related value) acquired in step S103 is collected. The set is collected and stored in the data storage unit 140.

[0059] If the object has not yet completed moving to any state corresponding to the current reverse motion pattern (Step S105: NO), the process returns to step S102, and the action generation unit 110 By outputting a control command to the robot device 200 (drive unit 210), the object is moved in a predetermined manner. Then, in step S103, the acquisition unit 120 acquires information (position In step S104, the data collection unit 130 is a control command of step S102 and information acquired in step S103 (position change The sets of the force change amount, the force change amount, and the velocity related value are collected and stored in the data storage unit 140. This process is repeated until the transition to the arbitrary state corresponding to the current backward motion pattern is completed. Repeat.

[0060] On the other hand, if the object has completed moving to an arbitrary state corresponding to the current backward motion pattern, (Step S105: YES), the motion generation unit 110 generates all of the multiple backward motion patterns. Check whether the operation has been completed for all patterns. If the operation has not been completed for all patterns ( Step S106: NO), then the process moves to the next reverse operation pattern (Step S107), and The process resumes from step S101.

[0061] If all patterns have been processed (step S106: YES), proceed to step S108. In this case, the setting acquisition unit 150 acquires the data collected by the data collection unit 130 (specifically, the data The robot device 200 performs a task based on the data stored in the data storage unit 140. Then, in step S109, the operating conditions (operating parameters) for the The setting acquisition unit 150 acquires the setting including the operating conditions (operating parameters) determined in step S108. The set information is stored in the library storage unit 160 as a set library.

[0062] By executing the flow shown in Figure 5 for each type of work, the settings for each type of work can be A constant library may be stored in the library storage unit 160.

[0063] (2.5) An example of a configuration library FIG. 6 is a diagram illustrating an example of a setting library stored in the control device 100 according to the embodiment. This is a diagram of the

[0064] In the example shown, the types of work include "mounting components on a circuit board," "packing food into boxes," and "tightening screws." "Pick and place" (so-called picking), and "AGV control." The work includes a plurality of steps.

[0065] For example, in the work type "mounting components on the board", Process 1 "Component recognition → Movement" → Process 2 "Hold the part" → Process 3 "Move the part towards the board" → Process 4 "Approach" → Process 5 "Platform the board" "Hole recognition" → Step 6 "Hole approach" → Step 7 "Hole tracing operation" → Step 8 "Hole insertion" → Step 9 "Hole insertion" On the other hand, the control device 100 performs the data collection and Sometimes, the operation of the robot device 200 may be controlled in the reverse order. In the case of "AGV control", a process of avoiding obstacles may be further included. The step of avoiding the above may be included in step 4 "moving the target object toward the target object."

[0066] The setting library contains the operation parameters for each work process, the transition destinations and transition conditions for each process. It includes information about the transition conditions (normal value, timeout value, error value, etc.) and the operation parameters. The data may include correspondence information that associates the measurement information (relative measurement information) with the control command. It includes correspondence information that associates measurement information (relative measurement information), speed relation values, and control commands. The operation parameters may include a control ratio between visual control and haptic control. The parameters may include information on the type of object (metal, resin, screw, connector, etc.) The motion parameters may include information about features on the object (holes, edges, connectors, etc.). stomach.

[0067] (3) Second embodiment The second embodiment will be described mainly focusing on the differences from the first embodiment. The system configuration is the same as that of the first embodiment (see FIG. 1). The following mainly describes an example of an embodiment based on the first embodiment. Furthermore, at least a part of the first embodiment does not have to be a prerequisite.

[0068] (3.1) Functional block configuration of the control device FIG. 7 is a block diagram showing a functional block configuration of a control device 100 according to the second embodiment. do.

[0069] The control device 100 according to the second embodiment is a device for measuring the operating state of the robot device 200. an acquisition unit 120 that acquires measurement information obtained using the sensor 300; and a control unit 170 that controls the operation of the robot device 200. The control unit 170 includes a visual sensor 310 and a force sensor 320. visual control, which is control based on visual measurement information obtained by the force sensor 320; The control ratio of the force sense control, which is a control based on force sense measurement information, and the force sense control, which is a control based on force sense measurement information, is Dynamic or step-by-step changes are made depending on the work situation, which allows for the benefits of visual control and By utilizing the advantages of force-sensation control, the movement of the robot device 200 can be controlled by both visual control and force-sensation control. This makes it possible to appropriately control the operation.

[0070] In the second embodiment, the control unit 170 refers to the visual measurement information and The visual control unit 171A generates first information indicating the content of control, and refers to the force sense measurement information. a force-sense control unit 171B that generates second information indicating the content of control of the robot device 200; and generating a control command for the robot device based on the first information and the second information. The robot device 200 is configured to receive visual measurement information and force measurement information according to the situation of the work performed by the robot device 200. By performing weighting processing between the first information and the second information, or weighting processing between the first information and the second information, and a weighting unit 173 for changing the control ratio. The control ratio can be appropriately changed.

[0071] The visual control performed by the visual control unit 171A is performed, for example, based on the operation parameters ( The visual control unit 171A may perform the determination based on the visual sensor 310. Based on the visual measurement information obtained using Then, the visual control unit 171A determines the corresponding positional relationship from the determined positional relationship. By using the information, the difference between the current position and the target position is reduced (i.e., the object is moved to the target position). and outputting the visual control command as the first information. At least part of such visual control may be performed using a trained model. stomach.

[0072] Similarly, the force-sense control performed by the force-sense control unit 171B is performed based on, for example, the operation parameters in the setting library. The force sense control unit 171B may be configured to perform the force sense control based on the force sensor (corresponding information, etc.). Based on the force measurement information obtained using the sensor 320, the current force (current reaction force) of the object and the target The difference between the target force and the target reaction force is identified, and the difference is calculated based on the correspondence information. Even if a force control command that reduces the force is generated and output as the second information, At least a part of such force control may be performed using a trained model. .

[0073] The control unit 170 (weighting unit 173) adjusts the weighting factor as the work using the robot device 200 progresses. The control ratio may be changed dynamically or stepwise as the robotic device 20 The work using 0 may include a plurality of predetermined steps. This allows the robot device 20 to be controlled at an appropriate control ratio for each process. You can control the behavior of 0.

[0074] The library storage unit 160 stores settings including settings related to the control ratios of each of a plurality of processes. The control unit 170 (weighting unit 173) may store a library (setting information). The control ratio may be changed for each process based on the setting library. An appropriate control ratio can be set.

[0075] The control unit 170 (weighting unit 173) controls the transition from one process to the next process in the work. At this time, the control ratio of one process is changed to the control ratio of the next process. This allows the control ratio to be changed gradually without sudden fluctuations, This enables precise operation control and reduces the occurrence of operation errors.

[0076] For example, if the control ratio (visual control: force sense control) of process A is "80:20", the next process of process A If the control ratio of process B is set to "50:50", When the transition condition is met, the control unit 170 (weighting unit 173) changes the weighting ratio from "80:20" to "7 5:25" → "70:30" → "65:35" → "60:40" → "55:45" → "5 The control ratio is changed in stages, for example, from "0:50" to "0:50". Section 173) is "80:20" → "79:21" → "78:22" → "77:22" → · The control ratio may be changed continuously in this way. The control may be performed in units of a control cycle, or in units of a time consisting of a plurality of control cycles.

[0077] The control unit 170 (weighting unit 173) performs weighting based on the measurement information obtained using the sensor 300. It determines whether the transition conditions from one process to the next in the work are met. The transition condition may be included as one of the behavior parameters in the configuration library. When the transition condition is satisfied, the control unit 170 (weighting unit 173) When the process moves from one step to the next step, the control ratio changes to the control ratio corresponding to the next step. You may change it.

[0078] After switching to the next process, the control unit 170 determines whether the control in the next process has converged. If the control ratio is not sufficient, the control ratio may be returned to the one process and changed to the control ratio corresponding to the one process. In this way, if the control does not converge, the process returns to the previous step and the operation is restarted. Therefore, it is expected that the control will converge when the process moves to the next step. "If the transition condition is not met" means that the measurement information does not meet the normal value of the transition condition, or a timeout occurs. At least one of the following conditions is met: a fault has occurred, or the measurement information has become an error value. It may also mean that

[0079] The control unit 170 (weighting unit 173) is in the first control state ( The second method uses visual control and force feedback control in cooperation with the first method. Control state (also called "visual + force-based") and force control are used as the priority. and a third control state (also referred to as "force-based"), "Priority is given to visual control" means, for example, that the visual control is the dominant control. This may mean that the ratio is approximately 65% ​​to 100%. "Control using both in coordination" means, for example, that visual control and force control are roughly 50:50. "Performing control by giving priority to force feedback control" may mean, for example, It may also mean that the control ratio accounted for by force sense control is approximately 65% ​​to 100%. The control unit 170 selects one of the first control state to the second control state based on the measurement information obtained using the sensor 300. The control unit 170 (weighting unit 173) may determine whether to switch the control in the three control states. ) is a control state in which the difference between the measurement information and the target value of the measurement information does not become equal to or less than a predetermined value. If the transition condition is not satisfied (it may be the case that the transition condition is not satisfied), the second control state or the third control state You can also switch to and refer to the force measurement information.

[0080] When an instruction is received from the outside, the control unit 170 (weighting unit 173) performs the following in response to the instruction. For example, a change in the control ratio may be instructed via the operation device 420. When this instruction is received, the control unit 170 (weighting unit 173) changes the control ratio in accordance with the instruction. Good too.

[0081] The control unit 170 controls the robot device 200 and the response to the control. Depending on the control result (sensor information), the robot device 200 is controlled accordingly. A prediction unit 174A predicts the control result to be obtained, and a control unit 174B corrects the subsequent control content according to the prediction. For example, the robot device 200 may further include a correction unit 174B. The operation specified by the control command may not be performed due to external factors such as the sensor itself or an error factor on the sensor 300 side. In the case where the robot device 200 does not perform the above, the prediction unit 174A calculates the error (i.e., (the difference between the control command and the control result) and predict the control result for the subsequent control content. Then, the correction unit 174B calculates the control command generated by the command generation unit 172 based on the prediction unit 174B. The correction unit 17 may correct the calculated value according to the prediction result of the correction unit 17 and output the corrected value to the driving unit 210. 4B may correct the control command to cancel the identified error. Even if there are error factors such as the above, more accurate robot control becomes possible.

[0082] The library storage unit 160 stores a plurality of setting libraries prepared for each type of work. Each of the plurality of setting libraries may contain setting information of the control ratio (operation parameters). Each of the plurality of setting libraries may include a set of data relating to each of the steps in the series. The control unit 170 selects the type of work to actually be performed from a plurality of setting libraries. The robot device 200 is controlled using the setting library selected according to the selection. This may be done by an operation input via the operation device 420. That is, the acquisition unit 120 (image recognition unit 121) acquires the image of the work site or the object observed by the visual sensor 310. The actual work to be performed may be estimated based on the visual measurement information obtained by observing the work. 170 displays the setting library corresponding to the estimated task on the display device 410. This may be suggested to the user (operator).

[0083] The library storage unit 160 stores a plurality of recognition libraries prepared for each type of work. Each of the plurality of recognition libraries may be used for image recognition processing of an object. The control unit 170 may select a trained model from among a plurality of recognition libraries. Then, image recognition processing is performed using a recognition library selected according to the type of work to be performed. The selection may be made by an operation input via the operation device 420. Prior to the input, the acquisition unit 120 (image recognition unit 121) recognizes the work site or the object using a visual sensor. 310 may estimate the actual work to be performed based on visual measurement information obtained by observing The control unit 170 displays the recognition library corresponding to the estimated task on the display device 410. The user (operator) may be prompted to indicate the desired result.

[0084] (3.2) Operation flow of the control device FIG. 8 is a flow chart for explaining an example of a control flow by the control device 100 according to the second embodiment. FIG.

[0085] In step S201, the control unit 170 starts the process n. The value is "1". At this time, the control unit 170 sets the setting information (including the control ratio) corresponding to the process n. (including operating parameters) are applied.

[0086] In step S202, the control unit 170 sets the setting information (control ratio) corresponding to the process n. The robot device 200 is controlled by applying the motion parameters (including the motion parameters).

[0087] In step S203, the control unit 170 determines whether the transition condition from step n to the next step is satisfied. If step n is completed (i.e., if step n is completed), If step S203: YES, the process proceeds to step S204. If not (step S203: NO), in step S205, the control unit 170 , it is determined whether the control of the step n has converged. If it is determined that the control of the step n has converged ( If the answer is YES in step S205, the process returns to step S202.

[0088] When it is determined that the control of the process n does not converge (step S205: NO), the control unit 170 In this case, the control unit 170 returns to the step (n-1) before the step n. Alternatively, the control ratio may be gradually changed from the control ratio of the step (n-1) to the control ratio of the step (n-2). 0, instead of returning to step (n-1), at least some of the operating parameters of step n are Step n may be restarted after being changed according to a predetermined rule. For example, If it is suspected that the speed is too high, the speed target value may be lowered by one level and process n may be restarted. stomach.

[0089] In step S204, the control unit 170 determines whether all steps of the work have been completed. If all steps are completed, this flow ends. If all steps are not completed (step Step S204: NO), in step S207, the control unit 170 executes the next step ( At this time, the control unit 170 determines the control ratio of the process (n+1) from the control ratio of the process n. Alternatively, the control ratio may be gradually changed to the control ratio.

[0090] (3.3) Specific examples of the operation of the control device FIG. 9 shows the operation of "mounting components on a board" as controlled by the control device 100 according to the second embodiment. 10A and 10B are diagrams for explaining specific examples of control during operation.

[0091] In the work type "Placing components on the board", process 1 "Component recognition → Move" → process 2 "Component "Grip" → Process 3 "Move the component towards the board" → Process 4 "Approach" → Process 5 "Board hole recognition" → Step 6 "Hole approach" → Step 7 "Hole copying operation" → Step 8 "Hole insertion" → Step 9 "Hole insertion completed" → Process 10 "Release grip" are performed in this order, and the operation parameters of each process are set It is included in the Lari.

[0092] Of these processes, the object (end effector (gripper 226), part In processes involving contact between the workpiece (workpiece W1) and the substrate (workpiece W2), force-based control is performed. The object (end effector, part) is moved at a certain speed or higher. Such processes are configured for vision-based control, directing objects toward contact between them. In processes where the robot is moved at low speed, the system is set to use visual and force-based control. The specific control ratios of each process are determined based on the data collected in the data collection according to the first embodiment. It is preferable that the setting be based on the

[0093] In the example shown, vision-based control is applied to step 1, "part recognition -> movement." However, The position of the part recognized by image recognition is the target position, and the current position of the end effector is When approaching the vicinity of the target position, vision and force-based control may be applied. Force-based control is applied to "part grasping."

[0094] Vision-based control is applied to step 3, "Move the component towards the board." Vision-based control is applied to the "near" position. When the current position of the part approaches the vicinity of the target position, the system uses visual and force-based Step 5, "Board Hole Recognition" may be applied with vision-based control. do.

[0095] Step 6, "Hole Approach," uses vision and force-based control. Step 7, "Hole Tracking" From step 10 "Grip Release", force-based control is applied.

[0096] (4) Third embodiment The third embodiment will be described mainly focusing on the differences from the first and second embodiments. The system configuration according to the third embodiment is the same as that of the first embodiment (see FIG. 1). The following mainly describes an example in which the present embodiment is based on the first and second embodiments. The embodiment does not necessarily have to be premised on at least a part of the first and second embodiments.

[0097] (4.1) Functional block configuration of the control device FIG. 10 is a block diagram showing the functional block configuration of a control device 100 according to the third embodiment. be.

[0098] The control device 100 according to the third embodiment is a device for measuring the operating state of the robot device 200. an acquisition unit 120 that acquires measurement information obtained using the sensor 300; A control unit 17 repeatedly generates control commands for the robot device 200 in each control period. The control unit 170 determines the robot device 200 based on the measurement information or the control command. and calculates a speed relation value for each control period according to the operating state of the robot device 200. A control command is generated to reduce the difference between the variable speed-related target value and the speed-related value. This allows control that prevents operational errors when working with the robot device 200. In addition, the speed-related target value is controlled according to the operating state (work situation) of the robot device 200. By making it variable for each cycle, it can also handle precise work involving nonlinear movements. As described above, the velocity-related value includes at least one of velocity, acceleration, and jerk.

[0099] In the third embodiment, the control unit 170 calculates the speed-related value from the measurement information or the control command at the control period. and a derivation unit 175 that derives the value for each operation state (work situation) of the robot device 200. A target setting unit 176 that variably sets a speed-related target value for each control period, A command generator generates a control command for each control period so as to reduce the difference between the speed-related target value and the control command. The target setting unit 176 sets the target value included in the operation parameters in the setting library. The speed-related target value may be variably set in accordance with the speed-related value included in the speed-related value.

[0100] The command generation unit 172 includes the visual control unit 171A and the force control unit 171B described in the second embodiment. The functions of the control unit 171B and the weighting unit 173 may be incorporated. 70, in addition to the command generation unit 172, the visual control unit 171A described in the second embodiment , a force-sense control unit 171B, and a weighting unit 173 may be provided.

[0101] The control unit 170 (command generation unit 172) receives measurement information and target measurement information (target value of the measurement information) and the difference between the speed-related value and the speed-related target value. Generate a directive.

[0102] For example, the control unit 170 (command generation unit 172) may The difference between the visual measurement information (current position) and the target visual measurement information (target position) is reduced, and the speed is increased. A control command may be generated to reduce the difference between the relation value and the speed-related target value. ,Control to reduce the difference between visual measurement information (current position) and target visual measurement information (target position) This is the same as the visual control described in the second embodiment.

[0103] The control unit 170 (command generation unit 172) uses the force measurement information obtained using the force sensor 320. The difference between the information (current force (reaction force)) and the target force measurement information (target force (reaction force)) is reduced, and A control command may be generated to reduce the difference between the speed-related value and the speed-related target value. Here, the difference between the force measurement information (current force (reaction force)) and the target force measurement information (target force (reaction force)) The control for reducing is the same as the force sense control described in the second embodiment.

[0104] The control unit 170 (target setting unit 176) calculates the difference between the current measurement information and the target measurement information (current For example, the control unit 170 (the relative measurement information) may change the speed-related target value. The target setting unit 176) stores the measurement information (relative measurement information) and the speed relationship included in the setting library. The velocity relation value corresponding to the current relative measurement information is calculated using the correspondence information that corresponds to the velocity relation value. Alternatively, the degree-related target value may be set.

[0105] For example, the control unit 170 (target setting unit 176) may use the visual sensor 310 to The speed-related target value may be changed according to the difference between the current visual measurement information and the target visual measurement information. The control unit 170 (target setting unit 176) uses the force sensor 320 to The velocity-related target value is changed according to the difference between the current force sense measurement information and the target force sense measurement information. That's fine.

[0106] The control unit 170 (deriving unit 175 and target setting unit 176) calculates the relationship between the measurement information and the target measurement information. Depending on the difference (current relative measurement information), it is used as a speed-related value (and speed-related target value). The value may be one or more of velocity, acceleration, and jerk. (Derivation unit 175 and target setting unit 176) are used to determine the velocity-related value (and On the other hand, for force-based control, You can also select jerk as the speed-related value (and speed-related target value). During control, acceleration may be selected as the speed-related value (and speed-related target value).

[0107] The control unit 170 controls the robot device 200 in the same manner as in the second embodiment. and the control result for the control content, the robot device 200 is controlled in the following manner. A prediction unit 174A predicts the control result for the content, and a control unit 174B determines the subsequent control content according to the prediction. and a correction unit 174B that performs the correction.

[0108] The library storage unit 160 stores a plurality of setting libraries prepared for each type of work. Each of the multiple setting libraries may contain setting information for speed-related values ​​(speed-related The control unit 170 may select the setting to be actually performed from among a plurality of setting libraries. The robot device 200 is controlled using a setting library selected according to the type of work to be performed. The selection may be made by an operation input via the operation device 420. Prior to the input of the operation, the acquisition unit 120 (image recognition unit 121) detects the work site or the object with a visual sensor. Even if the actual work to be performed is estimated based on the visual measurement information obtained by the observation of the user 310, The control unit 170 displays the setting library corresponding to the estimated work on the display device 410. By displaying this, a suggestion may be made to the user (operator).

[0109] The library storage unit 160 stores a plurality of recognition libraries prepared for each type of work. Each of the plurality of recognition libraries may be used for image recognition processing of an object. The control unit 170 may select a trained model from among a plurality of recognition libraries. Then, image recognition processing is performed using a recognition library selected according to the type of work to be performed. The selection may be made by an operation input via the operation device 420. Prior to the input, the acquisition unit 120 (image recognition unit 121) recognizes the work site or the object using a visual sensor. 310 may estimate the actual work to be performed based on visual measurement information obtained by observing The control unit 170 displays the recognition library corresponding to the estimated task on the display device 410. The user (operator) may be prompted to indicate the desired result.

[0110] (4.2) Specific examples of the operation of the control device FIG. 11 shows the control of the control device 100 according to the third embodiment, which is a control for "mounting components on a board." FIG. 10 is a diagram for explaining a specific example of control during work.

[0111] As mentioned above, in the work type "Placing components on the board", process 1 is "Component recognition → Movement" → Process 2 "Grab the part" → Process 3 "Move the part towards the board" → Process 4 "Approach" → Process 5 " "Board hole recognition" → Step 6 "Hole approach" → Step 7 "Hole tracing operation" → Step 8 "Hole insertion" → Step 9 The process is performed in the order of "hole insertion completed" → process 10 "release grip". The data is included in the settings library.

[0112] In each step, the control unit 170 (command generation unit 172) uses the sensor 300 to The difference between the measurement information and the target measurement information (target value of the measurement information) is reduced, and the speed-related value and A control command is generated to reduce the difference from the speed-related target value.

[0113] In the example shown in the figure, in process 1 "Part recognition → movement", the target measurement information is "part position", The speed-related target value may be "speed." In this case, the control unit 170 (command generating unit 1 72) is the current position of the end effector and the position of the part obtained using the visual sensor 310. and the difference between the velocity of the end effector and the velocity target value. Here, the control unit 170 (target setting unit 176) may generate a control command as follows. The difference between the current position of the end effector and the position of the part is calculated using the sensor 310. The speed target value may be changed for each control period.

[0114] For process 2 "Part gripping", "target reaction force" is set as the target measurement information, and "speed" is set as the speed-related value. In this case, the control unit 170 (command generation unit 172) uses the force sensor 320 and reducing the difference between the current reaction force acting on the end effector and the target reaction force, and generating a control command to reduce the difference between the velocity of the end effector and the velocity target value. Here, the control unit 170 (target setting unit 176) may The speed target value is controlled periodically according to the difference between the reaction force generated in the end effector and the target reaction force. It may be changed every time.

[0115] In process 3, "Move the component toward the board," the target measurement information is "board position" and speed function. In this case, the control unit 170 (command generating unit 172) uses the following: The current position of the end effector (or part) and the position of the board obtained using the visual sensor 310 The difference between the jerk of the end effector (or part) and the target jerk value is reduced. Here, the control unit 170 (target setting unit 17) may generate a control command to reduce the 6) is the current position of the end effector (or part) obtained using the visual sensor 310. The target jerk value may be changed for each control cycle according to the difference from the position of the substrate.

[0116] In step 4, "Approach," the target measurement information is "the difference between the end effector and the substrate position," The speed-related value may be "acceleration." In this case, the control unit 170 (command generating unit 17 2) is the current position of the end effector (or part) obtained using the visual sensor 310; The difference between the position of the substrate and the acceleration of the end effector is reduced, and the difference between the acceleration target value and the acceleration of the end effector is reduced. Here, the control unit 170 (target setting unit 17) may generate a control command to reduce the 6) is the current position of the end effector (or part) obtained using the visual sensor 310. The acceleration target value may be changed for each control cycle according to the difference from the position of the substrate.

[0117] In step 5 "Board Hole Recognition", "Board Hole" can be applied as target measurement information. In this case, the control unit 170 (command generation unit 172) recognizes the hole in the board using the visual sensor 310. do.

[0118] In step 6, "Hole Approach," the target measurement information is "Position of the hole on the board" and "Target reaction force," and the speed is In this case, the control unit 170 (command generating unit 172) can apply "acceleration" as the degree-related value. ) is calculated by subtracting the difference between the current position of the component obtained using the visual sensor 310 and the position of the hole on the board. and control the difference between the acceleration of the end effector and the target acceleration value. The control unit 170 (command generation unit 172) may generate a command. and reducing the difference between the current reaction force acting on the end effector and the target reaction force, At the same time, a control command is generated to reduce the difference between the acceleration of the end effector and the target acceleration value. It may be done.

[0119] For each process from process 7 "hole tracing operation" to process 10 "gripping release", target measurement information is In this case, the "target reaction force" for each process and the "jerk" as a speed-related value can be applied. The unit 170 (command generating unit 172) uses the current end-effect obtained using the force sensor 320. The difference between the reaction force generated in the end effector and the target reaction force for each process is reduced, and the end effector A control command may be generated to reduce the difference between the jerk and the target jerk value for each step. Here, the control unit 170 (target setting unit 176) uses the end point obtained by using the force sensor 320. The jerk target value is changed for each control cycle according to the difference between the reaction force generated in the effector and the target reaction force. That's fine.

[0120] (5) Other embodiments The operation flows and operation examples in the above-described embodiments do not necessarily follow the order described in the flow charts. For example, steps in an operation may not be executed in a chronological order as shown in a flow diagram. The operations may be performed in a different order than described, or may be performed in parallel. Some of the steps in the process may be removed and additional steps may be added to the process. .

[0121] A program for causing a computer to execute the operations according to the above-described embodiments may be provided. The program may be recorded on a computer-readable medium. The removable media allows you to install the program on your computer. Here, the computer-readable medium on which the program is recorded is a non-transitory storage medium. The non-transitory storage medium is not particularly limited, but may be, for example, a C The storage medium may be a D-ROM, a DVD-ROM, or the like.

[0122] As used in this disclosure, the terms "based on" and "depending on" are used unless otherwise specified. does not mean "based only on" or "in accordance with" unless the term "based on" is used in a "Based solely on" and "based at least in part on" are both used interchangeably. Similarly, the phrase "in response to" should be interpreted as "only in response to" and "at least partially in response to." " " Also, "include" and "comprise" ) and variations thereof do not mean to include only the listed items, but This means that the item may contain only the items listed, or may contain additional items in addition to the items listed. Additionally, the term "or" as used in this disclosure does not mean exclusive or. In this disclosure, for example, words such as a, an, and the in English are used interchangeably. If articles are added by translation, these articles should be used in a way that is not obvious from the context. Unless otherwise indicated, the plural is included.

[0123] The embodiment has been described in detail above with reference to the drawings, but the specific configuration has not been described above. The invention is not limited to the above, and various design changes can be made within the scope that does not deviate from the gist of the invention. It is possible.

[0124] (6) Supplementary notes The following additional notes are about the features of the above-described embodiment.

[0125] Appendix 1 Measurement information obtained using sensors for measuring the operating state of the robot device an acquisition unit that acquires the information; a control unit that controls the operation of the robot device based on the measurement information, the sensors include a visual sensor and a force sensor; The control unit is configured to control the visual measurement information obtained by using the visual sensor. and force sense control, which is the control based on force sense measurement information obtained using the force sensor. The control ratio of the robot is dynamically or stepwise changed depending on the situation of the work using the robot device. Change Control device.

[0126] Appendix 2 The control unit a visual control unit that references the visual measurement information and generates first information indicating the content of the control; , a force-sense control unit that references the force-sense measurement information and generates second information indicating the content of the control; , generating a control command for the robot device based on the first information and the second information; a command generation unit for Weighting processing between the visual measurement information and the force measurement information according to the work situation. Alternatively, the control ratio is set by performing weighting processing between the first information and the second information. and a weighting unit for changing the weight of the 10. The control device of claim 1.

[0127] Appendix 3 The control unit changes the control ratio dynamically or stepwise as the work progresses. R 3. The control device according to claim 1 or 2.

[0128] Appendix 4 The work includes a plurality of predetermined steps, The control unit changes the control ratio for each process. 4. A control device according to any one of appendices 1 to 3.

[0129] Appendix 5 storing a setting library containing settings relating to the control ratios of each of the plurality of processes; a library storage unit for storing the The control unit changes the control ratio for each process based on the setting library. 5. The control device according to claim 4.

[0130] Appendix 6 The control unit, when transitioning from one process to the next process in the work, Gradually change from the control ratio in one step to the control ratio in the next step. Change the control ratio 6. The control device according to claim 4 or 5.

[0131] Appendix 7 The control unit A transition condition from one process to the next process in the work is determined based on the measurement information. Determine whether it is satisfied or not When the transition condition is satisfied, the process transitions from the one process to the next process. and change the control ratio to that corresponding to the next process. 7. A control device according to any one of appendices 3 to 6.

[0132] Appendix 8 The control unit is configured to determine whether the control in the next process has converged after switching to the next process. If not, return to the one process and change the control ratio to the one process. do 8. The control device according to claim 7.

[0133] Appendix 9 The control unit is configured to have a first control state in which the visual control is given priority and a second control state in which the visual control is given priority. a second control state in which the force sense control is used in cooperation with the force sense control, and a second control state in which the force sense control is used in priority; and a third control state in which the control is performed using the 9. A control device according to any one of appendices 1 to 8.

[0134] Appendix 10 The control unit controls the first control state to the third control state based on the measurement information. Determine whether to switch the control 10. The control device according to claim 9.

[0135] Appendix 11 The control unit, in the first control state, If the difference does not become equal to or less than the predetermined value, the control state is switched to the second control state or the third control state. See kinesthetic measurement information 11. The control device according to claim 9 or 10.

[0136] Appendix 12 When an instruction is received from an external device, the control unit changes the control ratio in accordance with the instruction. R 12. A control device according to any one of appendices 1 to 11.

[0137] Appendix 13 The control unit Depending on the control content performed on the robot device and the control result for the control content, a prediction unit that predicts a control result for a subsequent control content for the robot device; a correction unit that corrects the subsequent control content in accordance with the prediction. 13. A control device according to any one of appendices 1 to 12.

[0138] Appendix 14 a library storage unit that stores a plurality of setting libraries prepared for each type of work; Further preparation, each of the plurality of setting libraries includes setting information relating to each of a series of steps; The control unit selects one of the plurality of setting libraries according to the type of work to be actually performed. The control is performed using the configured setting library. 14. A control device according to any one of appendices 1 to 13.

[0139] Appendix 15 Each of the plurality of setting libraries includes setting information of the control ratio. 15. The control device of claim 14.

[0140] Appendix 16 a library storage unit that stores a plurality of recognition libraries prepared for each type of work; Further preparation, Each of the plurality of recognition libraries contains a trained model used for image recognition processing of an object. Including The control unit selects one of the plurality of recognition libraries according to the type of work to be actually performed. The image recognition process is performed using the recognition library. 16. A control device according to any one of appendices 1 to 15.

[0141] Appendix 17 Measurement information obtained using sensors for measuring the operating state of the robot device To obtain and and controlling the operation of the robot device based on the measurement information. the sensors include a visual sensor and a force sensor; The control is performed based on visual measurement information obtained using the visual sensor. and the control is based on force measurement information obtained using the force sensor. The control ratio of the force sense control and the force sense control is dynamically or gradually changed depending on the situation of the work using the robot device. Including gradual changes Control method.

[0142] Appendix 18 The control device Measurement information obtained using sensors for measuring the operating state of the robot device To obtain and and controlling the operation of the robot device based on the measurement information. the sensors include a visual sensor and a force sensor; The control is performed based on visual measurement information obtained using the visual sensor. and the control is based on force measurement information obtained using the force sensor. The control ratio of the force sense control and the force sense control is dynamically or gradually changed depending on the situation of the work using the robot device. Including gradual changes program. [Explanation of symbols]

[0143] 100: Control device 101: Processor 102: Memory 103: External I / F 110: Motion generator 120: Acquisition section 121: Image recognition unit 130: Data collection section 140: Data storage unit 150: Setting acquisition section 160: Library storage section 170: Control unit 171A: Vision control unit 171B: Force sense control unit 172: Command generation section 173: Weighting section 174A: Prediction section 174B: Correction unit 175: Derivation part 176: Goal Setting Department 200: Robotic device 210: Drive unit 211: First joint 212: Second joint 213: Third joint 214: 4th joint 215: 5th joint 216: 6th joint 221: Base 222: Link 223: Link 224: Link 225: Link 226: Gripper 300: Sensor 310: Vision sensor 320: Force sensor 330: Other sensors 400: User I / F 410:Display device 420: Operating device 510:Transportation equipment

Claims

1. Measurement information obtained using sensors for measuring the operating state of the robot device an acquisition unit that acquires the information; a control unit that controls the operation of the robot device based on the measurement information, the sensors include a visual sensor and a force sensor; The control unit is configured to control the visual measurement information obtained by using the visual sensor. and force sense control, which is the control based on force sense measurement information obtained using the force sensor. The control ratio of the robot is dynamically or stepwise changed depending on the situation of the work using the robot device. Change Control device.

2. The control unit a visual control unit that references the visual measurement information and generates first information indicating the content of the control; 、 a force-sense control unit that references the force-sense measurement information and generates second information indicating the content of the control; 、 generating a control command for the robot device based on the first information and the second information; a command generation unit for Weighting processing between the visual measurement information and the force measurement information according to the work situation. Alternatively, the control ratio is set by performing weighting processing between the first information and the second information. and a weighting unit for changing the weight of the The control device according to claim 1 .

3. The control unit changes the control ratio dynamically or stepwise as the work progresses. R The control device according to claim 1 .

4. The work includes a plurality of predetermined steps, The control unit changes the control ratio for each process. The control device according to any one of claims 1 to 3.

5. storing a setting library containing settings relating to the control ratios of each of the plurality of processes; a library storage unit for storing the The control unit changes the control ratio for each process based on the setting library. The control device according to claim 4.

6. The control unit, when transitioning from one process to the next process in the work, Gradually change from the control ratio in one step to the control ratio in the next step. Change the control ratio The control device according to claim 4.

7. The control unit A transition condition from one process to the next process in the work is determined based on the measurement information. Determine whether it is satisfied or not When the transition condition is satisfied, the process transitions from the one process to the next process. and change the control ratio to that corresponding to the next process. The control device according to claim 3 .

8. The control unit is configured to determine whether the control in the next process has converged after switching to the next process. If not, return to the one process and change the control ratio to the one process. do The control device according to claim 7.

9. The control unit is configured to: a second control state in which the force sense control is used in cooperation with the force sense control, and a second control state in which the force sense control is used in cooperation with the force sense control, and a third control state in which the control is performed using the The control device according to any one of claims 1 to 3.

10. The control unit controls the first control state to the third control state based on the measurement information. Determine whether to switch the control The control device according to claim 9.

11. The control unit is configured to, in the first control state, If the difference does not become equal to or less than the predetermined value, the control state is switched to the second control state or the third control state. See kinesthetic measurement information The control device according to claim 9.

12. When an instruction is received from an external device, the control unit changes the control ratio in accordance with the instruction. R The control device according to any one of claims 1 to 3.

13. The control unit Depending on the control content performed on the robot device and the control result for the control content, a prediction unit that predicts a control result for a subsequent control content for the robot device; a correction unit that corrects the subsequent control content in accordance with the prediction. The control device according to any one of claims 1 to 3.

14. a library storage unit that stores a plurality of setting libraries prepared for each type of work; Further preparation, each of the plurality of setting libraries includes setting information relating to each of a series of steps; The control unit selects one of the plurality of setting libraries according to the type of work to be actually performed. The control is performed using the configured setting library. The control device according to any one of claims 1 to 3.

15. Each of the plurality of setting libraries includes setting information of the control ratio. The control device according to claim 14.

16. a library storage unit that stores a plurality of recognition libraries prepared for each type of work; Further preparation, Each of the plurality of recognition libraries contains a trained model used for image recognition processing of an object. Including The control unit selects one of the plurality of recognition libraries according to the type of work to be actually performed. The image recognition process is performed using the recognition library. The control device according to any one of claims 1 to 3.

17. Measurement information obtained using sensors for measuring the operating state of the robot device To obtain and and controlling the operation of the robot device based on the measurement information. the sensors include a visual sensor and a force sensor; The control is performed based on visual measurement information obtained using the visual sensor. and the control is based on force measurement information obtained using the force sensor. The control ratio of the force sense control and the force sense control is dynamically or gradually changed depending on the situation of the work using the robot device. Including gradual changes Control method.

18. The control device Measurement information obtained using sensors for measuring the operating state of the robot device To obtain and and controlling the operation of the robot device based on the measurement information. the sensors include a visual sensor and a force sensor; The control is performed based on visual measurement information obtained using the visual sensor. and the control is based on force measurement information obtained using the force sensor. The control ratio of the force sense control and the force sense control is dynamically or gradually changed depending on the situation of the work using the robot device. Including gradual changes program.

Images