Control device, control method, and program

The control device and method enable efficient data collection for robot control by executing reverse direction motions and using visual and force sensors, addressing the challenge of environmental changes in conventional systems.

JP7583490B1Active Publication Date: 2024-11-14COLAB CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024072901
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2024-04-26
Publication Date
2024-11-14
Estimated Expiration
2044-04-26

AI Technical Summary

Technical Problem

Conventional robot control systems require re-teaching when the work environment or object changes slightly, making it difficult to efficiently collect data for controlling the robot's operation.

Method used

A control device and method that includes a motion generation unit to execute reverse direction motions, an acquisition unit to gather sensor data, and a data collection unit to repeatedly collect sets of control commands and measurement information using visual and force sensors, enabling efficient data collection for robot control.

Benefits of technology

This approach allows for more appropriate and efficient control of robot devices by collecting data that can adapt to changes in the work environment, improving operational accuracy and versatility.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007583490000001_ABST
    Figure 0007583490000001_ABST
Patent Text Reader

Abstract

This allows for more appropriate control of the robot device. [Solution] The control device includes an operation generation unit that generates a control command to execute a reverse operation that changes the operating state of the robot device from a target state to an arbitrary state different from the target state, an acquisition unit that acquires measurement information obtained using a sensor for measuring the operating state of the robot device, and a data collection unit that repeatedly collects data including a set of the control command and the measurement information while the reverse operation is being executed in accordance with the control command, and the sensor includes at least one of a visual sensor and a force sensor.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a control device, a control method, and a program for controlling a robot device. [Background technology]

[0002] Conventionally, control devices for controlling robotic devices have been widely used. In such control devices, a technique is known in which a motion state of the robotic device is measured using a visual sensor and a force sensor, and the motion state of the robotic device is controlled to be a target state (for example, see Patent Document 1).

[0003] Conventionally, a teaching method has been adopted as a method for learning the movements of a robot device, in which the robot device is manually operated and a series of movements to be executed is recorded. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7295344 Summary of the Invention [Problem to be solved by the invention]

[0005] In the conventional technology, new teaching is required when the work environment and / or the target object change even slightly, which makes it difficult to efficiently collect data that can be used to control the operation of the robot device.

[0006] The present disclosure provides a control device, a control method, and a program that enable more appropriate control of a robot device. [Means for solving the problem]

[0007] A control device according to a first aspect of the present disclosure includes an operation generating unit that generates a control command for executing a reverse operation that changes the operating state of a robotic device from a target state to an arbitrary state different from the target state, an acquisition unit that acquires measurement information obtained using a sensor for measuring the operating state of the robotic device, and a data collection unit that repeatedly collects data including a set of the control command and the measurement information while the reverse operation is being executed in accordance with the control command, and the sensor includes at least one of a visual sensor and a force sensor.

[0008] A control method according to a second aspect of the present disclosure includes generating a control command for executing a reverse operation to change an operating state of a robotic device from a target state to an arbitrary state different from the target state, acquiring measurement information obtained using a sensor for measuring the operating state of the robotic device, and repeatedly collecting data including a set of the control command and the measurement information while the reverse operation is being executed in accordance with the control command, wherein the sensor includes at least one of a visual sensor and a force sensor.

[0009] A program according to a third aspect of the present disclosure causes a control device to generate a control command for executing a reverse operation to change an operating state of a robotic device from a target state to an arbitrary state different from the target state, acquire measurement information obtained using a sensor for measuring the operating state of the robotic device, and repeatedly collect data including a set of the control command and the measurement information while the reverse operation is being executed in accordance with the control command, the sensor including at least one of a visual sensor and a force sensor. Effect of the Invention

[0010] According to one aspect of the present disclosure, it is possible to provide a control device, a control method, and a program that enable more appropriate control of a robot device. [Brief description of the drawings]

[0011] [Figure 1]FIG. 1 is a diagram illustrating an example of a system configuration of a control system including a control device according to an embodiment. [Diagram 2] 2 is a block diagram showing a functional block configuration of a control device according to the first embodiment. FIG. [Diagram 3] FIG. 2 is a diagram for explaining an example of data collection by the control device according to the first embodiment. [Figure 4] FIG. 2 is a diagram for explaining an example of data collection by the control device according to the first embodiment. [Diagram 5] FIG. 4 is a flow chart for explaining an example of data collection by the control device according to the first embodiment. [Figure 6] 4 is a diagram for explaining an example of a setting library stored in a control device according to the embodiment; FIG. [Figure 7] FIG. 11 is a block diagram showing a functional block configuration of a control device according to a second embodiment. [Figure 8] FIG. 11 is a flow diagram for explaining an example of a control flow by a control device according to a second embodiment. [Figure 9] 13 is a diagram for explaining a specific example of control during an operation of "mounting components on a board" as control by the control device according to the second embodiment. FIG. [Figure 10] FIG. 11 is a block diagram showing a functional block configuration of a control device according to a third embodiment. [Figure 11] 13 is a diagram for explaining a specific example of control during an operation of "mounting components on a board" as control by the control device according to the third embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The embodiments will be described with reference to the drawings. In the description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0013] (1) Overview of the embodiment A control device according to an embodiment includes an operation generating unit that generates a control command for executing a reverse operation that changes the operating state of a robotic device from a target state to an arbitrary state different from the target state, an acquisition unit that acquires measurement information obtained using a sensor for measuring the operating state of the robotic device, and a data collection unit that repeatedly collects data including a set of the control command and the measurement information during execution of the reverse operation in accordance with the control command, and the sensor includes at least one of a visual sensor and a force sensor.

[0014] In this way, by repeatedly collecting data including a set of the control command and measurement information while performing a reverse motion according to the control command, it becomes possible to efficiently collect data that can be used to control the motion of the robot device during actual work (i.e., forward motion). For example, during actual work using the robot device, it becomes possible to appropriately control the motion of the robot device from an arbitrary state to a target state by generating a control command from the measurement information obtained during the work based on the collected data.

[0015] The "robot device" may be any device that can operate according to a control command output by a control device, and any robot device is applicable. For example, the robot device may be an industrial robot such as a manipulator, or may include a mobile body that can move automatically. Industrial robots include, for example, vertical articulated robots, SCARA robots, parallel link robots, Cartesian robots, and collaborative robots. Mobile bodies that can move automatically include, for example, drones, vehicles that are configured to be able to drive themselves, unmanned guided vehicles, or mobile robots, and further include combinations with the above industrial robots. In the embodiment described below, an example in which the robot device is a manipulator will be mainly described.

[0016] The "control command" is related to the control of the operation of the robotic device, and is, for example, a target control amount, an operation amount, etc. "Outputting a control command" may mean directly controlling the robotic device based on the control command, or, in a case where the robotic device is equipped with a controller, outputting a control command to the controller to cause the controller to control the operation of the robotic device.

[0017] An "operational state of a robotic device" refers to a state related to the operation of a part of the configuration of the robotic device (e.g., an end effector) and / or a state related to an object of work performed using the robotic device. Also, a "measurement related to the operational state of a robotic device" refers to a measurement of a state related to the operation of a part of the configuration of the robotic device (e.g., an end effector) and / or a measurement of a state related to an object of work performed using the robotic device. An "object" is an object that may be related to the operation of the robotic device, such as a workpiece. A part of the configuration of a robotic device (e.g., an end effector) may also be considered as an object.

[0018] An "operation" is a job to be performed by a robot device, and may include multiple steps. An operation is, for example, part transport, part fitting, screwing, processing, etc. An operation may be, for example, a simple job such as gripping a workpiece and releasing the workpiece. An operation may be given in advance, or may be given by designation by an operator.

[0019] A "goal state" is a state at the time when the purpose of a task (or process) is achieved. The goal state may vary depending on the task given. For example, assuming a task in which a robot device attaches an assembly part carried by an end effector to a workpiece, the goal state may be a state in which the assembly part is attached to the workpiece.

[0020] The "arbitrary state" may be any state that is different from the target state, and may be, for example, a state at the start or midway of a task performed using the robot device.

[0021] The "measurement information" is not limited to the measurement data of the sensor itself, but may be a feature amount calculated from the measurement data, etc. The "measurement information" may be, for example, information measured or calculated using an encoder and / or a servo motor that are present as a basic component of the robot.

[0022] (2) First embodiment In the first embodiment, data collection for collecting data before the actual work to be used for control during the actual work will be mainly described. Details of the control during the actual work will be described in the second and third embodiments.

[0023] (2.1) System Configuration 1 is a diagram showing an example of the system configuration of a control system including a control device 100 according to a first embodiment. Here, the description focuses on the hardware configuration of the control system.

[0024] In the illustrated example, the robot device 200 is a manipulator. Specifically, the robot device 200 (manipulator) is a six-axis vertical articulated industrial robot, and has a base 221 and six joints 211 to 216. Each of the joints 211 to 216 incorporates a servo motor (not shown) and is configured to be rotatable about each axis.

[0025] The first joint 211 is connected to the base 221 and rotates the tip portion around the axis of the base. Instead of the base 221, a moving mechanism capable of automatically moving (self-propelled) may be provided. The second joint 212 is connected to the first joint 211 and rotates the tip portion in the front-rear direction. The third joint 213 is connected to the second joint 212 via a link 222 and rotates the tip portion in the up-down direction. The fourth joint 214 is connected to the third joint 213 via a link 223 and rotates the tip portion around the axis of the link 223. The fifth joint 215 is connected to the fourth joint 214 via a link 224 and rotates the tip portion in the up-down direction. The sixth joint 216 is connected to the fifth joint 215 via a link 225 and rotates the tip portion around the axis of the link 225. A gripper 226 is attached to the tip side of the sixth joint 216 together with a force sensor 320. The gripper 226 is an example of an end effector.

[0026] The visual sensor 310 is a sensor that performs image measurement, and is disposed so as to observe each object (gripper 226, workpiece W1, workpiece W2) present in the environment (work space) in which the robot device 200 operates. In the illustrated example, the visual sensor 310 is attached to the link 225, and the visual sensor 310 is provided integrally with the robot device 200. However, the visual sensor 310 may be fixed to equipment in the work space, and the visual sensor 310 may be provided separately from the robot device 200. For example, a camera such as a digital camera or a video camera may be used as the visual sensor 310. Measurement data (i.e., image data) of the visual sensor 310 is an example of visual measurement information.

[0027] The force sensor 320 is a sensor that performs force measurement and is configured to measure the force and moment acting on the robot device 200 (specifically, the gripper 226). The force sensor 320 may be, for example, a six-axis force sensor that measures forces in three axial directions of the X-axis, Y-axis, and Z-axis acting on the gripper 226 and moments around the X-axis, Y-axis, and Z-axis. That is, the force sensor 320 can measure the force and moment caused by contact between an assembly part supported by the robot device 200 or the gripper 226 and an article. The measurement data of the force sensor 320 may be used to adjust the gripping force of the gripper 226 or to detect whether or not an abnormal force is acting on the gripper 226. In the following description, the term "force" is used as a term that also includes the meaning of "moment."

[0028] The force sensor 320 may be realized, for example, by a method of measuring forces in three axial directions of the X-axis, Y-axis, and Z-axis and moments around the X-axis, Y-axis, and Z-axis by measuring the current values ​​of motors (not shown) built into each of the joints 211 to 216, or may be a pressure sensor provided on the surface of the robot 200 or a sensor that utilizes a change in state of a jacket provided on the surface of the robot 200. The sensor may be one that detects a change in the flow rate of air and / or liquid, a change in capacitance, etc.

[0029] Each of the joints 211 to 216 may have an encoder (not shown) built in. The encoder is an example of a sensor. The encoder is configured to be able to measure the angle (control amount) of each of the joints 211 to 216. Measurement data of the encoder may be used to control the angle of each of the joints 211 to 216.

[0030] The control system may have a transport device 510 that transports the workpiece W2. The robot device 200 can perform a task using a gripper 226 (end effector) attached to the tip of the arm. The end effector is an external device that can be replaced depending on the application, and a welding gun, a tool, or the like may be attached instead of the gripper 226. The robot device 200 can perform a task using a force sensor 320 while tracking the workpiece W2 moving on the transport device 510 using a visual sensor 310. In the illustrated example, the robot device 200 performs a task of fitting the workpiece W1, which is an assembly part gripped by the robot device 200, into a hole in the workpiece W2 (part to be assembled, for example, a board) moving on the transport device 510.

[0031] The control device 100 includes a processor 101, a memory 102, and an external interface (I / F) 103. The processor 101 includes a central processing unit (CPU). The processor 101 may include at least one of a microprocessor, a field-programmable gate array (FPGA), and a digital signal processor (DSP). The memory 102 includes a random access memory (RAM), a read only memory (ROM), and an auxiliary storage device (e.g., a hard disk drive, a solid state drive). The processor 101 and the memory 102 constitute a computer. The control device 100 may be composed of a plurality of computers. The control device 100 is not limited to an information processing device designed specifically for the service provided, and may be a general-purpose information processing device such as a personal computer (PC), or a controller such as a programmable logic controller (PLC).

[0032] The memory 102 stores a program executed by the processor 101. The processor 101, together with the memory 102, executes the program stored in the memory 102 to realize the functions of each functional block described below. Although details will be described later, the memory 102 may store, for example, a recognition library including a trained model used in image recognition for recognizing an object, and a setting library including data collected by data collection before a task and setting information acquired based on the collected data. The recognition library and the setting library may be provided for each type of task that the robot device 200 can perform. The setting information included in the setting library may include a trained model generated based on the collected data.

[0033] The external I / F 103 is, for example, a Universal Serial Bus (USB) port or a dedicated port, and is an interface for communicatively connecting to an external device. The external I / F 103 may be connected to an external device (including the robot device 200) by wire or wirelessly. The type and number of the external I / F 103 may be appropriately selected according to the type and number of the external devices to be connected. In the illustrated example, the control device 100 is connected to the robot device 200, a visual sensor 310, and a user interface (I / F) 400 via the external I / F 103. As shown in FIG. 2, the user I / F 400 includes a display device 410 and an operation device 420. In the illustrated example, the user I / F 400 is provided separately from the control device 100, but the user I / F 400 may be provided integrally with the control device 100. The display device 410 may be a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like. The display device 410 may be a display equipped with a speaker. The operation device 420 is a device for performing operation input, such as a keyboard, a mouse, a touch panel, etc. The display device 410 and the operation device 420 may be integrated as a touch panel display. An operator can check the state of the control device 100 and operate the control device 100 by using the display device 410 and the operation device 420.

[0034] (2.2) Functional block configuration of the control device FIG. 2 is a block diagram showing a functional block configuration of the control device 100 according to the first embodiment. In the first embodiment, a functional block configuration related to data collection for collecting data to be used for control during work before the work is performed will be mainly described. Such data collection may be performed before a user (including an operator) starts using the control device 100 (and the robot device 200). For example, data collection may be performed in advance before the shipment of the control device 100 (and the robot device 200).

[0035] The control device 100 includes an action generating unit 110, an acquiring unit 120, a data collecting unit 130, a data storing unit 140, a setting acquiring unit 150, and a library storing unit 160. In this embodiment, the control device 100 may or may not include a control unit 170.

[0036] The motion generating unit 110 generates a control command for executing a forward motion for changing the motion state of the robotic device 200 from an arbitrary state to a target state. The motion generating unit 110 also generates a control command for executing a reverse motion for changing the motion state of the robotic device 200 from a target state to an arbitrary state different from the target state. The motion generating unit 110 outputs the generated control command to a driving unit 210 of the robotic device 200. The driving unit 210 includes servo motors provided in the joints 211 to 216 of the robotic device 200. The driving unit 210 may include a controller on the robotic device 200 side. The driving unit 210 drives the servo motors in accordance with the control command to operate the robotic device 200.

[0037] Here, the goal state refers to a state that is realized when the purpose of a task (or process) is achieved, or an intermediate state of the task. In the first embodiment, a series of tasks is assumed in which the robot device 200 attaches an assembly part (workpiece W1) carried by an end effector (gripper 226) to a sub-assembly part (workpiece W2), so the goal state is a state in which the assembly part is attached to the sub-assembly part, or an intermediate state before attachment. Specifically, the goal state is a state in which the workpiece W1 is fitted into a hole in the workpiece W2, or an intermediate state in which the workpiece W1 is in contact with the workpiece W2 at a position where it can be fitted into the hole.

[0038] The arbitrary state is a state different from the target state, for example, a state in which the workpiece W1 is located away from the hole in the workpiece W2. The arbitrary state may be set by operation input (user input) via the operation device 420. For example, the position of the arbitrary state may be set by operation input (user input) based on the position of the target state. Note that the term "position" may include not only the meaning of "coordinates" but also the meaning of "posture."

[0039] The acquisition unit 120 acquires measurement information obtained using a sensor 300 for measuring the operating state of the robot device 200. In the illustrated example, the sensor 300 includes a visual sensor 310 and a force sensor 320. The sensor 300 may further include another sensor 330 such as an encoder. The acquisition unit 120 may include an image recognition unit 121 that performs image recognition on the measurement data (i.e., image data) output by the visual sensor 310. The image recognition unit 121 may recognize an object (e.g., the gripper 226, the workpiece W1, the workpiece W2) by image recognition such as feature extraction, and acquire the position of the object (e.g., the coordinates of the feature part). Such position information is an example of visual measurement information.

[0040] The data collection unit 130 repeatedly collects data including a set of the control command output by the motion generation unit 110 and the measurement information acquired by the acquisition unit 120 during the execution of the reverse motion according to the control command output by the motion generation unit 110. This makes it possible to efficiently collect data that can be used to control the motion (forward motion) of the robot device 200 during actual work. For example, during actual work using the robot device 200, it becomes possible to appropriately control the motion of the robot device from an arbitrary state to a target state by generating a control command from the measurement information obtained during work based on the collected data. Note that the data collection unit 130 may repeatedly collect data including a set of the control command and the measurement information acquired by the acquisition unit 120 during the execution of the forward motion according to the control command output by the motion generation unit 110.

[0041] The motion generating unit 110 may generate control commands for executing a plurality of patterns of reverse motions that change the motion state of the robot device 200 from a target state to a plurality of arbitrary states different from each other. The data collecting unit 130 may repeatedly collect data for each of the plurality of patterns of reverse motions. This allows a plurality of patterns of data corresponding to a plurality of movement paths to be collected, thereby providing versatility to the motion control of the robot device 200 during actual work. For example, even if the workpiece W2 is not fixed or moves during work, it becomes easy to attach the workpiece W1 to the workpiece W2.

[0042] The motion generation unit 110 may generate a control command that collects more data in an area closer to the target state, and may generate a control command that collects less data in an area farther from the target state. The closer an area is to the target state, the more precise and accurate control is required during actual work. By increasing the amount of data collected in an area closer to the target state, it is possible to collect a sufficient amount of data to perform precise and accurate control. On the other hand, in an area farther from the target state, precise and accurate control is not so required. Therefore, by generating a control command that collects less data in an area farther from the target state, unnecessary data collection is suppressed, enabling efficient data collection.

[0043] The acquisition unit 120 shortens the acquisition period (e.g., sampling frequency) so that the amount of acquired measurement information increases in an area closer to the target state, and decreases in an area farther from the target state. To be less The acquisition period may be extended. This process also makes it possible to collect more data in an area closer to the target state and less data in an area farther from the target state.

[0044] The acquisition unit 120 may acquire, as relative measurement information, a difference between target measurement information obtained in the target state and current measurement information obtained during the execution of the reverse direction operation. The data collection unit 130 may repeatedly collect data including a set of the control command and the relative measurement information during the execution of the reverse direction operation according to the control command. For example, the acquisition unit 120 may acquire, as relative position information, a difference between visual measurement information (target position) of the object obtained by the visual sensor 310 in the target state and current measurement information (current position) of the object obtained during the execution of the reverse direction operation. In the environment of FIG. 1, the acquisition unit 120 may acquire, as a relative position, a difference between the target position and each current position, with the position of the hole in the workpiece W2 as the target position and each position on the movement path of the workpiece W1 as the current position. Thereby, a control command is associated with each relative positional relationship between the target position and the current position of the object. Therefore, during actual work, an appropriate control command can be generated from the relative positional relationship between the target position and the current position of the object. Furthermore, control using relative positional relationships is applicable even when the target object moves during work.

[0045] The acquisition unit 120 may acquire measurement information including a speed-related value related to the robot device 200 in response to the output of the sensor 300 or a control command. The speed-related value related to the robot device 200 may be at least one of the speed, acceleration, and jerk of the object. The acquisition unit 120 may derive the speed, acceleration, and jerk of the object from visual measurement information (position information of the object) acquired by the visual sensor 310. The acquisition unit 120 may derive the speed, acceleration, and jerk of the object from a control command output to the drive unit 210. The acquisition unit 120 may derive the speed, acceleration, and jerk of the object from measurement information acquired by another sensor 330 (e.g., an encoder). The data collection unit 130 may repeatedly collect data including a set of the control command and the measurement information including the speed-related value during the execution of the reverse direction operation according to the control command. The speed-related value collected in this manner can be used when setting a speed-related target value in the control during the actual work. The control using the speed-related target value will be described in the third embodiment.

[0046] The data storage unit 140 stores the data collected by the data collection unit 130. The data collected by the data collection unit 130 includes a plurality of sets of measurement information (relative measurement information) and control commands. Each set may include measurement information (relative measurement information), a speed-related value, and a control command. The data collected by the data collection unit 130 may include more sets of data in an area closer to the target state.

[0047] The setting acquisition unit 150 acquires setting information for performing a task using the robotic device 200 based on the data collected by the data collection unit 130 (specifically, the data stored in the data storage unit 140). Setting information for performing a certain task is referred to as a "setting library" for that task. For example, the setting acquisition unit 150 acquires the setting library after data collection by the data collection unit 130 is completed. The library storage unit 160 stores the setting library acquired by the setting acquisition unit 150. The control unit 170 controls the robotic device 200 using the setting library during actual work.

[0048] The setting library includes operation parameters for each work step, and information on the transition destination and transition condition for each step (such as normal value, timeout value, and error value of the transition condition). The operation parameters may include correspondence information that associates measurement information (relative measurement information) with a control command, or may include correspondence information that associates measurement information (relative measurement information), a speed relation value, and a control command. The operation parameters may include a control ratio between visual control, which is robot control based on visual measurement information obtained by the visual sensor 310, and force control, which is robot control based on force measurement information obtained by the force sensor 320. The operation parameters may include information on the type of object (metal, resin, screw, connector, etc.). The operation parameters may include information on features on the object (holes, edge surfaces, connectors, etc.). The setting acquisition unit 150 may acquire the setting library at least partially based on an operation input performed via the operation device 420.

[0049] The setting acquisition unit 150 may use the data collected by the data collection unit 130 as learning data to acquire a learned model for deriving a control command from measurement information by machine learning. In this case, the control unit 170 uses a setting library including the learned model during actual work to control the robot device 200 based on measurement information obtained during work. The type of the learned model (learning model) is not particularly limited as long as the ability to make inferences for generating control commands can be acquired by machine learning. The type of machine learning is not particularly limited, but is typically supervised learning or reinforcement learning. The learning model may be configured, for example, by a neural network such as a deep neural network (DNN). The learning model may be configured, for example, by a value function such as a state value function or an action value function. In this way, the setting acquisition unit 150 determines a calculation for inferring an optimal value, enables inference, and generates an optimal value variable of a control command by machine learning.

[0050] The setting acquisition unit 150 may acquire a setting library including the control ratio between visual control and haptic control for each process, for example. In this case, the control unit 170 uses the setting library during actual work to dynamically or stepwise change the control ratio according to the status of the work. Details of such control will be described in the second embodiment.

[0051] The setting acquisition unit 150 may acquire a setting library including a speed-related value related to the robotic device 200, for example, for each process. In this case, the control unit 170 may derive the speed-related value related to the robotic device 200 from measurement information or a control command obtained during work. Then, the control unit 170 may repeatedly perform control on the robotic device 200 to reduce the difference between the speed-related value and the speed-related target value, based on the setting library and the measurement information obtained during work. Details of such control will be described in the third embodiment.

[0052] The motion generation unit 110 may generate a control command for executing a reverse motion for each type of task performed by the robot device 200. Assuming the environment of FIG. 1, the task type is "mounting components on a board." Other examples of task types include "packing food into boxes," "screw tightening," "pick and place," and "AGV (Automated Guided Vehicle) control." The data collection unit 130 may collect data for each task type by repeatedly collecting data for each task type.

[0053] In this case, the setting acquisition unit 150 can acquire a setting library for each type of work based on the data collected by the data collection unit 130 for each type of work. Then, the control unit 170 controls the robot device 200 to perform the work by using a setting library corresponding to the type of work to be actually performed. For example, the control unit 170 may display a list of setting libraries stored in the library storage unit 160 on the display device 410, and control the robot device 200 by using a setting library selected from the list using the operation device 420. Note that, when the library storage unit 160 also stores a recognition library in association with the setting library, image recognition for performing the work may be performed by using the recognition library corresponding to the type of work to be actually performed.

[0054] (2.3) An example of data collection 3 and 4 are diagrams for explaining an example of data collection by the control device 100 according to the first embodiment.

[0055] As described above, the motion generating unit 110 generates control commands for executing a plurality of patterns of reverse motions that change the motion state of the robot device 200 from a target state to a plurality of arbitrary states different from each other. The data collecting unit 130 repeatedly collects data for each of the plurality of patterns of reverse motions. As shown in FIG. 3, the motion generating unit 110 generates control commands that collect more data in an area closer to the target state, and generates control commands that collect less data in an area farther from the target state. In FIG. 3, the x-axis and y-axis indicate directions that are orthogonal to each other in a horizontal plane, and the z-axis indicates a vertical direction. Also, in FIG. 3, the amount of collected data is indicated by shading. Specifically, the more the amount of collected data, the darker the color, and the less the amount of collected data, the lighter the color.

[0056] In the example of FIG. 4, the motion generating unit 110 causes the robot device 200 to execute reverse motions to any state of eight patterns P1 to P8 in total, with the state in which the workpiece W1 (component) is in the hole of the workpiece W2 (substrate) as the target state. Although the movement path of each pattern P1 to P8 is illustrated as being linear, each movement path does not have to be linear. For example, the movement path may be such that the workpiece W1 (component) is moved horizontally or diagonally upward after being pulled out of the hole of the workpiece W2 (substrate). In addition, among each pattern P1 to P8, the movement of P2, P4, and P8 ends in the region R close to the target state. As a result, the amount of data collected increases as the point approaches the position of the target state.

[0057] (2.4) Specific examples of the operation of the control device FIG. 5 is a flow diagram for explaining an example of data collection by the control device 100 according to the first embodiment.

[0058] In step S101, a user (e.g., a worker) moves an object (e.g., a workpiece W1 grasped or supported by an end effector) to a target state. The motion generation unit 110 also acquires sequence information of a control command corresponding to any one of a plurality of reverse motion patterns (i.e., any arbitrary state) and / or information on the position of the arbitrary state. Here, the acquisition unit 120 acquires the position of the object in the target state as a target position from visual measurement information acquired using the visual sensor 310. The acquisition unit 120 acquires the force (reaction force) applied to the object in the target state as a target force (target reaction force) from force measurement information acquired using the force sensor 320.

[0059] In step S102, the motion generator 110 outputs a control command to the robot device 200 (driver 210) based on the information acquired in step S101, thereby moving the target object by a predetermined movement amount.

[0060] In step S103, the acquisition unit 120 acquires current measurement information. The acquisition unit 120 acquires the current position of the object from visual measurement information acquired using the visual sensor 310. The acquisition unit 120 may acquire the difference between the current position of the object and the target position as a relative position (position change amount). The acquisition unit 120 also acquires a current force (reaction force) from force measurement information acquired using the force sensor 320. The acquisition unit 120 may acquire the difference between the current force (reaction force) and the target force (reaction force) as a relative force (force change amount). Furthermore, the acquisition unit 120 may acquire a velocity-related value (velocity, acceleration, and jerk) of the object from the current visual measurement information or the control command of step S102.

[0061] In step S104, the data collecting unit 130 collects a set of the control command in step S102 and the information acquired in step S103 (position change amount, force change amount, velocity related value), and causes the data storage unit 140 to store the set.

[0062] If the movement of the object to the arbitrary state corresponding to the current reverse motion pattern is not completed (step S105: NO), the process returns to step S102, and the motion generating unit 110 outputs a control command to the robot device 200 (drive unit 210) to move the object by a predetermined movement amount. Then, in step S103, the acquiring unit 120 acquires information (position change amount, force change amount, speed related value). In step S104, the data collecting unit 130 acquires a set of the control command of step S102 and the information acquired in step S103 (position change amount, force change amount, speed related value), and stores the set in the data storage unit 140. This process is repeated until the movement to the arbitrary state corresponding to the current reverse motion pattern is completed.

[0063] On the other hand, if the movement of the object to the arbitrary state corresponding to the current reverse motion pattern is completed (step S105: YES), the motion generation unit 110 checks whether the motions are completed for all of the multiple reverse motion patterns. If the motions are not completed for all patterns (step S106: NO), the process moves to the next reverse motion pattern (step S107) and resumes the process from step S101.

[0064] If all patterns have been completed (step S106: YES), in step S108, the setting acquisition unit 150 determines operation conditions (operation parameters) for performing a task using the robot device 200 based on the data collected by the data collection unit 130 (specifically, the data stored in the data storage unit 140). Then, in step S109, the setting acquisition unit 150 stores setting information including the operation conditions (operation parameters) determined in step S108 in the library storage unit 160 as a setting library.

[0065] Note that a setting library for each type of work may be stored in the library storage unit 160 by executing the flow shown in FIG.

[0066] (2.5) An example of a configuration library FIG. 6 is a diagram for explaining an example of a setting library stored in the control device 100 according to the embodiment.

[0067] In the illustrated example, the types of work include "mounting components on a circuit board," "packing food into boxes," "tightening screws," "pick and place" (also known as "picking"), and "AGV control." Each work includes multiple steps.

[0068] For example, in a task type of "mounting components on a board", the steps are performed in the following order: step 1 "recognize components and move" → step 2 "hold components" → step 3 "move components toward board" → step 4 "approach" → step 5 "recognize holes on board" → step 6 "approach holes" → step 7 "copy holes" → step 8 "insert holes" → step 9 "complete inserting holes" → step 10 "release grip". On the other hand, the control device 100 may control the operation of the robot device 200 in the reverse order during data collection. In the case of "AGV control", a step of avoiding obstacles may be further included. The step of avoiding obstacles may be included in step 4 "move object toward target".

[0069] The setting library includes operation parameters for each work process, and information on the transition destination and transition conditions for each process (normal value, timeout value, error value, etc. of the transition condition). The operation parameters may include correspondence information that associates measurement information (relative measurement information) with a control command, or may include correspondence information that associates measurement information (relative measurement information), a speed relationship value, and a control command. The operation parameters may include a control ratio between visual control and force control. The operation parameters may include information on the type of object (metal, resin, screw, connector, etc.). The operation parameters may include information on features on the object (holes, edges, connectors, etc.).

[0070] (3) Second embodiment The second embodiment will be described mainly with respect to differences from the first embodiment. The system configuration of the second embodiment is similar to that of the first embodiment (see FIG. 1). Note that the second embodiment will be described mainly as an example in which the second embodiment is based on the first embodiment, but the second embodiment does not necessarily have to be based on at least a part of the first embodiment.

[0071] (3.1) Functional block configuration of the control device FIG. 7 is a block diagram showing a functional block configuration of a control device 100 according to the second embodiment.

[0072] The control device 100 according to the second embodiment includes an acquisition unit 120 that acquires measurement information obtained using a sensor 300 for measuring the operating state of the robot device 200, and a control unit 170 that controls the operation of the robot device 200 based on the measurement information. The sensor 300 includes a visual sensor 310 and a force sensor 320. The control unit 170 dynamically or stepwise changes the control ratio between visual control, which is control based on visual measurement information obtained by the visual sensor 310, and force control, which is control based on force measurement information obtained by the force sensor 320, depending on the situation of the work using the robot device 200. This makes it possible to utilize the advantages of both visual control and force control, and appropriately control the operation of the robot device 200 by achieving both visual control and force control.

[0073] In the second embodiment, the control unit 170 includes a visual control unit 171A that generates first information indicating the content of control of the robot device 200 by referring to visual measurement information, a force-sense control unit 171B that generates second information indicating the content of control of the robot device 200 by referring to force-sense measurement information, a command generation unit 172 that generates a control command for the robot device 200 based on the first information and the second information, and a weighting unit 173 that changes the control ratio by performing weighting processing between the visual measurement information and the force-sense measurement information or between the first information and the second information according to the status of the work using the robot device 200. This makes it possible to appropriately change the control ratio by the weighting processing.

[0074] The visual control performed by the visual control unit 171A may be performed based on, for example, the operation parameters (association information, etc.) in the setting library. The visual control unit 171A identifies the relative positional relationship between the current position and the target position of the object based on the visual measurement information obtained using the visual sensor 310. Then, the visual control unit 171A may generate a visual control command from the identified positional relationship using the association information to reduce the difference between the current position and the target position (i.e., move the object closer to the target position), and output this visual control command as the first information. At least a part of such visual control may be performed using a trained model.

[0075] Similarly, the haptic control performed by the haptic control unit 171B may be performed based on, for example, the operation parameters (association information, etc.) in the setting library. The haptic control unit 171B may identify the difference between the current force (current reaction force) of the object and the target force (target reaction force) based on the haptic measurement information obtained using the force sensor 320, generate a haptic control command from the identified difference to reduce the difference using the association information, and output this haptic control command as the second information. At least a part of such haptic control may be performed using a trained model.

[0076] The control unit 170 (weighting unit 173) may dynamically or stepwise change the control ratio as the work using the robotic device 200 progresses. As described above, the work using the robotic device 200 may include a plurality of predetermined steps. The control unit 170 may change the control ratio for each step. This makes it possible to control the operation of the robotic device 200 with an appropriate control ratio for each step.

[0077] The library storage unit 160 may store a setting library (setting information) including settings related to the control ratios of each of a plurality of processes. The control unit 170 (weighting unit 173) may change the control ratio for each process based on the setting library. This allows an appropriate control ratio to be set for each process.

[0078] The control unit 170 (weighting unit 173) gradually changes the control ratio from the control ratio in one process to the control ratio in the next process when a task transitions from one process to the next process. This prevents abrupt changes in the control ratio, enabling precise operation control and suppressing the occurrence of operation errors.

[0079] For example, in a case where the control ratio (visual control: force sense control) of process A is set to "80:20" and the control ratio of process B following process A is set to "50:50", when a transition condition from process A to process B is satisfied, the control unit 170 (weighting unit 173) changes the control ratio stepwise, such as "80:20" → "75:25" → "70:30" → "65:35" → "60:40" → "55:45" → "50:50". Alternatively, the control unit 170 (weighting unit 173) may change the control ratio continuously, such as "80:20" → "79:21" → "78:22" → "77:22" → ... Such a change in the control ratio may be performed in units of a control cycle, or may be performed in units of a time consisting of a plurality of control cycles.

[0080] The control unit 170 (weighting unit 173) may determine whether or not a transition condition from one process to the next process in a task is satisfied based on measurement information obtained using the sensor 300. The transition condition may be included as one of the operation parameters in the setting library. In response to the satisfaction of the transition condition, the control unit 170 (weighting unit 173) may transition from the one process to the next process and change the control ratio corresponding to the next process.

[0081] If the control in the next step does not converge after switching to the next step, the control unit 170 may return to the step and change the control ratio to the control ratio corresponding to the step. In this way, if the control does not converge, it is expected that the control will converge when transitioning to the next step by returning to the previous step and repeating the operation. Note that "if the control does not converge" may mean that at least one of the following conditions is satisfied: the measurement information does not satisfy the normal value of the transition condition, a timeout has occurred, or the measurement information has become an error value.

[0082] The control unit 170 (weighting unit 173) may switch the control of the robot device 200 between a first control state (also referred to as "visual-based") in which visual control is given priority, a second control state (also referred to as "visual+force-based") in which visual control and force-sense control are used in cooperation, and a third control state (also referred to as "force-sense-based") in which force-sense control is given priority. "Control with visual control given priority" may mean, for example, that the control ratio occupied by visual control is about 65% to 100%. "Control with visual control and force-sense control used in cooperation" may mean, for example, that the visual control:force-sense control ratio is about 50:50. "Control with force-sense control given priority" may mean, for example, that the control ratio occupied by force-sense control is about 65% to 100%. The control unit 170 may determine switching of control among the first control state, second control state, and third control state based on the measurement information obtained using the sensor 300. When the difference between the measurement information and the target value of the measurement information does not become equal to or less than a predetermined value in the first control state (which may be the case when the transition condition is not satisfied), the control unit 170 (weighting unit 173) may switch to the second control state or the third control state and refer to the force sense measurement information.

[0083] When an instruction is given from the outside, the control unit 170 (weighting unit 173) may change the control ratio in response to the instruction. For example, when an instruction to change the control ratio is given via the operation device 420, the control unit 170 (weighting unit 173) may change the control ratio in response to the instruction.

[0084] The control unit 170 may further include a prediction unit 174A that predicts a control result for a subsequent control operation for the robot device 200 according to the control operation performed on the robot device 200 and the control result (sensor information) for the control operation, and a correction unit 174B that corrects the subsequent control operation according to the prediction. For example, when the robot device 200 does not perform an operation as specified by a control command due to an external factor such as an error factor on the robot device 200 side or an error factor on the sensor 300 side, the prediction unit 174A identifies the error (i.e., the difference between the content of the control command and the control result) and predicts the control result for the subsequent control operation. Then, the correction unit 174B may correct the control command generated by the command generation unit 172 according to the prediction result of the prediction unit 174A and output it to the drive unit 210. For example, the correction unit 174B may correct the control command so as to cancel the identified error. This makes it possible to perform more accurate robot control even when there is an external error factor.

[0085] The library storage unit 160 may store a plurality of setting libraries prepared for each type of work. Each of the plurality of setting libraries may include setting information (operation parameters) of the control ratio. Each of the plurality of setting libraries includes setting information related to each of a series of processes. The control unit 170 controls the robot device 200 using a setting library selected from the plurality of setting libraries according to the type of work to be actually performed. The selection may be performed by an operation input via the operation device 420. Prior to such an operation input, the acquisition unit 120 (image recognition unit 121) may estimate the work to be actually performed based on visual measurement information obtained by the visual sensor 310 observing the work site or the target object. The control unit 170 may suggest the setting library corresponding to the estimated work to the user (operator) by displaying it on the display device 410.

[0086] The library storage unit 160 may further store a plurality of recognition libraries prepared for each type of work. Each of the plurality of recognition libraries may include a trained model used for image recognition processing of an object. The control unit 170 performs image recognition processing using a recognition library selected from the plurality of recognition libraries according to the type of work to be actually performed. The selection may be performed by an operation input via the operation device 420. Prior to such an operation input, the acquisition unit 120 (image recognition unit 121) may estimate the work to be actually performed based on visual measurement information obtained by observing the work site or the object with the visual sensor 310. The control unit 170 may suggest the recognition library corresponding to the estimated work to the user (operator) by displaying it on the display device 410.

[0087] (3.2) Operation flow of the control device FIG. 8 is a flow diagram for explaining an example of a control flow by the control device 100 according to the second embodiment.

[0088] In step S201, the control unit 170 starts a process n. At the start of the work, the value of n is "1." At that time, the control unit 170 applies the setting information (operation parameters including the control ratio) corresponding to the process n.

[0089] In step S202, the control unit 170 controls the operation of the robot device 200 by applying the setting information (operation parameters including the control ratio) corresponding to the process n.

[0090] In step S203, the control unit 170 determines whether or not the transition condition from process n to the next process is satisfied (i.e., whether or not process n is completed). If process n is completed (step S203: YES), the process proceeds to step S204. On the other hand, if process n is not completed (step S203: NO), in step S205, the control unit 170 determines whether or not the control of process n has converged. If it is determined that the control of process n has converged (step S205: YES), the process returns to step S202.

[0091] When it is determined that the control of the process n does not converge (step S205: NO), the control unit 170 returns to the process (n-1) before the process n. At that time, the control unit 170 may gradually change the control ratio from the control ratio of the process n to the control ratio of the process (n-1). Alternatively, instead of returning to the process (n-1), the control unit 170 may change at least a part of the operation parameters of the process n according to a predetermined rule and then resume the process n. For example, when it is estimated that the speed-related target value is too high, the speed-related target value may be lowered by one step and the process n may be resumed.

[0092] In step S204, the control unit 170 determines whether or not all steps of the work have been completed. If all steps have been completed, this flow ends. If all steps have not been completed (step S204: NO), in step S207, the control unit 170 proceeds to step (n+1) next to step n. At that time, the control unit 170 may gradually change the control ratio from the control ratio of step n to the control ratio of step (n+1).

[0093] (3.3) Specific examples of the operation of the control device FIG. 9 is a diagram for explaining a specific example of control performed by the control device 100 according to the second embodiment during the operation of "mounting components on a board."

[0094] For the task type "mounting components on board," the steps are performed in the following order: process 1 "recognize and move component" → process 2 "grasp component" → process 3 "move component toward board" → process 4 "approach" → process 5 "recognize holes on board" → process 6 "approach hole" → process 7 "hole tracing operation" → process 8 "insert hole" → process 9 "hole insertion completed" → process 10 "release gripping," and the operating parameters for each of these processes are included in the setting library.

[0095] Of these processes, basically, force-based control is set for processes involving contact between objects (end effector (gripper 226), component (workpiece W1), board (workpiece W2)), visual-based control is set for processes in which objects (end effector, component) are moved at a certain speed or faster, and visual+force-based control is set for processes in which objects are moved at a low speed toward contact between the objects. However, it is preferable that the specific control ratios for each process are set based on data collected in the data collection according to the first embodiment.

[0096] In the illustrated example, vision-based control is applied to step 1 "recognize part → move". However, when the position of the part recognized by image recognition is set as the target position and the current position of the end effector approaches the vicinity of the target position, vision + force-based control may be applied. Force-based control is applied to step 2 "grasp part".

[0097] Vision-based control is applied to process 3, "Move the component towards the board." Vision-based control is applied to process 4, "Approach." However, when the current position of the component approaches the vicinity of the target position, which is determined by the position of the board recognized by image recognition, vision- and force-based control may be applied. Vision-based control is applied to process 5, "Recognize holes in the board."

[0098] Vision + force-based control is applied to step 6 "Hole approach". Force-based control is applied to steps 7 "Hole tracing" to 10 "Grip release".

[0099] (4) Third embodiment The third embodiment will be described mainly with respect to the differences from the first and second embodiments. The system configuration according to the third embodiment is similar to that of the first embodiment (see FIG. 1). Note that the third embodiment will be described mainly as an example in which the third embodiment is based on the first and second embodiments, but the third embodiment does not necessarily have to be based on at least a part of the first and second embodiments.

[0100] (4.1) Functional block configuration of the control device FIG. 10 is a block diagram showing a functional block configuration of a control device 100 according to the third embodiment.

[0101] The control device 100 according to the third embodiment includes an acquisition unit 120 that acquires measurement information obtained using a sensor 300 for measuring the operating state of the robot device 200, and a control unit 170 that repeatedly generates a control command for the robot device 200 for each control period based on the measurement information. The control unit 170 derives a speed-related value for the robot device 200 from the measurement information or the control command, and generates a control command so as to reduce a difference between a speed-related target value that is variable for each control period according to the operating state of the robot device 200 and the speed-related value. This enables control that does not cause an operation error when working with the robot device 200. In addition, by making the speed-related target value variable for each control period according to the operating state (work situation) of the robot device 200, it becomes possible to handle precise work involving nonlinear operations. As described above, the speed-related value includes at least one of the speed, acceleration, and jerk.

[0102] In the third embodiment, the control unit 170 has a derivation unit 175 that derives a speed-related value from measurement information or a control command for each control cycle, a target setting unit 176 that variably sets a speed-related target value for each control cycle according to the operation state (work situation) of the robot device 200, and a command generation unit 172 that generates a control command for each control cycle so as to reduce a difference between the speed-related value and the speed-related target value. The target setting unit 176 may variably set the speed-related target value according to a speed-related value included in an operation parameter in the setting library.

[0103] The command generating unit 172 may incorporate the functions of the visual control unit 171A, the force-sense control unit 171B, and the weighting unit 173 described in the second embodiment. Alternatively, the control unit 170 may be provided with the visual control unit 171A, the force-sense control unit 171B, and the weighting unit 173 described in the second embodiment, in addition to the command generating unit 172.

[0104] The control unit 170 (command generating unit 172) generates a control command to reduce the difference between the measurement information and target measurement information (target value of the measurement information) and to reduce the difference between the speed-related value and the speed-related target value.

[0105] For example, the control unit 170 (command generating unit 172) may generate a control command to reduce the difference between the visual measurement information (current position) obtained using the visual sensor 310 and the target visual measurement information (target position), and to reduce the difference between the speed related value and the speed related target value. Here, the control to reduce the difference between the visual measurement information (current position) and the target visual measurement information (target position) is similar to the visual control described in the second embodiment.

[0106] The control unit 170 (command generating unit 172) may generate a control command to reduce the difference between the force sense measurement information (current force (reaction force)) and the target force sense measurement information (target force (reaction force)) obtained using the force sensor 320, and to reduce the difference between the velocity related value and the velocity related target value. Here, the control to reduce the difference between the force sense measurement information (current force (reaction force)) and the target force sense measurement information (target force (reaction force)) is similar to the force sense control described in the second embodiment.

[0107] The control unit 170 (target setting unit 176) may change the speed-related target value according to the difference (current relative measurement information) between the current measurement information and the target measurement information. For example, the control unit 170 (target setting unit 176) may set the speed-related target value corresponding to the current relative measurement information by using association information that associates the measurement information (relative measurement information) with the speed-related value, which is included in the setting library.

[0108] For example, the control unit 170 (goal setting unit 176) may change the speed-related target value in accordance with the difference between current visual measurement information and target visual measurement information obtained using the visual sensor 310. The control unit 170 (goal setting unit 176) may change the speed-related target value in accordance with the difference between current force-sense measurement information and target force-sense measurement information obtained using the force sensor 320.

[0109] The control unit 170 (derivation unit 175 and goal setting unit 176) may select one or more values ​​to be used as the speed-related value (and the speed-related target value) from among speed, acceleration, and jerk, depending on the difference between the measurement information and the target measurement information (current relative measurement information). For example, the control unit 170 (derivation unit 175 and goal setting unit 176) may select speed as the speed-related value (and the speed-related target value) during visual-based control. On the other hand, during force-based control, the control unit 170 may select jerk as the speed-related value (and the speed-related target value). During visual+force-based control, the control unit 170 may select acceleration as the speed-related value (and the speed-related target value).

[0110] As in the second embodiment, the control unit 170 may have a prediction unit 174A that predicts a control result for a subsequent control operation performed on the robotic device 200 based on the control operation performed on the robotic device 200 and the control result for the control operation, and a correction unit 174B that corrects the subsequent control operation based on the prediction.

[0111] The library storage unit 160 may store a plurality of setting libraries prepared for each type of work. Each of the plurality of setting libraries may include setting information of speed-related values ​​(speed-related target values). The control unit 170 controls the robot device 200 using a setting library selected from the plurality of setting libraries according to the type of work to be actually performed. The selection may be performed by an operation input via the operation device 420. Prior to such an operation input, the acquisition unit 120 (image recognition unit 121) may estimate the work to be actually performed based on visual measurement information obtained by the visual sensor 310 observing the work site or the target object. The control unit 170 may suggest the setting library corresponding to the estimated work to the user (operator) by displaying it on the display device 410.

[0112] The library storage unit 160 may further store a plurality of recognition libraries prepared for each type of work. Each of the plurality of recognition libraries may include a trained model used for image recognition processing of an object. The control unit 170 performs image recognition processing using a recognition library selected from the plurality of recognition libraries according to the type of work to be actually performed. The selection may be performed by an operation input via the operation device 420. Prior to such an operation input, the acquisition unit 120 (image recognition unit 121) may estimate the work to be actually performed based on visual measurement information obtained by observing the work site or the object with the visual sensor 310. The control unit 170 may suggest the recognition library corresponding to the estimated work to the user (operator) by displaying it on the display device 410.

[0113] (4.2) Specific examples of the operation of the control device FIG. 11 is a diagram for explaining a specific example of control performed by the control device 100 according to the third embodiment during the operation of "mounting components on a board."

[0114] As described above, for the task type "mounting components on board," the steps are performed in the following order: step 1 "recognize and move components" → step 2 "grasp components" → step 3 "move components toward board" → step 4 "approach" → step 5 "recognize holes on board" → step 6 "approach holes" → step 7 "track holes" → step 8 "insert holes" → step 9 "hole insertion completed" → step 10 "release gripping," and the operating parameters for each of these steps are included in the setting library.

[0115] In each step, the control unit 170 (command generating unit 172) generates a control command to reduce the difference between the measurement information obtained using the sensor 300 and the target measurement information (target value of the measurement information), and to reduce the difference between the speed-related value and the speed-related target value.

[0116] In the illustrated example, in step 1 "recognize component→move", "component position" may be applied as the target measurement information, and "speed" may be applied as the speed-related target value. In this case, the control unit 170 (command generating unit 172) may generate a control command to reduce the difference between the current position of the end effector obtained using the visual sensor 310 and the position of the component, and to reduce the difference between the speed of the end effector and the speed target value. Here, the control unit 170 (target setting unit 176) may change the speed target value for each control cycle according to the difference between the current position of the end effector obtained using the visual sensor 310 and the position of the component.

[0117] In step 2 "part gripping", "target reaction force" may be applied as the target measurement information, and "speed" may be applied as the speed-related value. In this case, the control unit 170 (command generating unit 172) may generate a control command to reduce the difference between the target reaction force and the current reaction force acting on the end effector obtained using the force sensor 320, and to reduce the difference between the speed of the end effector and the speed target value. Here, the control unit 170 (target setting unit 176) may change the speed target value for each control cycle according to the difference between the target reaction force and the reaction force acting on the end effector obtained using the force sensor 320.

[0118] In step 3 "moving the component toward the board", "position of the board" may be applied as the target measurement information, and "jerk" may be applied as the speed-related value. In this case, the control unit 170 (command generating unit 172) may generate a control command to reduce the difference between the current position of the end effector (or the component) obtained using the visual sensor 310 and the position of the board, and to reduce the difference between the jerk of the end effector (or the component) and the jerk target value. Here, the control unit 170 (target setting unit 176) may change the jerk target value for each control cycle according to the difference between the current position of the end effector (or the component) obtained using the visual sensor 310 and the position of the board.

[0119] In step 4 "approach", "difference in position between the end effector and the board" may be applied as target measurement information, and "acceleration" may be applied as a speed-related value. In this case, the control unit 170 (command generation unit 172) may generate a control command to reduce the difference between the current position of the end effector (or component) obtained using the visual sensor 310 and the position of the board, and to reduce the difference between the acceleration of the end effector and the acceleration target value. Here, the control unit 170 (target setting unit 176) may change the acceleration target value for each control cycle according to the difference between the current position of the end effector (or component) obtained using the visual sensor 310 and the position of the board.

[0120] In step 5 “recognizing holes in board”, “holes in board” can be applied as target measurement information. In this case, the control unit 170 (command generating unit 172) uses the visual sensor 310 to recognize holes in the board.

[0121] In step 6 "approaching hole", "position of hole in board" and "target reaction force" may be applied as target measurement information, and "acceleration" may be applied as a speed-related value. In this case, the control unit 170 (command generating unit 172) may generate a control command to reduce the difference between the current position of the component obtained using the visual sensor 310 and the position of the hole in the board, and to reduce the difference between the acceleration of the end effector and the acceleration target value. The control unit 170 (command generating unit 172) may also generate a control command to reduce the difference between the current reaction force generated in the end effector obtained using the force sensor 320 and the target reaction force, and to reduce the difference between the acceleration of the end effector and the acceleration target value.

[0122] For each step from step 7 "hole tracing operation" to step 10 "grip release", a "target reaction force" for each step may be applied as target measurement information, and a "jerk" may be applied as a speed-related value. In this case, the control unit 170 (command generating unit 172) may generate a control command to reduce the difference between the current reaction force generated in the end effector obtained using the force sensor 320 and the target reaction force for each step, and to reduce the difference between the jerk of the end effector and the jerk target value for each step. Here, the control unit 170 (target setting unit 176) may change the jerk target value for each control cycle according to the difference between the reaction force generated in the end effector obtained using the force sensor 320 and the target reaction force.

[0123] (5) Other embodiments The operational flows and operational examples in the above-described embodiments do not necessarily have to be executed in chronological order according to the order depicted in the flow diagrams. For example, steps in the operations may be executed in an order different from that depicted in the flow diagrams, or in parallel. In addition, some of the steps in the operations may be deleted, and additional steps may be added to the process.

[0124] A program may be provided that causes a computer to execute the operations according to the above-described embodiments. The program may be recorded in a computer-readable medium. Using the computer-readable medium, it is possible to install the program in a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient storage medium. The non-transient storage medium is not particularly limited, and may be, for example, a storage medium such as a CD-ROM or a DVD-ROM.

[0125] As used in this disclosure, the terms "based on" and "depending on" do not mean "based only on" or "depending only on", unless otherwise specified. The term "based on" means both "based only on" and "based at least in part on". Similarly, the term "depending on" means both "based only on" and "depending at least in part on". In addition, the terms "include", "comprise", and variations thereof do not mean including only the items listed, but may include only the items listed, or may include additional items in addition to the items listed. In addition, the term "or" as used in this disclosure is not intended to be an exclusive or. In this disclosure, when articles are added by translation, such as a, an, and the in English, these articles are intended to include the plural unless the context clearly indicates otherwise.

[0126] The above describes the embodiments in detail with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes, etc. are possible without departing from the spirit of the invention.

[0127] (6) Additional Notes The following additional features relate to the above-described embodiment.

[0128] Appendix 1 a motion generating unit that generates a control command for executing a reverse motion that changes a motion state of the robot apparatus from a goal state to an arbitrary state different from the goal state; an acquisition unit that acquires measurement information obtained by using a sensor for measuring an operating state of the robot device; a data collection unit configured to repeatedly collect data including a set of the control command and the measurement information during the execution of the reverse direction operation in accordance with the control command; The sensor includes at least one of a visual sensor and a force sensor. Control device.

[0129] Appendix 2 the motion generating unit generates the control command for executing a plurality of patterns of reverse motions that change a motion state of the robot apparatus from the target state to a plurality of arbitrary states different from each other, The data collection unit repeatedly collects the data for each of the multiple patterns of reverse direction motion. 2. The control device according to claim 1.

[0130] Appendix 3 The action generation unit is generating the control command such that the amount of data collected increases in an area closer to the target state; The control command is generated so that the amount of collected data decreases in an area farther from the target state. 3. The control device according to claim 1 or 2.

[0131] Appendix 4 The acquisition unit is shortening an acquisition cycle so that an amount of the measurement information acquired increases in an area closer to the target state; The acquisition period is extended so that the amount of acquired measurement information increases in an area farther from the target state. 4. The control device according to claim 1,

[0132] Appendix 5 The acquisition unit acquires, as relative measurement information, a difference between target measurement information acquired in the target state and current measurement information acquired during execution of the reverse direction motion; The data collecting unit repeatedly collects the data including a set of the control command and the relative measurement information during the execution of the reverse direction operation according to the control command. 5. A control device according to any one of claims 1 to 4.

[0133] Appendix 6 the acquisition unit acquires the measurement information including a speed-related value related to the robot apparatus in response to an output of the sensor or the control command; The data collection unit repeatedly collects the data including a set of the control command and the measurement information including the speed-related value during the execution of the reverse direction operation according to the control command. 6. A control device according to any one of claims 1 to 5.

[0134] Appendix 7 a setting acquisition unit that acquires setting information for performing a task using the robot device based on the data collected by the data collection unit; a control unit that uses the setting information to control the robot device to perform the work. 7. A control device according to any one of claims 1 to 6.

[0135] Appendix 8 The setting acquisition unit acquires the setting information including a trained model for deriving the control command from the measurement information by machine learning using the data collected by the data collection unit as learning data; The control unit performs the control based on the measurement information obtained during the work, using the setting information including the trained model. 8. The control device according to claim 7.

[0136] Appendix 9 the sensors include the visual sensor and the force sensor, the setting acquisition unit acquires the setting information including information for setting a control ratio between visual control, which is the control based on visual measurement information obtained using the visual sensor, and force control, which is the control based on force measurement information obtained using the force sensor; and The control unit changes the control ratio dynamically or stepwise according to a status of a work using the robot device by using the setting information. 9. The control device according to claim 7 or 8.

[0137] Appendix 10 The control unit is deriving a velocity-related value for the robot device from the measurement information or control commands obtained during the work; Based on the setting information and the measurement information, the robot device is repeatedly controlled to reduce the difference between the speed-related value and the speed-related target value. 10. The control device according to any one of appendix 7 to 9.

[0138] Appendix 11 the motion generation unit generates the control command for executing the reverse motion for each type of work using the robot device; The data collection unit collects the data for each of the types by repeatedly collecting the data for each of the types. 11. A control device according to any one of claims 1 to 10.

[0139] Appendix 12 a setting acquisition unit that acquires setting information for performing a task using the robot device for each of the types based on the data collected by the data collection unit; a control unit that uses the setting information corresponding to a type of work to be actually performed to control the robot device to perform the work. 12. The control device according to claim 11.

[0140] Appendix 13 generating a control command for executing a reverse motion for changing a motion state of the robot apparatus from a target state to an arbitrary state different from the target state; acquiring measurement information obtained by using a sensor for measuring an operating state of the robot device; and repeatedly collecting data including a set of the control command and the measurement information during the execution of the reverse direction operation according to the control command; The sensor includes at least one of a visual sensor and a force sensor. Control methods.

[0141] Appendix 14 The control device includes: generating a control command for executing a reverse motion for changing a motion state of the robot apparatus from a target state to an arbitrary state different from the target state; acquiring measurement information obtained by using a sensor for measuring an operating state of the robot device; repeatedly collecting data including a set of the control command and the measurement information during the execution of the reverse direction operation according to the control command; The sensor includes at least one of a visual sensor and a force sensor. program. [Explanation of symbols]

[0142] 100: Control device 101: Processor 102: Memory 103: External I / F 110: Motion generator 120: Acquisition section 121: Image recognition unit 130: Data collection section 140: Data storage unit 150: Setting acquisition section 160: Library storage section 170: Control section 171A: Vision control unit 171B: Force sensor control unit 172: Command generation section 173: Weighting section 174A: Prediction Department 174B: Correction section 175: Derivation part 176: Goal Setting Department 200: Robot device 210: Drive unit 211: First joint 212: Second joint 213: Third joint 214: 4th joint 215: 5th joint 216: 6th joint 221: Base 222: Link 223: Link 224: Link 225: Link 226: Gripper 300: Sensor 310: Vision sensor 320: Force sensor 330: Other sensors 400: User I / F 410:Display device 420: Operating device 510:Transportation equipment

Claims

1. A motion generating unit that generates a control command for executing a reverse motion that changes a motion state of a manipulator from a target state to an arbitrary state different from the target state; an acquisition unit that acquires measurement information obtained using a sensor for measuring an operating state of the manipulator; a data collection unit configured to repeatedly collect data including a set of the control command and the measurement information during the execution of the reverse direction operation in accordance with the control command; The sensor includes at least one of a visual sensor and a force sensor. Control device.

2. A motion generating unit that generates a control command for executing a reverse motion that changes a motion state of a robot apparatus from a target state to an arbitrary state different from the target state; an acquisition unit that acquires measurement information obtained by using a sensor for measuring an operating state of the robot device; a data collection unit configured to repeatedly collect data including a set of the control command and the measurement information during the execution of the reverse direction operation in accordance with the control command; the sensor includes at least one of a visual sensor and a force sensor, the motion generating unit generates the control command for executing a plurality of patterns of reverse motions that change a motion state of the robot apparatus from the target state to a plurality of arbitrary states different from each other, The data collection unit repeatedly collects the data for each of the multiple patterns of reverse direction motion. Control device.

3. A motion generating unit that generates a control command for executing a reverse motion that changes a motion state of a robot device from a target state to an arbitrary state different from the target state; an acquisition unit that acquires measurement information obtained by using a sensor for measuring an operating state of the robot device; a data collection unit configured to repeatedly collect data including a set of the control command and the measurement information during the execution of the reverse direction operation in accordance with the control command; the sensor includes at least one of a visual sensor and a force sensor, The action generation unit is generating the control command such that the amount of data collected increases in an area closer to the target state; The control command is generated so that the amount of collected data decreases in an area farther from the target state. Control device.

4. A motion generating unit that generates a control command for executing a reverse motion that changes the motion state of a robot device from a target state to an arbitrary state different from the target state; an acquisition unit that acquires measurement information obtained by using a sensor for measuring an operating state of the robot device; a data collection unit configured to repeatedly collect data including a set of the control command and the measurement information during the execution of the reverse direction operation in accordance with the control command; the sensor includes at least one of a visual sensor and a force sensor, The acquisition unit is acquiring the measurement information so that the amount of information acquired increases as the area approaches the target state; The measurement information is acquired so that the amount of information acquired is smaller in an area farther from the target state. Control device.

5. A motion generating unit that generates a control command for executing a reverse motion that changes the motion state of the robot device from a target state to an arbitrary state different from the target state; an acquisition unit that acquires measurement information obtained by using a sensor for measuring an operating state of the robot device; a data collection unit configured to repeatedly collect data including a set of the control command and the measurement information during the execution of the reverse direction operation in accordance with the control command; the sensor includes at least one of a visual sensor and a force sensor, The acquisition unit acquires, as relative measurement information, a difference between target measurement information acquired in the target state and current measurement information acquired during execution of the reverse direction motion; The data collecting unit repeatedly collects the data including a set of the control command and the relative measurement information during the execution of the reverse direction operation according to the control command. Control device.

6. A motion generating unit that generates a control command for executing a reverse motion that changes the motion state of the robot device from a target state to an arbitrary state different from the target state; an acquisition unit that acquires measurement information obtained by using a sensor for measuring an operating state of the robot device; a data collection unit configured to repeatedly collect data including a set of the control command and the measurement information during the execution of the reverse direction operation in accordance with the control command; the sensor includes at least one of a visual sensor and a force sensor, the acquisition unit acquires the measurement information including a speed-related value related to the robot apparatus in response to an output of the sensor or the control command; The data collection unit repeatedly collects the data including a set of the control command and the measurement information including the speed-related value during the execution of the reverse direction operation according to the control command. Control device.

7. a setting acquisition unit that acquires setting information for performing a task using the robot device based on the data collected by the data collection unit; a control unit that uses the setting information to control the robot device to perform the work. A control device according to any one of claims 2 to 6.

8. The setting acquisition unit acquires the setting information including a trained model for deriving the control command from the measurement information by machine learning using the data collected by the data collection unit as learning data; The control unit performs the control based on the measurement information obtained during the work, using the setting information including the trained model. The control device according to claim 7.

9. the sensors include the visual sensor and the force sensor, the setting acquisition unit acquires the setting information including information for setting a control ratio between visual control, which is the control based on visual measurement information obtained using the visual sensor, and force control, which is the control based on force measurement information obtained using the force sensor; and The control unit changes the control ratio dynamically or stepwise according to a status of a work using the robot device by using the setting information. The control device according to claim 7.

10. The control unit is deriving a velocity-related value for the robot device from the measurement information or control commands obtained during the work; Based on the setting information and the measurement information, the robot device is repeatedly controlled to reduce the difference between the speed-related value and the speed-related target value. The control device according to claim 7.

11. A motion generating unit that generates a control command for executing a reverse motion that changes a motion state of a robot device from a target state to an arbitrary state different from the target state; an acquisition unit that acquires measurement information obtained by using a sensor for measuring an operating state of the robot device; a data collection unit configured to repeatedly collect data including a set of the control command and the measurement information during the execution of the reverse direction operation in accordance with the control command; the sensor includes at least one of a visual sensor and a force sensor, the motion generation unit generates the control command for executing the reverse motion for each type of work using the robot device; The data collection unit collects the data for each of the types by repeatedly collecting the data for each of the types. Control device.

12. a setting acquisition unit that acquires setting information for performing a task using the robot device for each of the types based on the data collected by the data collection unit; a control unit that uses the setting information corresponding to a type of work to be actually performed to control the robot device to perform the work. The control device according to claim 11.

13. Generating a control command for executing a reverse motion that changes a motion state of a manipulator from a target state to an arbitrary state different from the target state; acquiring measurement information obtained using a sensor for performing measurements related to an operating state of the manipulator; and repeatedly collecting data including a set of the control command and the measurement information during the execution of the reverse direction operation according to the control command; The sensor includes at least one of a visual sensor and a force sensor. Control methods.

14. The control device includes: generating a control command for executing a reverse motion to change a motion state of the manipulator from a target state to an arbitrary state different from the target state; acquiring measurement information obtained using a sensor for performing measurements related to an operating state of the manipulator; repeatedly collecting data including a set of the control command and the measurement information during the execution of the reverse direction operation according to the control command; The sensor includes at least one of a visual sensor and a force sensor. program.

15. A setting acquisition unit that acquires setting information for performing a task using the manipulator based on the data collected by the data collection unit; a control unit that uses the setting information to control the manipulator to perform the work. The control device according to claim 1 .

16. The setting acquisition unit acquires the setting information including a trained model for deriving the control command from the measurement information by machine learning using the data collected by the data collection unit as learning data; The control unit performs the control based on the measurement information obtained during the work, using the setting information including the trained model. The control device according to claim 15.

17. The sensor includes the visual sensor and the force sensor, the setting acquisition unit acquires the setting information including information for setting a control ratio between visual control, which is the control based on visual measurement information obtained using the visual sensor, and force control, which is the control based on force measurement information obtained using the force sensor; and The control unit changes the control ratio dynamically or stepwise according to a situation of a work using the manipulator, using the setting information. The control device according to claim 15.

18. The control unit: deriving a velocity-related value for the manipulator from the measurement information or control commands obtained during the operation; Based on the setting information and the measurement information, the manipulator is repeatedly controlled to reduce the difference between the speed-related value and the speed-related target value. The control device according to claim 15.

Citation Information

Patent Citations

  • Position-force reproducing method and position-force reproducing device

    JP2009279699A

  • Teaching method for traveling object, controlling device for traveling object and traveling object system

    JP2010146238A

  • Robot control device, robot system and robot control method

    JP2023072601A

  • Robot system and robot control device

    JP7295344B1

  • Aircraft, aircraft control device, aircraft control method, and aircraft control program

    WO2019049197A1