Information processing method, information processing device, control method for a work system, work system, method for manufacturing an article, program, and recording medium.

Abstract

To reduce the burden on users. [Solution] In an information processing method in which a processing unit processes information, the processing unit acquires instructions issued by the user to a work device that performs an operation (S101), extracts process information relating to the processing that the work device performs on an object from the instructions (S102), acquires object control information for controlling the object and device control information for controlling the work device associated with the process information (S103, S104), and generates command information to be commanded to the work device based on the acquired object control information and device control information (S107).

Description

Technical Field

[0001] The present invention relates to an information processing method, an information processing apparatus, a method for controlling a work system, a work system, a method for manufacturing an article, a program, and a recording medium.

Background Art

[0002] There has been proposed a technology that converts user voice information into text information, sets operation command information for a robot such as a cleaning robot based on the text information, and causes the robot to perform an operation according to the operation command information (see Patent Document 1). Also, in order to specify the work content to be executed by the robot from a library, job information including an identifier for specifying the library to be called is acquired. And there has been proposed a technology that causes the robot to execute the work content specified by the job information (see Patent Document 2). Further, in the technology of Patent Document 2, it is disclosed that the library includes an action library for defining the operation of the robot and an auxiliary library for recognizing the work object.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the invention described in Patent Document 1 lacks quantitative information regarding the operation command information, such as where and how to hold the object. Therefore, it is not possible to link user instructions to the robot's specific actions, meaning that the robot cannot be made to perform specific actions. Similarly, even if the invention described in Patent Document 2 can recognize the work object, it lacks quantitative information on how to make the robot move in relation to the work object, meaning that the robot cannot be made to perform specific actions. Consequently, it does not reduce the burden on the user by allowing the robot to perform specific actions simply by the user giving instructions.

[0005] Therefore, the present invention aims to provide an information processing method, an information processing device, a control method for a work system, a work system, a method for manufacturing articles, a program, and a recording medium that can reduce the burden on the user. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing method in which a processing unit processes information, characterized in that the processing unit acquires instructions issued by a user to an operating work device, extracts process information relating to processing performed by the work device on an object from the instructions, acquires object control information for controlling the object and device control information for controlling the work device associated with the process information, and generates command information to be commanded to the work device based on the acquired object control information and device control information.

[0007] One aspect of the present invention is an information processing apparatus having a processing unit for processing information, wherein the processing unit acquires instructions issued by a user to an operating work device, extracts process information relating to processing performed by the work device on an object from the instructions, acquires object control information for controlling the object and device control information for controlling the work device associated with the process information, and generates command information to be commanded to the work device based on the acquired object control information and device control information.

[0008] One aspect of the present invention is a control method for controlling a work system comprising an information processing device having a processing unit for processing information and a work device having a control unit, wherein the processing unit acquires instructions issued by a user to the work device that performs an operation, extracts process information relating to processing performed by the work device on an object from the instructions, acquires object control information for controlling the object and device control information for controlling the work device associated with the process information, generates command information to be issued to the work device based on the acquired object control information and device control information, and the control unit acquires the command information generated by the processing unit and controls the work device according to the command information.

[0009] One aspect of the present invention is a work system comprising an information processing device having a processing unit for processing information and a work device having a control unit, wherein the processing unit acquires instructions issued by a user to the work device that performs an operation, extracts process information relating to processing performed by the work device on an object from the instructions, acquires object control information for controlling the object and device control information for controlling the work device associated with the process information, generates command information to be commanded to the work device based on the acquired object control information and device control information, and the control unit acquires the command information generated by the processing unit and controls the work device according to the command information. [Effects of the Invention]

[0010] According to the present invention, the burden on the user can be reduced. [Brief explanation of the drawing]

[0011] [Figure 1] This is a schematic diagram showing a production line according to an embodiment. [Figure 2] This figure shows a robot control system according to an embodiment. [Figure 3] This is a block diagram showing an information processing device according to an embodiment. [Figure 4] This is a block diagram showing a robot controller according to an embodiment. [Figure 5] This is a perspective view showing the components and robot hand according to the embodiment. [Figure 6] This is a flowchart showing the control of the robot system according to the embodiment. [Figure 7] This is an explanatory diagram showing the flow of various types of information when removing a component according to the embodiment. [Figure 8] This is an explanatory diagram showing the contents of the robot control information according to the embodiment. [Figure 9] This is an explanatory diagram showing the contents of component control information according to the embodiment. [Figure 10] This is an explanatory diagram showing the contents of robot control information and component control information during component removal according to the embodiment. [Figure 11] This is an explanatory diagram showing the contents of the robot control information and the contents of the part control information with additional information added, according to the embodiment. [Figure 12] This is an explanatory diagram showing the flow of various types of information during the assembly of parts according to the embodiment. [Figure 13] This is an explanatory diagram showing the contents of robot control information and component control information during the assembly of a component according to the embodiment. [Figure 14] This is an explanatory diagram showing the operation of the robot during the assembly of parts according to the embodiment. [Figure 15]This figure shows the control information setting screen of the information processing device according to the embodiment. [Figure 16] This figure shows a flowchart for generating the operation when removing a component in an information processing device according to the embodiment. [Modes for carrying out the invention]

[0012] The embodiments for carrying out the invention will be described below with reference to the drawings.

[0013] [About the production line] First, the production line 1 according to the embodiment will be described using Figure 1. Figure 1 is a schematic diagram showing an example of a production line in a parts assembly plant according to the embodiment. As shown in Figure 1, the production line 1 is a production line that assembles and manufactures products 2 as articles, and is equipment in which a manufacturing method is carried out in which products 2 are manufactured by a robot system 100 as a work system. The production line 1 has areas divided into a parts area AR1, an assembly area AR2, and a finished product area AR3. The production line 1 is also equipped with a robot system 100 that can move across these parts area AR1, assembly area AR2, and finished product area AR3.

[0014] In the parts area AR1, the robot system 100 picks up the first part 401 from the returnable container 491 (part picking). Then, while holding the picked-up first part 401, the robot system 100 moves to the assembly area AR2 and assembles it onto the second part 402 placed on the assembly table 492 (part assembly). In the assembly area AR2, the robot system 100 assembles the second part 402, which has the first part 401 attached, onto product 2X, which does not have the second part 402 attached (assembly), thereby completing product 2 (manufacturing). Finally, the robot system 100 grasps the completed product 2, for example via a tray 3, and transports it from the assembly area AR2 to the finished product area AR3 and places it on the display table 4. In this embodiment, the user 9 instructs the robot system 100 to perform these tasks, and the information processing device 101 causes the robot system 100 to execute the instructed tasks.

[0015] [Robot System Configuration] Next, the configuration of the robot system 100 will be explained using Figures 2, 3, and 4. Figure 2 is a diagram showing the robot control system according to the embodiment. Figure 3 is a block diagram of the information processing device according to the embodiment. Figure 4 is a block diagram of the robot controller according to the embodiment.

[0016] As shown in Figure 2, the robot system 100 is broadly composed of an input device 500, an information processing device 101, and a mobile robot 200 as a work device. The robot 200 is further composed of a manipulator 200A, a mobile cart 200B, a robot controller 201, and a camera 300 as a recognition device. The robot system 100 is an automated assembly system that performs the work on the production line 1 described above by controlling the robot 200 with the robot controller 201 based on command information generated by information processing in the information processing device 101.

[0017] The robot system 100 may consist of one unit or multiple units. The information processing device 101 is a computer installed inside or outside the assembly plant, and the robot controller 201 is a computer mounted on, for example, the mobile cart 200B of the robot 200. The input device 500 and the camera 300 are connected to the information processing device 101 via wired or wireless communication to enable data communication. Furthermore, the information processing device 101 and the robot controller 201 are connected via wired or wireless communication to enable data communication.

[0018] The manipulator 200A of robot 200 is, for example, a 6-axis articulated robot arm, fixed to the top of a mobile carriage 200B that can travel on the floor. The manipulator 200A is also configured to be able to be fitted with a robot hand 202, which is a gripping device that is an end effector for gripping objects such as the first part 401, the second part 402, and the tray 3. That is, the robot hand 202 of robot 200 is attached to the mounting part of the manipulator 200A and used as a work unit. There are no particular restrictions on the shape or structure of the robot hand 202, as long as it is capable of performing tasks on objects required in the assembly process. In other words, the tasks are not limited to gripping and transporting, but can also include tasks such as cutting, attaching, painting, and disassembly, and it is sufficient if it has a work unit capable of performing such tasks. Furthermore, the manipulator 200A and the robot hand 202 may be equipped with sensors such as force sensors and distance sensors as needed.

[0019] Furthermore, at least one camera 300 is fixed to the manipulator 200A, and the camera 300 acquires images of the work area where parts picking and parts assembly are performed. In this case, the camera 300 may be a two-dimensional camera or a three-dimensional camera, as long as it has the function of outputting at least two-dimensional image data and three-dimensional point cloud data in addition to that. Also, the camera 300 may be an on-hand camera attached to any part of the manipulator 200A, or a fixed camera attached to the outside of the manipulator 200A, as long as it can send image data to the information processing device 101.

[0020] Image data captured by camera 300 is sent to information processing device 101 for information processing. This information processing involves calculating command information (hereinafter referred to as "operation command information IM") related to robot control for robot controller 201 to perform parts picking and parts assembly, essentially meaning calculating instruction values.

[0021] Furthermore, as shown in Figure 2, the robot 200 is not limited to one in which the manipulator 200A is mounted on a mobile cart 200B; it may also be fixed in place on the floor without being moved. If the manipulator 200A is fixed in place, the transport of parts and products can be carried out by the coordinated transport of the manipulators 200A of multiple robots 200.

[0022] (Configuration of information processing device) Next, the configuration of the information processing unit 101 of the robot system 100 will be explained using Figure 3. The information processing unit 101 includes a CPU (Central Processing Unit) 102, which is an example of a processor. The CPU 102 is an example of a processing unit. The information processing unit 101 also includes a ROM (Read Only Memory) 103, a RAM (Random Access Memory) 104, and an HDD (Hard Disk Drive) 105 as storage units. The information processing unit 101 also includes a recording disk drive 106, a display unit 108 which is an input / output interface, and a keyboard 109 and mouse 110 which are operation units.

[0023] The CPU 102, ROM 103, RAM 104, HDD 105, recording disk drive 106, display 108, keyboard 109, and mouse 110 are connected to each other by a bus so that they can communicate with one another. The ROM 103 stores the basic programs related to the operation of the computer. The RAM 104 is a storage device that temporarily stores various data, such as the results of calculations performed by the CPU 102. The HDD 105 stores the results of calculations performed by the CPU 102 and various data acquired from external sources, as well as a program 107 for executing various processes described later. Program 107 is application software that enables the CPU 102 to perform various processes described later. Therefore, the CPU 102 can perform various processes described later by executing program 107 stored in the HDD 105. The HDD 105 also has an area that serves as a database 120 for recording data such as various models obtained from the results of the various processes described later. The recording disk drive 106 can read various data and programs recorded on the recording disk 150. This database 120 also includes the robot model database 410, the parts model database 411, the robot control information database 420, and the parts control information database 421, which will be described in more detail later.

[0024] In this embodiment, the non-temporary recording medium readable by the computer is the HDD 105, and the program 107 is recorded on the HDD 105, but this is not the only possible representation. The program 107 may be recorded on any non-temporary recording medium readable by the computer. Examples of recording media that can be used to supply the program 107 to the computer include flexible disks, hard disks, optical disks, magneto-optical disks, magnetic tapes, non-volatile memory, and the like.

[0025] Furthermore, the information processing device 101 is connected to an input device 500 (described in detail later), a camera 300, and a robot controller 201. User 9's input via the input device 500 will be described later. The camera 300 sends image data to the information processing device 101, where the image data is processed by program 107. The processing result is output as motion command information IM (instruction value) for robot control and transmitted to the robot controller 201.

[0026] (Robot controller configuration) Next, the configuration of the robot controller 201 of the robot 200 will be explained using Figure 4. The robot controller 201 includes a CPU 204, which is an example of a processor. The CPU 204 is an example of a control unit. The robot controller 201 also includes a ROM 205, RAM 206, and HDD 207 as storage units. Furthermore, the robot controller 201 includes a recording disk drive 208 and an input / output interface 209.

[0027] The CPU 204, ROM 205, RAM 206, HDD 207, recording disk drive 208, and interface 209 are connected to each other by a bus, enabling communication. ROM 205 stores the basic program for the computer's operation. RAM 206 is a memory device that temporarily stores various data, such as the results of calculations performed by the CPU 204. HDD 207 stores the results of calculations performed by the CPU 204 and various data acquired from external sources, as well as a program 210 that instructs the CPU 204 to perform various processes. Program 210 is application software that enables the CPU 204 to perform various processes described later. Therefore, the CPU 204 can execute control processing by running program 210 stored in HDD 207, thereby controlling the operation of the manipulator 200A. The recording disk drive 208 can read various data and programs stored on the recording disk 250.

[0028] In this embodiment, the non-temporary recording medium readable by the computer is the HDD207, and the program 210 is recorded on the HDD207, but this is not the only possible representation. The program 210 may be recorded on any non-temporary recording medium readable by the computer. Examples of recording media that can be used to supply the program 210 to the computer include flexible disks, hard disks, optical disks, magneto-optical disks, magnetic tapes, non-volatile memory, and the like.

[0029] In this embodiment, information processing and control processing are performed by one computer, i.e., one CPU, but this is not the only option. Information processing and control processing may be performed by multiple computers, i.e., multiple CPUs 102, 204.

[0030] [Control of robot systems] Next, the control (control method) of the robot system 100 described above will be explained with reference to the drawings. First, the control of the robot system 100 when it picks up a first part 401 from a conduit box 491 in the part area AR1 will be explained using Figures 5 to 11. Figure 5 is a perspective view showing a part and a robot hand according to the embodiment. Figure 6 is a flowchart showing the control of the robot system according to the embodiment. Figure 7 is an explanatory diagram showing the flow of various information when a part is picked up according to the embodiment. Figure 8 is an explanatory diagram showing the contents of the robot control information according to the embodiment. Figure 9 is an explanatory diagram showing the contents of the part control information according to the embodiment. Figure 10 is an explanatory diagram showing the contents of the robot control information and the contents of the part control information when a part is picked up according to the embodiment. Figure 11 is an explanatory diagram showing the contents of the robot control information and the contents of the part control information with additional information added when a part is picked up according to the embodiment.

[0031] (Robot hand configuration) As shown in Figure 5, the robot hand 202 has a main body 202a that is attached to the manipulator 200A by a detachable part (not shown), and a plurality of finger parts 202b that are slidably positioned relative to the main body 202a and serve as gripping parts capable of grasping objects such as parts. In the embodiment shown in Figure 5, there are two finger parts 202b, but there may be three or more.

[0032] (Composition of the first component) On the other hand, the first part 401 is a part of type "XYZ-A-Plate," for example, and is also simply referred to as "part A." As shown in Figure 5, the first part 401 has a flat plate portion 401a and, for example, two protrusions 401b that are provided to protrude from one side of the plate portion 401a. These protrusions 401b are attached to the second part 402, which will be described in more detail later (see Figure 14). In addition, a recessed portion 401d is formed on the outer edge 401c of the plate portion 401a, and is shaped to be easily grasped by the fingers 202b of the robot hand 202.

[0033] (Control during parts removal) Next, we will explain the control when the robot system 100 removes the first part 401 (during removal). As shown in Figures 6 and 7, first, the user 9 gives instructions for the robot 200 to perform actions using linguistic or non-linguistic information via the input device 500, and inputs this to the information processing device 101 (S101). In other words, the information processing device 101 receives instructions for the robot 200 to perform actions from the user 9. The input device 500 could be a terminal device such as a tablet computer, and although not shown in the figures, it could have a microphone, an operation panel (touch panel), a camera, etc.

[0034] For example, in the case of linguistic information, it is conceivable that the input device 500 could input (record) the voice of the instruction "Pick up the first part" spoken by the user 9 for picking up the part using a microphone (not shown) or the like on the input device 500. Alternatively, in the case of linguistic information, it is conceivable that the user 9 could input the text instruction "Pick up the first part" using an operation panel (not shown) or the like on the input device 500. Furthermore, in the case of non-linguistic information, it is conceivable that the user 9 could input handwritten characters (including drawings, etc.) using an operation panel (not shown) or the like on the input device 500. In addition, in the case of non-linguistic information, it is conceivable that the user 9 could perform the action of "Pick up the first part" and that this action could be captured and input using a camera (not shown) or the like on the input device 500. It should be noted that the input of user 9's instructions via linguistic or non-linguistic information as described here is not limited to these methods, and various other methods are conceivable.

[0035] Here, regardless of how user 9's instructions are expressed, process information IPR regarding the process performed on the part, such as "take the first part," must be input by user 9. This process information IPR must include object information to identify the object (part) being worked on as "the first part," and action information IPRb representing the robot 200's movement, such as "take." The input device 500 has the function of extracting object information that identifies the object (part) and action information IPRb representing the robot 200's movement process from the information input by user 9, and converting them into text data (text information). For example, if user 9 inputs voice, the voice can be converted to text (so-called voice conversion). Alternatively, if handwritten characters are input, they can be converted to text using OCR (Optical Character Recognition). In other words, conversion to text data can be achieved by combining natural language processing (NLP) techniques, and various known techniques are available. Furthermore, in the case of non-verbal information, conversion to text data can be achieved using, for example, an RNN (Recurrent Neural Network) that uses machine learning to convert image data into text data. While this description explains an input device 500 that converts user 9's instructions into text data, the information processing device 101 may also convert user 9's instructions into text data.

[0036] Next, the input device 500 converts the text data obtained in step S101 into unique text data (S102). This is to address the fact that a part of type "XYZ-A-Plate" may be input using various expressions such as "part A," "part A," "first part," or "plate A part." In other words, the input device 500 (i.e., the robot system 100) processes ambiguous words to enable the system to identify that these are the same object. This only requires that information with ambiguous words registered in the database 120 is prepared, and it plays a role in uniquely determining which part the input object information (hereinafter referred to as "part information IPRa") refers to in the robot system 100. Similarly, the operation information IPRb, which represents an operation process such as "take," may also be expressed as "take," "grab," or "picking." Even in such cases, by matching it with the database 120 where ambiguous words are registered, it is possible to convert it into text data for the uniquely determined operation process of "take." Here, we describe an input device 500 that converts the input text data into unique text data, but it is also acceptable for the information processing device 101 to convert the input text data into unique text data.

[0037] As described above, the IPRa component information used to identify a component from the converted unique text data is essentially the IPRa component information for the component name "First Component" associated with the action process of "taking". In addition, the information representing the action process such as "taking" from the converted unique text data is the action information IPRb, which is the name of the action process related to the action of "taking".

[0038] Next, the CPU 102 of the information processing device 101 retrieves a robot model capable of handling a specific part from the robot model database 410 based on the part information IPRa (part name) corresponding to the unique text data, and loads it into the RAM 104 (S103). This robot model is, for example, a robot hand model 202M corresponding to the robot hand 202, and it is desirable that it is a model that contains information about its three-dimensional shape, such as a CAD (Computer-Aided Design) model.

[0039] At the same time, the CPU 102 of the information processing device 101 retrieves the corresponding robot control information IR from the robot control information database 420 based on the process information IPR (operation process name) which corresponds to unique text data such as "take". Then, it loads the robot control information IR into the RAM 104 (S103). The robot control information IR loaded here is associated with information for "taking" the first part 401 using the robot hand model 202M, as shown in Figure 8. The robot control information IR associated with this "taking" information constitutes device control information related to the control of the work device (robot) associated with the operation information IPRb.

[0040] Specifically, the robot control information IR includes the origin O, which is the reference point of the robot hand model 202M, as device position information. h_origin Opening degree D to "take" the first part model 401M open and degree of closure D close This information is included. Furthermore, as shown in Figure 9, the robot control information IR includes, as device position information, the gripping position center P, which is the information of the opening and closing position of the finger portion 202b of the robot hand 202 when "grabbing" the first part model 401M. h_center This is included. Furthermore, the robot control information IR includes the origin O of the robot hand model 202M as device position information. h_origin From the center of the gripping position P h_center The relative positional relationship P up to h_origin-h_center This includes the gripping position center, which may be treated as the working position center.

[0041] Here, the CPU 102 of the information processing apparatus 101 generates operation command information IM for operating the actual robot system 100 on a virtual space based on each model. However, in the actual robot system 100, the relative positional relationship between the robot hand 202 and the manipulator 200A is already known. That is, when the robot hand 202 is attached to the manipulator 200A, the attachment reference origin O of the robot hand in the manipulator 200A robot and the origin O of the robot hand 202 h_origin The relative positional relationship between them becomes known through assembly adjustment.

[0042] And the information on these origin positions and positional relationships (the information indicated by symbols O and P) can be expressed, for example, in a six-degree-of-freedom coordinate system, and vector expressions such as P(x, y, z, Rx, Ry, Rz) are possible. As is well known, these vector informations are commutative with an affine matrix by inverse matrix operation, so coordinate conversion for any six-degree-of-freedom coordinate system is possible. For example, for the gripping position center P shown in FIG. 8 h_center to move to an arbitrary position and orientation P any the positional relationship P to an arbitrary position is measured. Then, from the positional relationship P h_center-any the amount of movement of the gripping position center P h_center-any and the amount of movement of the origin O of the robot hand 202 h_center can be calculated by affine transformation using an affine matrix. h_origin That is, the CPU 102 of the information processing apparatus 101 uses the robot control information IR associated with "grasping" the first component 401, and specifically, only adds the result of measuring the reference position and orientation of the first component 401 described later and performs an affine transformation. Thereby, the information processing apparatus 101 can generate operation command information IM for moving the manipulator 200A to a position for "grasping" the first component 401 with the robot hand 202.

[0043] That is, the CPU 102 of the information processing apparatus 101 uses the robot control information IR associated with "taking" the first component 401, and specifically, only adds the result of measuring the reference position and orientation of the first component 401 described later and performs an affine transformation. Thereby, the information processing apparatus 101 can generate operation command information IM for moving the manipulator 200A to a position for "taking" the first component 401 with the robot hand 202.

[0044] Next, the CPU 102 of the information processing device 101 retrieves a part model from the part model database 411 based on the part name corresponding to the unique text data and loads it into the RAM 104 (S104). This part model is, for example, a part model 401M corresponding to the first part 401, and it is desirable that it is a model that contains information about the three-dimensional shape, such as a CAD model.

[0045] At the same time, the CPU 102 of the information processing device 101 retrieves the corresponding component control information IP from the component control information database 421 based on a uniquely determined operation process name such as "take" and loads it into the RAM 104 (S104). The component control information IP loaded here is associated with the component control information IP for "taking" the first component 401, as shown in Figure 9. That is, the component control information IP includes the origin O, which is the reference point of the first component 401, as object position information. a_origin , the position and orientation for which the first part 401 is taken (hereinafter referred to as the "gripping position and orientation") P a_center It also includes the origin O of the first part 401 as object position information. a_origin From the gripping position and orientation P a_center The relative positional relationship P up to a_origin-a_center This includes information such as the above.

[0046] Next, the CPU 102 of the information processing device 101 commands the CPU 204 of the robot controller 201 to move to the part area AR1 to allow the robot system 100 to work (S105). If the first part 401 is within the working range of the robot system 100 and is within the field of view of the camera 300, this step S105 can be omitted. If the first part 401 is within the working range of the robot system 100 but is not within the field of view of the camera 300, the mobile carriage 200B moves the robot 200 to a working area.

[0047] Next, the CPU 102 of the information processing device 101 performs measurements to recognize the position and orientation of the first part 401 (i.e., part recognition) (S106). Part recognition here refers to taking an image of the first part 401 using the camera 300 and performing image processing using the captured image and the aforementioned part model 401M, such as CAD matching.

[0048] In other words, if the CAD matching is completed successfully, as shown in Figure 10, the origin O, which is the reference point of the first part 401, is determined from the information of the part model 401M. a_origin This allows us to measure the position and orientation of the camera origin O. camera From the origin O of part 401 a_origin The relative positional relationship P up to a_recog It is possible to find this.

[0049] Also, camera origin O camera And the origin of the robot hand 202 O h_origin The relative positional relationship P hand_eye_calib This can be determined by a known camera calibration (hand-eye calibration). Therefore, the origin O of the first part 401 detected by camera 300 a_origin The position is the origin O of the robot hand 202. h_origin The position can be immediately converted using affine transformation to the position as seen from. Here, the acquired robot control information IR and component control information IP are treated as being consistent with the actual machine. Therefore, the origin O of the robot hand 202 a_origin Grasping position P from position a_center The relative positional relationship P up to a_origin-a_center This information is known from the aforementioned component control information IP.

[0050] And the origin of the robot hand 202 O h_origin From the position of finger part 202b to the center position P h_center The relative positional relationship up to this point is also known from the aforementioned robot control information IR. Therefore, as a result, the center position P of the finger part 202b is known. h_center and gripping position and orientation P a_center Difference P h_center-a_centerThis becomes computable. Therefore, this difference P h_center-a_center This can be generated as motion command information IM for the robot 200 to "take" the first part 401 (S107).

[0051] As described above, when the CPU 102 of the information processing device 101 generates motion command information IM for the robot 200, it outputs it to the robot controller 201. Upon receiving this, the CPU 204 of the robot controller 201 controls the manipulator 200A and robot hand 202 of the robot 200 based on the motion command information IM (S108). This allows the manipulator 200A and robot hand 202 to be controlled to move to a position for "grabbing" the first part 401, and then to "grab" the first part 401.

[0052] Furthermore, when generating the motion command information (IM) for the robot 200, various interpolation methods, such as linear interpolation to generate a straight trajectory or interpolation to draw an arc-shaped curve, may be specified in advance for the paths connecting coordinate points.

[0053] Furthermore, the above explanation described how to acquire robot control information IR and component control information IP that perform the minimum necessary actions to "remove" the first component 401. However, the explanation is not limited to this; additional control information IA may be added to the component control information IP, for example, in order to "remove" the first component 401.

[0054] Specifically, as shown in Figure 11, the transit position and orientation P is such that the first part 401 is approached via an arbitrary point. approach The following is provided as additional control information IA. And the center position P of the finger portion 202b. h_center The transit position and orientation P approach Grasping position and orientation P a_center To approach (move to) the target. The transit position and orientation P at this time. approachAdditional control information IA related to this can be obtained from the additional control information database 422, as shown in Figure 7. Furthermore, the CPU 102 can determine the center position P of the finger portion 202b, as shown in Figure 11. h_center From via position and orientation P approach Difference P up to h_center-approach The CPU 102 then calculates the transit position and orientation P. approach From the gripping position and orientation P a_center Difference P up to approach-a_center The CPU 102 then calculates the difference P. h_center-approach and difference P approach-a_center This generates motion command information IM for the robot 200 to "take" the first part 401. As a result, the robot hand 202 will reach a transit position and orientation P approach This allows the device to pass through the first part 401, then approach (move to) it, and "take" the first part 401.

[0055] The above explanation simply describes the process of "taking" the first part 401, but it is not limited to this. For example, if user 9 instructs to "quickly" "take" the first part 401, it is conceivable to associate the text information "quickly" with the information of the opening and closing speed V [m / s] of the finger portion 202b of the robot hand 202. It is also conceivable to associate the text information "quickly" with the information of the acceleration / deceleration A [m / s²] of the manipulator 200A (i.e., the robot hand 202) (i.e., the movement speed information, which is information on the movement side). Furthermore, if user 9 instructs to "firmly" "grasp", it is conceivable to associate the text information "firmly" with the information of the gripping force F [N] of the finger portion 202b (i.e., the gripping force information). In this case, it is conceivable to grip the first part 401 with a force sensor (not shown) provided on the robot hand 202 so that a constant gripping force is achieved. In short, all of this motion information IPRb regarding the behavior of the robot system 100 is included in the robot control information. In other words, as described above, it is conceivable to associate motion information IPRb, which has been converted into text information such as "take," with motion parameters and coordinate information using vector information with arbitrary degrees of freedom.

[0056] (Control during parts assembly) Next, the control of the assembly (fitting) of the first part 401 and the second part 402 by the robot system 100 will be explained using Figures 12 to 14. Figure 12 is an explanatory diagram showing the flow of various information during the assembly of parts according to the embodiment. Figure 13 is an explanatory diagram showing the contents of the robot control information and the contents of the part control information during the assembly of parts according to the embodiment. Figure 14 is an explanatory diagram showing the operation of the robot during the assembly of parts according to the embodiment.

[0057] As described above, after the manipulator 200A performs the action of "taking" the first part 401, for example, based on the action command of user 9, the manipulator 200A performs the action of "assembling" the first part 401 with the second part 402 while holding it. The control flow when this "assembly" action is performed by the robot system 100 is basically the same as the control when taking out a part ("taking") as shown in Figure 6 above.

[0058] First, user 9 gives instructions to the robot 200 via the input device 500 using linguistic or non-linguistic information, and inputs this to the information processing device 101 (S101). However, the control during part removal described above and the control during part assembly described below differ in the following respects. Specifically, as shown in Figure 12, the process information IPR input from user 9 differs in that there are two names for the target parts (i.e., part information IPRa), such as "assemble" "part 1" to "part 2". Furthermore, the control during part removal described above and the control during part assembly described below differ in that the operation process is operation information IPRb, which is "assemble". As mentioned above, this information is also converted into text data, and converted into text data that has uniqueness as the name of the part (part information IPRa) and the name of the operation process (operation information IPRb) (S102). The second part 402 is, for example, a part of the type "XYZ-B-Plate" and is also referred to as "part B".

[0059] Thus, in the control during component assembly, the control (processing) is the same as during component removal, except that the text data is divided into two parts: the name of the component being assembled (Part 1) and the name of the component being assembled (Part 2). Regarding the process of dividing the text data into the name of the component being assembled and the name of the component being assembled, known language processing (NLP) can be used, and various methods can be considered, such as algorithms for syntactic analysis and semantic analysis, or methods using machine learning.

[0060] Next, the CPU 102 of the information processing device 101 retrieves the robot model and robot control information IR from the database 120 based on the converted unique text data (operation information IPRb of the operation process name) and loads them into the RAM 104 (S103). Furthermore, the CPU 102 of the information processing device 101 retrieves the component model and component control information IP from the database 120 based on the converted unique text data and loads them into the RAM 104 (S104).

[0061] Here, regarding the robot control information IR, as shown in Figure 13, the origin O of the robot hand model 202M is used as the device position information. h_origin or its gripping position center P h_center It is sufficient if the following can be obtained. Also, for the component control information IP, since the first component model 401M will be newly assembled into the second component model 402M, it is sufficient if the object position information can be obtained. In other words, for the component control information IP, the object position information is the assembly reference P on the first component side. a_assy And, the gripping position center P h_center From the first component side assembly standard P a_assy Relative position P up to h_center-a_assy It is sufficient if the following can be obtained. Also, for the second part 402, the second part model 402M is used as the object position information, and the origin O b_origin It is sufficient if this can be obtained. Also, regarding the second part 402, the object position information is the second part side assembly reference P for assembling the first part model 401M to the second part model 402M. b_assyIt is sufficient if this can be obtained. Furthermore, for the second part 402, the object position information is the origin O of the second part model 402M. b_origin From the second component side assembly standard P b_assy Relative positional relationship up to b_origin-b_assy It's sufficient if that can be obtained.

[0062] Next, the CPU 102 of the information processing device 101 commands the CPU 204 of the robot controller 201 to move to the assembly area AR2 to allow the robot system 100 to work (S105). If the second part 402 is within the working range of the robot system 100 and is within the field of view of the camera 300, this step S105 can be omitted. If the second part 402 is within the working range of the robot system 100 but is not within the field of view of the camera 300, the mobile cart 200B moves the robot 200 to a working area.

[0063] Next, the CPU 102 of the information processing device 101 performs measurements to recognize the position and orientation of the second part 402 (i.e., part recognition) (S106). As shown in Figure 14, in this part recognition, as described above, the second part 402, which is the target object, is imaged using the camera 300, and the origin O of the second part 402 is determined using the imaged image and CAD matching. b_origin The position is measured. In this way, the camera origin O camera The origin of the second part 402 as seen from O b_origin The relative positional relationship P up to b_recog This is required. Here, similarly, the robot control information IR and component control information IP are treated as being consistent with the actual machine. Therefore, by using this information, the motion trajectory P for assembling the first component 401 to the second component 402 can be determined. a_assy-b_assyIn other words, the operation command information IM for the manipulator 200A (and robot hand 202) becomes computable (S107). Then, the CPU 102 of the information processing device 101 commands the CPU 204 of the robot controller 201 using the operation command information IM. As a result, the CPU 204 of the robot controller 201 controls the manipulator 200A (and robot hand 202) to perform the operation of "assembling" the first part 401 with the second part 402 (S108).

[0064] (Regarding control at the time of product completion) As described above, the second part 402, with the first part 401 assembled, is then assembled by the robot system 100 to product 2X, which does not yet have the second part 402 assembled, as shown in Figure 1, thereby completing product 2. The control of the robot system 100 for completing product 2 at this time can also be performed by the control shown in Figure 6.

[0065] Specifically, for example, user 9 inputs the instruction "assemble the second part to the product" to the input device 500 using linguistic or non-linguistic information (S101). The input information is then converted into unique text data and output to the information processing device 101 as process information IPR (S102). In response, the CPU 102 of the information processing device 101 acquires the robot model and robot control information IR associated with the process information IPR (the process of assembling the second part to the product) (S103). Furthermore, the CPU 102 acquires the part model (for example, the second part model 402M to which the first part model 401M is assembled) and part control information IP associated with the above process information IPR (S104). Note that the process information IPR here includes part information IPRa for "second part 402", part information IPRa for "product 2X", operation information IPRb for "assemble", etc.

[0066] Furthermore, the CPU 102 of the information processing device 101 commands the CPU 204 of the robot controller 201 to move the robot system 100 to a position where the product 2X can be imaged by the camera 300 (S105). After that, the product 2X is imaged (recognized) by the camera 300 and its position is measured (S106). Then, the CPU 102 uses the robot model, robot control information IR, component model, and component control information IP to generate operation command information IM for the manipulator 200A and the robot hand 202 (S107). The CPU 102 of the information processing device 101 commands the CPU 204 of the robot controller 201 to control the manipulator 200A and the robot hand 202 using this operation command information IM (S108). As a result, the robot system 100 assembles the second component 402, to which the first component 401 has been assembled, onto the product 2X, completing product 2.

[0067] (Control during the transport of finished products) As described above, the completed product 2 is loaded onto tray 3 by the robot system 100, transported to the stand 4, and placed there, as shown in Figure 1. The control of the robot system 100 for transporting product 2 can also be performed by the control shown in Figure 6.

[0068] Specifically, for example, user 9 inputs an instruction to "transport the product to the stand" to the input device 500 using linguistic or non-linguistic information (S101). The input information is then converted into unique text data and output to the information processing device 101 as process information IPR (S102). In response, the CPU 102 of the information processing device 101 acquires the robot model and robot control information IR associated with the process information IPR (S103). Specifically, the robot control information IR acquires information regarding actions such as loading product 2 onto tray 3, gripping and moving tray 3, and placing tray 3 on stand 4. Furthermore, the CPU 102 acquires the component model (for example, the model of product 2) and component control information IP associated with the above process information IPR (S104). The process information IPR here includes component information IPRa for "product 2", component information IPRa for "tray 3", component information IPRa for "stand 4", operation information IPRb for "transport", etc.

[0069] Furthermore, the CPU 102 of the information processing device 101 commands the CPU 204 of the robot controller 201 to move the robot system 100 to a position where the product 2 can be imaged by the camera 300 (S105). After that, the product 2 is imaged (recognized) by the camera 300 and its position is measured (S106). Then, the CPU 102 uses the robot model, robot control information IR, component model, and component control information IP to generate operation command information IM for the manipulator 200A and the robot hand 202 (S107). The CPU 102 of the information processing device 101 commands the CPU 204 of the robot controller 201 to control the manipulator 200A and the robot hand 202 using this operation command information IM (S108). As a result, the robot system 100 first loads the product 2 onto the tray 3, and then the manipulator 200A and robot hand 202 grasp and move the tray 3, transporting the product 2 placed on the tray 3 to the stand 4 and loading it.

[0070] [About the GUI of information processing devices] Next, the GUI (Graphical User Interface) for creating, setting, modifying, or checking the database 120 in the information processing device 101 of the robot system 100 will be explained using Figure 15. Figure 15 shows the control information setting screen of the information processing device according to the embodiment.

[0071] As described above, the information processing device 101 allows the user 9 to freely configure and verify the robot control information IR and component control information IP, which are obtained from the database 120 by being associated with process information IPR, using a GUI. Here, GUI refers to a configuration in which the screen displayed on the display 108 connected to the information processing device 101 is operated using a keyboard 109, mouse 110, etc. For example, the control information setting screen 600, which is a GUI shown in Figure 15, is provided with a component control information setting screen 610 and a robot control information setting screen 620.

[0072] On the component control information setting screen 610, a component selection box is provided where the component to be configured can be selected. In this case, the component name selected will be the component information IPRa, which is a uniquely determined component name. The component display screen 611 displays the component model (shown as the first component model 401M in Figure 15) in 3D format, and can display the configured coordinate information. Unique operation process names such as "take" and "assemble" can be selected, and origin information and coordinate information can be assigned to each operation process name. Regarding coordinate information, for example, when assigning 6 degrees of freedom, (x, y, z, Rx, Ry, Rz) is provided, and the meaning of each coordinate name is given by a flag. Flags include component reference (work reference) and extraction reference, and for the "take" process, at least one extraction reference is required. When generating the operation command information IM for the robot 200, it is necessary to set at least one reference coordinate for at least one operation process name, such as approaching the extraction reference. Regarding these reference coordinates, there may be one or more depending on the operation process. If there are multiple reference coordinates, it is necessary to set the relationships between them, such as their order.

[0073] The robot control information setting screen 620 has a robot selection field where you can select parts of the robot 200 to be controlled, such as a robot hand. The robot display screen 621 displays the model of the part selected in the robot selection field (in Figure 15, the robot hand model 202M is shown) in 3D format, and can display the set coordinate information. Unique operation process names such as "grab" and "assemble" can be selected, and origin information and coordinate information can be assigned to each operation process name. It is also possible to select the target parts that the robot hand will handle, and coordinate information can be set for each target part. Depending on the type of robot hand, the opening degree (open / close) can be set, and speed and force (not shown) can also be set as options. Regarding coordinate information, for example, when assigning 6 degrees of freedom, (x, y, z, Rx, Ry, Rz) are given, and the meaning of each coordinate name is given by a flag. Flags such as robot reference and gripping reference exist, and the "grab" process requires at least one gripping reference. When generating motion command information (IM) for robot 200, it is necessary to set at least one reference coordinate for at least one motion process name, such as by using a grasping criterion for the approach. This reference coordinate can be one or multiple depending on the motion process. If there are multiple reference coordinates, their relationships, such as their order, must be defined.

[0074] As described above, the information entered on the control information setting screen 600 is associated with robot control information IR and component control information IP and recorded in the database 120 (robot control information database 420 and component control information database 421 (see Figure 7)). This makes it possible to retrieve the robot control information IR and component control information IP recorded in the database 120 based on the uniquely determined process information IPR (text data) obtained from the user 9's instructions.

[0075] [About the Action Generation Flowchart] Next, we will explain the operation generation flowchart that can be used when generating the operation command information IM shown in Figure 6. Figure 16 is a diagram showing the operation generation flowchart when removing a component in the information processing device according to the embodiment.

[0076] In step S107 of Figure 6 described above, the CPU 102 of the information processing device 101 calculates and generates the operation command information IM, which means "to pick up the first part," based on the robot control information IR and the part control information IP. Here, we will explain how an operation generation flowchart for this calculation is created and recorded in the database 120 in advance, and how the operation command information IM is generated based on this operation generation flowchart and the acquired robot control information IR and part control information IP.

[0077] In other words, the motion generation flowchart shown in Figure 16 sets what specific actions the robot 200 should perform in step S107 of Figure 6, and describes the details of the generation of motion command information IM for the robot 200. Note that the motion generation flowchart shown in Figure 16 is an example of a motion generation flowchart for the case of "picking up the first part 401" using the robot hand 202, but it is not limited to this. For example, motion generation flowcharts for other actions, such as "assembling the first part 401 onto the second part 402," may be prepared in the database 120.

[0078] The CPU 102 of the information processing device 101 downloads the operation generation flowchart shown in Figure 16 from the database 120 and first performs the process of obtaining the reference coordinates [1] of the component control information IP (S201). Specifically, in order to obtain the reference coordinates [1], after the first component 401 is imaged by the camera 300 in step S106 of Figure 6, the origin O of the first component 401 is obtained by CAD matching. a_origin The position is measured. Then, from the component control information IP, for example, the origin O a_origin and the reference coordinate P of the first part 401 a_center Relative positional relationship P a_origin-a_centerCalculate and obtain it as the reference coordinate [1].

[0079] Next, the CPU 102 sets up to obtain the reference coordinates [2] from the robot control information IR (S202). Specifically, to obtain the reference coordinates [2], the CPU 102 obtains, for example, the origin O of the robot hand 202 from the robot control information IR. h_origin Refer to the gripping center P h_center The result is calculated and obtained as the reference coordinate [2]. Then, CPU 102 calculates the difference between the two obtained reference coordinates [1] and reference coordinate [2] (S203).

[0080] Next, CPU102 sets the interpolation method (interpolation type) from among various interpolation methods, such as linear interpolation which generates a straight trajectory, and interpolation which draws an arc-shaped curve (S204).

[0081] Next, the CPU 102 sets the operation to open the finger portion 202b of the robot hand 202 based on the robot control information IR (S205). Furthermore, the CPU 102 sets the trajectory to move the robot hand 202 from the above-calculated reference coordinate [1] to the reference coordinate [2] (approaching the first part 401) (S206). Then, the CPU 102 sets the operation to close the finger portion 202b of the robot hand 202 based on the robot control information IR (S207).

[0082] As described above, the CPU 102 generates specific motion command information IM to operate the robot 200 by performing concrete calculations using the motion generation flowchart. Then, the robot controller 201 executes control to operate the robot 200 (manipulator 200A and robot hand 202) based on this generated motion command information IM, thereby completing the operation in which the robot 200 picks up the first part 401.

[0083] Here, as an example, we have described a motion generation flowchart for generating motion command information (IM) for the "take" action of robot 200. However, the flowchart is not limited to this example; motion generation flowcharts can be freely written for generating motion command information (IM) for actions that can be performed using component control information (IP) and robot control information (IR). That is, each motion generation flowchart can be associated with a uniquely defined action process name, such as "take," and managed as a motion generation flowchart for generating motion command information for each action.

[0084] The motion generation flowchart described above only needs to be set once by, for example, a product designer or system designer, eliminating the need for user 9 to configure these settings each time the robot 200 performs an action. As a result, user 9 can use the robot system 100 simply by inputting linguistic or non-linguistic information (i.e., by issuing simple instructions including process information IPR).

[0085] [Summary of robot system control (information processing of information processing devices)] As described above, when user 9 issues an operation information IPRb (process information IPR) related to the operation of the robot system 100 using linguistic or non-linguistic information, the operation information IPRb is converted into text information. Subsequently, robot control information IR and component control information IP associated with the text information (operation information IPRb) are automatically acquired, and operation command information IM for the robot system 100 (especially the coordinates of the manipulator 200A and robot hand 202) is automatically generated. Then, the robot system 100 is automatically controlled by this operation command information IM. This makes it possible to make the robot system 100 perform specific actions according to user 9's commands, especially when making the robot system 100 perform actions involving components such as the first component 401 and the second component 402. Therefore, for example, a user 9 without specialized knowledge can simply give instructions to the robot system 100, and the robot system 100 will perform specific actions, thereby reducing the burden on user 9.

[0086] [Possibility of other embodiments] In the embodiments described above, the robot system 100 was described as having a robot 200 which is an industrial robot, as an example, but it is not limited to this. For example, as a work device that performs actions related to work, it could be a cleaning robot, a serving robot in a restaurant, a transport robot or delivery drone that carries luggage, a pet robot, or any other robot or mobile device that performs various tasks. In other words, it can be any work device that performs a process on an object as an action. For example, it could be any type of robot arm, such as a horizontal articulated robot arm, a parallel link robot arm, or a Cartesian robot. Furthermore, the present invention is also applicable to machines that can automatically perform actions such as extension and retraction, bending and straightening, vertical movement, horizontal movement, or rotation, or combinations thereof, based on information stored in a memory device provided in the control device.

[0087] Specifically, for example, a serving robot could be equipped with a hand, and based on user instructions (process information IPR), it could perform specific actions such as "distributing" or "removing (collecting)" objects like "food" or "dishes." In this case, the object control information could include information about the center of gravity, the position of the center of gravity, and the orientation (which may be converted to coordinates) of the "food" or "dishes" (object position information). The device control information could include information about the specific trajectory, position, and orientation (which may be converted to coordinates) of the hand performing actions such as "distributing" or "removing" (device position information). This would allow the robot to acquire specific object control information and device control information in response to user instructions (process information IPR), enabling it to perform specific actions that were not possible with conventional work devices.

[0088] Specifically, for example, a delivery drone could be equipped with a hand, and based on user instructions (process information IPR), it could perform specific actions such as "loading" or "delivering" objects like "packages." In this case, the object control information could include information about the center of gravity, the position of the center, and the orientation (which may be converted to coordinates) of the "package" (object position information). The device control information could include information about the specific position and orientation of the drone itself, the trajectory, position, and orientation (which may be converted to coordinates) of the hand (device position information). This would allow for the acquisition of specific object control information and device control information in response to user instructions (process information IPR), enabling the drone to perform specific actions that were not possible with conventional work devices.

[0089] Furthermore, in the control of the robot system 100 according to the embodiment (information processing method of the information processing device 101), it was explained that the process information IPR includes motion information IPRb and part information IPRa (object information). In other words, it was explained that process information IPR, which includes motion information IPRb such as "take" or "assemble" and part information IPRa such as "first part" or "second part," is extracted from the user's instructions. However, it is not limited to this, and it is also acceptable for the process information IPR not to include motion information IPRb, or not to include object information (part information IPRa). A concrete example of a process information that does not include object information is when a user gives an instruction to the robot that includes motion information such as "wave," and the robot performs the action of "waving" at an object (e.g., a person). Another concrete example of a process information that does not include motion information is when a user gives an instruction to the robot that includes object information such as "first part," and the robot automatically performs the action of "bringing the first part." In short, even if the user's instructions lack either the operation information (IPRb) or the object information, the work device can be made to operate automatically if the information itself is either missing or predetermined.

[0090] Furthermore, in the control of the robot system 100 according to the embodiment (information processing method of the information processing device 101), a control information setting screen 600 is described in which robot control information IR and component control information IP are displayed as a GUI, and both can be set. However, the invention is not limited to this, and it is also acceptable to display only one of the robot control information IR or component control information IP on the control information setting screen 600 and make it possible to set only one of them.

[0091] Furthermore, in the control of the robot system 100 according to the embodiment (information processing method of the information processing device 101), a method for generating an action generation flowchart based on robot control information IR and component control information IP, and thereby operating the robot 200, was described. However, it is not necessarily required to generate this action generation flowchart using both robot control information IR and component control information IP; it may be generated using only one of them.

[0092] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0093] Furthermore, the present invention is not limited to the embodiments described above, and many modifications are possible within the technical concept of the present invention. Also, two or more embodiments from the above-described embodiments may be combined and implemented. Moreover, the effects described in the embodiments are merely a list of the most preferred effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0094] Summary of this disclosure This disclosure includes at least the following: (Method 1) In an information processing method in which information is processed by a processing unit, The aforementioned processing unit, The instructions issued by the user to the work device performing the operation are acquired. From the above instructions, process information relating to the processing that the work device performs on the object is extracted, Acquire object control information for controlling the object and device control information for controlling the work device, associated with the process information. Based on the acquired object control information and device control information, command information is generated to be commanded to the work device. An information processing method characterized by the following: (Method 2) The aforementioned instructions are linguistic information, The processing unit generates text information by converting the language information into text, and extracts the process information from the text information. The information processing method according to Method 1, characterized in that (Method 3) The aforementioned instructions are nonverbal information, The processing unit generates text information by converting the non-verbal information into text, and extracts the process information from the text information. The information processing method according to method 1 or 2, characterized by the features described above. (Method 4) The process information includes operation information relating to the actions performed by the work device on the object. An information processing method according to any one of methods 1 to 3, characterized by the above. (Method 5) The process information includes object information for identifying the object. An information processing method according to any one of methods 1 to 4, characterized by the above. (Method 6) The object control information includes object position information, which is information about the position and orientation of the object. An information processing method according to any one of methods 1 to 5, characterized by the above. (Method 7) The device control information includes device position information, which is information about the position and orientation of the work device. An information processing method according to any one of methods 1 to 5, characterized by the above. (Method 8) The object control information includes object position information, which is information about the position and orientation of the object. The device control information includes device position information, which is information about the position and orientation of the work device. An information processing method according to any one of methods 1 to 5, characterized by the above. (Method 9) The object position information includes coordinate information for one or more degrees of freedom relating to the object. The information processing method according to method 8, characterized in that (Method 10) The object location information includes information on a reference point related to the object, The information processing method according to method 8 or 9, characterized by the features described above. (Method 11) The aforementioned reference point is information measured by the recognition device. The information processing method according to method 10, characterized in that (Method 12) The device position information includes coordinate information for one or more degrees of freedom relating to the work device. An information processing method according to any one of methods 8 to 11, characterized by the above. (Method 13) The work apparatus has a gripping device that includes a gripping portion capable of gripping the object, An information processing method according to any one of methods 8 to 12, characterized by the above. (Method 14) The device position information includes information on a reference point related to the gripping device. The information processing method according to method 13, characterized in that (Method 15) The device control information includes open / closed position information, which is information about the open / closed position of the gripping portion. The information processing method according to method 13 or 14, characterized by the features described herein. (Method 16) The device control information includes movement speed information, which is information about the movement speed of the gripping part. An information processing method according to any one of methods 13 to 15, characterized by the above. (Method 17) The device control information includes gripping force information, which is information about the gripping force of the gripping part. An information processing method according to any one of methods 13 to 16, characterized by the above. (Method 18) The object position information includes additional information which is additional position information that the gripping unit passes through before moving to the position where it grips the object. An information processing method according to any one of methods 13 to 17, characterized by the above. (Method 19) The processing unit displays the object control information and the device control information on the display unit, and allows setting one or both of the object control information and the device control information by operating the operation unit. An information processing method according to any one of methods 1 to 18, characterized by the above. (Method 20) The processing unit generates command information to be commanded to the work device based on the object control information, the device control information, and the operation generation flowchart. An information processing method according to any one of methods 1 to 19, characterized by the above. (Method 21) The processing unit displays the motion generation flowchart on the display unit, and the operation unit can set the motion generation flowchart using either or both of the object control information and the device control information. The information processing method according to method 20, characterized in that (Method 22) The aforementioned processing unit, The reference coordinates in the object control information and the reference coordinates in the device control information are obtained. The command information for operating the work device is generated from the reference coordinates of the device control information to the reference coordinates of the object control information. An information processing method according to any one of methods 1 to 21, characterized by the above. (Method 23) The aforementioned processing unit, The relative positional relationship between the origin set for the object and the reference position of the object is obtained as the reference coordinate in the object control information. The relative positional relationship between the origin set on the work unit of the work device and the center of the work position of the work unit is obtained as the reference coordinates for the device control information. The information processing method according to method 22, characterized in that (Method 24) The aforementioned work device is a robot. An information processing method according to any one of methods 1 to 23, characterized by the above. (Method 25) The aforementioned robot is an industrial robot. The information processing method according to method 24, characterized in that (Method 26) The robot has a multi-jointed robotic arm to which an end effector capable of working on an object can be attached. The information processing method according to method 24 or 25, characterized in that... (Method 27) The object is a part of an article manufactured by the work apparatus. An information processing method according to any one of methods 1 to 26, characterized by the above. (Composition 28) In an information processing device having a processing unit for processing information, The aforementioned processing unit, The instructions issued by the user to the work device performing the operation are acquired. From the above instructions, process information relating to the processing that the work device performs on the object is extracted, Acquire object control information for controlling the object and device control information for controlling the work device, associated with the process information. Based on the acquired object control information and device control information, command information is generated to be commanded to the work device. An information processing device characterized by the following: (Method 29) In a control method for controlling a work system comprising an information processing device having a processing unit for processing information and a work device having a control unit, The aforementioned processing unit, The instructions issued by the user to the work device performing the operation are acquired. From the above instructions, process information relating to the processing that the work device performs on the object is extracted, Acquire object control information for controlling the object and device control information for controlling the work device, associated with the process information. Based on the acquired object control information and device control information, command information is generated to be commanded to the work device. The control unit, The command information generated by the processing unit is acquired, and the work device is controlled according to the command information. A method for controlling a work system characterized by the following: (Composition 30) In a work system comprising an information processing device having a processing unit for processing information, and a work device having a control unit, The aforementioned processing unit, The instructions issued by the user to the work device performing the operation are acquired. From the above instructions, process information relating to the processing that the work device performs on the object is extracted, Acquire object control information for controlling the object and device control information for controlling the work device, associated with the process information. Based on the acquired object control information and device control information, command information is generated to be commanded to the work device. The control unit, The command information generated by the processing unit is acquired, and the work device is controlled according to the command information. A work system characterized by the following: (Method 31) A work apparatus is used to manufacture an article, which is controlled based on the command information generated by the information processing method described in any one of Methods 1 to 27. A method for manufacturing an article, characterized by the following: (Composition 32) A program for causing a computer to execute one of the information processing methods described in any one of Methods 1 through 27. (Composition 33) A computer-readable recording medium on which the program described in configuration 31 is recorded. [Explanation of symbols]

[0095] 2…Product (object, item) / 3…Tray (object) / 9…User / 100…Robot system (work system) / 101…Information processing device / 102…CPU (processing unit) / 108…Display (display unit) / 109…Keyboard (operation unit) / 110…Mouse (operation unit) / 200…Robot (work device, industrial robot) / 200A…Manipulator (articulated robot arm) / 202…Robot hand (end effector, gripping device) / 202b…Fingers (gripping unit) / 204…CPU (control unit) / 401…First part (part, object) / 402…Second part (part, object) / A…Acceleration / deceleration (movement speed information) / AI…Additional control information (additional information) / D open …Opening degree (device position information) / D close ...Closing degree (device position information) / F...Gripping force (gripping force information) / IM...Operation command information (command information) / IP...Component control information (object control information) / IPR...Process information / IPRa...Component information (object information) / IPRb...Operation information / IR...Device control information / O a_origin ...Origin (object position information, reference point) / O b_origin ...Origin (object position information, reference point) / O h_origin ...Origin (device position information) / P a_center ...Gripping position and orientation (object position information) / P a_origin-a_center ...Positional relationship (object position information) / P a_assy ...Assembly standard for the first component (object position information) / P b_assy ...Assembly standards for the second component (object position information) / P h_center ...Gripping position center (device position information) / P h_origin-h_center ...Positional relationship (device position information) / R b_origin-b_assy ...Positional relationship (object position information) / P h_center-a_assy ...relative position (object position information)

Claims

1. In an information processing method in which information is processed by a processing unit, The aforementioned processing unit, The instructions issued by the user to the work device performing the operation are acquired. From the above instructions, process information relating to the processing that the work device performs on the object is extracted, Acquire object control information for controlling the object and device control information for controlling the work device, associated with the process information. Based on the acquired object control information and device control information, command information is generated to be commanded to the work device. An information processing method characterized by the following:

2. The aforementioned instructions are linguistic information, The processing unit generates text information by converting the language information into text, and extracts the process information from the text information. The information processing method according to feature 1.

3. The aforementioned instructions are nonverbal information, The processing unit generates text information by converting the non-verbal information into text, and extracts the process information from the text information. The information processing method according to feature 1.

4. The process information includes operation information relating to the actions performed by the work device on the object. The information processing method according to feature 1.

5. The process information includes object information for identifying the object. The information processing method according to feature 1.

6. The object control information includes object position information, which is information about the position and orientation of the object. The information processing method according to feature 1.

7. The device control information includes device position information, which is information about the position and orientation of the work device. The information processing method according to feature 1.

8. The object control information includes object position information, which is information about the position and orientation of the object. The device control information includes device position information, which is information about the position and orientation of the work device. The information processing method according to feature 1.

9. The object position information includes information on the coordinates of one or more degrees of freedom relating to the object. The information processing method according to feature 8.

10. The object location information includes information on a reference point related to the object, The information processing method according to feature 8.

11. The aforementioned reference point is information measured by the recognition device. The information processing method according to feature 10.

12. The device position information includes coordinate information for one or more degrees of freedom relating to the work device. The information processing method according to feature 8.

13. The work apparatus has a gripping device that includes a gripping portion capable of gripping the object, The information processing method according to feature 8.

14. The device position information includes information on a reference point related to the gripping device. The information processing method according to feature 13.

15. The device control information includes open / closed position information, which is information about the open / closed position of the gripping portion. The information processing method according to feature 13.

16. The device control information includes movement speed information, which is information about the movement speed of the gripping part. The information processing method according to feature 13.

17. The device control information includes gripping force information, which is information about the gripping force of the gripping part. The information processing method according to feature 13.

18. The object position information includes additional information which is additional position information that the gripping unit passes through before moving to the position where it grips the object. The information processing method according to feature 13.

19. The processing unit displays the object control information and the device control information on the display unit, and allows setting one or both of the object control information and the device control information by operating the operation unit. The information processing method according to feature 1.

20. The processing unit generates command information to be commanded to the work device based on the object control information, the device control information, and the operation generation flowchart. The information processing method according to feature 1.

21. The processing unit displays the motion generation flowchart on the display unit, and the operation unit can set the motion generation flowchart using either or both of the object control information and the device control information. The information processing method according to claim 20.

22. The aforementioned processing unit, The reference coordinates in the object control information and the reference coordinates in the device control information are obtained. The command information for operating the work device is generated from the reference coordinates of the device control information to the reference coordinates of the object control information. The information processing method according to feature 1.

23. The aforementioned processing unit, The relative positional relationship between the origin set for the object and the reference position of the object is obtained as the reference coordinate in the object control information. The relative positional relationship between the origin set on the work unit of the work device and the center of the work position of the work unit is obtained as the reference coordinates for the device control information. The information processing method according to feature 22.

24. The aforementioned work device is a robot. The information processing method according to feature 1.

25. The aforementioned robot is an industrial robot. The information processing method according to feature 24.

26. The robot has a multi-jointed robotic arm to which an end effector capable of working on an object can be attached. The information processing method according to feature 24.

27. The object is a part of an article manufactured by the work apparatus. The information processing method according to feature 1.

28. In an information processing device having a processing unit for processing information, The aforementioned processing unit, The instructions issued by the user to the work device performing the operation are acquired. From the above instructions, process information relating to the processing that the work device performs on the object is extracted, Acquire object control information for controlling the object and device control information for controlling the work device, associated with the process information. Based on the acquired object control information and device control information, command information is generated to be commanded to the work device. An information processing device characterized by the following:

29. In a control method for controlling a work system comprising an information processing device having a processing unit for processing information and a work device having a control unit, The aforementioned processing unit, The instructions issued by the user to the work device performing the operation are acquired. From the above instructions, process information relating to the processing that the work device performs on the object is extracted, Acquire object control information for controlling the object and device control information for controlling the work device, associated with the process information. Based on the acquired object control information and device control information, command information is generated to be commanded to the work device. The control unit, The command information generated by the processing unit is acquired, and the work device is controlled according to the command information. A method for controlling a work system characterized by the following:

30. In a work system comprising an information processing device having a processing unit for processing information, and a work device having a control unit, The aforementioned processing unit, The instructions issued by the user to the work device performing the operation are acquired. From the above instructions, process information relating to the processing that the work device performs on the object is extracted, Acquire object control information for controlling the object and device control information for controlling the work device, associated with the process information. Based on the acquired object control information and device control information, command information is generated to be commanded to the work device. The control unit, The command information generated by the processing unit is acquired, and the work device is controlled according to the command information. A work system characterized by the following:

31. A work device is used to manufacture an article, which is controlled based on the command information generated by the information processing method described in any one of claims 1 to 27. A method for manufacturing an article characterized by the following:

32. A program for causing a computer to execute the information processing method described in claim 1.

33. A computer-readable recording medium on which the program described in claim 32 is recorded.

Images