Control device

Abstract

A control device according to an embodiment acquires, in order to control an industrial machine that performs a specified task on an object in collaboration with a worker, situation data including at least any of first sensor information, alarm information, second sensor information of a peripheral device around the industrial machine, and processing information related to the specified task during the operation of the industrial machine. When detecting an abnormal state of the industrial machine which is different from a state during a normal operation in the specified task on the basis of the situation data, the control device searches a database in which a document related to the industrial machine is stored, adds a related document similar to the situation data, and produces a first prompt to ask a generative language model about a problem with the situation data and a solution to the problem. The control device acquires a response to the first prompt from the generative language model, and notifies the worker.

Description

control device

[0001] An embodiment of the present disclosure relates to a technology for improving the efficiency of a system in which workers belong to the same group as industrial machines and work collaboratively.

[0002] In the line systems of various factories, industrial machines that work in collaboration with workers are often installed. Although industrial machines such as robots can sometimes work in collaboration with workers without the need for safety fences, even in this case, they can only work in designated locations and with designated objects, so instructions or follow-up from the worker are required in case of unforeseen circumstances.

[0003] Special Publication No. 2021-532467

[0004] Industrial robots, including collaborative robots, use visual information from cameras, for example, to recognize objects and perform tasks, but in the event of an unexpected situation, an alarm is displayed on a teaching control panel, etc., requiring the worker to temporarily stop work and check the teaching control panel, etc. In addition, because the control programs for industrial machinery are created to be able to respond to anticipated situations in advance, in situations that the worker did not anticipate, the industrial machinery will not operate appropriately and the worker will need to follow up, which places a burden on the worker.

[0005] Furthermore, when a worker is working with a robot on a line moving at the same speed, it is difficult for the worker to respond immediately even if the collaborative robot issues an alarm, and maintenance work must be performed after stopping the line to restore the robot's operation.Furthermore, when an alarm occurs, the worker must turn their attention to an alarm display medium such as a teaching control panel, which requires the line work to be stopped or increases the likelihood of human error, which creates other issues.

[0006] Therefore, there is a need for a system that can efficiently continue collaborative work even when the industrial machine that collaborates with the worker encounters a situation that was not anticipated by the control program.

[0007] In one embodiment, a control device acquires situation data including at least one of first sensor information during operation of the industrial machine, alarm information, second sensor information from peripheral devices of the industrial machine, and processing information related to the predetermined task in order to control industrial machinery that performs a predetermined task in collaboration with an operator. When the control device detects an abnormal state during the predetermined task that is different from normal operation of the industrial machine based on the situation data, the control device searches a database storing documents related to the industrial machine, adds related documents similar to the situation data, and creates a first prompt that queries a generative language model about problems in the situation data and solutions to the problems. The control device acquires a response to the first prompt from the generative language model and notifies the operator.

[0008] It is a block diagram showing the configuration of the robot system in the embodiment. It is a diagram showing a first procedure for solving an abnormal state in the robot system in the embodiment. It is a diagram showing a second procedure for solving an abnormal state in the robot system in the embodiment.

[0009] The following describes exemplary embodiments of the present disclosure. The control device of this embodiment is a device for controlling industrial machinery that performs predetermined tasks on objects in collaboration with an operator, and when the industrial machinery falls into a situation that the operator did not anticipate and that was not pre-programmed, the control device provides a solution to the situation in a timely manner using a generative language model.

[0010] The generative language model used in this embodiment is a machine learning model that has undergone additional learning (e.g., fine tuning or instruction tuning) to generate solutions to questions about abnormal conditions based on data such as industrial machinery manuals, specifications, past inquiries, and their responses. Furthermore, the control device of this embodiment uses search expansion and generation (RAG) technology to add various documents related to the industrial machinery to prompts and input them into the generative language model. Hereinafter, an interactive response system that can access this generative language model and generate answers will be referred to as generative AI.

[0011] Here, various types of industrial machinery are envisioned, such as industrial robots, service robots, and machines controlled by computer numerical control devices, but in this embodiment, an example will be described as a collaborative robot that works in collaboration with workers.

[0012] 1 is a block diagram showing the configuration of a robot system 1 according to this embodiment. The robot system 1 includes a control device 10 for controlling the operation of a collaborative robot that performs a predetermined task in collaboration with a worker. The control device 10 includes a generating AI and a reference database (DB), or is communicably connected to these to form an AI system.

[0013] The reference database stores various collaborative robot manuals, past failure cases, documents related to similar systems implemented in other factories, and even examples of control programs for resolving abnormal conditions in collaborative robots. For example, the various manuals include specifications, instructions, and alarm resolution manuals.

[0014] The control device 10 is an information processing device (computer) equipped with a control unit (processor) and a storage unit (memory), and controls the operation of the collaborative robot by executing a control program using information from the collaborative robot or various sensors installed around it as input.

[0015] Furthermore, if the collaborative robot is assigned the task of, for example, grabbing a necessary part from a parts shelf and moving it to a designated location, it can output various signals via the control device 10, such as outputting an alarm when the part runs out, or requesting a worker to replenish parts if there are only a few remaining.

[0016] The control unit of the control device 10 includes a data acquisition unit 11, an abnormality detection unit 12, a prompt creation unit 13, a notification unit 14, a reception unit 15, and an execution unit 16, and these functional units operate by the control unit reading and executing various software stored in the memory unit.

[0017] The data acquisition unit 11 acquires situation data including at least one of first sensor information during operation of the collaborative robot, alarm information, second sensor information of peripheral devices of the collaborative robot, and processing information related to a predetermined task.

[0018] Here, the first sensor information includes, for example, motor drive information, temperature information, force sensor measurement information, image information of an object, etc., which are observed in the collaborative robot. Furthermore, the second sensor information includes, for example, drive information of a transport device that transports the object, image information of an object or a worker's position, etc.

[0019] The processing information relating to a specific task performed by the collaborative robot may include, for example, measurement data such as the speed, size, position, number, and spacing of objects based on first sensor information, or information indicating the task step being performed by the collaborative robot.

[0020] The abnormality detection unit 12 detects an abnormal state that differs from the normal operation of the collaborative robot during a predetermined task based on the situation data acquired by the data acquisition unit 11. For example, the abnormality detection unit 12 detects an abnormal state by comparing reference image information of the object with image information of the object. Furthermore, the abnormality detection unit 12 detects an abnormal state according to pre-programmed conditions by, for example, comparing measurement data with model information of the object or comparing a standard value of first sensor information in a task step with actual first sensor information.

[0021] The prompt creation unit 13 searches a reference database storing documents related to collaborative robots, adds related documents similar to the situation data, and creates a first prompt that queries the generative language model about the problems and solutions to the situation data. The prompt creation unit 13 also searches the reference database, adds related documents similar to the instruction input from the worker received by the receiving unit 15 (described later), and creates a second prompt that queries about a program for controlling the collaborative robot.

[0022] The notification unit 14 obtains a response from the generated AI to the first prompt and notifies the worker by voice or text.

[0023] In response to the response from the generation AI, the reception unit 15 receives instructions from the worker regarding the work content of the collaborative robot, either by voice or text.

[0024] The execution unit 16 controls the collaborative robot by executing a program that is a response from the generation AI to the second prompt.

[0025] In this embodiment, the following production line is assumed as an example of the robot system 1, where a worker packs prepared foods into lunch boxes together with a collaborative robot. (1) Empty lunch boxes flow in order along the line in front of the collaborative robot and the worker. (2) A line carries prepared foods (fried chicken) in front of the collaborative robot, and a line carries other prepared foods in front of the worker. (3) If an abnormal state different from normal operation is detected while the collaborative robot is packing the fried chicken brought in front of it into an empty lunch box, the worker is notified of this.

[0026] Here, the following two cases will be exemplified as procedures for resolving an abnormal state, that is, a state that has not been anticipated in advance (a state that cannot be resolved by the control program alone).

[0027] 2 is a diagram showing a first procedure for resolving an abnormal state in the robot system 1 of this embodiment. In this case, two pieces of fried chicken are brought to the line in charge of the collaborative robot and are stuck together, which may result in a failure to pack them into a lunch box or the risk of two pieces being packed in at once. In this case, the control device 10 detects that the object (fried chicken) is different from normal based on, for example, a comparison between an image captured by a camera and a reference image or model information (size, weight, etc.).

[0028] Based on the situation data obtained from the collaborative robot and peripheral devices, the control device 10 adds related documents that are the results of searching the reference database, creates a first prompt that asks about the problems in the situation data and how to solve them, and inputs this to the generation AI.

[0029] When the generation AI responds to the first prompt with, for example, "There may be two pieces of fried chicken stuck together," the control device 10 notifies the worker of the response by voice, etc. In response to this response, the worker inputs instructions to the control device 10 by voice regarding the work content of the collaborative robot, such as "Please perform the action of 'separating the stuck fried chicken' before putting the fried chicken into the lunch box."

[0030] The control device 10 then generates a second prompt asking about a program for controlling the collaborative robot based on the instruction, adding related documents that are the results of searching the reference database, and inputs the second prompt to the generation AI. When the control device 10 receives a control program for the collaborative robot from the generation AI in response to the second prompt, the control device 10 notifies the worker that the control program will be executed, and then executes the program.

[0031] [Case 2] Figure 3 shows a second procedure for resolving an abnormal state in the robot system 1 of this embodiment. An alarm corresponding to a sensor value has been generated in the collaborative robot, requiring an operator to take action. In this case, the control device 10 searches the reference DB based on situation data including alarm information obtained from the collaborative robot and peripheral devices, adds related documents as search results, creates a first prompt asking about the problem with the situation data and how to resolve it, and inputs the first prompt to the generation AI.

[0032] When the generation AI responds to the first prompt with, for example, "An alarm has occurred. Based on similar situations, the cause is thought to be XX. The two best solutions are α and β," the control device 10 notifies the worker of this response by voice, etc. In response to this response, the worker inputs an instruction to the control device 10 by voice, such as "Please execute β," regarding the work content of the collaborative robot.

[0033] The control device 10 then generates a second prompt asking about a program for controlling the collaborative robot based on the instruction, adding related documents that are the results of searching the reference DB, and inputs the second prompt to the generation AI. When the control device 10 receives a control program for the collaborative robot, i.e., a program for implementing solution β, from the generation AI as a response to the second prompt, the control device 10 notifies the worker that the control program will be executed, and then executes the program.

[0034] In addition to the examples given in Cases 1 and 2, there are various other possible abnormal states that can be detected by the control device 10. For example, the control device 10 can respond to the following situations and improve the efficiency of collaborative work.

[0035] (1) In case 1, when a collaborative robot works on a single object (fried chicken), for example, if the object is easy to grasp and only one photograph is required, the process can be completed earlier than expected, resulting in a waiting time. This can be indicated to the generating AI, leading to a solution such as speeding up the line. Conversely, if additional actions are required, such as photographing the object from multiple angles and then removing the stuck fried chicken, this can be indicated to the generating AI, leading to a solution such as slowing down the line.

[0036] (2) In Case 1, when the control device 10 detects an abnormal condition, such as the object size exceeding a threshold, it receives a request for re-photographing and a re-photographing program from the generation AI as a response. If necessary, the control device 10 receives instructions from the operator and then adds the re-photographed image or detailed measurement data obtained from the image to the prompt, thereby obtaining a more appropriate response from the generation AI. Furthermore, if a re-photographing program is pre-installed in the control device 10, the control device 10 can automatically re-photograph the object if the object size exceeds a threshold.

[0037] (3) In case 2, if the generation AI obtains a plan γ that is more accurate than plans α and β, the control device 10 may notify the user, such as “We will attempt to solve the problem using plan γ,” and wait for instructions from the operator or automatically execute the plan after a certain period of time.

[0038] (4) In the case of an industrial machine that performs machining, such as a machining center, rather than a collaborative robot, the first sensor information includes the force sensor value gripping the workpiece metal, the surface temperature of the workpiece metal during machining, and an image of the finished product after machining. The control device 10 checks the error between the measurement data obtained from the image information and the target value, and if the error exceeds or is close to the allowable error, inputs data such as the force sensor value and surface temperature during machining, as well as related documents obtained from the reference database, into the generation AI. This allows the control device 10 to obtain a machining procedure that will result in an appropriate finish from the generation AI and propose it to the worker, and then execute the program according to instructions from the worker.

[0039] According to this embodiment, the control device 10 searches the reference database for documents related to similar situations based on situation data obtained from sensors on the industrial machinery, and adds the documents to the prompt. The control device then obtains an appropriate response from the AI ​​regarding the problem and a solution, and notifies the worker. This allows the worker to implement the optimal solution when an industrial machine encounters an unexpected situation without stopping the line or calling a maintenance technician. Furthermore, because the worker can search for and implement a solution from past cases in the reference database without having to reread an alarm manual, a user-friendly collaboration system can be built.

[0040] Furthermore, when an operator inputs instructions for the operation he or she wants the industrial machine to perform in response to a response from the generation AI, the control device 10 inputs this instruction input along with related documents to the generation AI, thereby obtaining a control program that is in line with the current situation as a response and executing a solution.

[0041] The generated language model can be additionally trained based on a reference DB. For example, solution cases, which are responses of the generated AI to abnormal conditions, are accumulated in response to instructions input by an operator, and these solution cases can be used as training data for additional training of the generated language model. The control device 10 may be equipped with a function for generating and training training data for additional training, or this may be performed by a separately prepared information processing device. As the training of the generated language model progresses, improvements in the accuracy and efficiency of responses to abnormal conditions can be expected.

[0042] The control device 10 may also be configured as an auxiliary device, separate from an existing control device of an industrial machine.

[0043] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the invention or the idea and intent of the present disclosure derived from the content described in the claims and their equivalents. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0044] The following supplementary note is further disclosed regarding the above-described embodiment: (Supplementary Note 1) A control device (10) for controlling industrial machinery that performs a predetermined task on an object in collaboration with an operator, the control device (10) comprising: a data acquisition unit (11) that acquires situation data including at least one of first sensor information, alarm information, second sensor information of a peripheral device of the industrial machinery while the industrial machinery is operating, and processing information related to the predetermined task; an abnormality detection unit (12) that detects an abnormal state that is different from normal operation of the industrial machinery during the predetermined task based on the situation data; a prompt creation unit (13) that searches a database storing documents related to the industrial machinery, adds related documents similar to the situation data, and creates a first prompt that asks a generative language model about problems in the situation data and solutions to the problems; and a notification unit (14) that acquires a response to the first prompt from the generative language model and notifies the operator.

[0045] (Supplementary Note 2) The control device (10) described in (Supplementary Note 1), wherein the first sensor information includes any of motor drive information, temperature information, measurement information of a force sensor, and image information of an image of the object, which are observed in the industrial machine.

[0046] (Appendix 3) The second sensor information includes either drive information of a transport device that transports the object, or image information capturing the position of the object or the worker. The auxiliary device (10) described in (Appendix 1) or (Appendix 2).

[0047] (Supplementary Note 4) The control device (10) described in (Supplementary Note 2), wherein the abnormality detection unit (12) detects the abnormal state by comparing reference image information of the object with image information of the object captured.

[0048] (Appendix 5) The control device (10) described in (Appendix 2) in which the processing information includes measurement data related to the object based on the first sensor information, and the abnormality detection unit (12) detects the abnormal state by comparing the measurement data with model information of the object.

[0049] (Appendix 6) The control device (10) described in (Appendix 2) in which the processing information includes information indicating a work step being performed by the industrial machine, and the abnormality detection unit detects the abnormal state by comparing a standard value of the first sensor information in the work step with actual first sensor information.

[0050] (Supplementary Note 7) The control device (10) according to any one of (Supplementary Note 1) to (Supplementary Note 6), wherein the notification unit (14) notifies the response from the generative language model by voice.

[0051] (Appendix 8) A control device (10) according to any one of (Appendix 1) to (Appendix 7), comprising: a reception unit (15) that receives instruction input from the worker regarding the work content of the industrial machinery in response to a response from the generated language model; the prompt creation unit (13) searches the database, adds related documents similar to the instruction input, and then creates a second prompt that asks about a program for controlling the industrial machinery; and an execution unit (16) that controls the industrial machinery based on the program that is the response from the generated language model to the second prompt.

[0052] (Supplementary Note 9) The control device (10) according to (Supplementary Note 8), wherein the accepting unit (15) accepts input by voice.

[0053] (Supplementary Note 10) The control device (10) according to any one of (Supplementary Note 1) to (Supplementary Note 9), wherein the industrial machine is any one of an industrial robot, a service robot, and a machine controlled by a computer numerical control device.

[0054] REFERENCE SIGNS LIST 1 Robot system 10 Control device 11 Data acquisition unit 12 Abnormality detection unit 13 Prompt creation unit 14 Notification unit 15 Reception unit 16 Execution unit

Claims

1. A control device for controlling industrial machinery that performs specified tasks on an object in collaboration with an operator, comprising: a data acquisition unit that acquires situation data including at least one of first sensor information, alarm information, second sensor information of peripheral devices of the industrial machinery while the industrial machinery is operating, and processing information related to the specified task; an abnormality detection unit that detects an abnormal state that is different from normal operation of the industrial machinery during the specified task based on the situation data; a prompt creation unit that searches a database storing documents related to the industrial machinery, adds related documents similar to the situation data, and creates a first prompt that asks a generative language model about problems in the situation data and solutions; and a notification unit that acquires a response to the first prompt from the generative language model and notifies the operator.

2. A control device as described in claim 1, wherein the first sensor information includes any of motor drive information, temperature information, force sensor measurement information, and image information of the object observed in the industrial machine.

3. An auxiliary device as described in claim 1 or claim 2, wherein the second sensor information includes either drive information of a conveying device that conveys the object, or image information capturing the position of the object or the worker.

4. The control device according to claim 2, wherein the abnormality detection unit detects the abnormal state by comparing reference image information of the object with image information of the object captured.

5. The control device described in claim 2, wherein the processing information includes measurement data regarding the object based on the first sensor information, and the abnormality detection unit detects the abnormal state by comparing the measurement data with model information of the object.

6. The control device described in claim 2, wherein the processing information includes information indicating the work step being performed by the industrial machine, and the abnormality detection unit detects the abnormal state by comparing a standard value of the first sensor information in the work step with the actual first sensor information.

7. The control device according to any one of claims 1 to 6, wherein the notification unit notifies the response from the generative language model by voice.

8. A control device as described in any one of claims 1 to 7, further comprising: a reception unit that receives instruction input from the worker regarding the work content of the industrial machinery in response to a response from the generated language model; the prompt creation unit searches the database, adds related documents similar to the instruction input, and creates a second prompt that asks about a program for controlling the industrial machinery; and an execution unit that controls the industrial machinery based on the program that is the response from the generated language model to the second prompt.

9. The control device according to claim 8, wherein the reception unit receives input by voice.

10. A control device according to any one of claims 1 to 9, wherein the industrial machine is one of an industrial robot, a service robot, and a machine controlled by a computer numerical control device.

Images