Control device for industrial robots, learning device for industrial robots, control system, learning method, control method, and program

Abstract

Even when tasks are automated by industrial robots, they should still be able to accept human input during execution. [Solution] A control device for controlling an industrial robot that performs work on an object comprises: an input unit for inputting user operations; a simulation unit for constructing a simulation environment for simulating the work performed by the industrial robot based on the operations; a teaching unit for teaching the industrial robot in the simulation environment; an execution unit for causing the industrial robot to perform the work in the actual environment based on the teaching results in the simulation environment; and a modification unit that, when it receives a modification operation from the user to change part or all of the work while the work is being executed in the actual environment, reflects the modification operation and modifies the work being executed in the actual environment.

Description

Technical Field

[0006] , , , , ,

[0005] ,

[0007] ,

[0001] The present invention relates to a control device for an industrial robot, a learning device for an industrial robot, a control system, a learning method, a control method, and a program.

Background Art

[0002] Efforts are being made to promote the Sustainable Development Goals (the 2030 Agenda for Sustainable Development, adopted at the United Nations Summit on September 25, 2015 (Heisei 27), hereinafter referred to as "SDGs"). Specifically, technology is required to achieve Goal "9": building resilient infrastructure, promoting inclusive and sustainable industrialization, and fostering innovation.

[0003] Conventionally, technology for remotely operating industrial robots has been widely known.

[0004] For example, first, a grinding robot equipped with a hand part equipped with a flaw detector, a grinder, and a grindstone contact detector is provided with an image processing device for detecting the position of a flaw. And, it is provided with a drive control device for operating the grinding robot by a signal from an arithmetic processing device. In this way, a technique for automatically grinding a flaw generated on the surface of a metal material is known (see, for example, Patent Document 1, etc.).

Prior Art Documents

Patent Documents

[0005] [[ID=??]]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] ?<000??1>? In the conventional technology, when work is automated by an industrial robot, there is a problem that it becomes difficult for a person to operate during execution. [[ID=?]]

[0007] It seems there are some unclear or incorrect tags in the original text which might cause issues during translation. I've translated it as accurately as possible based on the provided rules. If you can clarify those tags, it would be helpful for a more precise translation. The present invention aims to allow human intervention during the execution of tasks that have been automated by industrial robots. [Means for solving the problem]

[0008] To solve the above problems, a control device for controlling an industrial robot that performs work on an object, according to one aspect of the present invention, An input section for receiving user input, A simulation unit constructs a simulation environment for simulating the work performed by the industrial robot based on the above operation, The simulation environment includes a teaching unit that performs teaching on the industrial robot, An execution unit that causes the industrial robot to perform the task in the actual environment based on the teaching results in the simulation environment, When the user makes a change operation to modify part or all of the work while the work is being executed in the actual environment, the change unit modifies the work being executed in the actual environment to reflect the change operation. It is characterized by having the following features. [Effects of the Invention]

[0009] According to the present invention, even when work is automated by an industrial robot, it is possible to allow human intervention during execution. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of a control system for industrial robots. [Figure 2] This figure shows examples of hardware such as control devices. [Figure 3] This figure shows an example of a master-slave configuration environment. [Figure 4] This figure shows an example of a simulation environment. [Figure 5] This figure shows an example of a completed teaching session. [Figure 6] It is a diagram showing an example of a round nozzle. [Figure 7] It is a diagram showing an example of performing scarfing. [Figure 8] It is a diagram showing an example of a defect. [Figure 9] It is a diagram showing an example of scarfing settings. [Figure 10] It is a diagram showing an example of setting the height in scarfing. [[ID=I4]] [Figure 11] It is a diagram showing an example of setting the angle in scarfing. [Figure 12] It is a diagram showing an example of the depth output in scarfing. [Figure 13] It is a diagram showing an example of simulating scarfing marks. [Figure 14] It is a diagram showing an example of the simulation output of scarfing. [Figure 15] It is a diagram (part 1) showing an example of setting a virtual area. [Figure 16] It is a diagram (part 2) showing an example of setting a virtual area. [Figure 17] It is a diagram showing an example of a restriction by a virtual area. [Figure 18] It is a diagram showing the first setting example of a virtual permission area. [Figure 19] It is a diagram showing the second setting example of a virtual permission area. [Figure 20] 3 It is a diagram showing an example before a change operation is performed. [Figure 21] It is a diagram showing the first example of a change operation. [Figure 22] It is a diagram showing the second example of a change operation. [Figure 23] It is a diagram showing the third example of a change operation. [Figure 24] It is a diagram showing the fourth example of a change operation. [Figure 25] It is an overall processing example of the first embodiment. [Figure 26] It is a diagram showing the functional configuration example of the first embodiment. [Figure 27] It is a network configuration diagram showing the configuration example of AI. [Figure 28] This figure shows an example of pre-processing. [Figure 29] This figure shows an example of the execution process. [Figure 30] This figure shows examples of execution processes and simulations. [Figure 31] This figure shows an example of the entire process of AI learning and execution. [Figure 32] This diagram illustrates an example of how AI divides a task into multiple parts for learning and execution. [Figure 33] This is an example of the overall processing of the second embodiment. [Figure 34] This figure shows examples of the first and second learning processes. [Figure 35] This figure shows examples of the first execution process and the second execution process. [Figure 36] This figure shows an example of the functional configuration of the second embodiment. [Modes for carrying out the invention]

[0011] [First Embodiment] The following examples will be explained with reference to the attached drawings. In the following explanation, the reference numerals in the drawings refer to the same elements. Furthermore, the embodiments are not limited to the following examples, and embodiments may include elements other than those shown in the drawings.

[0012] The following explanation uses the example of an industrial robot that performs scarfing as its task. However, industrial robots may perform tasks other than scarfing. Also, a scarfing robot may be a robot that specializes in scarfing, or a robot that performs other tasks as well. For example, a scarfing robot and the scarfing task are as follows.

[0013] [About scarving and scarving robots] Scarfing (also called "scarfing" or "scaling") uses a gas containing multiple types of gases. The gas contains oxygen. Scarfing uses a gas mixture of oxygen with flammable gases such as hydrogen, acetylene (propane), and LPG (Liquefied Petroleum Gas).

[0014] Note that more than two types of gas may be mixed with oxygen. The gas is used for scarfing under high pressure. Specifically, scarfing is the process of removing defects from an object by blowing a gas under high pressure onto it. Hereinafter, the state in which the scarfing process has been performed will be referred to as the "removed state."

[0015] Furthermore, by removing the material through scarfing, it becomes possible to process the object by welding, gouging, cutting, or filleting.

[0016] The scarfing simulation results, i.e., the removal state, show, for example, information about the object, such as the "removal area," "depth," "removal amount," and "shape" of the object to be removed by the scarfing process.

[0017] The area removed refers to the area removed on a plane, for example, if the object being scarved is a plane.

[0018] The "depth" of removal refers to the distance from the plane to the bottom of the area being removed, for example, if the object being scarfed is a plane.

[0019] The removal amount is the quantity of material removed by scarfing (for example, expressed in units of volume or weight).

[0020] "Shape" refers to the shape of the area targeted by scarfing. For example, "shape" can be represented on the screen using a 3D model or similar.

[0021] Hereafter, the object to which scarfing is applied will be referred to as the "object." The object is, for example, metal. Specifically, the object is in the state of a slab material (also called a "flat plate" or "semi-finished product," etc.). For example, if the object is steel, in a continuous casting process, the molten steel produced in the converter is cooled and processed into a plate. Then, the steel is cut after it has cooled and solidified to become a slab material.

[0022] For example, scarfing is a process performed on slab material. However, the object is not limited to slab material; it can be in various shapes, such as billets. In other words, the object can be a square timber or a flat timber, and its shape is not restricted.

[0023] A defect refers to, for example, a part of the material that has become iron oxide. Alternatively, a defect refers to a part of the material that has a large amount of impurities mixed in, or a part that has scratches (including cracks, etc.).

[0024] For example, scarfing is a process performed in the manufacturing of an object. Therefore, scarfing is performed as part of the manufacturing process or near manufacturing equipment, and may be carried out in high-temperature environments.

[0025] Therefore, it is desirable for scarfing robots to be heat-resistant (the temperature they can withstand varies depending on the metals they handle). In other words, general industrial robots that are not heat-resistant may not be able to be installed in scarfing environments because they cannot operate in high-temperature environments.

[0026] Heat resistance can be achieved by covering the object with a heat-resistant jacket, for example. For instance, if the object contains steel, its melting point is 1580°C. Therefore, when installing a scarfing robot as part of the manufacturing process, it is desirable that the scarfing robot be capable of operating in environments close to such high temperatures.

[0027] Therefore, it is desirable that the scarfing robot has heat-resistant properties (including cases where a part of the scarfing robot, or the scarfing robot, temporarily acquires heat-resistant properties).

[0028] Parameters that affect the quality of scarfing include the speed at which the nozzle is moved, the angle between the nozzle and the object, the distance between the nozzle and the object, preheating conditions (such as the temperature at which the object was heated during pretreatment before the scarfing process), the mixing ratio of flammable gases and oxygen in the gas, the gas pressure, and the nozzle structure.

[0029] For example, scarfing uses three types of oxygen: high-pressure oxygen (hereinafter referred to as "high-pressure oxygen"), oxygen at a lower pressure than high-pressure oxygen (hereinafter referred to as "low-pressure oxygen"), and LPG. The following explanation will use an example with these three types, but scarfing may be performed using different types, numbers of types, and procedures other than those described below.

[0030] Furthermore, high-pressure oxygen only needs to be at a higher pressure than low-pressure oxygen; the exact pressure is not specified. In other words, the pressures of high-pressure and low-pressure oxygen are set appropriately depending on the object being tested.

[0031] In the first step, low-pressure oxygen and LPG are mixed, and a pilot light is ignited. In the second step, after the pilot light has been ignited in the first step, high-pressure oxygen is further mixed into the two-component gas mixture. Next, the third step adjusts the flame produced in the second step. Specifically, the third step adjusts the mixing ratio of low-pressure oxygen and LPG (or the output amount of each per unit time of low-pressure oxygen and LPG). For example, the third step is performed by observing the size of the flame produced in the second step. However, it is not limited to observing the size of the flame; adjustments may also be made based on flow meter readings, etc.

[0032] Note that in the third step, you may adjust only one of the two options. For example, in the third step, you may leave the LPG fully open and adjust only the low-pressure oxygen.

[0033] In the fourth step, the pressure of the hyperbaric oxygen is adjusted based on the flame that was adjusted in the third step.

[0034] As described above, scarfing is performed using the flame produced through the first to fourth steps. Therefore, if the mixing ratio of low-pressure oxygen and LPG adjusted in the third step, and the pressure of high-pressure oxygen adjusted in the fourth step can be set as parameters, the scarfing process and simulation will be easier to perform.

[0035] Note that parameters of types other than those listed above may also be set.

[0036] Furthermore, scarfing robots may have covers that protect against dirt and grime.

[0037] As described above, it is desirable for scarfing robots to be heat-resistant. Therefore, it is desirable for scarfing robots to be equipped with environmentally resistant materials such as heat-resistant jackets or covers. Environmentally resistant materials may be added later or be removable.

[0038] The objects can have various shapes. Therefore, in order to perform scarfing, the scarfing robot changes its posture by translating and rotating in accordance with the shape of the object. Specifically, the scarfing robot has degrees of freedom (DOF) that allow the nozzle to be translated in the left-right direction, translated in the up-down direction, translated in the depth direction (including diagonal movement by combining translation of two or more axes), rotate on the Roll axis, rotate on the Pitch axis, and rotate on the Yaw axis. In other words, the scarfing robot is a mechanism with at least 6DOF.

[0039] [Example System Configuration] Figure 1 shows an example of an industrial robot control system. For example, if the industrial robot is a scarfing robot, the scarfing robot control system (hereinafter referred to as "control system 100") includes a scarfing robot 11, a control panel 12, and a control device 10, etc. However, the robot and system configuration may differ depending on the type of work, etc.

[0040] The control system 100 may have a PLC 13 (Programmable Logic Controller) or the like, either internally or externally. Furthermore, the control system 100 may be connected to external systems such as a higher-level system 14.

[0041] The scarfing robot 11 is the robot controlled by the control panel 12 and the control device 10. There may be multiple scarfing robots 11. For simplicity, the following explanation will use a single scarfing robot 11 as an example.

[0042] Furthermore, the control system 100 may also control robots other than the scarfing robot 11. However, the robots controlled by the control system 100 will operate in harsh environments such as the surrounding environment of the manufacturing process. Therefore, they will be robots that can withstand the surrounding environment, such as heat resistance.

[0043] In addition, the control system 100 may have other devices externally or internally. Furthermore, each of the above devices may be composed of multiple devices. Moreover, each of the above devices may be, for example, an integrated scarfing robot 11 and control panel 12.

[0044] In the control system 100, for example, when a user 101 inputs an operation to the control device 10, the scarfing robot 11 operates based on that operation. In other words, in the control system 100, the scarfing robot 11 is the slave and the control device 10 is the master, forming a master-slave configuration.

[0045] However, the scarfing robot 11 does not necessarily need user 101 to operate (including partial operation, not the entire operation). For example, the scarfing robot 11 may operate based on a program or the like, even without user 101's input.

[0046] [Example of hardware configuration of control device 10] Figure 2 shows an example of hardware such as a control device. For example, the control device 10 has a hardware configuration that includes a CPU (Central Processing Unit, hereinafter referred to as "CPU10H1"), a storage device 10H2, an input device 10H3, an output device 10H4, and a communication device 10H5. In other words, the control device 10 is an information processing device such as a PC (Personal Computer) or a server. Note that the control device 10 may consist of multiple information processing devices.

[0047] The CPU 10H1 is an arithmetic unit and control unit. Therefore, the CPU 10H1 performs calculations in cooperation with the memory device 10H2 and other components to execute processing and control.

[0048] The storage device 10H2 is a memory device, etc. Therefore, the storage device 10H2 stores data, etc. The storage device 10H2 may also have an auxiliary storage device.

[0049] The input device 10H3 is, for example, a keyboard or a mouse. Thus, the input device 10H3 is a device that receives data from an external device or from human input.

[0050] The output device 10H4 is, for example, a display. Thus, the output device 10H4 is a device that outputs data to an external device or displays information to a person.

[0051] The communication device 10H5 is a device that transmits and receives data to and from external devices via wired or wireless communication.

[0052] The control device 10 may further include sensors, computing devices, control devices, input devices, output devices, memory devices, communication devices, or auxiliary devices, either internally or externally.

[0053] [Example of a master-slave configuration environment] Figure 3 shows an example of a master-slave configuration environment. The following explanation will use a metal 102 as the target object. Note that in the following system configuration example, the PLC 13 and other components will be omitted.

[0054] In metal 102, the plane on which scarfing is performed is defined as the "XY plane." Therefore, with respect to metal 102, the left-right direction (i.e., the horizontal direction) is defined as the "X-axis direction." Conversely, the up-down direction (i.e., the vertical direction) is defined as the "Y-axis direction." The depth direction is defined as the "Z-axis direction." Furthermore, in the following example, scarfing is assumed to be performed primarily in the direction of the Z-axis.

[0055] The scarfing robot 11 has nozzles 15. However, there may be two or more nozzles 15.

[0056] Hereinafter, the environment in which the scarfing robot 11 and the metal 102 actually exist as hardware, rather than in a virtual environment, will be referred to as the "real environment 201". Therefore, in the real environment 201, the scarfing robot 11 and the metal 102 are actually placed, and the scarfing robot 11 actually operates.

[0057] In the actual environment 201, the control device 10 is the master and the scarfing robot 11 is the slave. Furthermore, it is desirable that the master and slave are separated by distance, meaning the scarfing robot 11 is operated remotely by commands input to the control device 10.

[0058] In a real-world environment 201, the scarfing robot 11 is installed around metal 102. Therefore, the scarfing robot 11 often operates in environments that are harsh on humans, such as high temperatures. For this reason, it is desirable that the control device 10, i.e., the user 101, can remotely control the scarfing robot 11 from a location away from it.

[0059] For example, in the control device 10, the user 101 inputs an operation to instruct the scarfing robot 11 to perform operations such as translation, rotation, or scarfing. Based on such input, the scarfing robot 11 performs operations such as scarfing and translation.

[0060] [Simulation and simulation environment examples] Figure 4 shows an example of a simulation environment. For example, the control device 10 virtualizes the scarfing robot 11 in the real environment 201 to construct the simulation environment 202.

[0061] The simulation environment 202 is a virtual space that virtualizes the surrounding environment and a virtual model that performs the same operations as the scarfing robot 11 in the actual environment 201. Note that the virtual model and virtual space may differ from those in the actual environment 201.

[0062] Hereinafter, the virtual model of the scarfing robot 11 in the actual environment 201 will be referred to as "virtual robot 21". Similarly, the virtual model of the metal 102 in the actual environment 201 will be referred to as "virtual metal 112". Note that the simulation environment 202 may also include other devices, etc., that are virtualized in addition to the virtual robot 21 and virtual metal 112.

[0063] Furthermore, in the simulation environment 202, virtual models such as the virtual robot 21, which are displayed on the simulation environment 202 and operate based on user input, are sometimes referred to as "virtual operating entities."

[0064] Once the simulation environment 202 is established, just like in the real environment 201, when an operation to operate the scarfing robot 11 is input to the control device 10, the virtual robot 21 and other components will operate in the simulation environment 202 based on that operation. Therefore, even without the real environment 201, the user 101 can input an operation to operate the virtual robot 21 in the simulation environment 202 and confirm what kind of actions are performed based on that operation.

[0065] The state in which the simulation environment 202 is constructed is equivalent to the state in which the scarfing robot 11 is connected, even if the equipment of the actual environment 201, in particular the scarfing robot 11, is not present (however, even if the scarfing robot 11 actually exists and is connected to the control device 10 via a network, etc., it is sufficient if the power is not turned on or it is not accepting commands).

[0066] In simulation environment 202, teaching (also known as "robot teaching," "instruction," or "teaching operation") is performed.

[0067] Teaching is the process of inputting in advance how the scarfing robot 11 should operate in the real environment 201. For example, teaching is performed by operating a virtual robot 21 in the simulation environment 202. In other words, teaching is the process of reproducing how the scarfing robot 11 should operate in the real environment 201 under the simulation environment 202.

[0068] Figure 5 shows an example of completed teaching. Hereafter, it is assumed that "teaching data 103" is generated when teaching is performed. That is, if teaching data 103 is available, the scarfing robot 11 will perform the actions indicated by the teaching data 103 in the actual environment 201.

[0069] Therefore, by using the teaching data 103, the scarfing robot 11 can reproduce the same behavior in the real environment 201 as it did in the simulation environment 202.

[0070] The teaching data 103 (including cases where only a portion is generated or modified) may be generated in a format such as code input. For example, the teaching data 103 shows the position, velocity, acceleration, angle (such as changes in posture due to rotation), and the content of the work performed by the scarfing robot 11 in the actual environment 201 in chronological order.

[0071] Therefore, the teaching data 103 may be a collection of data indicating coordinate positions, etc. However, the format of the teaching data 103 is not limited as long as it can instruct the scarfing robot 11 on what to do. In addition, the teaching data 103 may be converted in format and optimized using conversion software, etc.

[0072] Therefore, in the simulation environment 202, the operation of the scarfing robot 11 in the actual environment 201 is considered, and teaching data 103 is generated. Next, once the teaching data 103 is generated, the scarfing robot 11 can operate without any operation by the user 101. For example, when performing similar tasks repeatedly, the user 101 does not need to repeatedly input information for each task.

[0073] Furthermore, in the actual environment 201, the scarfing robot 11 does not need to be connected to the control device 10 if teaching data 103 is available. In other words, after teaching, the scarfing robot 11 can operate unmanned and automatically in the actual environment 201.

[0074] However, even if the scarfing robot 11 is operating based on the teaching data 103, if the scarfing robot 11 receives an operation from the control device 10, it may be configured to prioritize and execute instructions from the control device 10.

[0075] [Example of nozzle 15] The nozzle 15 used for scarfing has, for example, the following configuration. However, the nozzle 15 may have a configuration other than that shown below.

[0076] Figure 6 shows an example of a round nozzle. Specifically, Figure 6(A) is a perspective view of a round nozzle, while Figure 6(B) is a front view of the round nozzle. Note that the figures are from Nippon Steel Corporation. I quote from Technical Report No. 364, "Automatic Slab Check Scarfing Device: Auto Defect-Detector with Scarfer for Slabs," published in 1997, authored by Hayato Kanayama, Kenji Mori, Jiro Matsuo, Katsumasa Konno, Mitsuru Sakakibara, and Shinichiro Sasamori.

[0077] The circular nozzle has a configuration in which, for example, multiple oxygen holes 151 for releasing oxygen and LPG holes 152 for releasing LPG are arranged in a circular pattern on the XY plane.

[0078] Furthermore, the round nozzle has a scarf oxygen pore 153 that releases scarf oxygen at the central position where the oxygen pores 151 and LPG pores 152 are located. In the round nozzle, the scarf oxygen pore 153 is round in shape.

[0079] In the nozzle 15 described above, the number, positional relationship, ratio, and diameter of the oxygen holes 151 and LPG holes 152 are not limited to those described above. Furthermore, the oxygen holes 151 and LPG holes 152 may also emit gases other than oxygen and LPG. Similarly, the shape, diameter, and position of the scarf oxygen holes 153 are not restricted.

[0080] The oxygen pores 151 and LPG pores 152 are holes that release gas to create a preheating flame for preheating the object. The preheating flame generated by the gas heats the object (sometimes only in certain areas). When the scarf oxygen released from the scarf oxygen pores 153 is blown onto the heated area, the defective area is removed.

[0081] [Example of scarfing] Figure 7 shows an example of scarfing. Hereafter, the object is assumed to be a plate-shaped metal 102 with its length in the Z-axis direction. Note that the nozzle 15 may not be in the orientation, size, shape, or installation position shown in the figure.

[0082] For example, while performing the scarfing operation, the scarfing robot 11 moves in the Z-axis direction, that is, along the longitudinal direction of the metal 102. And, if the nozzle 15 and other degrees of freedom are not moved, the scarfing work position moves in the Z-axis direction due to the translation of the scarfing robot 11.

[0083] In this way, the scarfing robot 11 can perform scarfing in a linear manner in the Z-axis direction.

[0084] However, the scarfing is not limited to the above in terms of position, range, and depth. For example, the scarfing robot 11 may perform scarfing other than that described above by combining translation and rotation. Furthermore, the scarfing robot 11 may perform scarfing other than that described above by changing the conditions and posture of the nozzle 15.

[0085] Specifically, the scarfing robot 11 may perform translational movements in two dimensions along the plane of the metal 102, i.e., the XZ plane. Alternatively, the scarfing robot 11 may be installed in a suspended ceiling configuration, for example.

[0086] Furthermore, it is desirable that sparks generated during scarving be captured by a camera. Therefore, it is desirable that the scarving robot 11 be equipped with a camera or other imaging device capable of capturing images of the area being worked on during scarving. It is also desirable that image data capturing the work situation, including sparks, be generated.

[0087] Sparks can occur when the flow of molten metal is obstructed, causing some of the metal to scatter into the air and react with high-pressure oxygen, etc., through oxidation. Therefore, sparks often emit a bright light. Furthermore, the luminescence of a spark is very short, lasting approximately 100 to 500 milliseconds.

[0088] Scarfing allows the quality of the work to be confirmed by observing the sparks. Therefore, if sparks are captured, the user 101 can confirm them even when operating remotely.

[0089] Furthermore, areas where sparks and scarving operations are performed are often brighter, or have higher luminance, compared to other areas. Therefore, a camera with a high dynamic range is desirable. In addition, since sparks often travel at high speeds, a camera capable of shooting at a high frame rate of 60fps or higher is desirable. Alternatively, high-luminance subjects such as sparks may be photographed and then processed. Another option is to place a filter in front of the camera to photograph the sparks.

[0090] Figure 8 shows an example of a defect. The following explanation will use an example of a defect in a plate-shaped metal 102. Specifically, if a defect is found in the metal 102, a "defect region 301" is set for the area of ​​the defect. For example, an area with a "flaw" on the surface of the metal 102 is set as a defect region 301.

[0091] The defect area 301 may be set by visual inspection or by detection using a sensor such as a camera. For example, the defect area 301 may be set based on an image of the metal 102 taken in the actual environment 201, or on data of the metal 102.

[0092] Once the defect area 301 is defined, for example, the location, range, and depth of the defect are determined. Note that there may be other settings for the defect area 301.

[0093] Furthermore, defects may occur on surfaces other than the flat surface of the metal 102. For example, defects may occur on the sides or edges of the metal 102.

[0094] Figure 9 shows an example of scarfing settings. For example, the following scarfing settings are applied to the defect region 301 shown in Figure 8.

[0095] For example, the area where scarfing will be performed is set with "arrow 302". As in this example, arrow 302 is set so that the defect area 301 is sufficiently subject to scarfing.

[0096] Arrow 302 indicates the direction in which the arrow 302 is directed on the plane (XZ plane) of the metal 102. Furthermore, the starting point of arrow 302 becomes the starting point for the scarfing operation. Therefore, the setting of arrow 302 determines the position, range, number of times, and direction of the scarfing operation.

[0097] Furthermore, settings such as the position, range, number of repetitions, and direction of the scarfing operation may be configured using a GUI (Graphical User Interface) other than arrow 302. For example, these settings may be configured by inputting coordinate values.

[0098] Figure 10 shows an example of setting the height in scarfing. For example, it is desirable that the "Height H" can be set as a setting item in scarfing, as shown below.

[0099] Height H is the distance between the nozzle 15 performing scarfing and the surface of the metal 102 (in the diagram, this is the distance on the Y-axis. Since this distance is the shortest distance between the nozzle 15 and the surface of the metal 102, the point of reference will differ depending on the orientation of the nozzle 15 and the metal 102). For example, Height H can be set numerically, such as "〇 millimeters," or it can be set by selecting from a predetermined number of levels (for example, three levels such as "far," "intermediate," or "close").

[0100] The height H may be set, for example, for each defect region 301, or for each scarfing operation (each arrow 302 in Figure 9). Alternatively, the height H may be set to be constant (fixed, i.e., common to operations set by multiple arrows 302) in the metal 102.

[0101] Being able to set the height H allows for a more accurate reproduction of the scarfing process results in the simulation.

[0102] Figure 11 shows an example of angle settings in scarfing. For example, it is desirable that the "angle θ" can be set as a setting item for scarfing, as shown below.

[0103] The angle θ is the relative angle between the nozzle 15 performing scarfing and the metal 102 (in the figure, the metal 102 is parallel to the XY plane, and the angle θ is set on the vertical axis. Depending on the orientation of the nozzle 15 and the metal 102, the angle referred to will differ). For example, the angle θ may be set numerically, such as "X degrees," or it may be set by selecting from a predetermined number of stages (for example, three stages such as "large incline," "medium," or "small incline").

[0104] The angle θ may be set, for example, for each defect region 301, or for each scarfing operation (each arrow 302 in Figure 9). Alternatively, the angle θ may be set to be constant (fixed, i.e., common to operations set by multiple arrows 302) in the metal 102.

[0105] Being able to set the angle θ allows for more accurate results from scarfing simulations.

[0106] Additionally, scarfing may be configured to operate while moving in parallel. The movement speed may be synchronized with the operation, or set using a speed coefficient, etc. Furthermore, the parallel movement may be configured in a way that allows setting a start point and an end point.

[0107] [Simulation Example] Figure 12 shows an example of depth output in scarfing. For example, depth can be represented by four levels (from "shallow" to "deep") using color. However, depth may be divided into fewer than four levels, or into five or more levels. Furthermore, depth is not limited to color; it may also be represented by hatching or other methods.

[0108] "Deep" refers to a removal state where a large amount of material is removed by scarfing. On the other hand, "shallow" refers to a removal state where a small amount of material is removed by scarfing.

[0109] The depth is determined by parameters such as the speed at which the nozzle 15 is moved, the angle between the nozzle 15 and the object, the distance between the nozzle 15 and the object, the preheating conditions, the mixing ratio of flammable gas and oxygen in the gas, and the nozzle structure. Therefore, by performing a simulation based on these conditions, it is possible to accurately simulate the depth that will be achieved during the scarfing process.

[0110] Figure 13 shows an example of a scarfing trace simulation. For example, if the settings are as shown in Figure 9, the result of the scarfing work (hereinafter, the area after scarfing will be referred to as "scarfing trace 303") will be simulated as follows.

[0111] The scarfing marks 303 are simulated, for example, by color-coding their depth as shown in Figure 12.

[0112] Thus, the simulation reproduces the location, extent, and depth of the scarfing marks 303. However, the simulation may also simulate other aspects.

[0113] Figure 14 shows an example of the output of a scarfing simulation. For example, it is desirable that the simulation output a magnified view of the area where scarfing is occurring.

[0114] In the simulation, it is desirable that the smoke and sparks produced by scarfing are hidden or made transparent. In the real environment 201, when scarfing occurs, smoke is produced, making it difficult to see with the naked eye.

[0115] On the other hand, in simulation environment 202, smoke is output as hidden. In this way, in simulation environment 202, even when work is being done, the view is not obstructed by smoke, making it easy to see where scarfing is being performed.

[0116] Furthermore, the sparks generated by scarving are very bright. Therefore, viewing these sparks in a real environment (201) may harm eye health. On the other hand, in the simulation environment (202), the sparks are either hidden or output at a light intensity that does not harm eye health.

[0117] Thus, in simulation environment 202, even when work is being done, there is no strong light, making it easy to see the areas where scarfing is being performed.

[0118] Furthermore, cross-sectional views and similar formats are difficult to view in the actual environment 201. Therefore, it is desirable that the simulation output can be displayed from viewpoints that are difficult to view in the actual environment 201, such as cross-sectional views.

[0119] [Example of virtual area] Figure 15 is a diagram (part 1) showing an example of setting up a virtual domain. The following explanation will use an example where the nozzle 15 (hereinafter, the trajectory of the nozzle 15's movement is referred to as the "nozzle trajectory") moves along the trajectory shown by the dotted line. For example, user 101 can view the simulation results on an output screen like the one shown, or perform operations related to the simulation.

[0120] Figure 15 shows an example of an operation screen for scarfing. As shown in the figure, the operation for the scarfing simulation involves inputting the defect area to be worked on, its position (the start and end points of the work), distance, angle, and speed. These input results are shown below as "movement arrows 16". Therefore, the above input items may be entered numerically as coordinates, or they may be entered in a format such as specifying the position using an input device.

[0121] As described above, when an instruction to perform scarfing is input, arrow 302, i.e., the simulation of the scarfing to be performed, is executed.

[0122] The starting point of the movement arrow 16 indicates the position where the operation was input. The length of the movement arrow 16 indicates the force, velocity, or acceleration applied to the nozzle 15. Furthermore, the ending point of the movement arrow 16 indicates the indicated direction.

[0123] For example, the virtual areas are configured as follows: Virtual Area 1 V1, Virtual Area 2 V2, and so on.

[0124] A virtual region is an area set up in the simulation environment 202 and does not exist in the real environment 201. Entry of virtual operating objects into the virtual region is restricted. Therefore, in this example, since the first virtual region V1 and the second virtual region V2 are set up, even if the movement arrow 16 is directed towards the first virtual region V1 and the second virtual region V2 (in the figure, the X-axis direction), it will be ineffective. Consequently, the nozzle 15 cannot enter the areas where the first virtual region V1 and the second virtual region V2 are set up, and no work is performed.

[0125] Thus, once a virtual region is created, it becomes possible to perform translation along any object (in this example, the surface of the first virtual region V1), a so-called "following motion." Note that this is not limited to translation; it could also be done by keeping the height (the position of the nozzle 15 in the Y-axis direction) constant, for example.

[0126] Furthermore, inputs that cause noise during user 101's operation, such as camera shake, can be canceled out.

[0127] Furthermore, during operation, the control device 10 may receive feedback when it is determined that it is in contact with the surface of the first virtual region V1. For example, if the input device 10H3 is equipped with an actuator, tactile feedback may be provided indicating that it is in contact with the surface of the first virtual region V1.

[0128] Figure 16 is a diagram (part 2) showing an example of setting up a virtual area. Compared to Figure 15, the X-axis position of the movement arrow 16 in Figure 16 is different from that in Figure 15. That is, Figure 16 is an input that performs scarfing separately from Figure 15, for example, after the scarfing that is performed based on the operation entered in Figure 15. Specifically, Figure 16 is an example of giving an instruction to perform scarfing in the Z-axis direction, similar to Figure 15. On the other hand, in Figure 16, the X-axis position is different from that of Figure 15; that is, the movement arrow 16 entered in Figure 16 is entered parallel to that of Figure 15.

[0129] In this case, the first virtual region V1 and the second virtual region V2 are moved to match the position where scarfing is performed, i.e., the position of the movement arrow 16. Therefore, in Figure 16, the positions of the first virtual region V1 and the second virtual region V2 are different from those in Figure 15.

[0130] In Figure 16, as in Figure 15, when performing a "copy operation," if the positions of the first virtual area V1 and the second virtual area V2 can be changed, as shown in the change from Figure 15 to Figure 16, it is possible to easily handle cases where the "copy operation" is performed multiple times.

[0131] Figure 17 shows an example of a limitation imposed by a virtual domain. For example, let's explain using an example where a second virtual domain V2 is set. Next, let's assume that the virtual operating object is moved in parallel to the second virtual domain V2 in the first movement direction DR1.

[0132] For such operations, the virtual operating object is not restricted from translation by the second virtual region V2 until it reaches a position where it contacts the surface of the second virtual region V2 (hereinafter, the position where the virtual operating object contacts the surface of the second virtual region V2 is referred to as the "contact point TP"). It then moves in the first movement direction DR1.

[0133] Next, when the virtual moving body moves in parallel to the contact point TP, that is, when the virtual moving body comes into contact with the surface of the second virtual region V2, the virtual moving body then moves in parallel in the second movement direction DR2 along the surface of the second virtual region V2. Hereafter, the corrected direction will be referred to as the "corrected direction".

[0134] In this example, the correction direction is the second movement direction DR2. That is, initially it was the first movement direction DR1, but after correction, it becomes the correction direction of the second movement direction DR2.

[0135] In this example, the first movement direction DR1 includes the directional component of the second movement direction DR2 as its second directional component. Therefore, based on the shape of the second virtual region V2 and the position it is set at, the virtual moving body is corrected so that the translation of the first movement direction DR1 is translated parallel to the second movement direction DR2 from the contact point TP, while limiting the first directional component. In this way, the virtual region can be set to various shapes and positions.

[0136] [Example of virtual permitted area] Figure 18 shows a first example of setting up a virtual permission area. For example, for the metal 102 shown in Figure 8, an area (hereinafter referred to as the "virtual permission area") may be set up for the locations where work is permitted, as follows.

[0137] The first virtual permitted area V10 is configured in the same way as the first virtual area V1 and the second virtual area V2. However, the difference is that while the first virtual area V1 and the second virtual area V2 restrict the nozzle 15 from entering the configured area, the virtual permitted area restricts the nozzle 15 from entering areas other than the configured area. Therefore, the configured area of ​​the virtual permitted area, such as the first virtual permitted area V10, is where scarfing operations can be performed.

[0138] Furthermore, the virtual permission area, like the virtual area, is set up in the simulation environment 202 and does not exist in the actual environment 201.

[0139] For example, the first virtual permission area V10 is set to have a predetermined "thickness" relative to the defect area 301. Hereinafter, "thickness" refers to the space created between the metal 102 and the first virtual permission area V10. The direction of the "thickness" can be set and any direction is acceptable.

[0140] Specifically, the first virtual permission area V10 is set to be larger than the defect area 301 by a set value of "thickness" in any of the X-axis, Y-axis, or Z-axis directions. In other words, the first virtual permission area V10 is set to cover the defect area 301. However, the area in which the nozzle 15 can enter may be set to be wider than the area in which the first virtual permission area V10 is set. The virtual permission area may have a so-called "margin" in relation to the width of the area to be set.

[0141] The "thickness," that is, how much larger the virtual permitted area should be than the defective area 301, can be set in advance, for example.

[0142] When the first virtual permission area V10 is set for the defective area 301, operations can be performed on the defective area 301, while operations on other areas are restricted. In this way, setting a virtual permission area prevents accidental operations that would cause operations on areas other than the defective area 301 to be performed incorrectly. Furthermore, if the defective area 301 is smaller in area than a normal area, i.e., an area where operations are not required, setting a virtual permission area requires setting a smaller area than setting a virtual area, thus reducing the amount of operations required.

[0143] Figure 19 shows a second example of the virtual permission area configuration. Compared to Figure 18, Figure 19 differs in that the shape of the virtual permission area is the same as that of the second virtual permission area V11. The following explanation will focus on the differences, omitting redundant explanations.

[0144] As shown in Figure 19, the virtual authorization area does not need to have equal areas in the X, Y, and Z directions. For example, a virtual authorization area that is frustoconical in shape, like the second virtual authorization area V11, is easy to use. Specifically, the second virtual authorization area V11 has a shape in which the circular area expands upward in the Y direction. Therefore, the second virtual authorization area V11 is wider at the top and the area of ​​the XZ plane narrows downward, i.e., as it approaches the defect area.

[0145] Furthermore, the second virtual permission area V11 is not limited to a frustum of a cone; for example, it may have steps. With the above configuration, a simulation environment 202 can be constructed that simulates the operation of the scarfing robot 11 in the actual environment 201. Then, when the movement of the nozzle 15, etc., can be input in the simulation environment 202, the scarfing robot 11 can be taught in the simulation environment 202.

[0146] Therefore, once teaching is performed in the simulation environment 202 and teaching data 103 is generated, the scarfing robot 11 can be automated.

[0147] [Example of a change operation] Figure 20 shows an example before any changes are made. The following explanation will use an example where the settings shown in the figure (hereinafter referred to as "initial settings 401") are used in the operation screen. Furthermore, in the following example, as in Figure 9, the position to be worked on will be entered using arrow 302.

[0148] For the sake of explanation, the following examples omit the assumption that the vertical position of the work (the Y-axis direction in Figure 9 and the height H parameter shown in Figure 10) is constant (for example, the work is performed at a pre-set height H). Therefore, the following operations will only involve inputting the depth direction (Z-axis direction) and the left-right direction (X-axis direction). However, other operations may also be used to modify these parameters.

[0149] For example, the initial setting 401 is an example where four arrows 302 are set. Hereafter, the four arrows 302 will be referred to as "first arrow 3021", "second arrow 3022", "third arrow 3023", and "fourth arrow 3024", respectively.

[0150] The first arrow 3021, the second arrow 3022, the third arrow 3023, and the fourth arrow 3024 all have the same starting and ending points in the Z-axis direction. On the other hand, the first arrow 3021, the second arrow 3022, the third arrow 3023, and the fourth arrow 3024 are in different positions in the X-axis direction. Therefore, the four scarfing operations are performed in parallel, with different positions in the X-axis direction.

[0151] Initial setting 401 is set, for example, by user 101 pre-entering coordinates indicating the starting and ending points. However, initial setting 401 may also be set by AI (Artificial Intelligence) or past performance data. Therefore, it does not matter how initial setting 401 is set.

[0152] Once initial settings such as 401 are configured, the simulation starts in simulation environment 202. Teaching is then performed on the simulation results to modify initial settings 401, etc.

[0153] Specifically, teaching involves the user 101 viewing the initial settings 401 in the simulation environment 202 and modifying them by adding, deleting, or changing arrows 302. The content of the teaching may be learned by the AI ​​and reflected in subsequent processes of performing the initial settings 401. Such learning allows for more accurate setting of the initial settings 401.

[0154] For example, the tasks to be simulated are executed in the simulation environment 202 in the order of the first arrow 3021, the second arrow 3022, the third arrow 3023, and the fourth arrow 3024 (this is an example of the order in the X-axis direction; however, the order of tasks may be set).

[0155] As described above, after teaching is performed in the simulation environment 202, the scarfing robot 11 starts executing its work in the real environment 201 based on the teaching results.

[0156] When the scarfing robot 11 is performing work in the actual environment 201 and a change operation is received to modify part or all of the work, the control device 10 performs a process to reflect the change operation in the work content after teaching (hereinafter referred to as "change processing"). For example, a change operation may be as follows:

[0157] [Example of a change operation] Figure 21 shows the first example of the change operation. Hereafter, we will assume that even after reflecting the teaching results, the initial setting 401 remains unchanged. That is, before the change operation, the initial setting was 401, and the explanation will be based on an example where the work is performed as shown in Figure 20.

[0158] Compared to Figure 20, that is, the initial setting 401, Figure 21 differs in that a change operation (hereinafter referred to as the "cancellation operation") has been performed to undo the third arrow 3023. The following explanation will focus on the changes from Figure 20.

[0159] A cancel operation is a modification operation that cancels part of a task, for example, by deleting arrow 302 on the GUI. For example, a cancel operation is accepted until the task to be canceled is scheduled to begin. Specifically, in the case of the third arrow 3023, a cancel operation to cancel the task by the third arrow 3023 is accepted until the execution of the task by the first arrow 3021 or the second arrow 3022, which are executed before the third arrow 3023. However, the acceptance period for modification operations may be set.

[0160] Note that the cancellation operation is not limited to canceling arrow 302. For example, the cancellation operation may also involve canceling other operations or canceling part of an operation.

[0161] Being able to cancel an operation in this way prevents unnecessary tasks from being executed, even if they are already in progress.

[0162] [Second example of a change operation] Figure 22 shows a second example of the change operation. Hereafter, we will assume that even after reflecting the teaching results, the initial setting 401 remains unchanged. That is, before the change operation, the initial setting was 401, and the explanation will be based on an example where the work is performed as shown in Figure 20.

[0163] Compared to Figure 20, i.e., the initial setting 401, Figure 22 differs in that a modification operation (hereinafter referred to as the "addition operation") has been performed to add a fifth arrow 3025 between the third arrow 3023 and the fourth arrow 3024. The following explanation will focus on the changes from Figure 20.

[0164] An additional operation is a modification operation that increases the scope of work by, for example, adding arrow 302 on the GUI. For example, an additional operation is accepted up until the start of the work to be added. Specifically, if the work of the fifth arrow 3025 is to be performed after the work of the third arrow 3023 and before the work of the fourth arrow 3024, an additional operation to add the work by the fifth arrow 3025 will be accepted until the third arrow 3023 is being executed.

[0165] When the fifth arrow 3025 is added, for example, a new operation will be performed between the third arrow 3023 and the fourth arrow 3024. However, the timing of the execution of the fifth arrow 3025 may be set in the add operation. Also, the added operation will be performed at the location indicated by the fifth arrow 3025.

[0166] Note that additional operations are not limited to adding arrow 302. For example, additional operations may include adding other tasks or adding a part of a task.

[0167] This ability to add operations allows you to add missing tasks even while the process is running.

[0168] [Example 3 of modification operations] Figure 23 shows a third example of the change operation. Hereafter, we will assume that even after reflecting the teaching results, the initial setting 401 remains unchanged. That is, before the change operation, the initial setting was 401, and the explanation will be based on an example where the work is performed as shown in Figure 20.

[0169] Compared to Figure 20, i.e., the initial setting 401, Figure 23 differs in that the operation to enlarge the work content of the third arrow 3023 (hereinafter referred to as "enlargement / reduction operation") has been performed. The following explanation will focus on the changes from Figure 20. Below, the result after the enlargement / reduction operation on the third arrow 3023 is shown as the "sixth arrow 3026".

[0170] A zoom in / out operation is a modification operation that changes the area being worked on by, for example, zooming in or out on arrow 302 on the GUI. For example, a zoom in / out operation is accepted until the start of the target work. Specifically, in the case of the third arrow 3023, a zoom in / out operation to enlarge the work performed by the third arrow 3023 is accepted until the work of the first arrow 3021 or the second arrow 3022, which is executed before the third arrow 3023, is being performed.

[0171] The sixth arrow 3026 is an example of a modification that expands the Z-axis direction compared to the third arrow 3023 (which is before the scaling operation), resulting in a wider area for the work to be performed. Therefore, when the sixth arrow 3026 is set, the third operation to be performed is modified to be performed more widely in the Z-axis direction than with the third arrow 3023, i.e., the initial setting 401.

[0172] Note that zooming in and out is not limited to zooming in using arrow 302. For example, zooming in and out may also involve canceling other operations or canceling part of an operation. Furthermore, zooming in and out is not limited to zooming in to enlarge the area being worked on, but may also include instructing an operation to shrink the area being worked on.

[0173] Being able to zoom in and out in this way allows you to change the process even while it's running, so that it adds missing parts to the work, or prevents you from working on unnecessary parts.

[0174] [Example 4 of modification operations] Figure 24 shows the fourth example of the change operation. Hereafter, we will assume that even after reflecting the teaching results, nothing has changed from the initial setting 401. That is, before the change operation, the initial setting was 401, and the explanation will be based on an example where the work is performed as shown in Figure 20.

[0175] Compared to Figure 20, i.e., the initial setting 401, Figure 24 differs in that an operation to change the trajectory of the third arrow 3023 (hereinafter referred to as "trajectory operation") has been performed. The following explanation will focus on the changes from Figure 20. The "seventh arrow 3027" will be shown below after the trajectory operation on the third arrow 3023.

[0176] Trajectory manipulation is a modification operation that changes the location where work is performed by altering the trajectory of arrow 302 on the GUI, for example. Trajectory manipulation is accepted up until the start of the target work. Specifically, in the case of the third arrow 3023, scaling operations to enlarge the work performed by the third arrow 3023 are accepted until the work of the first arrow 3021 or the second arrow 3022, which is executed before the third arrow 3023, is in progress.

[0177] Before accepting the change operation, the third arrow 3023 (before the track operation) was a straight line, whereas the seventh arrow 3027 differs in that its track curves midway. Thus, track operation is an operation that changes the location where work is performed, such as by bending arrow 302.

[0178] Note that track manipulation is not limited to bending arrow 302. For example, if the track manipulation involves changing the location where work is being performed, other methods of operation may also be used.

[0179] This ability to manipulate the trajectory allows for changes to the trajectory of the operation even while it is in progress.

[0180] Note that the modification operation is not limited to the examples above. For example, the modification operation may be a combination of the first to fourth examples above.

[0181] For example, the change operation could be something like changing the speed at which the work is performed.

[0182] [Overall processing example] Figure 25 shows an example of the overall processing in the first embodiment. For example, the control device 10 performs the following processing.

[0183] In step S01, the control device 10 constructs the simulation environment 202.

[0184] In step S02, the control device 10 performs teaching in the simulation environment 202. For example, user 101 performs teaching by inputting various operations on the simulation results in the simulation environment 202. Based on the results of this teaching, initial settings 401, etc., are changed.

[0185] In step S03, the control device 10 instructs the scarfing robot 11 to perform a task based on the teaching results in the simulation environment 202.

[0186] For example, steps S04 and S05 are executed in parallel with step S03, that is, while the work is being executed. However, steps S04 and S05 may also be executed by temporarily interrupting step S03.

[0187] In step S04, the control device 10 determines whether or not it has received a change operation. If it receives a change operation during the execution of the task (YES in step S04), the control device 10 proceeds to step S05. On the other hand, if it does not receive a change operation during the execution of the task (NO in step S04), the control device 10 proceeds to step S04, that is, it repeats step S04 and waits for input of a change operation during the execution of the task.

[0188] In step S05, the control device 10 modifies the work being performed in the actual environment 201 (i.e., the work being performed in step S03) to reflect the change operation.

[0189] [Example of Functional Configuration] Figure 26 shows an example of the functional configuration of the first embodiment. For example, the control system 100 includes an input unit 100F1, a simulation unit 100F2, a teaching unit 100F3, an execution unit 100F4, and a modification unit 100F5.

[0190] The input unit 100F1 performs an input procedure to receive operations from the user 101. For example, the input unit 100F1 can be implemented using an input device 10H3 or the like.

[0191] The simulation unit 100F2 performs a simulation procedure to construct the simulation environment 202 based on the operations performed in the input unit 100F1. For example, the simulation unit 100F2 is implemented using a CPU 10H1 or the like.

[0192] The teaching unit 100F3 performs teaching procedures for the industrial robot in the simulation environment 202. For example, the teaching unit 100F3 can be implemented using an input device 10H3, etc.

[0193] The execution unit 100F4 performs an execution procedure to cause the industrial robot to perform a task in the actual environment 201, based on the teaching results. For example, the execution unit 100F4 is implemented using a CPU 10H1 or the like.

[0194] The modification unit 100F5, upon receiving a change operation during the execution of a task, performs a change procedure that reflects the change operation and modifies the task currently running in the actual environment 201. For example, the modification unit 100F5 is implemented by a CPU 10H1 or the like.

[0195] With the above configuration, the control system 100 can teach the industrial robot remotely. Therefore, the user 101 can teach the industrial robot and automate the work even from a distance. Furthermore, even after the work is automated by the industrial robot, the control system 100 can accept human intervention to make changes during execution. Being able to make such changes allows the work to be modified in real time during execution.

[0196] In some cases, it can be difficult to determine the next steps in a task until a certain amount of work has been completed. For example, in scarfing, a part of the object may be removed, and the user 101 may look inside (or, in the case of remote operation, view images taken of the inside or the sensing results from sensors) to decide on the next steps.

[0197] Specifically, if there are defects inside the object, such as voids, cavities, or foreign matter, it is desirable for the control system 100 to perform additional work to remove the newly discovered defects inside. Therefore, if a modification operation is performed to add the removal of newly discovered defects inside at the stage when the inside can be inspected during the execution of the work, the quality of the work can be improved.

[0198] On the other hand, redoing all or most of a task reduces work efficiency. Therefore, as mentioned above, being able to make changes while the task is being executed can improve efficiency.

[0199] [Second Embodiment] The second embodiment will now be described. Components similar to those in the first embodiment will be denoted by the same reference numerals and their descriptions will be omitted. For example, the second embodiment uses a control device 10, etc., with the same system configuration and hardware configuration as the first embodiment. However, the second embodiment differs from the first embodiment in that it uses AI.

[0200] [About AI] The AI ​​used in this embodiment learns based on training data through "preprocessing." Hereinafter, the AI ​​that is the target of the learning stage, i.e., "preprocessing," will be referred to as "learning model A1." As learning progresses through "preprocessing," learning model A1 becomes "trained model A2." Hereinafter, the execution stage in which output processing is performed using trained model A2 will be referred to as "execution processing."

[0201] However, even with a pre-trained model A2, training may be performed after pre-processing, similar to the training model A1. Therefore, the AI ​​may, for example, perform "execution processing" and then perform "pre-processing" on the data used in the "execution processing" and add it to the training. Also, the term "AI" is sometimes used to refer to both the training model A1 and the pre-trained model A2 without distinction.

[0202] Figure 27 is a network configuration diagram showing an example of an AI configuration. The learning model A1 and the trained model A2 are AIs with the configuration shown in network configuration 300 below, for example.

[0203] In the following explanation, the learning model A1 and the trained model A2 will be described using an example where they are implemented on the cloud. However, some or all of the learning model A1 and the trained model A2 may be implemented on the control device 10, etc.

[0204] The network configuration 300 is, for example, a configuration having an input layer L1, an intermediate layer L2 (also called a "hidden layer," etc.), and an output layer L3, etc.

[0205] The input layer L1 is the layer into which data is input.

[0206] The hidden layer L2 transforms the data input to the input layer L1 based on weights (e.g., coefficients used for multiplication) and biases (e.g., adding constants). The results processed in the hidden layer L2 are then transmitted to the output layer L3.

[0207] The output layer L3 is the layer that outputs the output content, etc.

[0208] Through learning, the weight coefficients (for example, the coefficients for input characters or images are changed during learning) and the parameters that are changed during learning are optimized. Note that the network configuration 300 is not limited to the configuration shown in the figure. In other words, the AI ​​may be implemented using other machine learning methods.

[0209] For example, the AI ​​may be configured to perform preprocessing such as dimensionality reduction (for instance, transforming a relationship with three or more dimensions into a relationship that can be obtained through simplified calculations of three dimensions or less) using unsupervised machine learning. Ideally, the relationship between input and output should be processed using simple calculations such as linear equations. Such calculations can reduce computational costs.

[0210] Furthermore, the AI ​​may undergo processes to mitigate overfitting (also known as "overfitting" or "over-adjustment"), such as dropout. Other preprocessing steps, such as dimensionality reduction and normalization, may also be performed.

[0211] AI may have a network structure such as a CNN (Convolutional Neural Network). Alternatively, the network structure may also have configurations such as an RNN (Recurrent Neural Network) or LSTM (Long Short-Term Memory). In other words, AI may have a network structure other than deep learning.

[0212] Furthermore, the AI ​​may have a configuration that includes hyperparameters. That is, the AI ​​may be configured such that some settings are made by user 101 or the like. In addition, the AI ​​may specify the features to be learned, or user 101 may set some or all of the features to be learned.

[0213] Furthermore, the learning model A1 and the trained model A2 may utilize other machine learning methods. For example, the learning model A1 and the trained model A2 may undergo preprocessing such as normalization using unsupervised models. Moreover, the learning method may be reinforcement learning (a method in which the AI ​​is made to make choices and given evaluations (rewards) for those choices, and the learning method aims to increase the evaluation).

[0214] During training, data augmentation may be performed. That is, to increase the amount of training data D1 used to train the learning model A1, preprocessing may be performed to augment one experimental dataset or similar data into multiple training datasets D1. Increasing the amount of training data D1 in this way allows for further training of the learning model A1.

[0215] Furthermore, the learning model A1 and the trained model A2 may be configured to perform transfer learning or fine tuning. In other words, since the execution environment of the control device 10 often differs from device to device, the settings may differ for each device to match the execution environment. For example, the basic configuration of the AI ​​is learned on a separate information processing device. After that, each information processing device may undergo additional learning or configuration to optimize it for its respective execution environment.

[0216] As described above, AI may be applied to the present invention. For example, a trained model is generated by training a learning model with processes that are executed repeatedly. The training data used for training includes data that indicates the content of the process as the correct answer, and also includes data that indicates the target of the process. A learning model that performs deep learning is trained using such training data.

[0217] Furthermore, AI may be applied to various areas such as image recognition or input assistance.

[0218] "Preprocessing" is performed before "execution." However, "preprocessing," i.e., additional training of the trained model A2, may be performed immediately after "execution." The following example explains how to perform "preprocessing" and "execution" separately.

[0219] [Example of pre-processing] Figure 28 shows an example of preprocessing. For example, preprocessing is performed by the control device 10. However, preprocessing may be performed by devices other than the control device 10. For example, preprocessing may be performed by multiple information processing devices.

[0220] Learning model A1 learns by inputting the following learning data D1. In other words, learning model A1 performs what is known as "supervised learning."

[0221] The training data D1 has a data structure that includes, for example, object information D11, defect information D12, work content information D13, and correct answer data D20.

[0222] Object information D11 is data that indicates the characteristics or properties of the object. Specifically, object information D11 may include material, dimensions, manufacturing date, or quality grade. For example, object information D11 may be in the form of text, CAD (Computer-Aided Design) drawings, or images of the object.

[0223] Defect information D12 is data about defects occurring in the object. Specifically, defect information D12 includes the type, dimensions, grade, or location of the defect in the object.

[0224] For example, if defect information D12 is to show the type, dimensions, grade, and location numerically, it may be in a data format such as text. Alternatively, defect information D12 may be data indicating the location of the defect in an image, or data resulting from defect detection by a sensor, etc.

[0225] Work content information D13 is data that shows the details of the scarfing work performed for a defect. For example, work content information D13 is prepared separately for each defect.

[0226] However, if the work content information D13 involves work common to multiple defects (including cases where some information is common), the information may be standardized. Specifically, the work content information D13 consists of work conditions such as the starting point position where the work begins, the ending point position where the work ends, the height H at which the work is performed, the angle of the nozzle 15 performing the work, or the speed at which the nozzle 15 is moved. For example, the work content information D13 may be in data format such as text or a collection of coordinate values.

[0227] The correct answer data D20 is data that shows the pass / fail result of the work as a result of scarfing using the above combination of object information D11, defect information D12, and work content information D13. In other words, the correct answer data D20 is data that shows the evaluation result of the quality of the work as a result of performing the work under the work conditions etc. indicated by the work content information D13.

[0228] Ideally, the correct answer data D20 should include image data of sparks generated during scarving. Having such image data allows the AI ​​to learn about the work quality that can be assessed by the sparks.

[0229] Note that the correct answer data D20 may consist of one data point per object, or it may be divided into several points, such as by defect.

[0230] For example, the correct answer data D20 is data entered by the evaluator, such as "Pass" or "Fail" (these can be replaced with "Yes / No" or a binary value, etc.). Note that the correct answer data D20 may also include reasons, etc., in addition to "Pass" or "Fail". Furthermore, the correct answer data D20 may not be limited to a two-tiered system like "Pass" or "Fail", but may also be an evaluation score within a pre-set range, such as from "0" to "100".

[0231] Therefore, the correct answer data D20 is in the form of text or numerical data indicating "Pass" or "Fail," etc.

[0232] Furthermore, the training data D1 may also contain information such as the settings of the nozzle 15, the specifications of the nozzle 15, or the environment in which the work is performed (temperature, humidity, etc.). In addition, big data D4 may be used for preprocessing. For example, big data D4 may be publicly available information on the internet or a set of data acquired by sensors, etc.

[0233] The training data D1 may include text data, image data, audio data, video data, sensor data, gesture data, or a combination of these. In other words, the AI ​​may use multiple forms of data, a so-called multimodal approach.

[0234] In the preprocessing stage, the training data D1 is used to associate "correct answers" with the input combinations of object, defect, and work content. Hereinafter, combinations of object, defect, and work content for which the "correct answer" is known will be referred to as the "first work condition." The data showing the correct answer will be referred to as the correct answer data D20.

[0235] The following explanation will use a pair of examples where each piece of information constituting the training data D1 and each piece of ground truth data D20 are used. However, each piece of information constituting the training data D1 and each piece of ground truth data D20 may consist of multiple data points.

[0236] Furthermore, each piece of information constituting the training data D1, and the ground truth data D20, may be preprocessed during the preprocessing stage. For example, if each data set has multiple input formats, it is desirable to preprocess it by normalizing or standardizing it to the same expression. If the data is normalized and the numerical ranges are unified, the AI ​​can learn with greater accuracy.

[0237] Furthermore, if an expansion process that increases the training data D1 is performed as a preprocessing step, the amount of data that the learning model A1 uses for training can be increased. In this way, increasing the training data D1 allows the AI ​​to learn with greater accuracy.

[0238] Alternatively, preprocessing could involve separating the object into different material types based on object information D11 for learning. By dividing the object into categories based on material and other factors, the AI ​​can learn with greater accuracy.

[0239] Furthermore, if images are included, preprocessing such as filtering the images may be necessary.

[0240] Thus, in the training data D1, the first working condition, whose correct answer is known, is associated with the correct answer for the first working condition as shown in the correct answer data D20. Furthermore, training data D1 is data where the first working condition and the correct answer are paired. Therefore, the preprocessing is a process to train the learning model A1 to learn the correlation between the first working condition and the correct answer.

[0241] Once the above preprocessing is performed, the learning model A1 is trained and a pre-trained model A2 is generated. In the execution process below, the pre-trained model A2 generated in the preprocessing is used.

[0242] [Example of execution process] Figure 29 shows an example of execution processing. For example, the execution processing is performed by the control device 10. However, the execution processing may be performed by devices other than the control device 10. For example, the execution processing may be performed by multiple information processing devices.

[0243] In contrast to pre-processing, pre-processing associates correct data D20 with work conditions, which are a combination of object information D11, defect information D12, and work content information D13, whereas execution processing lacks correct data D20.

[0244] The unknown object D21 is, for example, information with a similar structure to the object information D11.

[0245] The unknown defect information D22 is, for example, information with a similar structure to the defect information D12.

[0246] Unknown work content information D23 is, for example, information with a similar structure to work content information D13.

[0247] Input data D2 is data that includes, for example, an unknown object D21, unknown defect information D22, and unknown work content information D23.

[0248] The trained model A2 is the state in which the learning model A1 has been trained through preprocessing. In other words, when preprocessing is performed on the learning model A1, the trained model A2 is generated. Thus, the trained model A2, which has been trained using training data D1 and big data D4, etc. as training data, is what is known as "generative AI".

[0249] When the trained model A2 receives input data D2, it uses the input of input data D2 as a trigger to generate output data D3.

[0250] Input data D2 represents a work condition whose "correct answer" is unknown; in other words, the "correct answer" for the work condition is unknown at the time of input. Hereafter, the work condition whose "correct answer" is unknown as indicated by input data D2 will be referred to as the "second work condition." Therefore, the data representing the second work condition is input data D2.

[0251] In training data D1, the "correct answer" is known in order to associate the correct answer with the first working condition, whereas in input data D2, the "correct answer" is unknown. Specifically, training data D1 includes the correct answer data D20, while input data D2 does not. Therefore, the relationship between the first working condition and the correct answer data D20 is known in training data D1.

[0252] On the other hand, the input data D2 does not contain the ground truth data D20, and the "ground truth" for input data D2 is unknown. The trained model A2 then generates output data D3 for input data D2 based on the correlation between the training data D1 and the ground truth data D20, which were learned in the preprocessing stage.

[0253] The output data D3 is, for example, teaching data 103. Therefore, by generating output data D3 and inputting it to the scarfing robot 11 in the real environment 201, the scarfing robot 11 can be operated.

[0254] Alternatively, output data D3 is data that shows, for example, how to operate the scarfing robot 11 in the simulation environment 202. In other words, output data D3 is data that constructs an environment for simulating the operation of the scarfing robot 11 in the simulation environment 202. For example, the simulation is performed as follows.

[0255] Figure 30 shows an example of execution processing and simulation. For example, when input data D2 is input, the trained model A2 constructs a simulation environment 202 for the input data D2 and simulates what actions the virtual robot 21 will perform. When such simulation results are output, the user 101 can check what kind of scarfing work will be performed in the simulation environment 202.

[0256] Furthermore, user 101 may be able to modify the simulation results. Since the so-called "initial settings" are generated by the AI, teaching can be started from a state where some teaching has already been completed, rather than starting from scratch, thus reducing the amount of teaching work.

[0257] As described above, when the simulation results are checked or corrected in the simulation environment 202, teaching data 103 is generated. Next, when the teaching data 103 is input to the scarfing robot 11, the scarfing robot 11 can be operated. In other words, teaching can be performed on the scarfing robot 11.

[0258] FIG. 31 is a diagram showing an overall processing example of AI learning and execution. Thus, the relationship between the preprocessing and the execution processing described above becomes the relationship between the generation and use of the learning model A1.

[0259] Note that the preprocessing and the execution processing do not necessarily have to be executed in a continuous order as illustrated in the figure. Therefore, it is not essential to make continuous the period for preparation by the preprocessing and the period for subsequent execution processing. Thus, the execution processing may be performed after a lapse of time from the preprocessing as long as the learned model A2 has been once created. Also, as long as the learned model A2 has been once generated, the learned model A2 may be diverted to perform the execution processing.

[0260] In the learning process and the execution process, the input data, that is, the learning data D1 and the input data D2 are different. Also, in the learning stage, the AI is the learning model A1, but as learning progresses to a certain extent, it becomes the learned model A2.

[0261] The first working condition indicated by the learning data D1 and the second working condition indicated by the input data D2 are both the same scarfing working conditions. That is, the working conditions are various settings and conditions for causing the scarfing robot 11 to execute the scarfing work.

[0262] For example, the working conditions are input by the user 101 in text or the like. Also, how the scarfing robot 11 is translated, rotated, or performs work, etc. may be part of the working conditions based on the results of the user 101's operation on the simulation environment 202.

[0263] Part of the execution processing may be replaced with processing using a table or the like. Thus, the preprocessing in the processing using a table or the like (so-called rule-based processing) is processing for preparing to input a table (also referred to as a look-up table (LUT), etc.) or a mathematical formula or the like.

[0264] [An example of AI learning and executing tasks by dividing them into multiple parts] Figure 32 illustrates an example of how AI learns and executes tasks by dividing them into multiple parts. Hereafter, the entire task will simply be referred to as "the task." On the other hand, a portion of the entire task will be referred to as "the first task." Similarly, a portion of the entire task that is performed separately from the first task will be referred to as "the second task." Note that the task may be divided into three or more parts. For the purpose of explanation, the following explanation will use an example where the entire task consists of only two parts: the first task and the second task.

[0265] For example, the second task is one that is performed after the first task. That is, the first task is the so-called "preceding process," and the second task is the so-called "following process." However, the relationship between the first and second tasks is not limited to time or sequence. In other words, the first and second tasks can be different tasks distinguished by settings or other means.

[0266] The following will explain the first and second operations using examples of preceding and succeeding processes. For example, once the first operation is completed, the object reaches a level of completion where it can perform its basic functions, a so-called "work in progress." In other words, even after the first operation is completed, the object is still insufficient as a finished product and will not pass inspection or shipment.

[0267] Next, the second task is performed on the object after the first task has been completed. For example, the second task is to eliminate any shortcomings at the completion of the first task and make the object a so-called "finished product." In this way, the first and second tasks have a relationship similar to that of "work in progress" and "finished product."

[0268] Alternatively, the relationship between the first and second tasks could be "before rework" and "after rework," for example. For instance, the quality of the object could be expressed on a scale from "0%" to "100%." ​​The object would not be considered a "passing product" and could not be shipped unless it reached "100%" quality based on pre-defined specifications.

[0269] Let's say the first step is to process the object to reach "100%" quality. However, due to various disturbances or variations, the object may not reach "80%", or the acceptable level, after the first step alone. In such cases, a second step, which is a "rework," is performed after the first step. Performing this second step raises the quality of the object from "80%" to "100%". In this way, by adding a second step when the first step was insufficient, it is possible to reduce defective products that do not meet the "acceptable" standards and improve the yield.

[0270] Furthermore, the terms "before rework" and "after rework" are not necessarily limited to the first and second tasks. The first task may be a "setting task" that raises the quality to a certain level, while the second task may be a "correction task" that further improves the quality after the first task, or raises an object that is of unacceptable quality to the level of an "acceptable product."

[0271] Whether or not the second task is performed may be decided during the first task or after the first task has been completed to a certain extent. In other words, the second task may be additional work performed to address newly discovered defects as a result of the first task. For example, defects inside an object may only be recognized after the first task is performed. Thus, the second task may be performed as work to further improve quality as a result of the first task. On the other hand, there may be cases where no additional work is necessary as a result of the first task. In such cases, the second task may not be performed.

[0272] Furthermore, the criteria for a "passing product" may not always be clearly defined in the specifications. For example, the person in charge of the work may decide whether the quality is sufficient at the time the work is performed based on their own judgment. In such cases, after the first task is performed, the quality of the object is checked, and if it is determined that the quality is insufficient, it is decided to perform additional work, i.e., the second task. In cases where the passing standards are not determined in advance, the first and second tasks may have a relationship where the first task is performed first, and the second task is performed after the necessity of the first task is completed is determined.

[0273] The result of the first task, that is, the basic parts of the object, is often consistently similar even if product requirements such as customer, specifications, or lot (production unit) differ. On the other hand, the result of the second task, that is, the finished parts of the object, often differs from customer to customer compared to the result of the first task, depending on customer preferences or specifications. Therefore, the "correct" answer for the second task differs depending on the product requirements. For this reason, it is desirable to use different AI for the first and second tasks; that is, to train the AI ​​separately without mixing the tasks.

[0274] Hereafter, the learning model A1 used for the first operation will be referred to as "first learning model A11". The learning data D1 that represents the content related to the first operation will be referred to as "first learning data D1W1". Furthermore, the trained model A2 generated by training the first learning model A11 with the first learning data D1W1 will be referred to as "first trained model A21".

[0275] Similarly, the learning model A1 for the second operation is called "second learning model A12". The learning data D1 that shows the content related to the second operation is called "second learning data D1W2". The trained model A2 generated by training the second learning model A12 with the second learning data D1W2 is called "second trained model A22".

[0276] Also, the data indicating product requirements is referred to as "identification data D30". However, the identification data D30 may not be necessary when the first operation and the second operation are in a relationship such as "before rework" and "rework".

[0277] However, the first learning data D1W1 may include other data, such as the second learning data D1W2, etc., as long as it includes the learning data D1 indicating the content related to the first operation. Similarly, the second learning data D1W2 may include other data, such as the first learning data D1W1, etc., as long as it includes the learning data D1 indicating the content related to the second operation.

[0278] Also, it is desirable that the identification data D30 is included in the second learning data D1W2 used for the learning of the second learning model A12. For example, the identification data D30 is data indicating a customer name (which may also be an identification number, etc.). In this case, the second operation is different for each customer. However, the second operation may also be different for non-customers, and whether to switch the second operation can be set with the identification data D30.

[0279] The first operation content information D13W1 is the operation content information D13 indicating the content related to the first operation.

[0280] The first correct data D20W1 is the correct data D20 for the first operation indicated by the first operation content information D13W1.

[0281] The second operation content information D13W2 is the operation content information D13 indicating the content related to the second operation.

[0282] The second correct data D20W2 is the correct data D20 for the second operation indicated by the second operation content information D13W2.

[0283] Note that whether the first operation content information D13W1 and the second operation content information D13W2 indicate which operation may be determined by AI or the like, or may be set with tags or the like.

[0284] For example, if the work content information D13 is a video taken during the execution of a work, user 101 sets a pre-configured number or similar to indicate whether the video is the first work or the second work that was filmed.

[0285] However, the video may be used to determine whether it is the first task or the second task by analyzing its content, and thus become the first task content information D13W1 or the second task content information D13W2.

[0286] For example, if analysis of a video or other means indicates that the object being worked on is already in a partially worked state, then it can be inferred that the work is the second task. On the other hand, if analysis of a video or other means indicates that the object being worked on is in its raw material state, that is, before all work has been done, then it can be inferred that the work is the first task.

[0287] Furthermore, the object information D11 and defect information D12 do not have to be information from before the work, i.e., before the first work; in the second learning data D1W2, they may also be information from after the first work (i.e., when it becomes "work in progress").

[0288] [Overall processing example] Figure 33 shows an example of the overall processing of the second embodiment. For example, the control device 10 performs the following processing.

[0289] In step S101, the control device 10 performs a first learning process that preprocesses the first learning model A11. Details of the first learning process will be described later.

[0290] In step S102, the control device 10 performs a first execution process using the first trained model A21 generated in step S101. Details of the first execution process will be described later.

[0291] In step S103, the control device 10 performs a second learning process, which involves pre-processing the second learning model A12. Details of the second learning process will be described later.

[0292] In step S104, the control device 10 performs a second execution process using the second trained model A22 generated in step S103. Details of the second execution process will be described later.

[0293] Note that the overall process is not limited to the processes described above. For example, the overall process may include processes other than those described above, or the pre-processing and execution processes do not have to be consecutive.

[0294] [Examples of the first and second learning processes] Figure 34 shows examples of the first and second learning processes. For example, steps S101 and S103 are processes as shown in the figure. The first and second learning processes are preprocessing steps.

[0295] The following explanation will use an example where the AI ​​learns from the execution of the entire process of the first embodiment shown in Figure 25. Therefore, processes similar to the overall process of the first embodiment are denoted by the same reference numerals, and redundant explanations are omitted.

[0296] Compared to the overall processing of the first embodiment, the overall processing of the second embodiment differs in that steps S201 and S202 are added.

[0297] In step S201, the control device 10 inputs the learning data D1. Alternatively, the learning data D1 may be input in advance (at the timing of step S201, i.e., before step S01, etc.), and the teaching results, etc., may be input to the learning model A1 as learning data D1 at the timing of step S02, etc. Thus, the timing of inputting the learning data D1 is not limited to step S201; it may be input in a distributed manner or at timings other than step S201.

[0298] In step S202, the control device 10 trains the learning model A1 to generate a trained model A2 that has learned the correlation between the work and the evaluation results.

[0299] In step S01, the control device 10 constructs the simulation environment 202. For example, the simulation environment 202 is constructed as shown in Figure 4. In constructing the simulation environment 202, parameters such as defect reflection, metal, end effector, robot, obstacles, virtual region, nozzle 15, or scarfing parameters (for example, parameters that affect the quality of scarfing, i.e., the speed at which the nozzle is moved, the angle between the nozzle and the object, the distance between the nozzle and the object, preheating conditions, the mixing ratio of flammable gas and oxygen in the gas, and the nozzle structure, etc.) may be set. The simulation environment 202 is constructed with these settings reflected.

[0300] In step S02, the control device 10 performs teaching on the scarfing robot 11. For example, the control device 10 accepts input from the user 101 to the scarfing robot 11, such as translation, rotation, or execution of various tasks. Based on such input, teaching data 103 is generated.

[0301] In step S03, the control device 10 simulates the removal state, etc. For example, the simulation result outputs metal 102 with defects removed after scarfing. Various settings such as the simulation time are set in advance. Other options include setting the preheating time, a timeline, simulation using 3D video, or a demonstration operation in the actual environment 201.

[0302] Furthermore, the system may input operations to modify the simulation results. If such modifications are made, the simulation results will be output again, reflecting the modifications.

[0303] Through the overall processing described above, once teaching data 103 is generated, the scarfing robot 11 can be operated in the actual environment 201 based on the teaching data 103. The user 101 can pre-verify how the robot will operate based on the teaching data 103 through simulation.

[0304] Once the simulation environment 202 is established, user 101 can teach the scarfing robot 11 in the simulation environment 202. Therefore, user 101 can perform teaching remotely, or without operating the scarfing robot 11 in the actual environment 201.

[0305] Furthermore, it is desirable that the first and second operations be divided into the first learning process and the second learning process, and that the first learning model A11 and the second learning model A12 be trained separately.

[0306] Specifically, in the first training process, training data D1 is the first training data D1W1. On the other hand, in the second training process, training data D1 is the second training data D1W2.

[0307] Similarly, in the first learning process, the simulation environment 202 constructed in step S01 is for the first working environment. On the other hand, in the second learning process, the simulation environment 202 constructed in step S01 is for the second working environment.

[0308] Furthermore, in the first learning process, the task performed in step S03 is the first task. On the other hand, in the second learning process, the task performed in step S03 is the second task.

[0309] Therefore, in the first learning process, the first operation is learned in the first learning model A11 in step S202. On the other hand, in the second learning process, the second operation is learned in the second learning model A12 in step S202.

[0310] In this way, when the pre-processing by the first learning process and the second learning process is executed, the first trained model A21 and the second trained model A22 are generated. Using the trained model A2 generated by this pre-processing, the control device 10 executes the first execution process and the second execution process, which are the execution processes.

[0311] [Examples of the first and second execution processes] Figure 35 shows examples of the first and second execution processes. For example, steps S102 and S104 are processes as shown in the figure. The first and second execution processes are execution processes.

[0312] Step S21 is when the control device 10 receives input data D2.

[0313] In step S22, the control device 10 generates output data D3 using the trained model A2.

[0314] In step S23, the control device 10 causes the scarfing robot 11 to perform scarfing based on the output data D3, or performs a simulation on the simulation environment 202.

[0315] Furthermore, if the simulation is to be performed on the simulation environment 202 in step S23, for example, step S01 is executed to construct the simulation environment 202.

[0316] It is desirable that the first and second operations be performed using different AIs, specifically the first pre-trained model A21 and the second pre-trained model A22, which are generated in advance during the pre-processing steps for the first and second learning processes, respectively.

[0317] Specifically, the first execution process uses the first pre-trained model A21. On the other hand, the second execution process uses the second pre-trained model A22.

[0318] Similarly, in the first execution process, the input data D2 entered in step S21 is for the first operation. Hereinafter, the input data D2 for the first operation will be referred to as "first input data". On the other hand, the input data D2 entered in step S21 is for the second operation. Hereinafter, the input data D2 for the second operation will be referred to as "second input data".

[0319] Therefore, in the first execution process, the task performed in step S23 is the first task. On the other hand, in the second execution process, the task performed in step S23 is the second task. That is, the output data D3 is generated separately for the first task (hereinafter referred to as "first output data") and for the second task (hereinafter referred to as "second output data").

[0320] Specifically, in step S23, the scarfing robot 11 performs a first operation based on the first output data generated when the first input data is input to the first trained model A21. Meanwhile, in step S23, the scarfing robot 11 performs a second operation based on the second output data generated when the second input data is input to the second trained model A22.

[0321] In this way, once the execution processes by the first execution process and the second execution process are executed, the scarfing robot 11 performs the first task and the second task.

[0322] [Example of Functional Configuration] Figure 36 shows an example of the functional configuration of the second embodiment. For example, in the control system 100, the learning device 30 includes an input unit 100F1, a simulation unit 100F2, a teaching unit 100F3, an initial execution unit 100F20, a modification unit 100F5, a learning data input unit 100F21, and a learning unit 100F22, etc. Also, in the control system 100, the control device 10 includes an input data input unit 100F23 and a control unit 100F24, etc. However, the control system 100 may have other functions besides those listed above.

[0323] Furthermore, the learning device 30 and the control device 10 may share similar functional configurations, or each device may have similar functions. For example, the input unit 100F1 may be located in the learning device 30 or the control device 10, or it may be located in either the learning device 30 or the control device 10.

[0324] The learning device 30 and the control device 10 may be different information processing devices or the same information processing device.

[0325] The input unit 100F1 performs an input procedure to receive operations from the user 101. For example, the input unit 100F1 can be implemented using an input device 10H3 or the like.

[0326] The simulation unit 100F2 performs a simulation procedure to construct a simulation environment 202 for simulating work performed by an industrial robot, based on the operations input by the input unit 100F1. For example, the simulation unit 100F2 is implemented using a CPU 10H1 or the like.

[0327] The teaching unit 100F3 performs teaching procedures for the industrial robot in the simulation environment 202. For example, the teaching unit 100F3 is implemented using a CPU 10H1 or the like.

[0328] The initial execution unit 100F20 performs an initial execution procedure to cause the industrial robot to perform a task in either the real environment 201 or the simulation environment 202, based on the teaching results in the simulation environment 202. For example, the initial execution unit 100F20 is implemented using a CPU 10H1 or the like.

[0329] The modification unit 100F5, upon receiving a change operation while work is being executed in the actual environment 201 or the simulation environment 202, performs a change procedure that reflects the change operation and modifies the work being executed in the actual environment 201 or the simulation environment 202. For example, the modification unit 100F5 is implemented by a CPU 10H1 or the like.

[0330] The learning data input unit 100F21 performs a learning data input procedure that involves inputting learning data D1, which includes a modification operation and the evaluation result for the task as correct data. For example, the learning data input unit 100F21 can be implemented by an input device 10H3 or the like.

[0331] The learning unit 100F22 performs a learning procedure in which it trains a learning model A1 using the learning data D1 and generates a trained model A2 that has learned the correlation between the work and evaluation results. For example, the learning unit 100F22 can be implemented using a CPU 10H1 or the like.

[0332] The input data input unit 100F23 performs a training data input procedure in which input data D2 is input to the trained model A2. For example, the input data input unit 100F23 can be implemented by an input device 10H3 or the like.

[0333] The control unit 100F24 performs control procedures to execute tasks in the real environment 201 based on the learned model A2. For example, the control unit 100F24 is implemented by a CPU 10H1 or the like.

[0334] With the above configuration, once the simulation environment 202 is established, the user 101 can teach the scarfing robot 11 in the simulation environment 202. Based on the teaching results, the scarfing robot 11 can then perform tasks based on the initial settings 401.

[0335] Thus, once the simulation environment 202 is established, the user 101 can teach the scarfing robot 11 in the simulation environment 202. Therefore, by teaching the scarfing robot 11 remotely, the automation of the scarfing robot 11 can be achieved.

[0336] Furthermore, using AI makes it possible to efficiently simulate the operation of the scarfing robot 11 in the simulation environment 202, or to teach the operation of the scarfing robot 11.

[0337] In particular, when modification operations are performed during execution, the AI ​​learns about those modification operations. An AI that has undergone such learning can perform tasks while taking modification operations into consideration.

[0338] Furthermore, it is desirable that learning and execution be carried out separately as a first operation and a second operation. Specifically, it is desirable that the learning unit 100F22 be equipped with a first learning unit for the first operation and a second learning unit for the second operation.

[0339] The first learning unit performs a first learning procedure for the first task, which involves training the first learning model with the first learning data to generate a first trained model that learns the correlation between the first task and the first evaluation result for the first task.

[0340] The second learning unit performs a second learning procedure for the second task, training the second learning model with the second learning data to generate a second trained model that has learned the correlation between the second task and the second evaluation results for the second task.

[0341] As described above, when multiple trained models are generated, it is desirable that the system executing the tasks also have a corresponding control unit for each task. Specifically, it is desirable that the control unit 100F24 comprises a first control unit for the first task and a second control unit for the second task.

[0342] The first control unit performs a first control procedure to cause the industrial robot to perform a first task in the real environment 201, based on a first learned model.

[0343] The second control unit performs a second control procedure to cause the industrial robot to perform the second task in the real environment 201, based on the second trained model.

[0344] As described above, having a learning unit corresponding to each task allows for the generation of a pre-trained model adapted to each task. For example, it may be desirable to distinguish between tasks such as "work in progress" and "finished work" and train the AI ​​accordingly. Note that a task may be divided into three or more parts. Therefore, the learning unit and execution unit may be configured with three or more units.

[0345] With this configuration, even if each customer has different requirements, the AI ​​can be trained to switch between tasks according to the individual customer's requirements, and the industrial robot can be made to perform tasks in a way that is appropriate for each customer.

[0346] [Calibration example] The simulation environment 202 and the actual environment 201 may differ in terms of the nozzle 15, target object, obstacles, or surrounding environment. Therefore, it is desirable to perform calibration to match the simulation environment 202 and the actual environment 201 and correct the teaching data 103, etc.

[0347] In scarfing operations, the initial position and dimensions of the object often differ each time. Therefore, it is desirable to calibrate each object so that its position and dimensions are reflected in the simulation environment 202.

[0348] The position and dimensions of the object are captured by cameras, measuring instruments, drones, etc., or measured by sensors such as touch sensors, and this data is used for calibration.

[0349] For example, the dimensions of the object, or mechanical positional deviations such as the orientation of the nozzle 15, may differ between the simulation environment 202 and the actual environment 201. These differences are detected, for example, by sensors. Specifically, if dimensions are measured by sensors, the dimensions of the object, etc., are corrected in the teaching data 103 based on the measurement results. Furthermore, it is desirable that related operations (for example, the range in which scarfing operations are performed, or the range in which the virtual area is set) are also corrected in conjunction with this correction.

[0350] Once this calibration is performed, the differences between the simulation environment 202 and the real environment 201 are adjusted, enabling highly accurate teaching.

[0351] [Variations of input devices] For teaching and other purposes, it is desirable that the input device 10H3 be hardware that has positional information and orientation (height H, angle θ, etc.). Specifically, the input device 10H3 may be a combination of a touch panel and a pen-shaped device that the user 101 holds in their hand and moves in three dimensions.

[0352] For scarfing, an input device 10H3 that allows for easy specification of a two-dimensional position is desirable. Therefore, if a point (which will be a coordinate) can be input via a touch panel to indicate the position or work object, operation will be easier.

[0353] Then, when the orientation of the input device 10H3 changes in three dimensions due to the operation of the user 101, the scarf-wing robot 11 performs a synchronized translation or rotation in the real environment 201 or the simulation environment 202. The input device 10H3 measures its position and angle in three dimensions in real time using a position sensor (for example, a gyroscope).

[0354] Furthermore, the input device 10H3 can be activated or deactivated (i.e., switched ON / OFF) by the user 101 pressing a switch.

[0355] Thus, it is desirable that the input device 10H3 has hardware that indicates a three-dimensional position and a switch that indicates the switching of operations. With such hardware, the user 101 can operate it intuitively, improving usability.

[0356] Furthermore, the input device 10H3 is not limited to the above, and it is desirable that a game controller or touch panel be applicable. Thus, it is desirable that the control device 10 be able to support multiple types of input devices 10H3.

[0357] Scarfing teaching involves actions not commonly performed in other work processes. For example, it frequently requires special actions such as maintaining a constant distance from the object while performing scarfing, or performing parallel movements such as sliding sideways while maintaining the distance. On the other hand, the speed or angle may be changed depending on the working position. An input device 10H3 that facilitates the instruction of such actions is desirable.

[0358] [Examples of industrial robots and their operations] The above embodiment is preferably applied to industrial robots that perform tasks other than scarfing. These industrial robots primarily work with metals or materials used in metalworking.

[0359] For example, industrial robots are robots that perform tasks such as spraying, sealing, jet nozzles, roller brushes, paintbrushes, brushing, roller herring, polishing, grinding, drilling, cutting, shoveling (scooping up), rakes, scraping, plastering with a trowel, or ladleing.

[0360] In addition, industrial robots may also be robots that perform tasks such as processing, assembly, welding, transport, inspection, pressing, painting, spraying, lettering, cleaning, cutting, arranging, or picking.

[0361] Spraying is the process of applying a liquid, powder, or solid substance to an object.

[0362] Sealing is the process of applying a sealant (often in paste form) to an object.

[0363] A jet nozzle is used, for example, to clean an object by spraying it with water at high pressure.

[0364] Roller brushing is the process of applying colored paint or other liquids to an object using a roller.

[0365] Brush painting is a process that uses a tool called a "brush," which is a bundle of bristles. The materials of the bristles and handle of the brush vary depending on its intended use. For example, brush painting is performed for purposes such as painting, cleaning, or applying various coatings. In brush painting, paints, adhesives, or cleaning agents are applied.

[0366] Painting is the process of applying paint or other materials to an object to give it color. However, painting may also be for purposes other than coloring, such as applying fire-resistant materials or coatings.

[0367] Lettering is the process of forming letters (which may also be numbers, pictures, or patterns) on an object, for example, by applying paint or engraving.

[0368] Spraying is the process of applying a gas, liquid, or gel-like substance to an object.

[0369] Brushing is the process of pressing a metal brush or similar object against an object and moving the brush. For example, brushing is performed to remove paint or rust from a surface. It can also be used to create grooves or to process a surface.

[0370] Roller hemming is a process in which the edges of car doors or bodywork are hemmed using a roller attached to the end of a robot while applying pressure.

[0371] Polishing is the process of smoothing the surface of an object by removing material using abrasives or similar substances.

[0372] Grinding is the process of removing material from the surface of an object using hard particles or powders, such as those found in abrasive wheels.

[0373] Cutting and scraping are processes that use tools to remove a portion of an object.

[0374] Drilling is the process of creating a hole in an object using a drill or similar tool.

[0375] Cutting is the process of dividing an object into multiple parts using a tool.

[0376] Gathering is the process of concentrating scattered objects in a specific location using tools such as a rake.

[0377] Trowel application is the process of applying a coating to an object using a tool commonly known as a "trowel." For example, trowel application is used for repairs, finishing, or painting large areas.

[0378] Scooping with a ladle and scooping are operations that involve using a ladle or other tool to scoop up an object.

[0379] Machining is the process of shaping an object. For example, machining can involve mechanically cutting, bending, punching, hammering, applying pressure, or shaping by placing the object in a mold. It can also involve creating or adding shapes using a 3D printer or similar technology.

[0380] Furthermore, processing is the process of changing the properties of an object. For example, processing can involve heating or surface treatment. Thus, processing is the process of performing mechanical, thermal, or chemical treatments.

[0381] Assembly is the process of combining multiple parts to create a single, complete assembly.

[0382] Welding is the process of joining parts of multiple components that make up an object by applying heat or pressure.

[0383] Transportation is the process of moving an object from its current location to another location.

[0384] Inspection is the process of determining whether the quality of an object that has been processed or otherwise modified fully meets the requirements. For example, inspection may involve checking for scratches on the surface of the object, checking for scratches using ultrasound, determining the surface profile (for example, by measurement as defined in ISO 8503), or checking for internal strain.

[0385] Pressing is the process of applying pressure to an object to deform it or otherwise alter it.

[0386] Cleaning is the process of removing dirt and other debris from an object by applying water or scraping it.

[0387] Picking is the process of retrieving an object.

[0388] Arranging refers to the process of placing multiple objects, such as those picked during picking, according to a certain rule or in specific locations.

[0389] Note that there may be other tasks besides those listed above. Also, a task may be a combination of multiple tasks.

[0390] [Other embodiments] This embodiment may also take the following forms.

[0391] [About the metaverse] The virtual space can also be what is known as the metaverse. The term "metaverse" is a combination of "meta" (transcendence) and "universe" (cosmos, world). The metaverse refers to a three-dimensional virtual space on a computer network that can accommodate many participants and allow them to act freely within it.

[0392] In the metaverse, multiple people can participate using avatars, for example. Within the metaverse space, transactions or processing may also occur within a space that utilizes three-dimensional image processing.

[0393] Transactions on the metaverse often utilize arbitrary tokens. These transactions can encompass a wide variety of activities, such as online games, virtual concerts, or e-commerce.

[0394] Furthermore, the metaverse can sometimes be realized using "XR" technologies such as AR (Augmented Reality) or VR (Virtual Reality). In addition, the metaverse can also be realized using technologies such as 3DCG, high-speed communication technology, AI, and blockchain.

[0395] Furthermore, each device does not necessarily have to be a single device. In other words, each device may be a combination of multiple devices.

[0396] The present invention may be implemented by a process for realizing the control method exemplified above, or by a program (including firmware and things equivalent to a program; hereinafter simply referred to as "program") that performs a process equivalent to the process described above.

[0397] In other words, the present invention may be implemented by a program written in a programming language or the like, which issues commands to a computer to obtain a predetermined result. The program may also be configured so that a part of the processing is executed by hardware such as an IC (integrated circuit).

[0398] A program causes the computer to perform the above-mentioned processes by having its arithmetic unit, control unit, and memory device work together. In other words, a program is loaded into main memory, issues commands to the arithmetic unit to perform calculations, and operates the computer.

[0399] Furthermore, the program may be provided on a computer-readable recording medium or via telecommunication lines such as a network.

[0400] The present invention may be implemented in a system composed of multiple devices. That is, an information processing system consisting of multiple computers may execute the above-described processes in a redundant, parallel, distributed, or combination thereof. Therefore, the present invention may be implemented in devices other than those described above, and in systems other than those described above.

[0401] [Regarding contributions to the SDGs] This invention realizes a technology for teaching industrial robots in a simulation environment. This provides a technology that contributes to the SDGs by enabling the construction of resilient infrastructure, the promotion of inclusive and sustainable industrialization, and the advancement of innovation, as targeted in Goal 9.

[0402] It should be noted that the present invention is not limited to the embodiments exemplified above. Therefore, the present invention can be modified by adding or changing components without departing from the technical spirit. Thus, all technical matters included in the technical concept described in the claims are covered by the present invention. The embodiments exemplified above are specific examples that are suitable for implementation. Furthermore, those skilled in the art can implement various modifications from the disclosed content, and such modifications are included in the technical scope described in the claims. [Explanation of Symbols]

[0403] 10: Control device 10H1:CPU 10H2: Storage device 10H3: Input device 10H4: Output device 10H5:Communication device 11: Scarfing Robot 12: Control Panel 14: Higher-level system 15: Nozzle 16: Movement Arrow 21: Virtual Robot 30: Learning device 100: Control System 100F1: Input section 100F2: Simulation Department 100F20: Initial execution unit 100F21: Learning data input section 100F22: Learning Department 100F23: Input data input section 100F24: Control Unit 100F3: Teaching Department 100F4: Execution Unit 100F5: Changes 101: User 102: Metal 103: Teaching Data 112: Virtual Metal 151: Oxygen pore 152 :LPG hole 153: Scarf oxygen pores 201: Real-world environment 202: Simulation Environment 300: Network Configuration 301: Defect area 302: Arrow 303: Scarfing marks 401: Initial settings 3021: First arrow 3022: Second arrow 3023: Third arrow 3024: Fourth arrow 3025: Fifth arrow 3026: 6th arrow 3027: 7th arrow A1: Learning Model A11: First Learning Model A12: Second Learning Model A2: Pre-trained model A21: First pre-trained model A22: Second pre-trained model D1: Training data D11: Object Information D12: Defect Information D13: Work details information D13W1: Information on the first work procedure D13W2: Information on the second work procedure D1W1: First training data D1W2: Second training data D2: Input data D20: Correct data D20W1: First correct answer data D20W2: Second correct answer data D21: Unknown object D22: Unknown defect information D23: Information on unknown work content D3: Output data D30: Identification Data D4: Big Data DR1: 1st movement direction DR2: 2nd movement direction H: Height L1: Input layer L2: Middle layer L3: Output layer TP:Touch point V1: First virtual domain V10: First Virtual Authorized Area V11: Second virtual permission area V2: Second virtual domain θ: angle

Claims

1. A control device for controlling an industrial robot that performs work on an object, An input section for receiving user input, A simulation unit constructs a simulation environment for simulating the work performed by the industrial robot based on the above operation, The simulation environment includes a teaching unit that performs teaching on the industrial robot, An execution unit that causes the industrial robot to perform the task in the actual environment based on the teaching results in the simulation environment, When the user makes a change operation to modify part or all of the work while the work is being executed in the actual environment, the change unit modifies the work being executed in the actual environment to reflect the change operation. A control device for industrial robots equipped with [a specific feature / feature].

2. The aforementioned simulation unit, The industrial robot is operated by a virtual operating body shown in the simulation environment in accordance with the operation. A virtual region is set in the simulation environment that restricts the movement of the virtual moving object, including translation or rotation. When the virtual domain is set in the simulation environment and it is determined that the virtual entity is located in the virtual domain, When the operation instructs a translation that includes a first directional component which is a directional component in the direction that restricts the virtual operating body within the virtual region, and a second directional component which is a directional component in the direction that aligns the virtual operating body with the surface of the virtual region, The direction in which the virtual moving object is translated is corrected based on the surface of the virtual region, and in the simulation environment, the virtual moving object is translated in the corrected direction. The control device for an industrial robot according to claim 1.

3. The system detects the difference between the simulation environment and the actual environment, and performs calibration to correct the teaching data generated by the teaching unit based on the difference. The control device for an industrial robot according to claim 1.

4. A learning device for learning teaching methods to be used for industrial robots that perform tasks on an object, An input section for receiving user input, A simulation unit constructs a simulation environment for simulating the work performed by the industrial robot based on the above operation, The simulation environment includes a teaching unit that performs teaching on the industrial robot, An initial execution unit that causes the industrial robot to perform the task in the actual environment or the simulation environment based on the teaching results in the simulation environment, When the user makes a change operation to modify part or all of the work while the work is being executed in the actual environment or the simulation environment, the change unit modifies the work being executed in the actual environment or the simulation environment to reflect the change operation. A learning data input unit inputs the aforementioned modification operation and learning data including the evaluation result for the operation as correct data, A learning unit that trains a learning model using the aforementioned training data and generates a trained model that has learned the correlation between the aforementioned operations and the aforementioned evaluation results. A learning device for industrial robots equipped with the following features.

5. The aforementioned learning unit, Regarding the first task, which is part of the aforementioned work, the first learning unit trains a first learning model using the first learning data related to the first task from the learning data, and generates a first trained model that learns the correlation between the first task and the first evaluation result for the first task. The system includes a second learning unit that, for a second operation which is part of the aforementioned operation and performed separately from the first operation, trains a second learning model using the second learning data related to the second operation from the aforementioned learning data, thereby generating a second trained model that learns the correlation between the second operation and the second evaluation result for the second operation. A learning device for an industrial robot according to claim 4.

6. The second operation described above is: Executed after the first operation described above. A learning device for an industrial robot according to claim 5.

7. A control device for controlling an industrial robot that performs work on an object, based on a trained model that has learned teaching to the industrial robot, An input section for receiving user input, A simulation unit constructs a simulation environment for simulating the work performed by the industrial robot based on the above operation, The simulation environment includes a teaching unit that performs teaching on the industrial robot, An initial execution unit that causes the industrial robot to perform the task in the actual environment or the simulation environment based on the teaching results in the simulation environment, When the user makes a change operation to modify part or all of the work while the work is being executed in the actual environment or the simulation environment, the change unit modifies the work being executed in the actual environment or the simulation environment to reflect the change operation. A control unit that performs the aforementioned modification operation in the actual environment based on the trained model which has learned the correlation between the aforementioned operation and the aforementioned evaluation result using the training data which includes the evaluation result for the aforementioned operation as ground truth data. A control device for industrial robots equipped with [a specific feature / feature].

8. The control unit, Regarding the first task, which is part of the aforementioned work, a first control unit executes the first task in the actual environment based on a first trained model that learns the correlation between the first task and the first evaluation result for the first task using the first training data related to the first task from the training data, The system includes a second control unit that, with respect to a second task which is part of the aforementioned work and performed separately from the first task, causes the second task to be executed in the actual environment based on a second trained model which has learned the correlation between the second task and the second evaluation result for the second task using second training data related to the second task from the training data. The control device for an industrial robot according to claim 7.

9. A control system comprising a learning device for learning teaching instructions to be given to an industrial robot performing work on an object, and a control device for controlling the industrial robot, The learning device is An input section for receiving user input, A simulation unit constructs a simulation environment for simulating the work performed by the industrial robot based on the above operation, The simulation environment includes a teaching unit that performs teaching on the industrial robot, An initial execution unit that causes the industrial robot to perform the task in the actual environment or the simulation environment based on the teaching results in the simulation environment, When the user makes a change operation to modify part or all of the work while the work is being executed in the actual environment or the simulation environment, the change unit modifies the work being executed in the actual environment or the simulation environment to reflect the change operation. A learning data input unit inputs the aforementioned modification operation and learning data including the evaluation result for the operation as correct data, A learning unit that trains a learning model using the aforementioned training data and generates a trained model that has learned the correlation between the aforementioned operations and the aforementioned evaluation results. Equipped with, The control device is a control device that controls the industrial robot based on the learned model, The control unit, based on the trained model, executes the operation in the actual environment. Equipped with Control system.

10. A control method performed by a control device that controls an industrial robot performing work on an object, Input procedure for receiving user input, A simulation procedure for constructing a simulation environment for simulating the work performed by the industrial robot based on the above operation, A teaching procedure for teaching the industrial robot in the aforementioned simulation environment, An execution procedure for causing the industrial robot to perform the task in the actual environment, based on the teaching results in the simulation environment, When the user makes a change operation to modify part or all of the work while the work is being executed in the actual environment, the change operation is reflected in the change procedure that modifies the work being executed in the actual environment. A control method including

11. A program for causing a computer to execute the control method described in claim 10.

12. A learning method performed by a learning device that learns teaching for an industrial robot that performs work on an object, Input procedure for receiving user input, A simulation procedure for constructing a simulation environment for simulating the work performed by the industrial robot based on the above operation, A teaching procedure for teaching the industrial robot in the aforementioned simulation environment, An initial execution procedure for causing the industrial robot to perform the task in the actual environment or the simulation environment, based on the teaching results in the simulation environment, When the system receives a change operation from the user to modify part or all of the work while the work is being executed in the actual environment or the simulation environment, the system implements a change procedure to modify the work being executed in the actual environment or the simulation environment to reflect the change operation. A learning data input procedure which involves inputting the aforementioned modification operation and learning data including the evaluation result for the aforementioned operation as correct data, A learning procedure to train a learning model using the aforementioned training data and generate a trained model that has learned the correlation between the aforementioned work and the aforementioned evaluation results. Learning methods that include this.

13. A program for causing a computer to execute the control method described in claim 12.

14. A control method performed by a control device that controls an industrial robot, based on a trained model that has learned teaching to the industrial robot performing work on an object, Input procedure for receiving user input, A simulation procedure for constructing a simulation environment for simulating the work performed by the industrial robot based on the above operation, A teaching procedure for teaching the industrial robot in the aforementioned simulation environment, An initial execution procedure for causing the industrial robot to perform the task in the actual environment or the simulation environment, based on the teaching results in the simulation environment, When the system receives a change operation from the user to modify part or all of the work while the work is being executed in the actual environment or the simulation environment, the system implements a change procedure to modify the work being executed in the actual environment or the simulation environment to reflect the change operation. A control procedure to execute the aforementioned modification operation in the actual environment based on the trained model which has learned the correlation between the aforementioned operation and the aforementioned evaluation result using the training data which includes the evaluation result for the aforementioned operation as ground truth data. A control method including

15. A program for causing a computer to execute the control method described in claim 14.

Images