Robot control system, robot control method and robot control program

Abstract

The robot control system comprises: a setting unit configured to first set the next position value in a current task for a robot located in a real workspace and performing the current task to machine a workpiece; a simulation unit configured to virtually execute the current task, in which the robot operates with the next position value to machine the workpiece, through simulation; an adjustment unit configured to adjust the next position value based on a prediction result obtained through the simulation; and a robot control unit configured to control the robot in the real workspace based on the adjusted next position value.

Description

Technical field

[0001] One aspect of the present disclosure relates to a robot control system, a robot control method, and a robot control program. State of the art

[0002] Patent literature 1 describes a robot system comprising a sensing unit that acquires initial input data predetermined as data influencing the operation of a robot, a computation unit that calculates computation costs for inference processing based on the initial input data, using a machine learning model that derives control data used to control the robot, an inference unit that derives the control data through the machine learning model determined according to the computation costs, and a drive control unit that controls the robot using the derived control data. List of cited documents Patent literature

[0003] [Patent document 1] Japanese Patent No. 7021158 Summary of the invention: Technical problem

[0004] A mechanism is required for the appropriate operation of a robot according to the current situation of an actual workspace. Solution to the problem

[0005] A robot control system according to one aspect of the present disclosure, comprising: a setting unit configured to initially set a next manipulation value in a current task for a robot located in a real workspace and performing the current task to machine a workpiece; a simulation unit configured to virtually perform the current task, in which the robot operates with the next setpoint to machine the workpiece, by simulation; an adjustment unit configured to adjust the next setpoint based on a prediction result obtained by the simulation; and a robot control unit configured to control the robot in the real workspace based on the adjusted next setpoint.

[0006] A robot control method according to one aspect of the present disclosure is executed by a robot control system comprising at least one processor. The robot control method comprises: initially setting a next manipulation value in a current task for a robot located in a real workspace, and executing the current task to machine a workpiece; virtually executing, by simulation, the current task in which the robot operates with the next manipulation value to machine the workpiece; adjusting the next manipulation value based on a prediction result obtained by the simulation; and controlling the robot in the real workspace based on the adjusted next manipulation value.

[0007] A robot control program according to one aspect of the present disclosure causes a computer to perform the following: initially setting a next manipulation value in a current task for a robot located in a real workspace, and executing the current task to machine a workpiece; virtually executing, by simulation, the current task in which the robot operates with the next manipulation value to machine the workpiece; adjusting the next manipulation value based on a prediction result obtained by the simulation; and controlling the robot in the real workspace based on the adjusted next manipulation value. Advantageous effects of the invention

[0008] According to one aspect of the present disclosure, it is possible to operate a robot appropriately according to the current situation of a real workspace. Brief description of the drawings Fig. Figure 1 is a diagram showing an application example of a robot control system. Fig. Figure 2 is a diagram showing an example of a functional configuration of the robot control system. Fig. Figure 3 is a diagram showing an example of a hardware configuration of a computer used for the robot control system. Fig. Figure 4 is a flowchart that shows an example of determining the next manipulation value and controlling a robot. Fig. Figure 5 is a diagram showing an architecture associated with determining the next manipulation value. Fig. Figure 6 is a diagram showing an example architecture in connection with the simulation. Fig. Figure 7 is a flowchart that shows an example of task control. Description of exemplary implementations

[0009] The following section describes in detail various examples of the present disclosure with reference to the accompanying drawings. In the description of the drawings, an identical or equivalent element is designated by an identical reference numeral, and redundant descriptions are omitted. [System overview]

[0010] A robot control system according to the present disclosure is a computer system for autonomously operating a real robot according to the current situation of a real workspace. In one example, the robot control system determines the next input value of a robot in a current task, where the robot is deployed in the real workspace and is performing the current task to machine a workpiece, and causes the robot to continue the current task based on the next input value. In the present disclosure, the task refers to an operation to be performed by the robot to achieve a specific purpose. For example, the task is to machine a workpiece. The robot performs the task, and a result desired by the user of the robot control system is achieved. The current task refers to a task that is currently being performed by the robot.In the present disclosure, the manipulated value or manipulated variable refers to information for generating a movement of the robot. Examples of manipulated values ​​include an angle of each joint of the robot (joint angle) and a torque at each joint (joint torque). The next manipulated value refers to a manipulated value of the robot at a predetermined time interval after the current time.

[0011] The robot control system does not determine the robot's next position based on a target or a pre-planned path, but rather based on the current state of the workspace, which is difficult to predict accurately in advance. For example, the robot control system identifies an attribute (e.g., type, condition, etc.) of the workpiece being processed as the current state of the workspace and derives the next position from this. Such control enables robot-controlled operation that is directly tied to the workpiece. For instance, based on the current state of a workpiece whose state transitions are not reproducible, the robot control system determines the next position of the robot processing the workpiece.Alternatively, based on the current state of a workpiece with an undefined appearance, the robot control system determines the next position of the robot that processes the workpiece. The robot control system then instructs the robot to execute the current task based on this determined next position.

[0012] In the present disclosure, the term “workpiece” refers to a tangible object that is directly or indirectly affected by a movement of the robot. The workpiece may be a tangible object that is directly processed by the robot or another tangible object that is in the vicinity of the tangible object that is directly processed by the robot. For example, if the current task is to open a package that encloses a specific product, the workpiece may be at least the package or the product. As another example, in a case where the current task is a process for packaging a product of indeterminate appearance into a container, the workpiece may be at least the product or the container.The term "workpiece whose state transition is not reproducible" refers to a workpiece for which it is difficult to predict which state will be reached next or last. One could say that a "workpiece whose state transition is not reproducible" is a workpiece whose state changes irregularly. An example of a workpiece whose state transition is not reproducible is a tangible object, such as packaging material or a soft plastic bag, whose external shape changes irregularly due to an external force (for example, the movement of a robot). The term "workpiece with indeterminate appearance" refers to the fact that the appearance of individual workpieces is not completely identical. Examples of material objects with indeterminate appearance include fresh food items such as vegetables, fruits, fish, and meat.

[0013] To control the robot robustly according to the current situation, the robot control system first determines the next position and then virtually executes the current task, in which the robot processes the workpiece with this next position, through simulation. Simulation is a process in which a real robot located in the actual workspace is not actually operated, but rather its movement is simulated on a computer. The robot control system adjusts the next position based on a prediction obtained through the simulation and controls the real robot accordingly. In other words, the robot control system predicts the state of the workpiece at a slightly later time and adjusts and sets the next position based on this prediction.

[0014] In one example, the robot control system determines, based on the execution status of the current task, whether the current task continues without changing the robot's action position (where it interacts with the workpiece) or continues after a change in the action position. The action position is, for example, the position where the robot holds the workpiece with an end effector. In another example, the robot control system determines whether the current task continues or not based on its execution status. The robot control system can also plan a next task based on the execution status and terminate the current task accordingly. These control scenarios also illustrate the autonomous operation of a real robot based on the current situation in the actual workspace. [System configuration]

[0015] Fig. Figure 1 is a diagram illustrating an application example of the robot control system. In this example, the robot control system 1 causes a real robot 2, located in a real workspace 9 and processing a real workpiece 8, to operate autonomously according to the current situation of the workspace 9. The robot control system 1 is connected via a communication network to a robot controller 3, which controls the robot 2, and a camera 4, which records the workspace 9. The communication network can be a wired or wireless network. The communication network must include at least internet or intranet access. Alternatively, the communication network can be implemented simply using a single communication cable.

[0016] The example in Fig. Figure 1 shows a product 81 and a layered packaging material 82 enclosing the product 81 as workpieces 8. In the current task, the robot 2 opens the packaging material 82 enclosing the product 81 while changing its holding position within the packaging material 82. Therefore, in the current task, the packaging material 82 is a workpiece directly processed by the robot 2, and the product 81 is a workpiece indirectly affected by a movement of the robot 2 (i.e., work performed by the robot 2). In the next task, the robot 2 can directly process the product 81, for example, by moving the product 81 out of the packaging material 82 to a different location.

[0017] Robot 2 is a device that receives energy, performs a predetermined operation according to a purpose, and does useful work. In one example, Robot 2 comprises a plurality of joints, an arm, and an end effector 2a attached to the end of the arm. Robot 2 uses the end effector 2a to perform unpacking operations and, in another example, can also perform additional operations. Examples of the end effector 2a include a gripper, a suction hand, and a magnetic hand. A joint axis is defined for each of the plurality of joints. Some components of Robot 2, such as the arm and a swivel unit, rotate about the joint axis, allowing Robot 2 to change the position and orientation of the end effector 2a within a predetermined range. In another example, Robot 2 is a multi-axis, serially linked, vertically articulated robot.Robot 2 can be a six-axis vertically articulated robot or a seven-axis vertically articulated robot with a redundant axis added to the six axes. Robot 2 can be a mobile robot, such as an autonomous mobile robot (AMR) or a robot carried by an automated guided vehicle (AGV). Alternatively, Robot 2 can be a stationary robot fixed in a predetermined location.

[0018] The robot controller 3 is a device that controls the robot 2 according to a previously generated operating program. In one example, the robot controller 3 receives a robot position signal from the robot control system 1 to adjust the position and orientation of the end effector to a setpoint specified by the operating program and controls the robot 2 accordingly. Furthermore, the robot controller 3 transmits the position signal back to the robot control system 1. As described above, examples of the position signal include the joint angle (the angle of each joint) and the joint torque (the torque at each joint).

[0019] Camera 4 is a device that captures at least a portion of the area within the workspace 9 and generates image data that displays the current state of the workpiece in that area. For example, camera 4 captures at least the workpiece 8 being processed by robot 2 and generates a situational image showing the current state of the workpiece 8. Camera 4 transmits this situational image to the robot control system 1. Camera 4 can be mounted on a mast, a roof, or similar structure, or positioned near the tip of robot 2's arm.

[0020] In the present disclosure, image data and various images can be a still image or a group of one or more individual images selected from a multitude of individual images that form a video.

[0021] Fig. Figure 2 is a diagram showing an example of a functional configuration of the robot control system 1. In this example, the robot control system 1 comprises a sensing unit 11, a setting unit 12, a simulation unit 13, a prediction evaluation unit 14, an adjustment unit 15, an iteration control unit 16, a status evaluation unit 17, a planning unit 18, a decision unit 19, a robot control unit 20, a data generation unit 21, a sample database 22, and a training unit 23 as functional components.

[0022] The acquisition unit 11 is a functional module that acquires data from the robot controller 3 and the camera 4, which is used to determine the next setpoint in the current task. The setting unit 12 is a functional module that initially sets the next setpoint. The simulation unit 13 is a functional module that virtually executes the current task, in which the robot 2 processes the workpiece 8 with the next setpoint, by means of simulation. The prediction evaluation unit 14 is a functional module that calculates an evaluation value for a prediction result of the simulation based on a target value specified in connection with the workpiece 8. In this disclosure, this evaluation value is also referred to as the "prediction evaluation value". The adjustment unit 15 is a functional module that adjusts the next setpoint based on the prediction evaluation value.The iteration control unit 16 is a functional module that controls the simulation unit 13, the prediction evaluation unit 14, and the adjustment unit 15 such that the simulation, the calculation of the prediction evaluation value, and the adjustment of the next setpoint are repeated. The status evaluation unit 17 is a functional module that calculates an evaluation value with respect to an execution status of the current task (e.g., a current state of the workpiece 8 to be processed) based on the target value preset and associated with the workpiece 8. In this disclosure, this evaluation value is also referred to as the "status evaluation value." The planning unit 18 is a functional module that plans the next task based on the execution status of the current task.Decision unit 19 is a functional module that completes the next operation of robot 2 based on at least the adjusted next setpoint, the execution status of the current task, and the plan for the next task. Robot control unit 20 is a functional module that controls robot 2 based on the specified parameters.

[0023] The data generation unit 21, the sample database 22, and the training unit 23 are functional modules for generating a trained model used to control the robot 2. The trained model is generated by machine learning, a method for autonomously discovering a law or rule through iterative learning based on given information. The data generation unit 21 is a functional module that generates at least some of the training data used in machine learning based on the operation of the robot 2, which is currently performing the task, or the state of the workpiece 8, which is currently being processed in the current task. The sample database 22 is a functional module that stores the training data generated by the data generation unit 21 and the training data collected by the robot 2 before the execution of the current task.This means that the sample database 22 can store both pre-collected training data and training data acquired by the robot 2 during the execution of the current task. The training unit 23 is a functional module that generates the trained model through machine learning using the training data in the sample database 22. For example, the training unit 23 generates at least one control model used by the setting unit 12, one state prediction model used by the simulation unit 13, one evaluation model used by the prediction evaluation unit 14 and the state evaluation unit 17, and one planning model used by the scheduling unit 18. These trained models are implemented, for example, by a neural network such as a deep neural network (DNN).By generating the trained model through machine learning, it is possible to quantify the evaluation of workpiece 8 or the task based on implicit knowledge (knowledge based on human experience or intuition) and to control the robot 2 accordingly.

[0024] The robot control system 1 can be implemented by any type of computer. The computer can be a general-purpose computer such as a personal computer or a business server, or it can be integrated into a special device that performs specific processing operations.

[0025] Fig. Figure 3 is a diagram showing an example of a hardware configuration for a computer 100 used for the robot control system 1. In this example, the computer 100 comprises a main unit 110, a monitor 120, and an input device 130.

[0026] The main body 110 is a device with a circuit 160. The circuit 160 has a processor 161, a data storage device 162, a memory 163, an input / output port 164, and a communication port 165. The number of each hardware component can be 1, 2, or more. The memory 163 stores a program for configuring each functional module of the main body 110. The memory 163 is a computer-readable recording medium, such as a hard disk, non-volatile semiconductor memory, a magnetic disk, or an optical disk. The data storage device 162 temporarily stores a program loaded from the memory 163, calculation results of the processor 161, and the like. The processor 161 configures each functional module by executing the program in conjunction with the data storage device 162.The input / output port 164 inputs electrical signals into the monitor 120 or the input device 130 and outputs them from these devices in response to commands from the processor 161. The communication port 165 performs data communication with other devices, such as the robot controller 3, via the communication network N according to commands from the processor 161.

[0027] The monitor 120 is a device for displaying information output by the main body 110. For example, the monitor 120 is a device that can enable a graphic display, such as a liquid crystal display.

[0028] The input device 130 is a device for inputting information into the main body 110. Examples of the input device 130 include operating interfaces such as a keyboard, a mouse, and a manipulation control.

[0029] The monitor 120 and the input device 130 can be integrated as a touch panel. For example, the main body 110, the monitor 120, and the input device 130 can be integrated like a tablet computer.

[0030] Each functional module in the robot control system 1 is implemented by loading a robot control program into processor 161 or memory 162 and executing the program in processor 161. The robot control program includes code for implementing each functional module of the robot control system 1. Processor 161 operates the input / output port 164 and the communication port 165 according to the robot control program and performs reading and writing of data to data memory 162 or memory 163.

[0031] The robot control program can be provided by recording it on a non-volatile recording medium such as a CD-ROM, a DVD-ROM, or semiconductor memory. Alternatively, the robot control program can be provided over a communication network as data signals superimposed on carrier waves. [Robot control method](Robot control based on the nearest setpoint)

[0032] Examples of the robot control method according to the present disclosure are based on the following: Fig. Four to six examples of controlling the robot by determining the next setpoint are described. Fig. Figure 4 is a flowchart showing the sequence of processes as processing flow S1. This means that the robot control system 1 executes processing flow S1. Fig. Figure 5 is a diagram showing an architecture associated with determining the next setpoint. Fig. 5 is the time (t - 1) the current time and the time t is a time at which the robot control is executed based on the next setpoint, i.e. a time shortly after the current time. Fig. Figure 6 is a diagram showing an example of an architecture in the context of simulation.

[0033] In step S11, the acquisition unit 11 records observation data indicating the current state of the workspace 9. For example, the acquisition unit 11 records a position value of the robot 2, which is processing the workpiece 8, as the current position value from the robot controller 3, and records a situational image from the camera 4 showing the workpiece 8 being processed by the robot 2. That is, the observation data can include the current position value and the situational image.

[0034] In step S12, the setting unit 12 first sets the next setpoint OP. initThe robot 2's position in the current task is determined based on the observation data. The setting unit 12 inputs the situational image and the current setpoint into a control model 12a to determine the next setpoint OP. init The control model 12a is a trained model that is trained to calculate a second position value of robot 2 at a second time point after the first time point, based on an example image that specifies a workpiece at a first time point and a first position value of robot 2 at the first time point.

[0035] In step S13, simulation unit 13 performs a simulation based on the specified next setpoint. In the first loop processing, simulation unit 13 virtually executes the current task through the simulation, in which robot 2 is assigned the next setpoint OP. initThe simulation unit 13 works to machine workpiece 8. In one example, the simulation unit 13 uses a robot model that specifies robot 2 and a context relating to an element that forms the workspace 9 (hereinafter also referred to as a "component"). The robot model is electronic data that specifies requirements relating to robot 2 and end effector 2a. The specifications may include parameters relating to the structures of robot 2 and end effector 2a, such as shape, dimensions, etc., as well as parameters relating to the functions of robot 2 and end effector 2a, such as the range of motion of each joint, capabilities of end effector 2a, etc. The context refers to electronic data that specifies various properties of each or all components of the workspace 9 and may be expressed, for example, by text (i.e., natural language).The element that constitutes workspace 9 can be described as a tangible object existing within workspace 9. The context can include various attributes of workpiece 8, such as its type, shape, physical properties, dimensions, and color. Alternatively, the context can include various attributes of robot 2 or end effector 2a, such as its type, shape, size, and color. Alternatively, the context can include attributes of the environment of both robot 2 and workpiece 8. Examples of environment attributes include the type, shape, and color of the worktable, the type and color of the floor, and the type and color of the walls. As described above, the context can include at least workpiece information relating to workpiece 8, robot information (robot model) relating to robot 2, and environment information relating to the surrounding environment.Based on the robot model, the context, and the next setpoint, the simulation unit 13 generates a prediction result that includes a predicted state of the workpiece 8 within a predetermined time interval in the future, including time t. The prediction result can also include a movement of the robot 2 within this time interval.

[0036] An example of the simulation is given with reference to Fig. Section 6 describes this in detail. In this example, simulation unit 13 performs kinematic / dynamic calculations based on the next control input to generate a virtual motion of robot 2 operating in response to that input. This processing generates a motion that takes into account the geometric (kinematic) and mechanical (dynamic) constraints of robot 2. Subsequently, simulation unit 13 uses a renderer to generate a motion image Pm, which depicts the virtual motion of robot 2. Since the virtual motion is generated based on the next control input, the renderer that renders the virtual motion can be considered a process based on the next control input. In one example, simulation unit 13 uses differentiable kinematics / dynamics and a differentiable renderer to generate the motion image Pm from the next control input.This example can be implemented to make a series of processes differentiable from the input of the next setpoint to the output of the prediction evaluation value, in order to use backward propagation to reduce the prediction evaluation value.

[0037] The simulation unit 13 inputs the virtual motion specified by the motion image Pm and the context into a state prediction model 13a and generates a state of the workpiece 8, which is being processed by the robot 2 operating with the next control value, as the predicted state. The predicted state can include a temporal change in the situation of the workpiece 8 within a predetermined time interval in the future, encompassing time t. The predicted state can further specify a motion of the robot 2 within this time interval. In an example, the state prediction model 13a generates a predicted image Pr showing the predicted state. The state prediction model 13a is a trained model, trained to predict a state of the workpiece 8 based on the motion of the robot 2 and the context.The simulation unit 13 can generate a temporal change in the virtual appearance of the workpiece 8 as the predicted state (the predicted image Pr) due to the virtual movement of the robot 2. The appearance of the workpiece refers, for example, to the shape of its appearance.

[0038] Back to the Fig. 4 and Fig. 5. In step S14, the forecast evaluation unit 14 evaluates the forecast result obtained from the simulation. In an example, the forecast evaluation unit 14 calculates a forecast evaluation value E. pred, which is an evaluation value of the predicted state of workpiece 8, based on a preset target value relating to workpiece 8. In an example, the target value is represented by a target image, which is an image indicating a predetermined state of workpiece 8 to be compared with the predicted state. The target value can be a final state of workpiece 8 in the current task, and in this case, the target image indicates the final state. Alternatively, the target value can be a state of workpiece 8 at a point in time in the middle of the current task (intermediate state), and, for example, an intermediate state of workpiece 8 at a time when the next setpoint is actually applied (time t in the example of ). Fig. 5) In this case, the target image indicates the intermediate state. The prediction evaluation value E predIndicates how close the predicted state of workpiece 8 is to the target value. The smaller the prediction evaluation value E pred The closer the predicted state is to the target value, the higher the predicted value. In an example, the prediction evaluation unit 14 inputs the predicted image Pr and the target image into an evaluation model 14a to determine the prediction evaluation value E. pred to calculate. The evaluation model 14a is a trained model that is trained to calculate an evaluation value based on a state of the workpiece 8 and a target value (for example, based on an image showing a state of the workpiece 8 and a target image showing a target value).

[0039] In step S15, the adjustment unit 15 adjusts the next setpoint based on the evaluation of the prediction result (predicted state). For example, the adjustment unit 15 adjusts the next setpoint based on an evaluation of a temporal change in the virtual appearance state of workpiece 8. The adjustment unit 15 can set the next setpoint so that the state of workpiece 8 is closer to the target value than the predicted state, and adjust the next setpoint OP. adj The adjustment unit 15 can increase the adjustment amount of the next setpoint if the prediction evaluation value E pred increases, i.e., when the predicted state deviates from the target value.

[0040] In step S16, the iteration control unit 16 determines, based on a predetermined termination condition, whether the setting of the next setpoint should be terminated or not. The termination condition can be that the iteration process has been repeated a predetermined number of times or that a predetermined calculation time has elapsed. Alternatively, the termination condition can be that the difference between the previously obtained prediction evaluation value E pred and the currently obtained forecast evaluation value E pred becomes equal to or less than a predetermined threshold, i.e., the prediction evaluation value E pred remains the same or converges.

[0041] In a case where the next setpoint is to be further adjusted (NO in step S16), the process returns to step S13. In the repeated step S13, simulation unit 13 performs the simulation based on the set next setpoint OP.adj The simulation unit 13 executes the simulation based on the set next setpoint OP. adj and the context, in order to generate at least one predicted state of workpiece 8 within a predetermined time span in the future, encompassing time t. Since the next setpoint OP adjSince the predicted state obtained in the current loop processing differs from each subsequent setpoint used in the previous loop processing, the predicted state obtained in the current loop processing can also differ from each predicted state used in the previous loop processing. As described above, the simulation unit 13 can generate the predicted image Pr, which displays the predicted state. In the repeated step S14, the prediction evaluation unit 14 inputs the predicted state (predicted image Pr) and the target value (target image) obtained this time into the evaluation model 14a to determine the prediction evaluation value E. pred to calculate. In the repeated step S15, the adjustment unit 15 adjusts the next setpoint based on the prediction evaluation value E. predfurther. Through such an iterative process, a multitude of adjusted next setpoints are generated (OP). adj receive.

[0042] In a case where the adjustment is to be completed (YES in step S16), the process proceeds to step S17. In step S17, decision unit 19 selects the next setpoint OP from the multitude of possible values. adj one last next value OP final For example, decision unit 19 determines the next setpoint OP. adj , which was finally obtained through the iterative process, as the next place value OP final Alternatively, decision unit 19 can set the next position value OP. adj , where the state of workpiece 8 is expected to converge to the target value assigned to workpiece 8, as the next setpoint OP final define. For example, decision unit 19 sets OP as the next setpoint. final the next value OP adjfixed, at which workpiece 8 is expected to converge to the target value at the earliest.

[0043] In step S18, the robot control unit 20 controls the actual robot 2 in the workspace 9 based on the next setpoint OP. final Since the next setpoint OP final one of the many next values ​​OP adj It can be said that the robot control unit 20 controls the robot 2 based on the adjusted next setpoint OP. adj controls. The robot control unit 20 transmits the next setpoint OP. final to robot controller 3 to control robot 2. Robot controller 3 controls robot 2 according to the setpoint OP. final . Robot 2 continues to execute the current task according to the control system in order to further process workpiece 8.

[0044] The robot control system 1 can repeatedly execute the processing sequence S1 at predefined time intervals. In the example of Fig. 5. The robot control system 1 executes the processing sequence S1 based on the observation data at time (t - 1) to determine the next setpoint at time t. The actual robot 2 processes the actual workpiece 8 based on this setpoint.

[0045] The robot control system 1 receives the setpoint at time t from the robot controller 3 and captures the situational image, which shows the state of the workpiece 8 at time t, from camera 4. Based on this observational data, the robot control system 1 executes the processing sequence S1 to determine the next setpoint at time (t + 1). The actual robot 2 continues processing the actual workpiece 8 based on the setpoint. The robot control system 1 instructs the robot 2 to perform the current task, while simultaneously generating the next setpoint sequentially by repeating this processing. (Task control)

[0046] Examples of the robot control method according to the present disclosure are examples of task control with reference to Fig. 7 described. Fig.Figure 7 is a flowchart showing a series of task control procedures as a processing sequence S2. This means that the robot control system 1 executes the processing sequence S2. In one example, the robot control system 1 executes the processing sequences S1 and S2 in parallel.

[0047] In step S21, the acquisition unit 11 records the observation data indicating the current state of workspace 9. This process corresponds to step S11. As described above, the acquisition unit 11 can record the current setpoint and the situational picture as observation data.

[0048] In step S22, decision unit 19 decides whether the current task should be continued or not. For this determination, status evaluation unit 17 calculates a status evaluation value, which is an evaluation value relating to the execution status of the current task, based on the target value that is preset in connection with workpiece 8. In one example, the target value is represented by a target image, which is an image indicating a predetermined state of workpiece 8 that is to be compared with the current state of workpiece 8, represented by the situation image. The target value can be a final state of workpiece 8 in the current task, and in this case, the target image indicates the final state. The status evaluation value indicates how close the execution status of the current task (e.g., the current state of workpiece 8) is to the target value.In the present disclosure, the smaller the status evaluation value, the closer the execution status of the current task (e.g., the current state of workpiece 8) is to the target value. In an example, the status evaluation unit 17 inputs the situation and target states into the evaluation model to calculate the status evaluation value. Based on the status evaluation value, the decision unit 19 determines whether the current task is continued or not. Therefore, the decision unit 19 also functions as a determining unit. For example, the decision unit 19 determines whether to continue the current task if the status evaluation value is greater than or equal to a predetermined threshold, and determines whether to terminate the current task if the status evaluation value is less than the threshold.In a case where the current task is to be continued (YES in step S22), the process moves to step S23, and in a case where the current task is to be terminated (NO in step S22), the process moves to step S26.

[0049] In step S23, decision unit 19 determines whether the action position in the current task should be changed. For this determination, status evaluation unit 17 calculates a status evaluation value, which is an evaluation value related to the execution status of the current task, based on a target value that is preset in connection with workpiece 8. Similar to step S22, status evaluation unit 17 can calculate the evaluation value for the current state of workpiece 8 as the execution status of the current task. Unlike step S22, the target value in step S23 can be an ideal state of workpiece 8 (an intermediate state) at a point in time in the middle of the current task. In this case, the target image displays the intermediate state. In an example, status evaluation unit 17 inputs the current image and the target image into the evaluation model to calculate the status evaluation value.Decision unit 19 determines, based on the status evaluation value, whether the action position should be changed from its current position. For example, decision unit 19 decides to change the action position if the status evaluation value is greater than or equal to a predetermined threshold, and decides not to change the action position if the status evaluation value is less than the threshold. In a case where the action position should be changed (YES in step S23), the process proceeds to step S24, and in a case where the action position should not be changed (NO in step S24), the process proceeds to step S25.

[0050] In step S24, the robot control unit 20 controls the robot 2 so that it changes its action position and continues the current task. For example, the robot control unit 20 analyzes the situation to search for and determine a new action position. Then, the robot control unit 20 generates a command to change the action position from the current position to the new position and transmits the command to the robot controller 3. The robot controller 3 controls the robot 2 according to the command. As directed by this control, the robot 2 changes its action position from the current position to the new position and continues executing the current task.

[0051] In step S25, the robot control unit 20 controls the robot 2 so that the current task continues without changing the action position. This process corresponds to step S18 described above. The robot control unit 20 controls the robot 2 based on the next setpoint OP. final , which was determined by processing sequence S1. The robot control unit 20 transmits the next setpoint OP. final to robot controller 3 to control robot 2. Robot controller 3 controls robot 2 according to the setpoint OP. final . According to this control, robot 2 continues the current task without changing its action position in order to continue processing workpiece 8.

[0052] In step S26, the robot control unit 20 controls robot 2 to complete the current task. In one example, for this processing, the planning unit 18 inputs the current situation into a planning model to create a plan for the next task following the current one. The planning model is a trained model that is trained to plan the next task based on the current situation of workpiece 8. According to a result of the plan, the robot control unit 20 controls robot 2 to complete the current task. For example, the plan for the next task might include a plan for an operation of the robot within the next task, and the robot control unit 20 might control the position of robot 2 at the end of the current task so that robot 2 can transition smoothly to this operation.The robot control unit 20 sends a command to the robot controller 3 to instruct the real robot 2 to complete its current task. The robot controller 3 instructs the robot 2 to complete its current task according to the command. In an example, the robot control unit 20 further transmits a command for the next task to the robot controller 3. The robot controller 3 instructs the robot 2 to begin the next task according to this command.

[0053] As shown in the processing flow S2, the robot control unit 20 can control the robot 2 based on a circuit (decision) as to whether the current task should be continued or not, or a decision as to whether the action position should be changed or not.

[0054] The robot control system 1 can repeatedly execute the processing sequence S2 at predefined time intervals. As a result of this repetition, the robot 2 continues the current task, changing its action position as needed to process the workpiece 8, and finally completes the current task. [Machine Learning]

[0055] In one example, training unit 23 creates or updates the one or more trained models used in robot control system 1 through supervised learning. Supervised learning uses training data (sample data) comprising a variety of datasets that specify a combination of input data to be processed by a machine learning model and the basic truth of output data from the machine learning model. For each dataset of the training data, training unit 23 performs the following processing: It inputs the input data specified by the dataset into the machine learning model. Training unit 23 then performs backward propagation based on an error between the output data estimated by the machine learning model and the basic truth specified by the dataset, and updates the parameters in the machine learning model.Training Unit 23 repeats the process for each data set until a predetermined termination condition is met, at which point the trained model is generated or updated. The termination condition might be to process all records of the training data. It's important to note that each generated or updated trained model is a computational model that is estimated to be optimal, and not necessarily an "actually optimal computational model."

[0056] The generation or updating of the control model is described. In an example, the data generation unit 21 generates a data set that includes a combination of the current setpoint, the situational information obtained from the acquisition unit 11, and the next setpoint (e.g., the final determined next setpoint) set based on the current setpoint. The data generation unit 21 stores the data set in the sample database 22 as at least part of the training data. The training unit 23 updates the control model using machine learning with the data set. In this machine learning process, the training unit 23 uses the adjusted next setpoint (e.g., the final determined next setpoint) as the baseline.

[0057] As another example, data generation unit 21 creates a training image from the predicted image Pr (state prediction model) generated by simulation unit 13. Data generation unit 21 modifies the predicted image based on change information regarding the scene specified by the predicted image—that is, the scene representing the predicted state—and obtains a training image representing a different state, distinct from the predicted state. The change information can be information about modifying the workpiece represented by the predicted image. For example, the change information can be information about modifying a predicted image showing a scene where a plastic bag is being processed into a training image showing a scene where a hemp sack is being processed.Alternatively, the change information can be information on changing the environment of robot 2 and workpiece 8. For example, the change information can be information on changing a predicted image showing a scene in which a workpiece placed on a worktable is being processed, to a training image showing a scene in which a workpiece placed on a floor is being processed. The data generation unit 21 can generate a data record that includes the current setpoint, the next setpoint set based on the current setpoint (e.g., the final setpoint), and the training image. The data generation unit 21 stores the data record in the sample database 22 as at least part of the training data.Training Unit 23 can update the control model using machine learning with the dataset or generate a different control model for the initial setting of the next setpoint. In either case, during such machine learning, Training Unit 23 uses the adjusted next setpoint (e.g., the finally determined next setpoint) as the baseline.

[0058] The generation and updating of the state prediction model is described. In an example, the data generation unit 21 generates a data set that includes a combination of the adjusted next setpoint (e.g., the finally determined next setpoint) and an actual state, which is a state of the actual workpiece 8 being processed by the actual robot 2, controlled by the robot control unit 20 based on the setpoint. That is, the data generation unit 21 generates a data set that includes a combination of the adjusted next setpoint and the resulting situation picture obtained as a result of this setpoint. The data generation unit 21 stores the data set in the sample database 22 as at least part of the training data. The training unit 23 can update the state prediction model using machine learning with the data set or generate a new state prediction model.In this machine learning process, training unit 23 uses kinematics / dynamics and a renderer to generate a virtual movement of robot 2 from the next input value specified by the training data, and then inputs the generated movement and a predefined context into the machine learning model. Training unit 23 uses the situational image as its basic truth.

[0059] As another example, in a case where the context is expressed by text, training unit 23 can receive the text specifying the context, compare the text with the predicted state generated by the state prediction model, and update the state prediction model through machine learning based on the result of the comparison. For example, training unit 23 inputs the predicted image into a coder model that converts an image-specified situation into text, generating text that specifies the predicted situation. Then, training unit 23 can compare the text specifying the context with the text specifying the predicted situation and update the state prediction model through machine learning using a difference (i.e., a loss) between the two texts.Alternatively, training unit 23 can compute a latent variable from both the text providing the context and the predicted state (predicted image), and update the state prediction model using machine learning based on a difference (loss) between the two latent variables. Alternatively, training unit 23 can use a predetermined comparison model that compares the text providing the context with the predicted state (predicted image), and update the state prediction model using machine learning based on a comparison result obtained from the comparison model.

[0060] The generation of the evaluation model is described. In one example, the sample database 22 stores a large number of data records in advance as training data. Each record contains a combination of image data indicating the state of a workpiece processed at a specific time in the past, a target value predefined in relation to the workpiece, and an evaluation value defined for the workpiece's state. The training unit 23 generates the evaluation model using machine learning with this training data. In this machine learning process, the training unit 23 uses the evaluation value specified by the training data as the baseline truth.

[0061] The generation of the planning model is described. In one example, the sample database 22 stores a large number of data records in advance as training data. Each data record contains a combination of image data showing the state of a workpiece at a specific point in the past and a plan for the next task related to the workpiece. The plan for the next task may include a plan for the movement of robot 2 during the next task. The training unit 23 generates the planning model using machine learning with this training data. In this machine learning process, the training unit 23 uses the plan for the next task specified by the training data as the basic truth.

[0062] Creating the trained model corresponds to a learning phase of machine learning. Prediction or estimation using the created trained model corresponds to an operational phase of machine learning. The processing flows S1 and S2 mentioned above correspond to the operational phase.

[0063] In the examples above, a combination of the control model, the state prediction model, and the evaluation model can be described as a command generation model trained to output specific position data in a case where at least image data (situational image) is input. This data indicates the robot's position at a second time point after the first time point at which the image data is acquired. The next control value can be interpreted as the specified position data. [Modifications]

[0064] The technology according to the present disclosure has been described in detail above with reference to various examples. However, the present disclosure is not limited to the examples mentioned above. Various modifications can be made to the technology according to the present disclosure without altering its core message.

[0065] The robot control system can control at least one of a plurality of real robots that cooperatively process a workpiece according to the current situation of a real workspace in which the plurality of real robots are arranged. For example, the robot control system controls each six-axis robot in a process where two six-axis robots work together to open packaging material. The robot control system can execute the processing sequences S1 and S2 described above for at least one of the plurality of robots, for example, for each robot.

[0066] The control model can be trained to calculate a second robot position at a second time point, based on a sample image showing the workpiece at a first time point and a first robot position at that first time point. In a case where the control model is used, the setting unit inputs one of the current position values ​​and the situation image into the control model to initially set the next position value. Alternatively, the control model can be trained to calculate the second position value based on at least one of the following elements: context, target value (which specifies the final or intermediate target with respect to the workpiece), and blank point, in addition to at least one of the following elements: sample image and first position value.In a case where the control model is used, the setting unit inputs at least one of the current setpoint and situation picture and at least one of the context, setpoint and blank point into the control model in order to initially set the next setpoint.

[0067] The simulation method and the configuration of the state prediction model are not limited to the examples above. For instance, the simulation unit can input the next setpoint into the state prediction model, which is trained to predict the workpiece's state based on that next setpoint, in order to generate the predicted workpiece state. Therefore, the simulation unit can generate the predicted state without using the kinematics / dynamics and the renderer.

[0068] The trained model is transferable between computer systems. The robot control system may not include functional modules corresponding to the data generation unit 21, the sample database 22, and the training unit 23, and may use a trained model generated by another computer system.

[0069] The adjustment unit can adjust the initially set next position value, and the robot control unit can control the robot based on the adjusted next position value. Therefore, the robot control system may not include a function module corresponding to the iteration control unit 16.

[0070] The adjustment unit can adjust the next setpoint without using the prediction evaluation value. For example, the adjustment unit can calculate a difference between the target image displaying the setpoint and the predicted image, and adjust the next setpoint based on this difference. For example, the adjustment unit can increase the adjustment amount of the next setpoint as the difference increases. In such a modification, the robot control system may not include a function module corresponding to the prediction evaluation unit 14.

[0071] The robot control system cannot perform the process of determining whether the current task should be completed and controlling the robot. Alternatively, the robot control system cannot perform the process of determining whether the action position in the current task should be changed and controlling the robot. Alternatively, the robot control system must not perform the process of planning the next task and completing the current task according to a planning result. Therefore, the robot control system must not include a functional module that corresponds to at least one of the status evaluation unit 17, the determination unit (part of the decision unit 19), and the planning unit 18.

[0072] In the examples above, camera 4 captures the current situation of the workspace 9, but a different sensor type than the camera, for example a laser sensor, can capture the current situation of the actual workspace.

[0073] The hardware configuration of the system is not limited to a single aspect where each functional module is implemented by executing a program. For example, at least some of the functional modules described above may be configured by a logic circuit specialized for that function, or by an application-specific integrated circuit (ASIC) in which the logic circuit is integrated.

[0074] The processing procedure performed by the at least one processor is not limited to the example above. For instance, some of the steps or processes described above can be omitted, or the steps can be executed in a different order. Furthermore, two or more of the steps described above can be combined, or some of the steps can be modified or deleted. Alternatively, additional steps can be performed alongside those described above.

[0075] When comparing the magnitude of two numerical values ​​in a computer system or computer, one of two criteria, "equal to or greater than" and "greater than", can be used, and one of two criteria, "equal to or less than" and "less than", can be used. [Attachment]

[0076] As can be seen from the various examples described above, the present revelation includes the following aspects.

[0077] (Annex 1) Robot control system, comprising: a setting unit configured to first set the next position value in a current task for a robot that is located in a real workspace and is performing the current task of machining a workpiece; a simulation unit configured to virtually perform the current task, in which the robot works with the next setpoint to process the workpiece, through simulation; an adjustment unit configured to adjust the next setpoint based on a prediction result obtained through simulation; and a robot control unit configured to control the robot in the real workspace based on the adjusted next setpoint.

[0078] (Annex 2) Robot control system according to Annex 1, wherein the prediction result includes a predicted state, which is a state of the workpiece that was processed by the robot with the next setpoint, and the adjustment unit is configured to adjust the next setpoint at least on the basis of the predicted state.

[0079] (Annex 3) Robot control system according to Annex 2, further comprising an evaluation unit configured to calculate an evaluation value of the predicted state of the workpiece based on a setpoint preset in connection with the workpiece, the adjustment unit is configured to adjust the next setpoint based on the evaluation value.

[0080] (Annex 4) Robot control system as per Annex 3, further comprising: an iteration control unit configured to control the simulation unit, the evaluation unit, and the adjustment unit so that the simulation, the calculation of the evaluation value, and the adjustment of the next setpoint based on the evaluation value are repeated; and a decision unit configured to determine a final next setpoint from a multitude of adjusted next setpoints obtained through repetition, where the robot control unit is configured to control the robot based on the final next setpoint.

[0081] (Annex 5) Robot control system according to one of Annexes 1 to 4, wherein the setting unit is configured to first set the next setpoint based on image data showing the workpiece being processed by the robot in the real workspace.

[0082] (Annex 6) Robot control system according to any one of Annexes 1 to 5, wherein the setting unit is configured to input a current position value of the robot processing the workpiece into a control model which is trained to calculate a second position value at a second time after the first time based on a first position value of the robot at a first time, and first sets the next position value.

[0083] (Annex 7) Robot control system according to one of Annexes 2 to 4, wherein the simulation unit is configured to: generates a virtual movement of the robot, which operates with the next setpoint; and The generated virtual motion is fed into a state prediction model that is trained to predict a state of the workpiece based on a movement of the robot, and the predicted state is generated.

[0084] (Annex 8) Robot control system according to Annex 7, wherein the simulation unit is configured to generate as a predicted state a temporal change of a virtual appearance state of the workpiece caused by the virtual movement, and wherein the adjustment unit is configured to adjust the next setpoint at least on the basis of the temporal change in the virtual appearance state of the workpiece.

[0085] (Annex 9) Robot control system according to Annex 7 or 8, wherein the simulation unit is configured to input the generated virtual motion and a context relating to an element forming the workspace into a state prediction model trained to further predict a state of the workpiece on the basis of the context and to generate the predicted state.

[0086] (Annex 10) Robot control system according to Annexes 7 to 9, further comprising a training unit configured to update the state prediction model by machine learning using training data comprising a combination of the adjusted next setpoint and an actual state, which is a state of the workpiece processed by the robot under the control of the robot control unit.

[0087] (Annex 11) Robot control system according to Annex 10, wherein the training unit is configured to: a text receives as context that refers to an element that forms the workspace; compares the text and the predicted state and updates the state prediction model through machine learning based on the result of the comparison.

[0088] (Annex 12) Robot control system according to Annexes 7 to 11, wherein the simulation unit is configured to generate an image showing the virtual motion using a renderer based on the nearest input value.

[0089] (Annex 13) Robot control system according to any one of Annexes 1 to 12, further comprising: an evaluation unit configured to calculate an evaluation value regarding the execution status of the current task based on a target value preset in conjunction with the workpiece; and a determination unit configured to switch, based on the evaluation value, whether the current task continues or not, the robot control unit is configured to control the robot based on the switching.

[0090] (Annex 14) Robot control system according to any one of Annexes 1 to 13, further comprising: an evaluation unit configured to calculate an evaluation value regarding the execution status of the current task based on a target value preset in conjunction with the workpiece; and a determination unit configured to determine, based on the evaluation value, whether an action position should be changed from a current position or not, where the action position is a position at which the robot acts on the workpiece in the current task, wherein the robot control unit is configured such that, in a case where it is determined that the action position should be changed from the current position, it causes the robot to change the action position from the current position to a new position and continue the current task.

[0091] (Annex 15) Robot control system according to any one of Annexes 1 to 14, further comprising a planning unit configured to plan a next task following the current task based on a planning model and image data, wherein the image data specifies the workpiece being processed by the robot in the real workspace, and wherein the planning model is trained to output a plan for the next task in response to the input image data, wherein the robot control unit is configured to control the robot according to a result of the planning by the planning unit in such a way that the current task is completed.

[0092] (Annex 16) Robot control system as set out in Annex 6, further comprising a training unit configured to update the control model by machine learning using training data comprising a combination of the current setpoint and the adjusted next setpoint.

[0093] (Annex 17) Robot control system as specified in Annex 16, further comprising a data generation unit configured to generate the training data, wherein the simulation unit is configured to generate a predicted image that displays the predicted state of the workpiece based on a state prediction model and the next setpoint, wherein the state prediction model is trained to generate the predicted image based on a movement of the robot working with the next setpoint and a context relating to an element that forms the workspace, where the data generation unit is configured such that it: The predicted image is modified based on change information to change a scene that displays the predicted state, and a training image is generated that displays a different state that differs from the predicted state; the training data generated, which includes a combination of the current setpoint, the adjusted next setpoint, and the training image, and wherein the training unit is configured to update the control model or generate a different control model for initially setting the next setpoint by performing machine learning using the training data, which further includes the training image.

[0094] (Annex 18) Robot control method executable by a robot control system having at least one processor, the method comprising: Initial setting of the next setpoint in a current task for a robot located in a real workspace, and execution of the current task to process a workpiece; Virtual execution of the current task, in which the robot works with the next setpoint to process the workpiece, through simulation; Adjusting the next setpoint based on a prediction result obtained through the simulation; and Controlling the robot in the real workspace based on the adjusted next setpoint.

[0095] (Appendix 19) Robot control program to cause a computer to perform the following: Initial setting of the next setpoint in a current task for a robot arranged in a real workspace and execution of the current task to process a workpiece; Virtual execution, through simulation, of the current task, in which the robot works with the next setpoint to process the workpiece; Adjusting the next setpoint based on a prediction result obtained through the simulation; and Controlling the robot in the real workspace based on the adjusted next setpoint.

[0096] (Annex 20) Robot control system, comprising: a robot configured to perform a current task on a workpiece; a capture unit configured to sequentially capture image data showing the workpiece during the execution of the current task; a command generation unit configured to sequentially output, based on a command generation model trained to do so, certain position data indicating the position of the robot at a second time point after a first time point at which the image data is acquired, wherein at least the image data is input, the certain position data corresponding to the sequentially acquired image data; a robot control unit configured to control the robot to perform the current task based on sequentially generated specific position data.

[0097] (Annex 21) Robot control system as specified in Annex 20, further comprising: an evaluation unit configured to evaluate an execution status of the current task at a time when the image data is acquired, based on an evaluation model trained to output an evaluation value relating to an execution status of the current task in a case where at least the image data is input; a determination unit configured to switch, based on the generated specific position data and according to a result of the evaluation by the evaluation unit, whether the control of the robot continues or not.

[0098] (Annex 22) Robot control system according to Annex 21, further comprising an action point extraction unit configured to extract a new robot action point on the workpiece, wherein the robot control unit is configured to control the robot so that it performs the current task while acting on the workpiece at the new action point, in a case where the control of the robot is not continued.

[0099] (Annex 23) Robot control system according to Annex 20, further comprising a planning unit configured to output a plan for the next task following the current task, based on the acquired image data and a planning model trained to respond to at least the input image data, planning the next task, wherein the robot control unit is configured to terminate the execution of the current task by the robot according to a result of the planning by the planning unit.

[0100] According to Annexes 1, 18, and 19, the simulation, based on the originally defined next setpoint, predicts how the robot will next process the workpiece in the currently executed task. The next setpoint is then adjusted based on the prediction result, and the robot is controlled in the actual workspace based on this adjusted next setpoint. Because the next setpoint for further robot control is adjusted according to the prediction made by the simulation of the current task, the robot can be operated appropriately according to the current situation in the actual workspace. Furthermore, such appropriate robot control enables the current task and the workpiece to converge toward a desired target state.

[0101] According to Annex 2, the simulation predicts the state the workpiece will change to during the current task, and the next setpoint is adjusted based on this prediction. The state of the workpiece being processed by the robot is directly related to the success of the current task. By adjusting the next setpoint based on the slightly later state of the workpiece, the actual robot can therefore be instructed to process the workpiece appropriately according to the current situation in the real workspace.

[0102] According to Annex 3, a subsequent state of the workpiece obtained through simulation is evaluated against the target value associated with the workpiece, and a subsequent setpoint is adjusted based on this evaluation. The target value can be said to specify the desired state of the workpiece. Since the next setpoint is adjusted taking the target value into account, the actual robot can be instructed to process the real workpiece according to the current situation in the real workspace, bringing the real workpiece into the desired state.

[0103] According to Annex 4, the adjustment of the next setpoint is repeated based on the simulation and the evaluation of the prediction result, and then the next setpoint for controlling the robot is finally determined. By repeating the adjustment, the actual robot can be controlled with a more suitable next setpoint.

[0104] According to Annex 5, the next setpoint is initially determined based on image data that depicts the workpiece actually being machined. By using image data that clearly indicates the current state of the workpiece, the next setpoint can initially be appropriately determined based on the situation. Therefore, it can also be expected that the next setpoint to be adjusted will be a more appropriate value.

[0105] According to Annex 6, the next setpoint is first determined by the control model (trained model) based on the current setpoint of the real robot. This processing is expected to ensure that the next setpoint, which exhibits continuity with the current setpoint—that is, the next setpoint for the smooth operation of the real robot—is obtained more reliably. Therefore, it can be expected that the next setpoint to be adjusted will also be a suitable value that achieves smooth robot control, where the position of the actual robot does not change rapidly.

[0106] According to Annex 7, a virtual robot movement is generated, which is executed at the next setpoint value, and this movement is fed into a state prediction model (trained model) to predict the state of the workpiece being processed by the robot. By generating the predicted state from the virtual movement using the state prediction model, the state of the workpiece can be accurately predicted.

[0107] According to Annex 8, a temporal change in the virtual appearance of the workpiece is generated as a predicted state, and the next setpoint is adjusted based on this temporal change. Generally, with a workpiece whose appearance is changing, it is difficult to predict how this appearance will change shortly thereafter. By adjusting the next setpoint after predicting the change using simulation, the robot can be instructed to appropriately process the workpiece, whose appearance changes irregularly, according to the current situation.

[0108] According to Annex 9, a virtual robot movement occurring at the next setpoint and the context relating to a workspace element are input into the state prediction model, and the state of the workpiece being processed by the robot is predicted. Since the state prediction model receives the context input and generates the predicted state, the predicted state can be generated for various types of workpieces. By introducing a universal state prediction model capable of handling a wide variety of workpiece types and individually executing the robot movement and the generation of the predicted workpiece state in the simulation, universal robot control becomes possible that is independent of any workspace configuration element.Furthermore, since it is not necessary to prepare the state prediction model for each configuration element of the workspace, the number of steps required to prepare the state prediction model can be reduced or eliminated.

[0109] According to Annex 10, the state prediction model for predicting the workpiece state can be updated by machine learning based on the actual state of the workpiece being processed by the robot, which is actually controlled based on the adjusted next setpoint. The accuracy of the state prediction model can be further improved by machine learning using new data obtained from the actual robot control.

[0110] According to Annex 11, the state prediction model is updated by machine learning based on the comparison between the text providing the context and the predicted state of the workpiece. This machine learning can implement the state prediction model, which generates the predicted state according to the context specified in text form.

[0111] According to Annex 12, the image depicting the robot's virtual movement is generated by the renderer. Using the renderer allows the robot's three-dimensional structure and movement to be accurately represented in a single image. This enables a more precise prediction from the simulation.

[0112] According to Annexes 13 and 21, the execution status of the current task is evaluated based on the target value for the workpiece, and this evaluation determines whether or not to continue the current task. Since the decision to continue the current task is made taking into account the target value, which can be considered the desired state of the workpiece, the current task can be appropriately continued or terminated according to the current situation in the actual workspace.

[0113] According to Annexes 14 and 22, the execution status of the current task is evaluated based on the target value for the workpiece, and this evaluation determines whether or not the workpiece's action position should be changed. Since the action position in the current task is controlled taking into account the target value, which can be considered the desired state of the workpiece, the workpiece can be appropriately processed in the current task according to the actual situation in the workspace.

[0114] According to Annexes 15 and 23, the image data specifying the workpiece being processed by the current task is processed by the planning model (trained model). The next task following the current task is planned, and the current task is controlled according to the plan's outcome. By controlling the current task based on the plan for the next task, rather than the current task itself, a series of processes can be executed smoothly from the current task to the next.

[0115] According to Annex 16, the control model for the initial setting of the next setpoint is updated by machine learning based on the current setpoint and the adjusted next setpoint. The accuracy of the control model can be further improved by machine learning using the next setpoint actually used for robot control.

[0116] According to Annex 17, the training image, which represents a different state than the predicted state, is generated from the predicted image (which indicates the predicted state of the workpiece and is created by the state prediction model in the simulation). The control model can then be updated or regenerated using machine learning based on the combination of the current setpoint, the adjusted next setpoint, and the training image. By using machine learning with the training image generated from the predicted image, the accuracy of the control model can be improved, and a new control model can be created according to a variation element in the workspace. Furthermore, the number of steps required to create the control model can be reduced or eliminated.

[0117] According to Annex 20, the image data specifying the workpiece at the first time point processed by the current task is generated based on the command generation model, and the intended position data for the second time point, which occurs after the first, is generated. The robot is then controlled to continue executing the current task based on the intended position data. Since the intended position data for continuous robot control is generated according to the current situation of the current task, the robot can be operated appropriately according to the current situation of the actual workspace. Furthermore, such appropriate robot control allows the current task and the workpiece to converge toward a desired target state. Reference symbol list

[0118] 1: Robot control system, 2: Robot, 2a: End effector, 3: Robot controller, 4: Camera, 8: Workpiece, 9: Workspace, 11: Acquisition unit, 12: Setting unit, 12a: Control model, 13: Simulation unit, 13a: State prediction model, 14: Prediction evaluation unit, 14a: Evaluation model, 15: Adjustment unit, 16: Iteration control unit, 17: Status evaluation unit, 18: Planning unit, 19: Decision unit, 20: Robot control unit, 21: Data generation unit, 22: Sample database, 23: Training unit, Pm: Motion image, Pr: Predicted image. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 7021158

[0003]

Claims

[1] Robot control system, comprising: a setting unit configured to initially set the next position value in a current task for a robot located in a real workspace and performing the current task to machine a workpiece; a simulation unit configured to virtually perform the current task, in which the robot works with the next setpoint to process the workpiece, through simulation; an adjustment unit configured to adjust the next setpoint based on a prediction result obtained through simulation; and a robot control unit configured to control the robot in the real workspace based on the adjusted next setpoint. [2] Robot control system according to claim 1, wherein the prediction result includes a predicted state, which is a state of the workpiece that was processed by the robot with the next setpoint, and the adjustment unit is configured to adjust the next setpoint at least on the basis of the predicted state. [3] Robot control system according to claim 2, further comprising an evaluation unit configured to calculate an evaluation value of the predicted state of the workpiece based on a setpoint preset in connection with the workpiece, wherein the adjustment unit is configured to adjust the next setpoint based on the evaluation value. [4] Robot control system according to claim 3, further comprising: an iteration control unit configured to control the simulation unit, the evaluation unit, and the adjustment unit so that the simulation, the calculation of the evaluation value, and the adjustment of the next setpoint based on the evaluation value are repeated; and a decision unit configured to determine a final next setpoint from a multitude of adjusted next setpoints obtained through repetition, where the robot control unit is configured to control the robot based on the final next setpoint. [5] Robot control system according to any one of claims 1 to 4, wherein the setting unit is configured to first set the next setpoint on the basis of image data showing the workpiece being processed by the robot in the real workspace. [6] Robot control system according to any one of claims 1 to 4, wherein the setting unit is configured to input a current position value of the robot processing the workpiece into a control model which is trained to calculate a second position value at a second time after the first time based on a first position value of the robot at a first time, and first sets the next position value. [7] Robot control system according to any one of claims 2 to 4, wherein the simulation unit is configured such that it: generates a virtual movement of the robot, which operates with the next setpoint; and The generated virtual motion is fed into a state prediction model that is trained to predict a state of the workpiece based on a movement of the robot, and the predicted state is generated. [8] Robot control system according to claim 7, wherein the simulation unit is configured to generate as a predicted state a temporal change of a virtual appearance state of the workpiece caused by the virtual movement, and wherein the adjustment unit is configured to adjust the next setpoint at least on the basis of the temporal change in the virtual appearance state of the workpiece. [9] Robot control system according to claim 7, wherein the simulation unit is configured to input the generated virtual motion and a context relating to an element forming the workspace into a state prediction model which is trained to further predict a state of the workpiece on the basis of the context and generates the predicted state. [10] Robot control system according to claim 7, further comprising a training unit configured to update the state prediction model by machine learning using training data comprising a combination of the adjusted next setpoint and an actual state, which is a state of the workpiece processed by the robot controlled by the robot control unit. [11] Robot control system according to claim 10, wherein the training unit is configured such that it: a text receives as context that refers to an element that forms the workspace; compares the text and the predicted state and updates the state prediction model through machine learning based on the result of the comparison. [12] Robot control system according to claim 7, wherein the simulation unit is configured to generate an image showing the virtual movement using a renderer based on the nearest input value. [13] Robot control system according to any one of claims 1 to 4, further comprising: an evaluation unit configured to calculate an evaluation value regarding the execution status of the current task based on a target value preset in conjunction with the workpiece; and a determination unit configured to switch, based on the evaluation value, whether the current task continues or not, the robot control unit is configured to control the robot based on the switching. [14] Robot control system according to any one of claims 1 to 4, further comprising: an evaluation unit configured to calculate an evaluation value regarding the execution status of the current task based on a target value preset in conjunction with the workpiece; and a determination unit configured to determine, based on the evaluation value, whether an action position should be changed from a current position or not, where the action position is a position at which the robot acts on the workpiece in the current task, wherein the robot control unit is configured such that, in a case where it is determined that the action position should be changed from the current position, it causes the robot to change the action position from the current position to a new position and continue the current task. [15] Robot control system according to any one of claims 1 to 14, further comprising a planning unit configured to plan a next task following the current task based on a planning model and image data, wherein the image data specifies the workpiece being processed by the robot in the real workspace, and wherein the planning model is trained to output a plan for the next task in response to the input image data, wherein the robot control unit is configured to control the robot according to a result of the planning by the planning unit in such a way that the current task is completed. [16] Robot control system according to claim 6, further comprising a training unit configured to update the control model by machine learning using training data comprising a combination of the current setpoint and the adjusted next setpoint. [17] Robot control system according to claim 16, further comprising a data generation unit configured to generate the training data, wherein the simulation unit is configured to generate a predicted image that displays the predicted state of the workpiece based on a state prediction model and the next setpoint, wherein the state prediction model is trained to generate the predicted image based on a movement of the robot working with the next setpoint and a context relating to an element that forms the workspace, where the data generation unit is configured such that it: The predicted image is modified based on change information to change a scene that displays the predicted state, and a training image is generated that displays a different state that differs from the predicted state; the training data is generated, comprising a combination of the current setpoint, the adjusted next setpoint, and the training image, and wherein the training unit is configured to update the control model or generate a different control model for the initial setting of the next setpoint by performing machine learning using the training data, which further comprises the training image. [18] Robot control method executable by a robot control system having at least one processor, the method comprising: Initial setting of the next setpoint in a current task for a robot located in a real workspace, and execution of the current task to process a workpiece; Virtual execution of the current task, in which the robot works with the next setpoint to process the workpiece, through simulation; Adjusting the next setpoint based on a prediction result obtained through simulation; and controlling the robot in the real workspace based on the adjusted next setpoint. [19] Robot control program to cause a computer to perform the following: Initial setting of the next setpoint in a current task for a robot arranged in a real workspace and execution of the current task to process a workpiece; Virtual execution, through simulation, of the current task, in which the robot works with the next setpoint to process the workpiece; Adjusting the next setpoint based on a prediction result obtained through the simulation; and Controlling the robot in the real workspace based on the adjusted next setpoint.

Images