Information processing device, robot controller, information processing system, and information processing method

Abstract

To provide an information processing device, a robot controller, an information processing system, and an information processing method capable of improving efficiency when a robot is used.SOLUTION: An information processing device includes a control unit that estimates a first control program to be executed by a first robot, from at least one control program recorded in a storage unit and including a processing pattern of a robot. The control unit acquires first environment information indicating at least a part of a work environment of the first robot, and estimates the first control program by estimating at least one candidate of the first processing pattern to be executed by the first robot, on the basis of the first environment information.SELECTED DRAWING: Figure 1

Description

Cross-reference to related applications

[0001] This application claims priority to Japanese Patent Application No. 2021-158478 (filed September 28, 2021), the entire disclosure of which is incorporated herein by reference. [Technical Field]

[0002] The present disclosure relates to an information processing device, a robot controller, an information processing system, and an information processing method. [Background technology]

[0003] Conventionally, there is known a device that causes a robot to execute a group of tasks even if instructions from a user are insufficient (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-30407 Summary of the Invention

[0005] An information processing device according to an embodiment of the present disclosure includes a control unit that estimates a first control program to be executed by a first robot from at least one control program including a robot processing pattern stored in a storage unit. The control unit estimates the first control program by acquiring first environmental information that indicates at least a part of a working environment of the first robot, and estimating at least one candidate first processing pattern to be executed by the first robot based on the first environmental information.

[0006] A robot controller according to an embodiment of the present disclosure controls the first robot based on the first control program output from the information processing device.

[0007] An information processing system according to an embodiment of the present disclosure includes the information processing device, and a database connected to the information processing device and recording a control program for executing the processing pattern.

[0008] An information processing method according to an embodiment of the present disclosure is executed by an information processing device that estimates a first control program to be executed by a first robot from at least one control program recorded in the information processing device and including a robot processing pattern. The information processing method includes the information processing device acquiring first environmental information that indicates at least a portion of a work environment of the first robot. The information processing method includes the information processing device estimating the first control program by estimating at least one candidate first processing pattern to be executed by the first robot based on the first environmental information. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of the configuration of a robot control system according to an embodiment. [Figure 2] 1 is a block diagram illustrating an example of the configuration of a robot control system according to an embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of a work environment image. [Figure 4] 10 is a table showing an example of attribute information of objects located in a work environment. [Figure 5] 10 is a table showing an example of program candidates extracted based on a work environment image. [Figure 6] 10 is a flowchart showing an example of a procedure for associating a control program with previously recorded information. [Figure 7] 1 is a flowchart illustrating an example of a procedure of an information processing method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] (Overview of robot control system 100) 1 and 2, a robot control system 100 according to one embodiment includes a robot 1, a robot controller 2, an information acquisition device 3, a terminal device 9, and an information processing device 10. The robot control system 100 causes the robot 1 to perform a task on an object placed on a workbench 5 located in a work environment 4. Specifically, for example, the robot 1 can move a workpiece 8 stored in a first tray 6 to a second tray 7.

[0011] The program for controlling the movement of the robot 1 may be created, for example, by a user while actually operating the robot 1, and may be stored in the robot controller 2 or the information processing device 10. The program for controlling the movement of the robot 1 is also referred to as a control program. The control program may be created using a robot simulator or the like. In this case, for example, the control program may be created without being connected to a network (offline).

[0012] The robot 1 performs a task by operating in accordance with a predetermined processing pattern. The processing pattern is expressed, for example, as a combination of actions for the robot 1 to accomplish a task instructed by the robot 1. For example, the processing pattern can be expressed as a combination of an action to grasp the workpiece 8, an action to transport the grasped workpiece 8, and an action to place the transported workpiece 8 in a predetermined position. A task is a concept that defines a goal to be achieved by executing a series of actions included in the processing pattern. For example, the task may be a concept that defines a goal of transporting the workpiece 8 from point A to point B. For example, the task may be a concept that defines a goal of placing the workpiece 8 at point C. For example, the task may be a concept that defines a goal of combining the workpiece 8 with another object. The processing pattern can also be considered a combination of actions for the robot 1 to complete a task. The control program can also be considered a set of instructions that causes the robot 1 to execute a combination of actions represented by the processing pattern so that the robot 1 can complete the task. In other words, the control program can also be considered software in which a processing pattern for completing a predetermined task is programmed. The plurality of control programs may be configured to include control programs in which different operations to be performed on the same type of workpiece 8 are respectively recorded, or may be configured to include control programs in which the same type of operations to be performed on different types of workpiece 8 are respectively recorded. Also, one type of operation for one type of workpiece 8 may be programmed in one control program.

[0013] When the robot 1 is to be made to perform one task, one control program corresponding to the task to be performed is prepared in advance. When the robot 1 is to be made to perform multiple tasks, multiple control programs corresponding to the respective tasks are prepared in advance. The user selects a control program corresponding to the task that the robot 1 is to be made to perform and instructs the robot controller 2 to make the robot 1 perform the task.

[0014] If the number of prepared control programs increases, the burden on the user of selecting a control program may increase. For example, the user may spend a lot of time searching for a control program corresponding to the task that the user wants the robot 1 to perform. Furthermore, the user may select the wrong control program.

[0015] As a comparative example, a robot controller can estimate the workpieces present in the work environment based on images of the workpieces in the work environment, and estimate the work to be performed by the robot on the workpieces present in the work environment. However, with the work estimation method according to the comparative example, when multiple different tasks can be performed on the same workpiece, it is difficult for the robot controller 2 to narrow down the task to one. Furthermore, if a user places an incorrect workpiece in the work environment, the task to be performed by the robot is estimated based on the incorrectly placed workpiece. In this case, a task different from the one intended by the user may be performed without the user having a chance to notice the error.

[0016] The information processing device 10 of the robot control system 100 according to an embodiment of the present disclosure acquires information about the work environment 4 using the information acquisition device 3. The information processing device 10 of the robot control system 100 compares the information about the work environment 4 with information associated with a control program stored in the information processing device 10 or the robot controller 2. The information processing device 10 of the robot control system 100 extracts, from among a plurality of control programs, control programs whose information associated with the control program is similar to the information about the work environment 4, as candidates for control programs to be executed by the robot 1. This reduces the burden on the user of selecting a control program. It also reduces the possibility that the user will select the wrong control program.

[0017] Furthermore, the information processing device 10 of the robot control system 100 of the present disclosure determines the suitability of at least one candidate processing pattern to be executed by the robot 1 to be controlled from at least one processing pattern of the robot 1 recorded in the storage unit 12 of the information processing device 10. By doing so, even when control programs for executing similar tasks, such as stacking the workpieces 8 in bulk and arranging the workpieces 8 in an aligning orientation when transporting the workpieces 8 to a target location, are stored, the accuracy of selecting the control program can be improved.

[0018] The robot 1 to be controlled is also referred to simply as the first robot for the sake of distinction. Also, a processing pattern representing an operation to be performed by the first robot is also referred to simply as the first processing pattern for the sake of distinction. Also, a control program executed by the robot controller 2 to cause the first robot to perform the operation represented by the first processing pattern is also referred to simply as the first control program for the sake of distinction.

[0019] (Configuration example of robot control system 100) As shown in FIGS. 1 and 2 , a robot control system 100 according to one embodiment includes a robot 1, a robot controller 2, an information acquisition device 3, a terminal device 9, and an information processing device 10. At least one component of the robot control system 100 may be communicatively connected via a network 80, or may be communicatively connected without using the network 80. At least one component of the robot control system 100 may be communicatively connected via a wired or wireless connection. At least one component of the robot control system 100 may be communicatively connected via a dedicated line. At least one component of the robot control system 100 may be communicatively connected to each other in various other forms, without being limited to these examples. Each component of the robot control system 100 will be described in detail below.

[0020] <Information processing device 10> The information processing device 10 includes a control unit 11 and a storage unit 12. The information processing device 10 is communicably connected to other components of the robot control system 100 via a network 80 or directly without via the network 80.

[0021] The control unit 11 may be configured to include at least one processor to realize various functions of the information processing device 10. The processor may execute programs that realize various functions of the information processing device 10. The processor may be realized as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). The processor may be realized as multiple integrated circuits and discrete circuits that are connected to each other in a communicative manner. The processor may be configured to include a CPU (Central Processing Unit). The processor may be configured to include a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit). The processor may be realized based on various other known technologies.

[0022] The information processing device 10 further includes a storage unit 12. The storage unit 12 may include an electromagnetic storage medium such as a magnetic disk, or may include a memory such as a semiconductor memory or a magnetic memory. The storage unit 12 may be configured as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage unit 12 stores various information and programs executed by the control unit 11. The storage unit 12 may function as a work memory for the control unit 11. At least a part of the storage unit 12 may be included in the control unit 11. At least a part of the storage unit 12 may be configured as a storage device separate from the information processing device 10.

[0023] The information processing device 10 may be configured to include a communication device configured to be able to communicate by wire or wirelessly. The communication device may be configured to be able to communicate by a communication method based on various communication standards.

[0024] The information processing device 10 may be configured to include one or more servers. The information processing device 10 may be configured to cause multiple servers to execute parallel processing. The information processing device 10 does not need to be configured to include a physical housing, and may be configured based on virtualization technology such as a virtual machine or a container orchestration system. The information processing device 10 may be configured using cloud services. When the information processing device 10 is configured using cloud services, it may be configured by combining managed services. In other words, the functions of the information processing device 10 may be realized as cloud services.

[0025] The information processing device 10 may include at least one server group and at least one database group. The server group functions as a control unit 11. The database group functions as a storage unit 12. The number of server groups may be one or two or more. When there is one server group, the functions realized by one server group include the functions realized by each server group. The server groups are connected to each other so as to be able to communicate with each other via wired or wireless communication. The number of database groups may be one or two or more. The number of database groups may be increased or decreased as appropriate based on the amount of data managed by the information processing device 10 and the availability requirements required of the information processing device 10. The database groups are connected to each server group so as to be able to communicate with each other via wired or wireless communication. The information processing device 10 may be connected to an external database. An information processing system may be configured that includes the information processing device 10 and an external database.

[0026] Although the information processing device 10 is depicted as a single configuration in FIGS. 1 and 2, multiple configurations can be operated as a single system as needed. In other words, the information processing device 10 is configured as a platform with variable capacity. By using multiple configurations as the information processing device 10, even if one configuration becomes inoperable due to an unforeseen event such as a natural disaster, the system can continue to operate using the other configurations. In this case, each of the multiple configurations is connected by a line, whether wired or wireless, and is configured to be able to communicate with each other. The multiple configurations may be configured across a cloud environment and an on-premises environment.

[0027] Furthermore, the information processing device 10 is communicably connected to at least one component of the robot control system 100 via a line, whether wired or wireless. The information processing device 10 and at least one component of the robot control system 100 are provided with an interface that uses a mutually standard protocol, enabling two-way communication.

[0028] <Terminal Device 9> The terminal device 9 is communicatively connected to at least one of the robot controller 2 and the information processing device 10 of the robot control system 100. The terminal device 9 may also be communicatively connected to other components of the robot control system 100. The terminal device 9 and at least one component of the robot control system 100 are communicatively connected via the network 80 or directly without via the network 80.

[0029] The terminal device 9 may be configured to include at least one processor. The processor of the terminal device 9 may be configured identically or similarly to the processor of the control unit 11 of the information processing device 10. The terminal device 9 may be configured to include a storage device. The storage device of the terminal device 9 may be configured identically or similarly to the storage unit 12 of the information processing device 10. The terminal device 9 may be configured to include a communication device. The communication device of the terminal device 9 may be configured identically or similarly to the communication device of the information processing device 10.

[0030] The terminal device 9 may include an input device. The input device may include, for example, a touch panel or a touch sensor, or a pointing device such as a mouse. The input device may include physical keys. The input device may include an audio input device such as a microphone. The input device is not limited to these examples and may include various other devices.

[0031] The terminal device 9 may include an output device. The output device may include a display device. The display device may include, for example, a liquid crystal display (LCD), an organic electroluminescence (EL) display, an inorganic EL display, or a plasma display panel (PDP). The display device is not limited to these displays and may include various other display types. The display device may include a light-emitting device such as an LED (light-emitting diode). The display device may include various other devices. The output device may include an audio output device such as a speaker that outputs auditory information such as sound. The output device is not limited to these examples and may include various other devices.

[0032] The number of terminal devices 9 included in the robot control system 100 is not limited to one, and may be two or more. When the robot control system 100 includes multiple terminal devices 9, each terminal device 9 may receive input from a user. The terminal device 9 may be configured as a tablet terminal. The terminal device 9 may be configured as a mobile phone terminal such as a feature phone or a smartphone. The terminal device 9 may be configured as a PC terminal such as a desktop PC (Personal Computer) or a notebook PC. The terminal device 9 is not limited to these examples, and may be configured as various devices that can provide a GUI (Graphical User Interface) and communication functions.

[0033] The terminal device 9 may be used by a user to store a control program in advance in the information processing device 10. The terminal device 9 may also be used to monitor the state of the robot 1. The terminal device 9 is not limited to these examples and may provide various other functions. The terminal device 9 may be provided alongside the robot controller 2. When the terminal device 9 is provided alongside the robot controller 2, the robot controller 2 itself may have an input device or an output device. The robot controller 2 may be included in the terminal device 9.

[0034] <Robot Controller 2> The robot controller 2 downloads a control program from the information processing device 10. By executing the downloaded control program, the robot controller 2 outputs information for controlling the operation of the robot 1 to the robot 1 and causes the robot 1 to perform the task specified in the control program. The robot controller 2 may execute a control program stored in the robot controller 2 itself. As illustrated in FIG. 1 , the robot controller 2 may cause the robot 1 to perform the task of moving a workpiece 8 from a first tray 6 to a second tray 7. The robot controller 2 may cause the robot 1 to perform various tasks, without being limited to this. The robot controller 2 may or may not be connected to a cloud computing environment. When the robot controller 2 is not connected to a cloud computing environment, the operation of the robot controller 2 is completed in an on-premises environment. When the operation of the robot controller 2 is completed in an on-premises environment, the operation of the information processing device 10 is executed by the robot controller 2.

[0035] The robot controller 2 may include a communication device that downloads the control program from the information processing device 10. The communication device of the robot controller 2 may be configured the same as or similar to the communication device of the information processing device 10. The robot controller 2 may include a processor that generates information for controlling the operation of the robot 1 by executing the control program. The processor of the robot controller 2 may be configured the same as or similar to the processor of the control unit 11 of the information processing device 10.

[0036] In the configuration illustrated in FIG. 1, one robot controller 2 is connected to one robot 1. One robot controller 2 may be connected to two or more robots 1. One robot controller 2 may control only one robot 1, or two or more robots 1. The number of robot controllers 2 and robots 1 is not limited to two, and may be one, or three or more. Furthermore, the robot controller 2 may be integrated with the information processing device 10, and the function of the robot controller 2 may be realized as one function of the information processing device 10.

[0037] <Robot 1> The robot 1 may be configured as a robot arm having an arm. The arm may be configured as, for example, a 6-axis or 7-axis vertical articulated robot. The arm may be configured as a 3-axis or 4-axis horizontal articulated robot or a SCARA robot. The arm may be configured as a 2-axis or 3-axis Cartesian robot. The arm may be configured as a parallel link robot or the like. The number of axes configuring the arm is not limited to those exemplified.

[0038] The robot 1 may be equipped with an end effector attached to the arm. The end effector may include, for example, a gripping hand configured to grip a workpiece. The gripping hand may have multiple fingers. The gripping hand may have two or more fingers. The fingers of the gripping hand may have one or more joints. The end effector may include a suction hand configured to pick up a workpiece by suction. The end effector may include a scooping hand configured to scoop up a workpiece. The end effector may include a tool such as a drill and be configured to perform various processes, such as drilling a hole in a workpiece. The end effector is not limited to these examples and may be configured to perform various other operations.

[0039] The robot 1 can control the position of the end effector by operating the arm. The end effector may have an axis that serves as a reference for the direction in which it acts on a workpiece. If the end effector has an axis, the robot 1 can control the direction of the axis of the end effector by operating the arm. The robot 1 controls the start and end of the operation of the end effector acting on the workpiece. The robot 1 can move or process a workpiece by controlling the operation of the end effector while controlling the position of the end effector or the direction of the axis of the end effector.

[0040] The robot 1 may be configured as an automated guided vehicle (AGV). The robot 1 may be configured as a drone. The robot 1 is not limited to a robot arm or an AGV, and may be configured in various other forms such as a vehicle, an electronic device, or a controlled machine.

[0041] The robot 1 may further include a sensor that detects the state of at least one component of the robot 1. The sensor may detect information regarding the actual position or posture of at least one component of the robot 1, or the velocity or acceleration of at least one component of the robot 1. The sensor may detect a force acting on at least one component of the robot 1. The sensor may detect a current flowing through a motor that drives at least one component of the robot 1, or a torque of the motor. The sensor can detect information obtained as a result of the actual operation of the robot 1. The robot controller 2 can grasp the result of the actual operation of the robot 1 by obtaining the detection results of the sensor.

[0042] Unlike devices that perform predetermined tasks on products and process different types of products by setting conditions according to the type of product, the robot 1 is capable of generating a control program tailored to the task, and by generating a control program, it can operate to perform various tasks.

[0043] <Information acquisition device 3> The information acquisition device 3 acquires information about at least a portion of the work environment 4 of the robot 1. The information acquisition device 3 may include a camera that captures images of at least a portion of the work environment 4. The information acquisition device 3 may include a sensor that measures the position, shape, or size of an object present in at least a portion of the work environment 4. The information acquisition device 3 may include a 3D sensor, a distance sensor, or the like. The information acquisition device 3 may include a sensor that measures the temperature, humidity, or the like of at least a portion of the work environment 4. If at least a portion of the work environment 4 is a clean room, the information acquisition device 3 may include a sensor that measures particle density within the clean room. The information about at least a portion of the work environment 4 may include classification information for the clean room. The information acquisition device 3 may include a sensor that measures noise levels in at least a portion of the work environment 4. The information acquisition device 3 may also include a sensor, such as a current sensor, that measures current, etc., indicating the operation of peripheral equipment. The information acquisition device 3 may be fixed in a position where it can acquire information about at least a portion of the work environment 4 of the robot 1. The information acquisition device 3 may be attached to a robot arm or an end effector. The number of information acquisition devices 3 is not limited to one and may be two or more. The multiple information acquisition devices 3 may be communicatively connected to each other. At least one information acquisition device 3 among the multiple information acquisition devices 3 may be communicatively connected to the robot controller 2 or the information processing device 10. The information acquisition device 3 may include multiple information acquisition units that are physically separated from each other. The multiple information acquisition units may be communicatively connected to each other.

[0044] The information acquired by the information acquisition device 3 as information on at least a portion of the work environment 4 may include information on the work range of the robot 1, such as the range where the work table 5 and objects placed on the work table 5 are located, as illustrated in FIG. 1 . The information acquired by the information acquisition device 3 as information on at least a portion of the work environment 4 may include information other than the work target object itself or the robot 1 itself. Furthermore, the information acquired by the information acquisition device 3 as information on at least a portion of the work environment 4 is not limited to these pieces of information and may include information outside the work range of the robot 1. Specifically, the information acquired by the information acquisition device 3 as information on at least a portion of the work environment 4 may include information on the range where peripheral devices are located. The peripheral devices may include a parts feeder on which the workpiece 8 is placed, or a tray changer on which the first tray 6 or the second tray 7 is placed, etc. The peripheral devices may include a workpiece inverting device that inverts the workpiece 8. The peripheral devices may include a tool changer that replaces the end effector of the robot 1. The peripheral devices may include manufacturing equipment such as a polishing machine that processes the workpiece 8. The peripheral devices are not limited to these and may include various devices. The information acquired by the information acquisition device 3 as information on at least a portion of the work environment 4 may include information on an area where the workpiece 8 is not located. The information acquired by the information acquisition device 3 as information on at least a portion of the work environment 4 may include attribute information of the end effector of the robot 1 or attribute information of the arm. The attribute information of the end effector may include information specifying whether the end effector is a hand or a suction cup, etc. The attribute information of the arm may include information specifying the number of joints or the range of motion of the arm, or the length of the arm.

[0045] The range in which the information acquisition device 3 acquires information may be the operating range of the robot 1. Furthermore, the range in which the information acquisition device 3 acquires information is not limited to the operating range of the robot 1, but may include a range outside the operating range of the robot 1 or a range surrounding the operating range. The information acquisition device 3 may acquire information about the range outside the operating range of the robot 1 or the range surrounding the operating range as information about the work environment 4. The range surrounding the operating range may be within a predetermined distance from the operating range. The predetermined distance may be set as the distance between two points, such as within the same factory building, the same room, or the same production line. The peripheral equipment may include a belt conveyor that transports workpieces 8, etc., toward the operating range of the robot 1. The robot control system 100 may select a control program based on the state of the workpieces 8 on the belt conveyor located outside the operating range of the robot 1.

[0046] (Operation example of robot control system 100) The robot control system 100 may store a control program (processing pattern) in the information processing device 10 in association with a program number, a program name, or a task name. The robot control system 100 associates a program number, etc. with at least a portion of information about the work environment 4 in which the control program is executed. That is, a control program is associated with at least a portion of information about the work environment 4 in which the control program is executed. At least a portion of information about the work environment 4 prepared by the user is also referred to as preparation environment information or first environment information. At least a portion of information about the work environment 4 associated with a control program is at least a portion of information about the work environment 4 when the control program was previously recorded, and is also referred to as past record information or second environment information. The past record information or second environment information corresponds to at least a portion of information about the work environment 4 when the robot 1 performed a task by executing the control program with the robot controller 2. The past record information or second environment information may include at least a portion of information about the work environment 4 registered by the user. In the robot control system 100, the terminal device 9 may be configured to receive an input from the user to register at least a portion of information about the work environment 4. That is, the robot control system 100 may associate information simulated by a user with the number of a control program (processing pattern), etc. A user who can register the past record information or second environmental information of the first robot may be, for example, a user who has had the first robot perform a task in the past, or may be a user of another robot. A user who can register the past record information or second environmental information of the first robot may be a manager of a factory, etc., or a worker, etc.

[0047] The previously recorded information may include information that can be compared with the preparatory environment information. That is, the previously recorded information may include information that corresponds to the preparatory environment information. For example, if the preparatory environment information includes an image, the previously recorded information may also include an image, and if the preparatory environment information includes numerical data such as a sensor output, the previously recorded information may also include numerical data. In other words, the preparatory environment information may include information that can be compared with the previously recorded information.

[0048] The previously recorded information may include, for example, an image of the work environment 4 taken when the control program is executed. The previously recorded information may include attribute information of objects present in the work environment 4 obtained by analyzing the image of the work environment 4. The attribute information of an object may include information on the appearance of the object, such as the shape of the object, such as the outline of the object, the color or texture of the object, or the size of the object. The attribute information of an object may include two-dimensional information captured in the image of the work environment 4, or three-dimensional information based on depth information of the work environment 4. The attribute information is not limited to information on the appearance of the object, but may also include information on the material or density of the object. The previously recorded information may include position information, such as the coordinates of an object present in the work environment 4. The position information may represent a relative position from a reference position set in the work environment 4. The position information may represent an absolute position set in the information acquisition device 3. The position information may represent the positional relationship of multiple objects present in the robot 1. The position information may represent the positional relationship of multiple objects present in the work environment 4. The position information may represent the positional relationship between the workpiece 8 and an object present in the vicinity of the workpiece 8. The past record information may include the type of camera that captured the work environment 4, or the capturing conditions such as the position or orientation of the camera.

[0049] As described above, the robot control system 100 associates a control program associated with a program number or a program name with past record information of the control program and stores the control program in the information processing device 10. Specifically, the robot control system 100 may store the following information in the information processing device 10: (1) Program number (2) Control program (processing pattern) (3) Image of working environment 4 (4) Attribute information and location information of objects recognized from images of the work environment 4

[0050] 3 shows an example of an image captured of a work environment 4. A work table 5 is located in the work environment 4. A first tray 6 and a second tray 7 are located on the work table 5. Furthermore, a workpiece 8 is stored in the first tray 6.

[0051] Figure 4 shows an example of attribute information and position information of objects recognized from an image captured of the work environment 4. A total of 11 objects were recognized from the image. There are four types of objects. To identify the objects by type, each type is assigned an object ID numbered 1 to 4. The workbench 5 is associated with number 1. The second tray 7 is associated with number 2. The first tray 6 is associated with number 3. The workpiece 8 is associated with number 4. There is one workbench 5, one first tray 6, and one second tray 7 each. There are eight workpieces 8.

[0052] The color information of each object is expressed as average color information obtained by averaging the color information of each part of the object. The color information is expressed as a numerical value corresponding to the gradation of each RGB (Red Green Blue) component of the image. The average color information of the workbench 5 (object ID = 1) is expressed as (R1, G1, B1). The average color information of the second tray 7 (object ID = 2) is expressed as (R2, G2, B2). The average color information of the first tray 6 (object ID = 3) is expressed as (R3, G3, B3). The average color information of the workpiece 8 (object ID = 4) is expressed as (R4, G4, B4).

[0053] The contours and texture of each object are represented by its image.

[0054] In this embodiment, the size of each object is expressed by its width (W) x depth (D) x height (H). The size of the workbench 5 (object ID = 1) is expressed as W1 x D1 x H1. The size of the second tray 7 (object ID = 2) is expressed as W2 x D2 x H2. The size of the first tray 6 (object ID = 3) is expressed as W3 x D3 x H3. The size of the workpiece 8 (object ID = 4) is expressed as W4 x D4 x H4.

[0055] The coordinates of each object may be expressed by relative coordinates with respect to the coordinates of a reference position set in the work environment 4, or by absolute coordinates (for example, pixel positions in a captured image) set in the information acquisition device 3. In this embodiment, the coordinates are expressed in XYZ coordinates. The coordinates of the work table 5 (object ID=1) are expressed as (X1, Y1, Z1). The coordinates of the second tray 7 (object ID=2) are expressed as (X2, Y2, Z2). The coordinates of the first tray 6 (object ID=3) are expressed as (X3, Y3, Z3). The coordinates of the workpiece 8 (object ID=4) are expressed as (X4, Y4, Z4).

[0056] <Associating with past record information when a control program is executed> The robot control system 100 stores a control program in the information processing device 10 in advance. If the control program is not yet associated with previously recorded information when it is executed, the robot control system 100 may acquire information about the work environment 4 when the control program is executed. The robot control system 100 may associate the acquired information about the work environment 4 with the control program as previously recorded information. For example, when the control program is executed for the first time, the robot control system 100 may acquire information about the work environment 4 as previously recorded information and associate it with the control program. It can also be said that the robot control system 100 associates previously recorded information with the task corresponding to the control program. The previously recorded information may include at least a portion of the information about the work environment 4 registered by the user. In the robot control system 100, the terminal device 9 may be configured to accept an input from the user to register at least a portion of the information about the work environment 4. In other words, the robot control system 100 may associate information simulated by the user with the number of the robot control program (processing pattern), etc. The user who can register the past record information of the first robot may be, for example, a user who has had the first robot perform a task in the past, or may be a user of another robot. The user who can register the past record information of the first robot may be a manager of a factory or the like, or a worker, etc.

[0057] Specifically, the control program can be stored in the information processing device 10 in association with the previously recorded information in the following procedure.

[0058] In the robot control system 100, the workpieces 8 and other items required for the robot 1 to execute a task are prepared as shown in FIG. 3. For example, the user places the first tray 6 and the second tray 7 in the work environment 4. The user places the workpieces 8 in the first tray 6. In other words, the workpieces 8 and the first tray 6 and the second tray 7 may be placed by the user. As another example, the workpieces 8 and the first tray 6 and the second tray 7 may be placed by a peripheral device.

[0059] After preparing a workpiece 8 and the like in the work environment 4, the user selects a control program to be executed by the robot controller 2 using the terminal device 9. The terminal device 9 outputs the selected control program to the robot controller 2. The robot controller 2 controls the robot 1 by executing the acquired control program, and causes the robot 1 to perform a task.

[0060] When the robot 1 performs a task, the robot 1 or the robot controller 2 acquires information about the work environment 4 using the information acquisition device 3. The information acquisition device 3 may, for example, photograph the work environment 4 and acquire the photographed image of the work environment 4 as information about the work environment 4. If the information acquisition device 3 is attached to the arm or end effector of the robot 1, the robot 1 may move the information acquisition device 3 to a predetermined photographing position and cause the information acquisition device 3 to photograph the work environment 4. The information acquisition device 3 may acquire not only photographed images of the work environment 4 but also various other information such as depth information of the work environment 4 as information about the work environment 4.

[0061] The information acquisition device 3 outputs the acquired information of the work environment 4 to the information processing device 10. The control unit 11 of the information processing device 10 recognizes objects such as the workpiece 8 present in the work environment 4 by analyzing the information of the work environment 4. The control unit 11 may recognize each of the workpiece 8, the first tray 6, and the second tray 7, for example, by image analysis of a captured image of the work environment 4. The control unit 11 may recognize each of the workpiece 8, the first tray 6, and the second tray 7, for example, by analyzing depth information of the work environment 4.

[0062] The control unit 11 acquires attribute information and position information of each object recognized from information on the work environment 4. The control unit 11 associates the attribute information and position information of each object as previously recorded information with a control program stored in the storage unit 12. As described above, the control program is stored in the information processing device 10 in association with the previously recorded information.

[0063] <Execution of control programs associated with past record information> By associating the control program with the past record information, the robot control system 100 can easily select, by the following procedure, a control program to be used to make the robot 1 perform a task in the prepared work environment 4. It can also be said that the robot control system 100 selects a processing pattern corresponding to the control program.

[0064] In the robot control system 100, the workpieces 8 and other items required for the robot 1 to execute a task are prepared as shown in FIG. 3. For example, the user places the first tray 6 and the second tray 7 in the work environment 4. The user places the workpieces 8 in the first tray 6. In other words, the workpieces 8 and the first tray 6 and the second tray 7 may be placed by the user. As another example, the workpieces 8 and the first tray 6 and the second tray 7 may be placed by a peripheral device.

[0065] After preparing a workpiece 8 and the like in the work environment 4, the user uses the terminal device 9 to search for a control program to be executed by the robot controller 2. The robot control system 100 acquires information about the work environment 4 using the information acquisition device 3 based on a search instruction for the control program input to the terminal device 9. When the information acquisition device 3 is attached to the arm or end effector of the robot 1, the robot controller 2 controls the operation of the robot 1 to move the information acquisition device 3 to a predetermined position or orient it in a predetermined direction in order to acquire information about the work environment 4.

[0066] The information acquisition device 3 outputs the acquired information of the work environment 4 to the information processing device 10. The control unit 11 of the information processing device 10 recognizes objects such as the workpiece 8 present in the work environment 4 by analyzing the information of the work environment 4. The control unit 11 may recognize each of the workpiece 8, the first tray 6, and the second tray 7, for example, by image analysis of a captured image of the work environment 4. The control unit 11 may recognize each of the workpiece 8, the first tray 6, and the second tray 7, for example, by analyzing depth information of the work environment 4.

[0067] The control unit 11 acquires attribute information and position information of each object recognized from the information of the work environment 4. The control unit 11 compares the attribute information and position information of each object with previously recorded information associated with the control program stored in the memory unit 12 of the information processing device 10. The control unit 11 extracts a control program that matches the information of the work environment 4 and presents it to the user via the terminal device 9. It can also be said that the control unit 11 extracts a processing pattern that matches the information of the work environment 4. The user inputs into the terminal device 9 whether to execute or reject the presented control program. If multiple control programs are presented, the user may input into the terminal device 9 an instruction to select a control program to execute. When a control program to execute is selected, the terminal device 9 outputs information identifying the selected control program to the robot controller 2. The robot controller 2 executes the selected control program and causes the robot 1 to perform a task.

[0068] <<Control Program Extraction Based on Similarity>> The control unit 11 may calculate the similarity between the information of the work environment 4 and the previously recorded information in order to extract a control program or processing pattern that matches the information of the work environment 4. The similarity is calculated as 100% when the two pieces of information completely match, and as 0% when the two pieces of information completely mismatch. The control unit 11 may extract a control program or processing pattern with a high similarity as a candidate and output it to the terminal device 9 to present it to the user. The control unit 11 may extract one or more control programs or processing patterns with a similarity equal to or greater than a predetermined value as candidates and output it to the terminal device 9 to present it to the user. The control unit 11 may extract at least one candidate for the first processing pattern that has a predetermined similarity and output it to the terminal device 9 to present it to the user. The control unit 11 may present to the user a list of control programs arranged in descending order of similarity, as shown in FIG. 5, for example. The control program with a higher similarity is displayed higher in the candidate ranking.

[0069] The control unit 11 may calculate the similarity of attribute information such as the shape or color of objects present in the work environment 4 as the similarity. Specifically, the control unit 11 compares the attribute information of each object recognized from the information of the work environment 4 with the attribute information included in the previously recorded information associated with the control program stored in the information processing device 10. The control unit 11 may calculate the similarity as a numerical value using template matching or a trained model generated by machine learning or deep learning. The control unit 11 calculates the similarity for each type of recognized object. The control unit 11 calculates the similarity for all types of recognized objects. The objects recognized in the work environment 4 illustrated in FIG. 3 are divided into four types. The control unit 11 calculates the similarity for each of the four types. The control unit 11 calculates the average value and standard deviation of the four similarities calculated for each type. If the average value is equal to or greater than a predetermined average value determination threshold and the standard deviation is equal to or less than a predetermined standard deviation determination threshold, the control unit 11 extracts the control program associated with the previously recorded information for which the similarity is to be calculated. The average value determination threshold may be set to, for example, 60. The standard deviation determination threshold may be set to, for example, 10. The trained model can be generated by machine learning using data relating to multiple pieces of previously recorded information as a training dataset.

[0070] The control unit 11 may calculate the similarity between the positions of objects present in the work environment 4 as the similarity. Specifically, the control unit 11 calculates the distance between each object based on the position information of each object recognized from the information of the work environment 4. The position information of each object includes the coordinates of the center of gravity of each object. The position information of each object may also include the coordinates of the edge of each object (e.g., the minimum or maximum X, Y, and Z coordinates of the range in which each object exists). The control unit 11 calculates the distance between each object based on the coordinates of the center of gravity of each object. In the work environment 4 illustrated in FIG. 3, the control unit 11 calculates the distances between the work table 5 (object ID = 1), the second tray 7 (object ID = 2), the first tray 6 (object ID = 3), and the workpiece 8 (object ID = 4). In the example of FIG. 3, there are eight objects classified as the same type of workpiece 8 (object ID = 4). The control unit 11 considers multiple objects classified as the same type as a single object, calculates the coordinates of the center of gravity, and calculates the distance between each object. If there are four types of objects in the working environment 4, there are six combinations for calculating the distance between each object. Therefore, six distances between each object are calculated.

[0071] The control unit 11 calculates the distance between each object included in the previously recorded information based on the position information included in the previously recorded information associated with the control program stored in the information processing device 10. The control unit 11 may calculate the distance between each object for previously recorded information including four types of objects. The control unit 11 may also calculate the distance between each object for previously recorded information including objects corresponding to objects present in the work environment 4. In other words, the control unit 11 may extract previously recorded information that matches the attribute information of objects located in the work environment 4 in advance, and calculate the distance between each object only for the extracted previously recorded information.

[0072] The control unit 11 compares the distances between the objects recognized from the information of the work environment 4 with the distances between the objects in the previously recorded information associated with the control program. The control unit 11 selects two objects from the objects recognized from the information of the work environment 4. The control unit 11 also selects two objects from the objects included in the previously recorded information that correspond to the two objects selected from the objects recognized from the information of the work environment 4. The control unit 11 calculates the absolute value of the difference between the distances between the two objects selected from the objects recognized from the information of the work environment 4 and the distances between the two objects selected from the objects included in the previously recorded information. For each of the distances calculated for the six combinations in the example of FIG. 3, the control unit 11 calculates the absolute value of the difference between the distances between the two objects included in the information of the work environment 4 and the distances between the two objects included in the previously recorded information. The control unit 11 calculates the average value and standard deviation of the absolute values ​​of the distance differences calculated for all combinations as the position similarity. The control unit 11 may assign higher priorities to the control programs associated with the previously recorded information for which the position similarity is to be calculated in ascending order of the average value or standard deviation. The control unit 11 may assign higher priorities to the control programs associated with the previously recorded information for which the position similarity is to be calculated in descending order of the average value.

[0073] The control unit 11 displays the extracted control programs on the terminal device 9. As shown in FIG. 5, the control unit 11 may prioritize the extracted control programs and display them on the terminal device 9. The user selects one control program from the control programs displayed by the terminal device 9. The terminal device 9 outputs information identifying the control program selected by the user's selection input to the robot controller 2. The robot controller 2 executes the selected control program and causes the robot 1 to perform a task.

[0074] The type of object or the arrangement of the object in the work environment 4 may not match any of the previously recorded information associated with the control program. A mispreparation of the work environment 4 may result in the information in the work environment 4 not matching the previously recorded information. Mispreparation of the work environment 4 may include, for example, the user forgetting to place a workpiece 8 or other object necessary for the robot 1's work or placing an extra object not used in the robot 1's work. The control unit 11 may display an alarm indicating that the work environment 4 is incompatible on the terminal device 9 if any of the similarities calculated for the previously recorded information are below a predetermined value. The control unit 11 may estimate the cause of the work environment 4 being incompatible and notify the user via the terminal device 9. For example, the control unit 11 may notify the user of items with similarities lower than a predetermined threshold as the cause of the work environment 4 being incompatible. Specifically, the control unit 11 may notify the user that the workpiece 8 is incorrect if the similarity between the color information or shape information, such as the outline, of the workpiece 8 is below a predetermined threshold. Furthermore, the control unit 11 may notify that the object placement is incorrect, for example, when the same objects as those in the previously recorded information have been prepared but the placement of the objects is different. Furthermore, for example, when there is a surplus or deficiency of prepared objects compared with the objects in the previously recorded information, the control unit 11 may estimate an object not in the previously recorded information as an extra object, or may estimate an object not in the work environment 4 as a missing object. The control unit 11 may superimpose the estimation result of the cause of the work environment 4 being unsuitable on the terminal device 9, and display it on the image of the work environment 4.

[0075] <Generating a control program tailored to work environment 4> In order to have the robot 1 perform a task in a work environment 4 where the shape of the work object has been changed, etc., the robot control system 100 may need to create a new control program corresponding to the changed work environment 4, even if the task content remains the same. In this case, the control program may be created by editing an existing control program. The robot control system 100 is configured to be able to create a control program to be executed in the prepared work environment 4 by editing a control program associated with past record information. It can also be said that the robot control system 100 generates a new processing pattern corresponding to the control program by editing an existing processing pattern.

[0076] In the robot control system 100, it is assumed that the workpiece 8 and other items required for the robot 1 to perform a task are prepared as shown in Fig. 3. The robot control system 100 can create a new control program for causing the robot 1 to perform a task in the prepared work environment 4 by following the procedure below.

[0077] Similar to the method for extracting a control program associated with previously recorded information described above, the robot control system 100 acquires information about the work environment 4, recognizes objects present in the work environment 4, and acquires attribute information and position information for each object. The robot control system 100 also extracts a control program that matches the information about the work environment 4 and presents it to the user via the terminal device 9. The user selects a control program to be edited to create a control program corresponding to a task the user wants the robot 1 to perform in the prepared work environment 4 by editing an existing control program. The user edits the control program on the terminal device 9. The terminal device 9 associates the newly created control program edited by the user with information about the prepared work environment 4 as previously recorded information and stores it in the storage unit 12 of the information processing device 10. In this way, a new control program can be easily created.

[0078] (Example of information processing procedure) The robot control system 100 may execute an information processing method including the procedures of the flowcharts illustrated in Figures 6 and 7. The information processing method may be realized as an information processing program executed by a processor constituting the control unit 11 of the information processing device 10, the robot controller 2, or the terminal device 9. The information processing program may be stored in a non-transitory computer-readable medium.

[0079] The control unit 11 may associate previously recorded information with a control program by executing the procedure of the flowchart in Fig. 6. The control unit 11 acquires a selection of a control program from a user (step S1). Specifically, the terminal device 9 accepts an input from the user to select a control program. The control unit 11 acquires the selection of the control program from the terminal device 9.

[0080] The control unit 11 acquires information about at least a portion of the work environment 4 (step S2). Specifically, the information acquisition device 3 acquires, as the information about at least a portion of the work environment 4, an image of at least a portion of the work environment 4, or depth information about at least a portion of the work environment 4. The control unit 11 acquires the information about at least a portion of the work environment 4 from the information acquisition device 3.

[0081] The control unit 11 recognizes an object in at least a part of the work environment 4 (step S3). The control unit 11 acquires attribute information and position information of the recognized object (step S4). The control unit 11 associates previously recorded information including attribute information and position information of the object recognized in at least a part of the work environment 4 with the control program selected by the user and stores the previously recorded information in the memory unit 12 (step S5). After executing the procedure of step S5, the control unit 11 ends execution of the procedure of the flowchart in FIG. 6.

[0082] The control unit 11 may extract a control program associated with previously recorded information similar to the work environment 4 by executing the procedure of the flowchart in FIG. 7 so that the user can easily select a control program.

[0083] The control unit 11 receives an instruction to search for a control program from the user (step S11). Specifically, the terminal device 9 receives an input from the user instructing to search for a control program. The control unit 11 receives the instruction to search for a control program from the terminal device 9.

[0084] The control unit 11 acquires information about at least a portion of the work environment 4 (step S12). The control unit 11 recognizes an object in at least a portion of the work environment 4 (step S13). The control unit 11 acquires attribute information and position information about the recognized object (step S14).

[0085] The control unit 11 compares the previously recorded information associated with the control program with the attribute information and position information of the recognized object (step S15). The control unit 11 extracts a control program based on the comparison result (step S16). Specifically, the control unit 11 may calculate the similarity between the previously recorded information and the attribute information and position information of the object, and extract the control program based on the similarity. The control unit 11 may prioritize the control programs before extracting them.

[0086] The control unit 11 displays the extracted program on the terminal device 9 (step S17). The control unit 11 acquires a selection of a control program from the user (step S18). Specifically, the terminal device 9 accepts an input from the user to select a control program. The control unit 11 acquires the selection of a control program from the terminal device 9.

[0087] The control unit 11 determines whether to execute or edit the selected control program (step S19). Specifically, the terminal device 9 receives an input from the user to determine whether to execute the selected control program or edit the selected control program to generate a new control program. The control unit 11 obtains information from the terminal device 9 specifying whether to execute or edit the control program.

[0088] When the control unit 11 acquires information specifying that a control program is to be executed from the terminal device 9, it determines that the selected control program is to be executed (step S19: "Execute"), and causes the robot controller 2 to execute the selected control program (step S20). After executing the procedure of step S20, the control unit 11 ends the execution of the procedure of the flowchart in FIG. 7.

[0089] When the control unit 11 acquires information specifying that the control program is to be edited from the terminal device 9, it determines that the control program is to be edited (step S19: "Edit") and generates a new control program based on the editing input from the user (step S21). Specifically, the terminal device 9 accepts input of the edit content of the control program from the user and outputs it to the information processing device 10. The control unit 11 edits the control program based on the input of the edit content acquired from the terminal device 9, generates a new control program, and stores it in the storage unit 12. After executing the procedure of step S21, the control unit 11 ends execution of the procedure of the flowchart in FIG. 7.

[0090] (summary) As described above, according to the robot control system 100, information processing device 10, and information processing method of this embodiment, candidates for a control program to be executed are extracted based on the prepared work environment 4. Extracting the candidates makes it easier for the user to select a control program. Furthermore, the user is less likely to select the wrong control program. As a result, the burden on the user can be reduced.

[0091] Furthermore, if a user selects a task and causes the robot 1 to execute a processing pattern for completing the selected task, there may be multiple different work environments 4 for the selected task. For example, even for the same type of task, the optimal processing pattern may differ depending on the production line's characteristics, etc. In this case, it is difficult to select a processing pattern to be executed by the robot 1. According to this embodiment, the processing pattern itself can be identified and candidate control programs can be extracted, allowing the user to easily select a control program corresponding to the processing pattern. Furthermore, the user is less likely to select the wrong control program. As a result, the burden on the user can be reduced.

[0092] Furthermore, according to this embodiment, it is possible to estimate the habits of the upstream process of the production line, including the habits of the user when placing the workpiece 8 etc. in the work environment 4. Based on the estimation result, a control program corresponding to the processing pattern required to complete the work can be selected. By executing the selected control program, it is possible to make the robot 1 complete the work based on the estimation result.

[0093] Furthermore, even if the work content is the same, the processing pattern required to complete the work is not limited to the same processing pattern. For example, various modes can be selected to align the workpieces 8, such as stacking them in bulk or aligning them in the same orientation, so as to suit the next process. According to this embodiment, candidate control programs can be extracted depending on the difference in processing pattern.

[0094] Furthermore, the efficiency of using the robot 1 can be improved. Furthermore, the efficiency of collaborative work including the robot 1 can be improved. Collaborative work includes work performed in cooperation between a person and the robot 1, work performed in cooperation between robots 1, or work performed in cooperation between the robot 1 and another machine. Collaborative work includes collaborative work performed in cooperation between a person and the robot 1.

[0095] (Other embodiments) The robot control system 100 according to this embodiment can be deployed in industrial production sites, food processing sites that handle food ingredients, or sites that manufacture cosmetics or pharmaceuticals.

[0096] The robot control system 100 according to this embodiment can also be used to control a robot 1, such as a communication robot or a service robot, that interacts with a user. For example, if the robot 1 has an information acquisition device 3 such as a camera, the robot control system 100 may extract a control program based on the results of recognizing the face of a user who is the target of the interaction or service provision. The robot control system 100 can be applied when it is necessary to change the control program in response to a change in the situation, such as when the user is wearing glasses or a mask.

[0097] The robot control system 100 according to this embodiment calculates a similarity between at least a portion of the information about the work environment 4 prepared by the user (preparation environment information or first environment information) and previously recorded information associated with the control program, and displays at least one control program in descending order of likelihood. The robot control system 100 may calculate other indices in addition to the similarity. For example, the robot control system 100 may acquire information, such as attribute information about the robot 1 that the user plans to use, information about the user who plans to use the robot 1, or a recognized work object through the robot controller 2, the terminal device 9, the information acquisition device 3, or the like, and calculate an index called a relevance to extract control programs frequently executed within a predetermined period for the robot 1 that the user plans to use, a robot of the same type as the robot 1 that the user plans to use, or the work object that the user plans to work on. The control unit 11 of the information processing device 10 may calculate the relevance so that a higher priority is given to control programs frequently executed within a predetermined period going back to the present. The control unit 11 may calculate the relevance so that a higher priority is given to a control program that was executed most recently. The control unit 11 may calculate the relevance so that a higher priority is given to a control program that was executed most recently by a user who is preparing the work environment 4 and causing the robot 1 to perform a task, or a control program that the user has executed frequently. The control unit 11 may extract a control program based on the calculated value of an index such as similarity or relevance. The control unit 11 may store the calculated value of an index such as similarity or relevance in the memory unit 12. The control unit 11 may set an index that combines multiple indexes such as the calculated similarity and relevance. The control unit 11 may calculate the value of an index that combines similarity and relevance, and extract a control program based on the calculated value. The control unit 11 may calculate the value of an index that combines multiple indexes such as similarity and relevance, and extract a control program based on the calculated value. The control unit 11 may calculate the value of an index that combines three or more indexes, and extract a control program based on the calculated value.

[0098] When newly associating information about the working environment 4 prepared by the user with the control program as previously recorded information, the control unit 11 may compare the information about the working environment 4 prepared by the user with the previously recorded information already associated with the control program. When the information about the working environment 4 prepared by the user does not match the previously recorded information already associated with the control program, the control unit 11 may record the information about the working environment 4 prepared by the user as previously recorded information, associating it with the control program. Furthermore, when the similarity between the information about the working environment 4 prepared by the user and the previously recorded information already associated with the control program is less than a predetermined value, the control unit 11 may record the information about the working environment 4 prepared by the user as previously recorded information, associating it with the control program. Note that the comparison of the new previously recorded information with the known previously recorded information may be triggered by the user selecting a control program to be executed by the robot.

[0099] As a method for associating the information of the work environment 4 with the robot control processing pattern as previously recorded information, template matching or a learned model of previously recorded information generated by machine learning or deep learning may be performed by relearning using the information of the work environment 4.

[0100] The control unit 11 may create a new robot control processing pattern based on at least one robot control processing pattern candidate and externally input information. The control unit 11 may extract, from among the robot control processing patterns stored in the storage unit 12, a robot control processing pattern associated with previously recorded information that is most similar to the information of the work environment 4 prepared by the user, as a candidate. The control unit 11 may create a new robot control processing pattern using information on the editing content input by the user to the terminal device 9 as externally input information.

[0101] An embodiment of the robot control system 100 has been described above, but the present disclosure can also be embodied as a method or program for implementing the system or device, as well as a storage medium on which a program is recorded (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card). The program may be stored in a non-transitory computer-readable medium.

[0102] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module incorporated into an operating system. Furthermore, the program may or may not be configured so that all processing is performed only by the CPU on the control board. The program may be executed by another processing unit mounted on an expansion board or expansion unit added to the board as needed. Therefore, it may be configured so that some or all of these are implemented.

[0103] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions included in at least one component can be rearranged so as not to cause logical inconsistencies, and multiple components can be combined or divided into one.

[0104] All of the features described in this disclosure and / or all steps of all of the disclosed methods or processes may be combined in any combination except combinations in which these features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by an alternative feature serving the same, equivalent, or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is only one example of a generic series of identical or equivalent features.

[0105] Furthermore, embodiments of the present disclosure are not limited to the specific configurations of any of the above-described embodiments, but rather extend to any novel feature or combination thereof described herein, or any novel method or process step or combination thereof described herein.

[0106] In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. In this disclosure, the configurations distinguished by descriptions such as "first" and "second" can have their numbers interchanged. For example, the first tray 6 can exchange the identifiers "first" and "second" with the second tray 7. The identifiers are exchanged simultaneously. The configurations remain distinguished even after the identifiers are exchanged. The identifiers may be deleted. A configuration from which the identifiers have been deleted is distinguished by a symbol. The identifiers "first" and "second" in this disclosure should not be used solely to interpret the order of the configurations or to justify the existence of an identifier with a smaller number. [Explanation of symbols]

[0107] 100 Robot control system (1: Robot, 2: Robot controller, 3: Information acquisition device, 4: Work environment, 5: Work table, 6: First tray, 7: Second tray, 8: Work) 9 Terminal Equipment 10 Information processing device (11: control unit, 12: storage unit) 80 Network

Claims

[Claim 1] a control unit that estimates a first control program to be executed by the first robot from at least one control program including a processing pattern of the robot, the control program being stored in a storage unit; The control unit acquiring first environment information indicating at least a part of a working environment of the first robot; an information processing device that estimates the first control program by estimating at least one candidate first processing pattern to be executed by the first robot based on the first environmental information;

Images