Information processing device, information processing method, information processing program, robot, control method, control program, and information processing system

Abstract

This invention provides an information processing device, information processing method, information processing program, robot, control method, control program, and information processing system that make it easier than ever to introduce robots capable of automating tasks that can handle a variety of objects. [Solution] An information processing device is provided that can be connected to a robot capable of autonomously working on objects, and comprises a control unit, which is configured to perform receiving processing, identifying processing, and transmitting processing, wherein in the receiving processing, it receives image data from the robot that includes a mark attached to an object and having object information relating to the object, in the identifying processing, it identifies work instruction data for causing the robot to work autonomously based on the object information and a first trained model, the first trained model is a model that has been trained to understand the relationship between object information and work instruction data, and in the transmitting processing, it transmits the work instruction data to the robot.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, an information processing program, a robot, a control method, a control program, and an information processing system.

Background Art

[0002] Since a robot can be used more generally than a machine tool used for a single purpose, it can contribute to labor saving in a manufacturing site. However, when moving a robot, it is necessary to perform so-called teaching in which what kind of operation is to be performed on the robot is recorded in advance, and there has been a certain hurdle to the introduction of robots into the manufacturing site. Therefore, the following conventional techniques exist.

[0003] There is disclosed a robot including a work mechanism that grasps and moves a work object, a reading unit that reads identification information of an identifier installed at a work location, a storage unit that stores a work pattern corresponding to the identification information, and a control unit that controls a machine tool based on the work pattern corresponding to the identification information read by the reading unit (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, since the robot disclosed in Patent Document 1 reads identification information installed at a work location, it is premised on work corresponding to a predetermined work location and not on work corresponding to various objects.

[0006] In view of the above circumstances, the present invention provides an information processing device, an information processing method, an information processing program, a robot, a control method, a control program, and an information processing system, etc., that enable easier introduction of robots capable of automating tasks that can handle a variety of objects than in the past. [Means for solving the problem]

[0007] According to one aspect of the present invention, an information processing device is provided that can be connected to a robot capable of autonomously working on an object, and comprises a control unit, which is configured to perform receiving processing, identifying processing, and transmitting processing, wherein in the receiving processing, it receives image data from the robot that includes a mark attached to an object and having object information relating to the object, in the identifying processing, it identifies work instruction data for causing the robot to work autonomously based on the object information and a first trained model, the first trained model is a model that has been trained to understand the relationship between object information and work instruction data, and in the transmitting processing, it transmits the work instruction data to the robot.

[0008] Furthermore, according to one aspect of the present invention, a robot is provided that is connected to an information processing device and capable of autonomously working on an object, comprising a control unit, the control unit being configured to perform acquisition processing, connection processing, transmission processing, reception processing, identification processing, and execution processing, wherein in the acquisition processing, it captures an image of a mark attached to an object that has object information relating to the object and has connection information for connecting to the information processing device, and acquires image data including the mark; in the connection processing, it connects to the information processing device based on the connection information; in the transmission processing, it transmits the image data to the information processing device; in the reception processing, it receives work instruction data identified by the information processing device, which is work instruction data for autonomous work, from the information processing device; in the identification processing, it decodes the work instruction data and identifies the task to be performed; and in the execution processing, it performs the identified task.

[0009] The information processing device in the above embodiment uses a trained model to identify work instruction data that defines the actions a robot should take to perform a task according to a specific work procedure. Therefore, the identified work instruction data will have more "ambiguity" than work instruction data identified using rule-based reference information. In other words, rather than strictly controlling the robot's movements, a certain degree of ambiguity is left, improving adaptability to various objects and allowing for more flexible configuration of the environment in which the robot is deployed.

[0010] Furthermore, the information processing device in the above embodiment can have a trained model infer what results will occur when an operation is performed on an object, and reflect this inference when identifying work instruction data. Therefore, the robot can suitably perform autonomous operations on objects.

[0011] Thus, the information processing device in the above embodiment can improve the functions of a computer to achieve at least one of the following (1) to (4): (1) The computer's processing speed can be increased. (2) The computer's power consumption can be reduced. (3) The computer's communication speed can be increased. (4) The resources saved in the computer can be used for other core functions.

[0012] Furthermore, since the robot according to the above embodiment is connected to the information processing device according to the above embodiment and can perform tasks autonomously, it can be easily introduced into the manufacturing site even if one does not have sufficient knowledge about robots.

[0013] This configuration makes it easier than before to introduce robots capable of automating tasks that handle a variety of objects. [Brief explanation of the drawing]

[0014] [Figure 1] This is a diagram showing the configuration of information processing system 100. [Figure 2]It is a block diagram showing the hardware configuration of the information processing apparatus 200. [Figure 3] It is a block diagram showing the hardware configuration of the robot 300. [Figure 4] It is a block diagram showing the functions realized by the information processing apparatus 200 (control unit 210). [Figure 5] It is a block diagram showing the functions realized by the robot 300 (control unit 310). [Figure 6] It is an activity diagram showing the flow of information processing executed by the information processing apparatus 200 and the robot 300. [Figure 7] It is an activity diagram showing the flow of information processing executed by the information processing apparatus 200 and the robot 300. [Figure 8] It is a diagram showing the workpiece 410 and the mark 420 attached to the workpiece 410. [Figure 9] It is a diagram showing an example of the workpiece information 450. [Figure 10] It is a diagram showing an example of the processing in the activity A200. [Figure 11] It is a diagram showing an example of the work instruction data 460. [Figure 12] It is a diagram showing an example of the work data 470. [Figure 13] It is a diagram showing an example of the processing in the activity A280. [Figure 14] It is a diagram showing an example of the change information 480.

Embodiments for Carrying Out the Invention

[0015] <First Embodiment> Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various characteristic matters shown in the following embodiments can be combined with each other.

[0016] Incidentally, a program for realizing the software that appears in one embodiment may be provided as a non-transitory computer-readable medium readable by a computer, may be provided so as to be downloadable from an external server, or may be provided such that the program is launched on an external computer to realize its function on a client terminal (so-called cloud computing).

[0017] Also, in various information processes according to one embodiment, an input and an output corresponding to the input can be realized. Here, if an output is obtained as a result of the input, the form of information (hereinafter referred to as reference information) referred to in such information processing is not limited. The reference information may be, for example, rule-based information such as a database, a lookup table, a predetermined function (including a determination formula such as a regression formula constructed by a statistical method), a learned model in which the correlation between the input and the output has been learned in advance, or a large language model capable of outputting a desired result by inputting a prompt.

[0018] Also, in one embodiment, a "part" may include, for example, hardware resources implemented by a circuit in a broad sense and information processing of software that can be specifically realized by these hardware resources. Also, in one embodiment, various types of information are handled, and these information are represented, for example, by physical values of signal values representing voltage and current, the high and low of signal values as a set of binary bits composed of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculation can be executed on a circuit in a broad sense.

[0019] Furthermore, a circuit in a broad sense is a circuit realized by combining at least a suitable combination of circuits, circuits, processors, and memory. The processor may be a general-purpose processor or a dedicated circuit. In other words, it includes application-specific integrated circuits (ASICs), programmable logic devices (for example, simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)), etc.

[0020] 1. Hardware Configuration Section 1 describes the hardware configuration of this embodiment.

[0021] 1-1. Information Processing System 100 Figure 1 is a diagram showing the configuration of the information processing system 100. The information processing system 100 comprises an information processing device 200 and a robot 300, which are connected via a network. The information processing device 200 and the robot 300 are interconnected and configured to perform autonomous tasks on objects by exchanging various types of information. These components will be explained further. Here, the system exemplified in the information processing system 100 consists of one or more devices or components.

[0022] 1-2. Information processing device 200 Figure 2 is a block diagram showing the hardware configuration of the information processing device 200. The information processing device 200 is an information processing device that can be connected to a robot 300 capable of autonomously working on objects. In this embodiment, the information processing device 200 will be described as a server. The information processing device 200 has a control unit 210, a storage unit 220, and a communication unit 250, and these components are electrically connected within the information processing device 200 via a communication bus 260. Each component will be described further.

[0023] The control unit 210 performs processing and control of the overall operation related to the information processing device 200. The control unit 210 is, for example, a Central Processing Unit (CPU) (not shown). The control unit 210 realizes various functions related to the information processing device 200 by reading predetermined programs stored in the storage unit 220. That is, information processing by software stored in the storage unit 220 is concretely realized by the control unit 210, which is an example of hardware, and can be executed as each functional unit included in the control unit 210. These will be explained further in Section 2. Note that the control unit 210 is not limited to being a single unit, and may be implemented with multiple control units 210 for each function, or a combination thereof.

[0024] The storage unit 220 stores various information necessary for information processing by the information processing device 200. This can be done, for example, as a storage device such as a solid-state drive (SSD) that stores various programs related to the information processing device 200 executed by the control unit 210, or as memory such as random access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to program calculations. A combination of these may also be used.

[0025] The communication unit 250 preferably uses wired communication methods such as USB, IEEE1394, Thunderbolt®, and wired LAN network communication, but may also include wireless LAN network communication, mobile communication such as 5G / LTE / 3G, and Bluetooth® communication as needed. In other words, it is more preferable to implement it as a collection of these multiple communication methods. Specifically, the information processing device 200 communicates various information with the robot 300 via the network through the communication unit 250.

[0026] 1-3. Robot 300 Figure 3 is a block diagram showing the hardware configuration of the robot 300. The robot 300 is configured to be connectable to the information processing device 200 and is capable of autonomously working on objects. Here, a robot is an intelligent mechanical system having three elemental technologies: sensors, an intelligent control system, and a drive system. In this embodiment, the robot 300 is described as an industrial robot deployed in a manufacturing site. The robot 300 has a control unit 310, a storage unit 320, a display unit 330, an input unit 340, a communication unit 350, an arm 370, an end-effector 380, and an imaging unit 390, and these components are electrically connected within the robot 300 via a communication bus 360. The descriptions of the control unit 310, storage unit 320, and communication unit 350 are substantially the same as the descriptions of the control unit 210, storage unit 220, and communication unit 250 in the information processing device 200, so they are omitted here.

[0027] The display unit 330 may be included in the housing of the robot 300 or it may be an external component. The display unit 330 displays a graphical user interface (GUI) screen that can be operated by the user. This is preferably done by using a display device such as a CRT display, liquid crystal display, organic EL display, or plasma display, depending on the type of robot 300. In the following description, the display unit 330 will be described as being included in the housing of the robot 300.

[0028] The input unit 340 may be included in the housing of the robot 300 or it may be external. For example, the input unit 340 may be integrated with the display unit 330 and implemented as a touch panel. If it is a touch panel, the user can input tap operations, swipe operations, etc. Of course, a switch button, mouse, QWERTY keyboard, etc. may be used instead of a touch panel. In other words, the input unit 340 receives operation input made by the user. This input is transmitted as a command signal to the control unit 310 via the communication bus 360. The control unit 310 can then perform predetermined controls and calculations as needed.

[0029] The arm 370 is a well-known multi-joint (multi-axis) robot arm, and its tip (not shown) can be freely displaced in three dimensions by automatic control by the control unit 310. An end-effector 380 is attached to the tip of the arm 370.

[0030] The end-effector 380 is a component that constitutes the tip of the arm 370, i.e., the end-effector. The end-effector 380 can be made of a rigid metal, for example, and can be made of a component similar to a human end-effector. The tip of the end-effector 380 is fitted with, for example, a spindle that rotates a tool to allow the robot 300 to perform a predetermined task, a gripper hand for grasping the workpiece 410, and a sensor of an appropriate type, such as a laser or camera, for acquiring information about the workpiece 410. Here, information about the workpiece 410 corresponds to "object information about an object" in the claim, and refers to at least one of any information that is useful for the robot 300 to perform work on the workpiece 410, such as whether or not the workpiece 410 is present, the position of the workpiece 410, the properties and shape of the workpiece 410, etc. Furthermore, a space is formed inside the end-effector 380 for passing electrical wiring and air piping used for the spindle, gripper hand, sensor, etc.

[0031] The imaging unit 390 is a digital camera, for example, including a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and is mounted on the robot 300 so that it can capture information necessary for the robot's operation. When the robot 300 moves autonomously, the imaging unit 390 is used to detect pathways and obstacles using the captured images.

[0032] 2. Functional Configuration Section 2 will describe the functional configuration of this embodiment.

[0033] 2-1. Functional Configuration (Information Processing Device 200) As mentioned above, the information processing performed by the software stored in the memory unit 220 is concretely realized by the control unit 210, which is an example of hardware, and can be executed as each functional unit included in the control unit 210.

[0034] Figure 4 is a block diagram showing the functions realized by the information processing device 200 (control unit 210). As described above, the information processing device 200 includes a control unit 210. The information processing device 200 (control unit 210) includes a receiving unit 211, a identifying unit 212, a transmitting unit 213, and a learning unit 214. Here, the control unit 210 is configured to perform receiving processing, identifying processing, transmitting processing, and learning processing in correspondence with each of the above units.

[0035] The receiving unit 211 is configured to receive various types of information. The receiving unit 211 is configured to perform receiving processing. For example, the receiving unit 211 receives image data from the robot 300 that includes a mark attached to an object and contains object information about that object.

[0036] Here, "things" are not particularly limited to those that are handled by the robot 300, and may include, for example, food, vehicles such as cars and trains, mechanical products, logistics goods, animals, plants, people, etc. Subsequent definitions of "things" can be similar.

[0037] Furthermore, the mark is not particularly limited as long as it includes object information, connection information for connecting to the information processing device 200, etc. For example, it may be a one-dimensional code such as a barcode, a stacked two-dimensional code such as PDF417 or CODE49, or a matrix two-dimensional code such as QR code (registered trademark), DataMatrix, or VeriCode. Subsequent marks may be defined in the same manner.

[0038] Furthermore, image data is data captured by imaging devices such as digital cameras and scanners, in which visual information is represented in a digital format. Image data is mainly composed of numerical data such as color, shape, and brightness using pixels or vectors, and may also include non-numerical data such as text data, metadata, embedded information, and steganography. The format of image data is not particularly limited; for example, it may be a raster image or a vector image. Examples of raster images include JPEG, GIF, and PNG. Examples of vector images include SVG, EPS, AI, and PDF.

[0039] The identification unit 212 is configured to identify various types of information. The identification unit 212 is configured to perform specific processing. For example, based on object information and a first trained model, the identification unit 212 identifies work instruction data to allow the robot 300 to perform tasks autonomously.

[0040] Here, the first pre-trained model is a model that has been trained using machine learning to understand the relationship between object information and work instruction data. In other words, the first pre-trained model is configured to output the corresponding work instruction data when object information is input.

[0041] Furthermore, work instruction data is data that refers to actions in robot language for the robot 300 to perform a task according to a specific work procedure. For example, it refers to data that refers to various actions such as starting the robot 300, returning to the origin, stabilizing, machining, and constraints. Subsequent work instruction data can be defined in a similar manner.

[0042] The transmitting unit 213 is configured to transmit various types of information. The transmitting unit 213 is configured to perform transmission processing. For example, the transmitting unit 213 transmits work instruction data to the robot 300.

[0043] The learning unit 214 is configured to learn various kinds of information. The learning unit 214 is configured to execute learning processes. For example, the learning unit 214 retrains the second trained model using working data.

[0044] Here, the second pre-trained model is a model that has been trained to learn the relationship between work data and change information for modifying the control parameters of the robot 300. In other words, the second pre-trained model is configured to output corresponding change information when work data is input.

[0045] Furthermore, the work data is data relating to the work performed on objects by the robot 300, and includes, for example, at least one of the following: the number of workpieces flowing in per unit time, the timing of workpiece recognition, and the time required to process one workpiece. Subsequent work data can be defined in a similar manner.

[0046] 2-2. Functional Configuration (Robot 300) As mentioned above, the information processing performed by the software stored in the memory unit 320 is concretely realized by the control unit 310, which is an example of hardware, and can be executed as each functional unit included in the control unit 310.

[0047] Figure 5 is a block diagram showing the functions realized by the robot 300 (control unit 310). As described above, the robot 300 includes a control unit 310. The robot 300 (control unit 310) includes an acquisition unit 311, a connection unit 312, a transmission unit 313, a reception unit 314, a specification unit 315, an execution unit 316, and a collection unit 317. Here, the control unit 310 is configured to perform acquisition processing, connection processing, transmission processing, reception processing, specification processing, execution processing, and collection processing in correspondence with each of the above units.

[0048] The acquisition unit 311 is configured to acquire various types of information. The acquisition unit 311 is configured to perform acquisition processing. For example, the acquisition unit 311 captures an image of a mark attached to an object that has object information relating to the object and has connection information for connecting to the information processing device 200, and acquires image data including the mark.

[0049] The connection unit 312 is configured to connect the robot 300 to other devices. The connection unit 312 is configured to perform connection processing. For example, the connection unit 312 connects to the information processing device 200 based on connection information.

[0050] The transmitting unit 313 is configured to transmit various types of information. The transmitting unit 313 is configured to perform transmission processing. For example, the transmitting unit 313 transmits image data to the information processing device 200.

[0051] The receiving unit 314 is configured to receive various types of information. The receiving unit 314 is configured to perform receiving processing. For example, the receiving unit 314 receives work instruction data from the information processing device 200, which is work instruction data for autonomous work identified by the information processing device 200.

[0052] The identification unit 315 is configured to identify various types of information. The identification unit 315 is configured to perform specific processing. For example, the identification unit 315 decodes work instruction data and identifies the action to be performed.

[0053] The execution unit 316 is configured to cause the robot 300 to perform various actions. The execution unit 316 is configured to execute execution processes. For example, the execution unit 316 performs a specified action.

[0054] The collection unit 317 is configured to collect various types of information. The collection unit 317 is configured to perform collection processing. For example, the collection unit 317 collects work data related to work on objects.

[0055] 3. Information Processing Methods Section 3 describes the flow of information processing performed by the information processing device 200 and the robot 300 described above. The information processing method performed by the information processing device 200 comprises the various processes in the information processing device 200. The control method performed by the robot 300 comprises the various processes in the robot 300.

[0056] Figures 6 and 7 are activity diagrams showing the flow of information processing performed by the information processing device 200 and the robot 300. The following explanation will follow each activity in these activity diagrams.

[0057] In this embodiment, the robot 300 is described as performing one step in the manufacturing process of a diesel particulate filter (DPF). The object that the robot 300 works on is described as a workpiece 410, which is a component of the DPF. In this activity, the robot 300 is positioned at the location corresponding to the above step in the manufacturing line and is described as having grasped the workpiece 410 that has flown in from the previous step with an end-effector 380 attached to an arm 370.

[0058] First, the control unit 310 in the robot 300 causes the imaging unit 390 to image the mark 420 attached to the workpiece 410 (activity A110). In other words, the acquisition process images the mark 420 attached to the workpiece 410, which has object information related to the workpiece 410 and connection information for connecting to the information processing device 200. In activity A110, for example, the following two stages of information processing are performed: (1) The control unit 310 reads a predetermined module from the storage unit 320. (2) The control unit 310 uses the module to perform the acquisition process and causes the imaging unit 390 to image the mark 420.

[0059] Next, the control unit 310 in the robot 300 acquires the image data 440 captured by the imaging unit 390 (activity A120). In other words, the acquisition process acquires the image data 440 including the mark 420. In activity A120, for example, the following information processing is performed: The control unit 310 stores the image data 440 captured by the imaging unit 390 in the storage unit 320.

[0060] Figure 8 shows a workpiece 410 and a mark 420 attached to the workpiece 410. As shown in Figure 8(A), the mark 420 is attached to a region 430 of the workpiece 410. The mark 420 may be placed anywhere on the workpiece 410, as long as it is a location that can be imaged by the imaging unit 390. When the region 430 is imaged by the imaging unit 390, image data 440 including the mark 420 can be acquired, as shown in Figure 8(B).

[0061] Returning to the explanation of Figure 6, the control unit 310 in the robot 300 then sends a connection request to the information processing device 200 based on the connection information for connecting to the information processing device 200 (Activity A130). In other words, the connection process connects to the information processing device 200 based on the connection information for connecting to the information processing device 200. In Activity A130, for example, the following five stages of information processing are executed: (1) The control unit 310 reads a predetermined module and image data 440 from the storage unit 320. (2) The control unit 310 executes the module and extracts connection information from the image data 440. (3) The control unit 310 reads a predetermined module from the storage unit 320. (4) The control unit 310 executes the connection process using the module and the connection information. (5) The communication unit 350 sends a connection request to the information processing device 200.

[0062] Next, the control unit 210 in the information processing device 200 receives a connection request transmitted from the robot 300 (activity A140). In activity A140, for example, the following two stages of information processing are performed: (1) The communication unit 250 receives the connection request transmitted from the robot 300. (2) The control unit 210 stores the connection request in the storage unit 220.

[0063] Next, the control unit 210 in the information processing device 200 processes the connection to the robot 300 (activity A150). In activity A150, for example, the following four stages of information processing are performed: (1) The control unit 210 reads the connection request and a predetermined module from the storage unit 220. (2) The control unit 210 performs the connection process using the connection request and the predetermined module. (3) The control unit 210 enables the exchange of information with the robot 300 via the communication unit 250. (4) The control unit 210 stores the result that a connection state has been achieved that enables the exchange of information with the robot 300 (hereinafter also referred to as the "connection result") in the storage unit 220.

[0064] Next, the control unit 210 in the information processing device 200 transmits the connection result to the robot 300 (activity A160). In activity A160, for example, the following two stages of information processing are performed: (1) The control unit 210 reads the connection result from the storage unit 220. (2) The control unit 210 transmits the connection result to the robot 300 via the communication unit 250.

[0065] Next, the control unit 310 in the robot 300 receives the connection result from the information processing device 200 (activity A170). In activity A170, for example, the following two stages of information processing are performed: (1) The communication unit 350 receives the connection result transmitted from the information processing device 200. (2) The control unit 310 stores the connection result in the storage unit 320.

[0066] Next, the control unit 310 in the robot 300 transmits the image data 440 captured by the imaging unit 390 to the information processing device 200 (activity A180). In other words, the transmission process transmits the image data 440 to the information processing device 200. In activity A180, for example, the following three stages of information processing are performed: (1) The control unit 310 reads the image data 440, connection results, and predetermined modules from the storage unit 320. (2) The control unit 310 performs the transmission process using the image data 440, connection results, and predetermined modules. (3) The control unit 310 transmits the image data 440 to the information processing device 200 via the communication unit 350.

[0067] Next, the control unit 210 in the information processing device 200 receives image data 440 transmitted from the robot 300 (activity A190). In other words, in the reception process, image data 440 is received from the robot 300, which includes a mark 420 attached to the workpiece 410 and which has workpiece information 450 (corresponding to "object information" in the claims) relating to the workpiece 410. In activity A190, for example, the following two stages of information processing are performed: (1) The communication unit 250 receives the image data 440 transmitted from the robot 300. (2) The control unit 210 stores the image data 440 in the storage unit 220.

[0068] Figure 9 shows an example of work information 450. Work information 450 includes, for example, the application of work 410, the arrangement of work 410 corresponding to that application, the function of work 410, the material of work 410, the shape of work 410, etc. Here, the application of work 410 is described as, for example, an automotive application. The arrangement of work 410 is described as, for example, being placed in the engine compartment of an automobile. The function of work 410 is described as, for example, a filter for removing dust. The material of work 410 is described as, for example, ceramic. The shape of work 410 is described as, for example, a rectangular parallelepiped with a length (long side) of 15 cm, a width (short side) of 5 cm, and a height of 5 cm.

[0069] Returning to the explanation of Figure 6, the control unit 210 in the information processing device 200 generates work instruction data 460 to allow the robot 300 to perform tasks autonomously (Activity A200). In other words, in a specific process, work instruction data 460 is generated based on work information 450 and a first trained model 510. Here, the first trained model 510 is a model that has been trained to learn the relationship between object information such as work information 450 and work instruction data. In Activity A200, for example, the following five stages of information processing are executed: (1) The control unit 210 reads image data 440 and a predetermined module from the storage unit 220. (2) The control unit 210 executes the predetermined module and extracts work information 450 from the marks 420 contained in the image data 440. (3) The control unit 210 reads the first trained model 510 from the storage unit 220. (4) The control unit 210 inputs the work information 450 into the first trained model 510 and causes the first trained model 510 to generate work instruction data 460. (5) The control unit 210 stores the work instruction data 460 in the storage unit 220. According to activity A200, there is no need to prepare work instruction data in advance, and work that can be handled by various objects can be automated.

[0070] Figure 10 shows an example of processing in activity A200. The control unit 210 inputs work information 450 to the first trained model 510. The first trained model 510 processes the work information 450 internally and outputs (generates) work instruction data 460 corresponding to the work 410.

[0071] Figure 11 shows an example of work instruction data 460. Work instruction data 460 includes work information, environmental information, control information (movement), control information (work), and control information (inspection), etc.

[0072] Work information includes work content, work process, work conditions, safety conditions, and quality conditions. Work content, for example, describes the work performed by robot 300 on workpiece 410, more specifically, the alternating sealing of square clusters of bundled cell structures. Work process, for example, describes the flow of robot 300's movements on workpiece 410, more specifically, the flow of movements in which workpiece 410 flowing from the previous process is grasped by end-effector 380, a mask member (not shown) is brought into contact with the end face of workpiece 410, ceramic material is injected through the mask member, and the alternatingly sealed workpiece 410 is sent to the next process. Work conditions, for example, describe the conditions necessary for robot 300 to perform the work with high precision, more specifically, the injection pressure (pump pressure) when injecting ceramic material into the end face of workpiece 410. Safety conditions, for example, indicate conditions to ensure the safety of surrounding workers in the event of a malfunction of the robot 300, and more specifically, indicate that the robot 300 will stop working if a force exceeding a predetermined level is applied to the arm 370 or end-effector 380. Quality conditions, for example, indicate conditions to prevent workpieces 410 that have become defective due to an error in the robot 300 from proceeding to subsequent processes, and more specifically, indicate that if a predetermined light is shone on one end face of the workpiece 410 and a light level exceeding a threshold is detected from the other end face of the workpiece 410, the sealing will be deemed insufficient and the workpiece 410 will be discarded as a defective product.

[0073] Environmental information includes the work environment, tools, parts and materials, and inventory management. The work environment indicates, for example, the amount of work that needs to be performed by the robot 300, and more specifically, the number of workpieces 410 that need to be processed per unit time. The tools indicate, for example, the types of tools required for the robot 300's work, and more specifically, the types of mask members according to the type of DPF. The parts and materials indicate, for example, the parts and materials used in the workpiece 410, and more specifically, the types of ceramic materials injected from the end face of the workpiece 410. Inventory management indicates, for example, the inventory of tools and parts and materials required for the robot 300's work, and more specifically, the inventory of mask members, ceramic raw materials, etc.

[0074] Control information (movement) refers to the autonomous control program for movement required when robot 300 performs a task. Control information (work) refers to the autonomous control program for work required when robot 300 performs a task. Control information (inspection) refers to the autonomous control program for inspection required when robot 300 performs a task.

[0075] Returning to the explanation of Figure 6, the control unit 210 in the information processing device 200 then transmits the work instruction data 460 to the robot 300 (activity A210). In other words, the transmission process transmits the work instruction data 460 to the robot 300. In activity A210, for example, the following two stages of information processing are performed: (1) The control unit 210 reads the work instruction data 460 from the storage unit 220. (2) The control unit 210 transmits the work instruction data 460 to the robot 300 via the communication unit 250.

[0076] Next, the control unit 310 of the robot 300 receives work instruction data 460 from the information processing unit 200 (activity A220). In other words, in the receiving process, the robot 300 receives work instruction data 460, which has been identified by the information processing unit 200 and is used to make the robot 300 perform the work autonomously. In activity A220, for example, the following two stages of information processing are performed: (1) The communication unit 350 receives the work instruction data 460 from the information processing unit 200. (2) The control unit 310 stores the work instruction data 460 in the storage unit 320.

[0077] Next, the control unit 310 in the robot 300 decodes the work instruction data 460 and identifies the action to be performed (activity A230). In other words, the identification process decodes the work instruction data 460 and identifies the action to be performed. In activity A230, for example, the following four stages of information processing are performed: (1) The control unit 310 reads the work instruction data 460 and a predetermined module from the storage unit 320. (2) The control unit 310 executes the predetermined module and decodes the work instruction data 460. (3) The control unit 310 performs identification processing on the decoded content and identifies the action to be performed. (4) The control unit 310 stores the information of the identified action in the storage unit 320.

[0078] Now, let's move on to the explanation of Figure 7. Next, the control unit 310 in the robot 300 executes the specified action (activity A240). In other words, the execution process executes the specified action. In activity A240, for example, the following two stages of information processing are performed: (1) The control unit 310 reads the information of the specified action from the storage unit 320. (2) The control unit 310 executes the execution process and controls the entire robot 300 to execute the specified action.

[0079] Furthermore, the control unit 310 in the robot 300 collects work data 470 related to the work on the workpiece 410 in parallel with the processing of activity A240 (activity A250). In other words, the collection process collects work data 470 related to the work on the workpiece 410. Activity A250 performs, for example, the following three stages of information processing: (1) The control unit 310 reads a predetermined module from the storage unit 320. (2) The control unit 310 executes the predetermined module and collects work data 470 using sensors and imaging units 390 attached to the end-effector 380. (3) The control unit 310 stores the work data 470 in the storage unit 320.

[0080] Figure 12 shows an example of work data 470. The work data 470 includes, for example, (1) the number of workpieces 410 flowing per unit time, (2) the timing of workpiece recognition, and (3) the time required to process one workpiece 410. (1) For example, assume that 6 workpieces 410 flow per minute. (2) For example, assume that the workpiece 410 is recognized at a position 30 cm away from the robot 300 body. (3) For example, assume that it takes 11 seconds.

[0081] Returning to the explanation of Figure 7, the control unit 310 in the robot 300 then transmits the work data 470 to the information processing device 200 (activity A260). In other words, the transmission process sends the work data 470 to the information processing device 200. In activity A260, for example, the following two stages of information processing are performed: (1) The control unit 310 reads the work data 470 from the storage unit 320. (2) The control unit 310 transmits the work data 470 to the information processing device 200 via the communication unit 350. Activities A250 to A260 allow for the automation of tasks that can be performed on various objects in cooperation with the information processing device.

[0082] Next, the control unit 210 in the information processing device 200 receives work data 470 from the robot 300 (activity A270). In other words, the receiving process receives work data 470 related to the work performed by the robot 300, which is work data 470 collected by the robot 300. In activity A270, for example, the following two stages of information processing are performed: (1) The communication unit 250 receives work data 470 from the robot 300. (2) The control unit 210 stores the work data 470 in the storage unit 220.

[0083] Next, the control unit 210 in the information processing device 200 identifies change information 480 for changing the control amount of the robot 300 (activity A280). In other words, in the identification process, the change information 480 for changing the control amount of the robot 300 is identified based on the work data 470 and the second trained model 520. Here, the second trained model 520 is a model that has been trained to learn the relationship between work data and change information. In activity A280, for example, the following three stages of information processing are executed: (1) The control unit 210 reads the work data 470 and the second trained model 520 from the storage unit 220. (2) The control unit 210 inputs the work data 470 to the second trained model 520 and has the second trained model 520 identify the change information 480. (3) The control unit 210 stores the change information 480 in the storage unit 220.

[0084] Figure 13 shows an example of processing in activity A280. The control unit 210 inputs the work data 470 to the second trained model 520. The second trained model 520 processes the work data 470 internally and identifies and outputs the change information 480 corresponding to the work data 470.

[0085] Figure 14 shows an example of change information 480. The change information 480 includes (1) the timing of recognizing the workpiece 410, (2) the movement speed of the arm 370, and (3) the processing time of the workpiece 410. (1) For example, adjust the angle of the sensor on the end-effector 380 so that the workpiece 410 is recognized at a position 25 cm away from the robot body 300. (2) For example, change the movement speed of the arm 370 from 30 cm / second to 31 cm / second. (3) For example, slightly increase the injection pressure of the ceramic material to shorten the processing time from 11 seconds to 10 seconds.

[0086] Returning to the explanation of Figure 7, in parallel with activity A280, the control unit 210 in the information processing device 200 retrains the second trained model 520 using the work data 470 (activity A290). In other words, the training process retrains the second trained model 520 using the work data 470. The retraining method is not particularly limited and may be, for example, transfer learning or fine tuning. In activity A290, for example, the following three stages of information processing are performed: (1) The control unit 210 reads the work data 470 and the second trained model 520 from the storage unit 220. (2) The control unit 210 applies the work data 470 to the second trained model 520 to generate a retrained second trained model. (3) The control unit 210 stores the retrained second trained model in the storage unit 220. Activity A290 can improve the adaptability of the robot's autonomous control.

[0087] Next, the control unit 210 in the information processing device 200 transmits the change information 480 to the robot 300 (activity A300). In other words, the transmission process transmits the change information 480 to the robot 300. In activity A300, for example, the following two stages of information processing are performed: (1) The control unit 210 reads the change information 480 from the storage unit 220. (2) The control unit 210 transmits the change information 480 to the robot 300 via the communication unit 250. Activities A270, A280, and A300 enable feedback control of the robot using the work data collected by the robot.

[0088] Next, the control unit 310 in the robot 300 receives change information 480 from the information processing unit 200 (activity A310). In other words, in the receiving process, change information 480 that changes the control amount of the robot 300's movement is received from the information processing unit 200. In activity A310, for example, the following two stages of information processing are performed: (1) The communication unit 350 receives change information 480 from the information processing unit 200. (2) The control unit 310 stores the change information 480 in the storage unit 320.

[0089] Next, the control unit 310 in the robot 300 decodes the change information 480 and identifies the operation to be changed (activity A320). In other words, the identification process decodes the change information 480 and identifies the operation to be changed. In activity A320, for example, the following three stages of information processing are performed: (1) The control unit 310 reads the change information 480 and a predetermined module from the storage unit 320. (2) The control unit 310 executes the predetermined module and extracts information about the operation to be changed from the change information 480. (3) The control unit 310 stores the information about the operation to be changed in the storage unit 320.

[0090] Next, when the control unit 310 in the robot 300 determines that autonomous control of the workpiece 410 has been completed, it terminates the information processing of this embodiment. On the other hand, when the control unit 310 determines that there is still work to be done on the workpiece 410, it proceeds to the processing of activities A240 and A250. Here, when the control unit 310 proceeds to the processing of activity A240, it modifies and executes the work using the information of the work to be changed (activity A240). In other words, the execution process modifies and executes the specified operation. According to activity A240, feedback control can be performed using the collected work data.

[0091] According to the above information processing, it becomes easier than before to introduce robots capable of automating tasks that handle a variety of objects.

[0092] <Second Embodiment> In the second embodiment, explanations that overlap with the description of the first embodiment will be omitted as appropriate.

[0093] 4. Information Processing Methods Section 4 will describe the information processing flow according to the second embodiment.

[0094] In the first embodiment, the first trained model 510 used to identify or generate work instruction data in activity A200 may be a model that has been trained on a minimum dataset to the extent that it can output corresponding work instruction data when object information is input. In this case, the marks attached to objects include not only information related to objects (hereinafter also referred to as "knowledge information") as shown in Figure 9, but also, for example, prompts that serve as instructions for requests and constraints to the first trained model (hereinafter also referred to as "instruction information"), descriptions for defining the thinking patterns and information structure of the first trained model (hereinafter also referred to as "schema information"), and programs or APIs for executing the first trained model (hereinafter also referred to as "execution information").

[0095] For example, if a user wants to know the brand of wine they have picked up, the following information processing steps are performed. First, the user has the robot 300's imaging unit 390 capture an image of the mark on the wine and transmit it (corresponding to activities A110 to A180). Next, the control unit 210 of the information processing device 200 inputs schema information and execution information contained in the mark, as well as knowledge information related to the wine and the instruction information at that time, into the first trained model (corresponding to activities A190 to A200). Subsequently, the first trained model learns the input knowledge information based on the input instruction information, and identifies and transmits work instruction data based on the input instruction information, the input schema information, and the input execution information (corresponding to activities A200 to A210). Next, the control unit 310 in the robot 300 decodes the work instruction data, displays a chat screen on the display unit 330, displays information about the wine brand as an answer on the chat screen, and accepts input from the user (corresponding to activities A220 to A240).

[0096] In this way, by minimizing the dataset used to train the first pre-trained model, the first pre-trained model becomes lightweight. Therefore, the generation of various data and task processing by the first pre-trained model can be optimized more accurately and in a shorter time than before.

[0097] Although various embodiments of the present invention have been described above, these are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0098] 5. Variations Section 5 describes modifications of this embodiment. The following modifications can be combined as appropriate.

[0099] The embodiments of this model may also be programs. The information processing program is configured to cause a computer, such as the information processing device 200, to execute each process in the information processing device 200. The control program is configured to cause the robot 300 to execute each process in the robot 300.

[0100] The control unit 210 writes (stores) and reads various data and information to the storage unit 220, but is not limited to this. For example, it may also use registers or cache memory within the control unit 210 to perform information processing for each activity.

[0101] The control unit 310 writes (stores) and reads various data and information to the storage unit 320, but is not limited to this. For example, it may also use registers or cache memory within the control unit 310 to perform information processing for each activity.

[0102] In this embodiment, a CPU is given as an example for the control unit 210 in the information processing device 200 and the control unit 310 in the robot 300, but it is not limited to this. The control unit 210 and the control unit 310 may be a CPU, a graphics processing unit (GPU), a neural processing unit (NPU), or a tensor processing unit (TPU), respectively, or a combination of these processors. In other words, the control unit 210 and the control unit 310 refer to one or more of the above processors, and these processors may work together to execute the information processing method of this embodiment.

[0103] In this embodiment, the first trained model 510 and the second trained model 520 have been described as being stored in the storage unit 220 of the information processing device 200, but the embodiment is not limited to this. For example, the first trained model 510 and the second trained model 520 may each be stored in an external server other than the information processing device 200 and used by the control unit 210 of the information processing device 200.

[0104] Activity A200 generates work instruction data 460, but it is not limited to this. In a specific process, work instruction data for autonomously performing tasks can be identified based on object information and a first trained model. For example, the first trained model may access a specific database and extract work instruction data from that database that matches the object information.

[0105] The activities shown in Figures 6 and 7 do not have to be executed in the order described in this embodiment. Therefore, the activities may be executed in any order, almost simultaneously, or some activities may be omitted.

[0106] Work instruction data may include at least one of the following: work information, environmental information, movement information, work information, and inspection information. In other words, work instruction data can be set appropriately depending on the type of object and the content of the work. This configuration allows for the use of information necessary for autonomous control of the robot.

[0107] In this embodiment, the robot 300 has been described as an industrial robot, but it is not limited to this. The type of robot 300 is not particularly limited and may include, for example, robots for infrastructure maintenance and inspection, surgical robots, robots for transporting people and goods, robots for communicating with people, robot cars such as autonomous vehicles and self-driving cars, and robot drones related to autonomously operating drones. The type of work performed by the robot 300 is not particularly limited and may include, for example, processing goods, transporting goods, inspecting goods, purchasing goods (payment processing), etc.

[0108] In this embodiment, at least one of the first trained model 510 and the second trained model 520 may perform a process of identification and prediction on newly input data using AI inference.

[0109] The marks in this embodiment may be captured by an imaging device owned by the user (for example, a camera mounted on a smartphone), or by an imaging unit 390 in the robot 300.

[0110] The work instruction data in this embodiment may include work instructions that combine work instructions for autonomously executing, operating, and moving the robot 300, and work instructions that autonomously respond to the user. Here, we will explain using the work performed by the robot 300 located in a store as an example. For example, if a user takes an image of a mark attached to a product in the store with the camera mounted on their smartphone and sends it to the robot 300, the robot 300 may exchange information with the information processing device 200, display information about the product (origin, nutritional information, etc.) on the display unit 330, and if the user's intention to purchase the product is received by the input unit 340, the robot 300 may perform the payment processing for the product.

[0111] 6. Others The product may be provided in any of the following embodiments.

[0112] (1) An information processing device that can be connected to a robot capable of autonomously working on an object, comprising a control unit, the control unit being configured to perform a receiving process, a identifying process, and a transmitting process, wherein the receiving process receives image data from the robot that includes a mark attached to the object and having object information relating to the object, the identifying process identifies work instruction data for causing the robot to work autonomously based on the object information and a first trained model, the first trained model is a model that has been trained to understand the relationship between the object information and the work instruction data, and the transmitting process transmits the work instruction data to the robot.

[0113] This configuration makes it easier than before to introduce robots capable of automating tasks that handle a variety of objects.

[0114] (2) Information processing device as described in (1) above, wherein the receiving process further receives work data related to work performed by the robot and work data collected by the robot, the identifying process identifies change information for changing the control amount of the robot based on the work data and a second trained model, the second trained model is a model that has been trained to learn the relationship between the work data and the change information, and the transmitting process transmits the change information to the robot.

[0115] In this configuration, the robot can be controlled using work data collected by the robot.

[0116] (3) An information processing device as described in (2) above, wherein the control unit is further configured to perform a learning process, and in the learning process, the second trained model is retrained using the work data.

[0117] This configuration can improve the adaptability of autonomous control of robots.

[0118] (4) An information processing device according to any one of (1) to (3) above, wherein in the specific processing, the information processing device generates the work instruction data based on the object information and the first trained model.

[0119] In this configuration, there is no need to prepare work instruction data in advance, and tasks that can handle various objects can be automated.

[0120] (5) An information processing device according to any one of (1) to (4) above, wherein the work instruction data includes at least one of work information, environmental information, movement information, work information, and inspection information.

[0121] In this configuration, information necessary for autonomously controlling the robot can be used.

[0122] (6) An information processing method comprising each of the processes in the information processing device described in any one of (1) to (5) above.

[0123] This configuration makes it easier than before to introduce robots capable of automating tasks that handle a variety of objects.

[0124] (7) An information processing program configured to cause a computer to perform each of the processes in the information processing device described in any one of (1) to (5) above.

[0125] This configuration makes it easier than before to introduce robots capable of automating tasks that handle a variety of objects.

[0126] (8) A robot configured to be connectable to an information processing device and capable of autonomously working on an object, comprising a control unit, the control unit being configured to perform an acquisition process, a connection process, a transmission process, a reception process, a specification process, and an execution process, wherein the acquisition process involves capturing an image of a mark attached to the object that has object information relating to the object and has connection information for connecting to the information processing device, and acquiring image data including the mark; the connection process involves connecting to the information processing device based on the connection information; the transmission process involves transmitting the image data to the information processing device; the reception process involves receiving work instruction data, which has been specified by the information processing device and is work instruction data for autonomous work, from the information processing device; the specification process involves decoding the work instruction data and specifying an action to be performed; and the execution process involves performing the specified action.

[0127] This configuration makes it easier than before to introduce robots capable of automating tasks that handle a variety of objects.

[0128] (9) A robot as described in (8) above, wherein the control unit is further configured to perform a collection process, wherein the collection process collects work data relating to work on the object, and the transmission process transmits the work data to the information processing device.

[0129] In this configuration, it is possible to automate tasks related to various objects in cooperation with information processing equipment.

[0130] (10) A robot as described in (9) above, wherein the receiving process receives change information from the information processing device to change the amount of control of the operation, the identifying process decodes the change information to identify the operation to be changed, and the execution process changes and executes the identified operation.

[0131] In this configuration, the collected work data can be used for feedback control.

[0132] (11) A control method comprising each of the processes in a robot described in any one of (8) to (10) above.

[0133] This configuration makes it easier than before to introduce robots capable of automating tasks that handle a variety of objects.

[0134] (12) A control program configured to cause the robot to perform any one of the processes described in (8) to (10) above.

[0135] This configuration makes it easier than before to introduce robots capable of automating tasks that handle a variety of objects.

[0136] (13) An information processing system comprising an information processing device described in any one of (1) to (5) above and a robot described in any one of (8) to (10) above, wherein the information processing device and the robot are interconnected and configured to autonomously perform work on the object by exchanging various kinds of information.

[0137] This configuration makes it easier than before to introduce robots capable of automating tasks that handle a variety of objects. Of course, this is not always the case. [Explanation of symbols]

[0138] 100: Information Processing Systems 200: Information Processing Device 210: Control Unit 211: Receiving unit 212: Specific part 213: Transmitter 214: Learning Department 220: Storage section 250: Communications Department 260: Communications bus 300: Robot 310: Control Unit 311: Acquisition Department 312: Connection part 313: Transmitter 314: Receiving unit 315: Specific part 316: Execution Department 317: Collection Department 320: Storage section 330: Display section 340: Input section 350: Communications Department 360: Communications Bus 370: Arm 380: Endpiece component 390: Imaging Unit 410: Work 420: Mark 430: area 440: Image data 450: Work Information 460: Work instruction data 470: Work data 480: Change Information 510: First pre-trained model 520: Second pre-trained model

Claims

1. An information processing device that can be connected to a robot capable of autonomously working on objects, Equipped with a control unit, The control unit is configured to perform receiving processing, identifying processing, and transmitting processing. In the aforementioned receiving process, image data including a mark attached to the object and containing object information relating to the object is received from the robot. In the aforementioned specific processing, work instruction data for autonomously performing tasks is identified based on the object information and the first trained model. The first trained model is a model that has been trained to learn the relationship between the object information and the work instruction data. In the transmission process, the work instruction data is transmitted to the robot. Information processing device.

2. In the information processing apparatus according to claim 1, In the aforementioned receiving process, the robot further receives work data related to the work performed by the robot, which is work data collected by the robot. In the aforementioned specific processing, based on the work data and the second trained model, change information for changing the control amount of the robot is identified. The second pre-trained model is a model that has been trained to learn the relationship between the work data and the change information. In the transmission process, the change information is transmitted to the robot. Information processing device.

3. In the information processing apparatus according to claim 2, The control unit is further configured to perform a learning process, In the aforementioned learning process, the second trained model is retrained using the aforementioned working data. Information processing device.

4. In the information processing apparatus according to claim 1, In the aforementioned specific processing, the work instruction data is generated based on the object information and the first trained model. Information processing device.

5. In the information processing apparatus according to claim 1, The aforementioned work instruction data includes at least one of the following: work information, environmental information, movement information, work information, and inspection information. Information processing device.

6. Information processing method, The information processing device comprises each of the processes described in any one of claims 1 to 5. Information processing methods.

7. It is an information processing program, The information processing device described in any one of claims 1 to 5 is configured to cause a computer to execute each of the processes described in that device. Information processing program.

8. A robot configured to be connectable to an information processing device and capable of autonomously working on objects, Equipped with a control unit, The control unit is configured to perform acquisition processing, connection processing, transmission processing, reception processing, identification processing, and execution processing. In the acquisition process described above, a mark attached to the object, which has object information relating to the object and has connection information for connecting to the information processing device, is captured, and image data including the mark is acquired. In the connection process, based on the connection information, the information processing device is connected, In the transmission process, the image data is transmitted to the information processing device. In the reception process, work instruction data identified by the information processing device, which is work instruction data for autonomous work, is received from the information processing device. In the aforementioned specific process, the work instruction data is decoded to identify the action to be performed. In the execution process described above, the identified operation is performed. robot.

9. In the robot according to claim 8, The control unit is further configured to perform collection processing, In the aforementioned data collection process, work data related to the work performed on the object is collected. In the transmission process, the work data is transmitted to the information processing device. robot.

10. In the robot according to claim 9, In the reception process, change information that modifies the control amount of the operation is received from the information processing device, In the aforementioned specific process, the change information is decoded to identify the action to be changed. In the execution process described above, the identified operation is modified and executed. robot.

11. A control method, A robot comprising each of the processes described in any one of claims 8 to 10, Control method.

12. It is a control program, The robot is configured to perform each of the processes described in any one of claims 8 to 10. Control program.

13. An information processing system, The present invention comprises an information processing device according to any one of claims 1 to 5 and a robot according to any one of claims 8 to 10. The information processing device and the robot are interconnected and configured to perform autonomous tasks on the object by exchanging various types of information. Information processing system.

Images