Robots, processing systems, processing methods, and programs

Abstract

This robot comprises: an operation means for operating an interface in which information about a system in which the robot moves is displayed; and a control means for identifying, in the interface, information relating to the motion of the robot, and controlling the motion of the robot.

Description

ROBOT, PROCESSING SYSTEM, PROCESSING METHOD, AND RECORDING MEDIUM

[0001] The present disclosure relates to a robot, a processing system, a processing method, and a recording medium.

[0002] Robots are used in various fields such as logistics. Patent Document 1 discloses a related technology relating to a robot hand that can improve maintainability.

[0003] Japanese Patent Application Laid-Open No. 2021-122925

[0004] In the robot system described in Patent Document 1 and the like, the robot and a worker may collaborate and perform work on behalf of the worker. In such cases, the robot is required to perform the same operations as the worker. Therefore, when the robot and the worker collaborate and the robot performs work on behalf of the worker, there is a need for technology that enables the robot and software that displays information about the system in which the robot operates on an interface to interact (link) appropriately.

[0005] One of the objectives of each aspect of the present disclosure is to provide a robot, a processing system, a processing method, and a recording medium that can solve the above-mentioned problems.

[0006] According to one aspect of the present disclosure, a robot includes an operation means for operating an interface that displays information about a system in which the robot operates, and a control means for identifying information about the operation of the robot in the interface and controlling the operation of the robot.

[0007] According to another aspect of the present disclosure, a processing system includes the robot described above and a management system that causes the interface to display information about a system in which the robot operates.

[0008] According to another aspect of the present disclosure, a processing method operates an interface that displays information about a system in which a robot operates, identifies information about the operation of the robot in the interface, and controls the operation of the robot.

[0009] According to another aspect of the present disclosure, a recording medium stores a program that causes a computer to operate an interface that displays information about a system in which a robot operates, identify information about the operation of the robot in the interface, and control the operation of the robot.

[0010] Aspects of the present disclosure allow for appropriate interaction between a robot and software that causes an interface to display information about the system in which the robot operates.

[0011] 1 is a diagram illustrating an example of a configuration of a processing system according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a configuration of a management system according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of a database stored in a storage unit according to an embodiment of the present disclosure. FIG. 4 is a diagram illustrating a first example of a display on a display unit according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating a second example of a display on a display unit according to an embodiment of the present disclosure. FIG. 6 is a diagram illustrating a third example of a display on a display unit according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating an example of a configuration of a robot system according to an embodiment of the present disclosure. FIG. 8 is a diagram illustrating an example of a sequence of a plan generated by a generation unit according to an embodiment of the present disclosure. FIG. 9 is a diagram illustrating an example of a control signal for a plan generated by a controller according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating an example of a processing flow of a processing system according to an embodiment of the present disclosure. FIG. 11 is an image diagram of the surrounding environment of a processing system according to an embodiment of the present disclosure. FIG. 12 is a diagram illustrating an example of a configuration of a management system according to an embodiment of the present disclosure. FIG. 13 is a diagram illustrating an example of a database stored in a storage unit according to an embodiment of the present disclosure. FIG. 14 is a diagram illustrating an example of a processing flow of a processing system according to an embodiment of the present disclosure. FIG. 1 is a diagram illustrating an example of the configuration of a robot system according to an embodiment of the present disclosure; FIG. 2 is a diagram illustrating an example of a processing flow of a processing system according to an embodiment of the present disclosure; FIG. 3 is an image diagram of the surrounding environment of a processing system according to an embodiment of the present disclosure; FIG. 4 is a diagram illustrating an example of the configuration of a management system according to an embodiment of the present disclosure; FIG. 5 is a diagram illustrating an example of the configuration of a robot system according to an embodiment of the present disclosure; FIG. 6 is a diagram illustrating an example of a sound source and a light source that can be used by a worker in an embodiment of the present disclosure; FIG. 7 is a diagram illustrating an example of a robot with a minimum configuration according to an embodiment of the present disclosure; FIG. 8 is a diagram illustrating an example of a processing flow of a robot with a minimum configuration according to an embodiment of the present disclosure; and FIG. 9 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment.

[0012]

[0023] Hereinafter, the embodiments will be described in detail with reference to the drawings. <First Embodiment> Fig. 1 is a diagram showing an example of the configuration of a processing system 1 according to an embodiment of the present disclosure. Below, the processing system 1 will be described using, as a specific example of work, a sorting operation in which products M in cardboard boxes C carried by a belt conveyor are transferred to trays T, and the trays T are placed on another belt conveyor and moved to a predetermined position.

[0013] As shown in FIG. 1, the processing system 1 includes a management system 10 and a robot system 20 .

[0014] 2 is a diagram illustrating an example of the configuration of the management system 10 according to an embodiment of the present disclosure. As shown in FIG. 2, the management system 10 includes a management unit 101, a storage unit 102, a control unit 103, and a display unit 104.

[0015] The management unit 101 writes the status of each task in the storage unit 102. Specifically, for example, the management unit 101 determines whether a completion button (described later) displayed on the display unit 104 has been touched. If the management unit 101 determines that the completion button has been touched, it determines that the task corresponding to the completion button has been completed. Then, the management unit 101 writes a completion flag indicating that the task has been completed in association with the task determined to have been completed in the storage unit 102.

[0016] The memory unit 102 stores various information necessary for the processing performed by the management system 10. For example, the memory unit 102 stores the status of each task. Specifically, for example, the memory unit 102 stores a flag of 0 (incomplete) or 1 (completed) associated with the task. Furthermore, for example, the memory unit 102 stores information that the control unit 103 causes the display unit 104 to display according to the status of each task. FIG. 3 is a diagram illustrating an example of a database DB1 stored in the memory unit 102 according to an embodiment of the present disclosure. The information that the control unit 103 causes the display unit 104 to display according to the status of each task is a specific graphic representing information corresponding to the status of each task, and information to be notified to a worker that corresponds one-to-one to the specific graphic. As shown in FIG. 3, the memory unit 102 stores the specific graphic representing information corresponding to the status of each task and the information to be notified to a worker in a one-to-one association as the database DB1.

[0017] The information that the control unit 103 causes the display unit 104 to display is a specific graphic that corresponds to the status of each task, and information to be notified to the worker. Examples of the specific graphic include an Augmented Reality (AR) marker, a Quick Response (QR) code, and geometric shapes such as a circle (○) or a triangle (△). Examples of information to be notified to the worker include a notification inquiring about the completion of the task, the details of the next task, etc.

[0018] The control unit 103 controls the display of the display unit 104. For example, the control unit 103 causes the display unit 104 to display a specific graphic, which is information corresponding to the status of each task stored in the storage unit 102, and information to be notified to the worker, which is stored in association with the specific graphic.

[0019] The display unit 104, under the control of the control unit 103, displays specific graphics, which are information corresponding to the status of each task, and information to be notified to the worker. For example, the display unit 104 is a display with a touch panel function. For example, the display unit 104 is installed at a work location. FIG. 4 is a diagram illustrating a first example of a display on the display unit 104 according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating a second example of a display on the display unit 104 according to an embodiment of the present disclosure. FIG. 6 is a diagram illustrating a third example of a display on the display unit 104 according to an embodiment of the present disclosure. For example, under the control of the control unit 103, the display unit 104 displays, within a frame, a button that displays an AR marker, which is a specific graphic corresponding to a selection content for selecting "yes" or "no" as to whether the task is completed. Furthermore, for example, under the control of the control unit 103, the display unit 104 displays, within a frame, the next task content and an AR marker, which is a specific graphic corresponding to the content, as shown in FIG. 5. Also, for example, under the control of the control unit 103, the display unit 104 displays, as shown in Figure 6, the next work content, as well as a button showing an AR marker, which is a specific shape corresponding to the selection content for selecting "yes" or "no" as to whether the work is completed, within a frame.

[0020] 7 is a diagram illustrating an example of the configuration of a robot system 20 according to an embodiment of the present disclosure. As illustrated in FIG. 7 , the robot system 20 includes a task instruction device 201, a generation unit 202, a camera 203, a controller 204, a robot 205, a determination unit 206, a storage unit 207, a recognition unit 208, and an identification unit 209.

[0021] The task instruction device 201 instructs the controller 204 on a task for causing the robot 205 to perform a desired task. For example, the task instruction device 201 instructs the controller 204 on a task via a network (or, in some cases, by using a cloud service).

[0022] For example, if the desired task is to move products M in a cardboard box C to a destination tray T, determine whether the task of moving the products M to the tray T is complete, and, if it is determined that the task is complete, perform a sorting task of moving the tray T to a predetermined position on a conveyor belt, the task instruction device 201 inputs task goals and constraint conditions to the generation unit 202 as a procedure for instructing the task. Examples of task goals include information indicating the type of products M, the number of products M to be moved, the source of the products M, and the destination of the products M, as well as information indicating a predetermined task unit for confirming the completion of the task. Examples of constraint conditions include no-entry areas when moving the products M, areas that deviate from the range of motion of the robot 205, and further conditions on the surface of the products M regarding gripping the products M, releasing gripping of the products M, or changing the products M. The task instruction device 201 may receive an input from the worker as a task goal, such as "move six parts A from cardboard box C to tray T," and specify that the type of product M to be moved is part A, the quantity of product M to be moved is six, the source of product M is cardboard box C, and the destination of product M is tray T. The task instruction device 201 may then input the specified information to the generation unit 202. Furthermore, the task instruction device 201 may receive, as a task goal, an input from the worker, for example, that "the units of predetermined tasks for confirming completion of the tasks are a first task of stopping the belt conveyor for moving the cardboard box C at the position of the robot 205, a second task of moving the cardboard box C from the stop position of the belt conveyor onto the work table F, a third task of moving the product M in the cardboard box C on the work table F to the tray T, a fourth task of stopping the belt conveyor for moving the tray T, a fifth task of moving the tray T onto the belt conveyor for moving the tray T, and a sixth task of moving the belt conveyor for moving the tray T," and cause the generation unit 202 to generate a process for confirming completion of the task after each of the first to sixth tasks. Furthermore, the task instruction device 201 may set the position of the product M identified in information acquired by the camera 203, which will be described later, as the source of movement of the product M.Furthermore, the task instruction device 201 may receive, for example, from a worker, the positions of obstacles along the path of moving the product M from the start point to the destination point as constraint conditions indicating no-entry areas, and input the information to the generation unit 202. Alternatively, a file indicating the constraint conditions may be stored in, for example, the storage unit 207, and the task instruction device 201 may input the constraint conditions indicated in the file to the generation unit 202, or the generation unit 202 may read the constraint conditions directly from the file, or both. In other words, any method of acquisition may be used as long as the generation unit 202 can acquire the necessary task goals and necessary constraint conditions.

[0023] The generation unit 202 generates a plan indicating the flow of operations of the robot 205 based on the task goal and constraints input by the task instruction device 201 .

[0024] For example, when the task goal and constraint conditions are input by the task instruction device 201, the generation unit 202 generates a first sequence that determines whether the cardboard box C has moved to the position of the robot 205, continues executing the determination until it is determined that the cardboard box C has moved, and stops the belt conveyor for moving the cardboard box C when it is determined that the cardboard box C has moved. The generation unit 202 also generates a second sequence that confirms whether the work performed by the first sequence has been completed. The first sequence corresponds to the first work described above. The second sequence is a sequence generated by the generation unit 202 in accordance with the first work, which is a predetermined unit of work for confirming completion of the work.

[0025] Furthermore, for example, the generation unit 202 generates a third sequence for moving the cardboard box C from the stop position of the belt conveyor onto the work table F. The generation unit 202 also generates a fourth sequence for confirming whether or not the work performed by the third sequence has been completed. The third sequence is a sequence corresponding to the second work described above. The fourth sequence is a sequence generated by the generation unit 202 in accordance with the second work, which is a predetermined unit of work for confirming the completion of the work.

[0026] Furthermore, for example, the generation unit 202 generates a fifth sequence for moving the product M in the cardboard box C on the work table F to the tray T. The generation unit 202 also generates a sixth sequence for confirming whether the work according to the fifth sequence has been completed. The fifth sequence is a sequence corresponding to the third work described above. The sixth sequence is also a sequence generated by the generation unit 202 in accordance with the third work, which is a predetermined unit of work for confirming the completion of the work.

[0027] More specifically, the generation unit 202 acquires, from the camera 203, two-dimensional and three-dimensional information (described later) of the origin of the product M indicated by the task target. The generation unit 202 can recognize the state (i.e., position and posture) of the product M at the origin of the movement from the two-dimensional and three-dimensional information acquired from the camera 203. The generation unit 202 generates, for example, by simulation, a movement path including the state of the product M from the state of the product M at the origin of the movement to the state of the product M at the destination of the movement. Information representing the movement path is information necessary for the controller 204 to generate a control signal for controlling the robot 205. The generation unit 202 then generates, for example, by simulation, information (i.e., a fifth sequence) representing each state of the robot 205 at each time step during the movement (such as the type (including the shape) of the product M, the position and posture of the robot 205, and the behavior of the robot 205 (such as the grip strength of the product M)).

[0028] Furthermore, for example, the generation unit 202 generates a seventh sequence that stops the belt conveyor for moving the tray T. The generation unit 202 also generates an eighth sequence that checks whether the work according to the seventh sequence has been completed. The seventh sequence is a sequence that corresponds to the above-mentioned fourth work. The eighth sequence is a sequence generated by the generation unit 202 in accordance with the fourth work, which is a predetermined work unit that checks whether the work has been completed.

[0029] Furthermore, for example, the generation unit 202 generates a ninth sequence for moving the tray T on a belt conveyor for moving the tray T. The generation unit 202 also generates a tenth sequence for confirming whether or not the work according to the ninth sequence has been completed. The ninth sequence is a sequence corresponding to the fifth work described above. The tenth sequence is a sequence generated by the generation unit 202 in accordance with the fifth work, which is a predetermined work unit for confirming the completion of the work.

[0030] Furthermore, for example, the generation unit 202 generates an eleventh sequence for moving a belt conveyor for moving trays T. The generation unit 202 also generates a twelfth sequence for confirming whether or not the work performed by the eleventh sequence has been completed. The eleventh sequence corresponds to the sixth work described above. The twelfth sequence is a sequence generated by the generation unit 202 in accordance with the sixth work, which is a predetermined unit of work for confirming the completion of the work. The generation unit 202 outputs the generated sequence to the controller 204 each time it generates each of the first to twelfth sequences.

[0031] 8 is a diagram illustrating an example of a sequence TBL1 of a plan generated by the generation unit 202 according to an embodiment of the present disclosure. For example, the sequence TBL1 of a plan generated by the generation unit 202 is a sequence indicating each state of the robot 205 for each n time step from the movement origin of the product M to the destination, as shown in FIG.

[0032] The camera 203 is provided at a position (for example, above the robot 205) that allows a bird's-eye view of the work performed by the robot system 20. The camera 203 includes a two-dimensional camera 2031 and a depth camera 2032.

[0033] The two-dimensional camera 2031 acquires two-dimensional information (i.e., a planar image) of the object being photographed, and the depth camera 2032 acquires three-dimensional information of the object being photographed, including depth information.

[0034] The controller 204 generates a control signal for controlling the robot 205 based on the sequence output by the generation unit 202. Note that the controller 204 may generate a control signal that optimizes an evaluation function when generating the control signal. Examples of the evaluation function include a function that represents the amount of energy consumed by the robot 205 when moving the product M, and a function that represents the distance along the path along which the product M is moved. The controller 204 outputs the generated control signal to the robot 205.

[0035] Furthermore, when the determination unit 206 determines that the work has been completed in determining whether the work to be performed for each predetermined unit of work in the plan has been completed, the controller 204 generates a control signal that causes the robot 205 to touch, for example, a touch pen 205 a (shown in FIG. 7 ) to a specific figure in the three-dimensional space of the work location identified by the identification unit 209. Then, the controller 204 outputs the generated control signal to the robot 205.

[0036] 9 is a diagram illustrating an example of the control signal Cnt of the plan generated by the controller 204 according to an embodiment of the present disclosure. For example, as shown in FIG. 9 , the control signal Cnt of the plan generated by the controller 204 is, for example, each control signal for n time steps from the movement origin of the product M to the destination.

[0037] The robot 205 operates in response to a control signal generated by the controller 204. For example, if the control signal is a control signal for moving the product M from the origin to the destination, the robot 205 moves the grasped product M from the origin to the destination in response to the control signal. In each embodiment of the present disclosure, "grasping" includes "suction" in which the object M is sucked by a vacuum or the like, and "clamping" in which an object is pinched between two or more pseudo-fingers that mimic the fingers of a human or animal. For example, if the control signal is a control signal for touching a specific figure in the three-dimensional space of the work area with the touch pen 205a, the robot 205 touches the specific figure with the touch pen 205a in response to the control signal. The touch pen 205a may be retractable within the robot 205 and may appear from its storage location when the specific figure is touched with the touch pen 205a.

[0038] The determination unit 206 determines whether or not each predetermined unit of work in the plan has been completed. For example, when the determination unit 206 makes a determination regarding the third work described above, in accordance with the sixth sequence, the determination unit 206 determines whether or not the product M is within a predetermined range based on the information about the product M acquired by the camera 203 at the destination. Then, for example, if the determination unit 206 determines that the product M is within the predetermined range based on the information about the product M acquired by the camera 203 at the destination, the determination unit 206 determines that the third work has been completed. Furthermore, for example, if the determination unit 206 determines that the product M is not within the predetermined range based on the information about the product M acquired by the camera 203 at the destination, the determination unit 206 determines that the work controlled by the plan has not been completed.

[0039] When the determination unit 206 determines that the work under the control of the plan has been completed, the controller 204 generates, for example, a control signal that causes the robot 205 to touch a specific figure corresponding to the completion of the work with the touch pen 205 a. Then, the controller 204 outputs the generated control signal to the robot 205.

[0040] Furthermore, when the determination unit 206 determines that the work under the control of the plan has not been completed, the controller 204 may generate a control signal to move the product M again so that it falls within a predetermined range, for example, based on information about the product M acquired at the destination by the camera 203. Then, the controller 204 may output the generated control signal to the robot 205.

[0041] If the determination unit 206 determines that the work under the control of the plan has not been completed, the controller 204 may, instead of the above-described processing, notify the worker that the work has not been completed, for example, by notifying the task instruction device 201, etc. Then, if the worker confirms that the work has not been completed, the worker may instruct the processing system 1 to move the product M so that it falls within a predetermined range, or the worker may move the product M into the predetermined range.

[0042] The storage unit 207 stores various information necessary for processing performed by the robot system 20. For example, the storage unit 207 stores files indicating constraint conditions, reference figures used by the recognition unit 208 to identify specific figures, etc. Furthermore, for example, the storage unit 207 stores a database DB1 similar to the database DB1 stored in the storage unit 102.

[0043] Recognition unit 208 identifies a specific graphic displayed on display unit 104 in the information (i.e., the planar image) acquired by camera 203. For example, recognition unit 208 compares information on each reference graphic stored in storage unit 207 with the information acquired by camera 203. Then, when the degree of match in the comparison result is equal to or greater than a predetermined degree, recognition unit 208 identifies the matching graphic and the position of the matching graphic in the information acquired by camera 203.

[0044] The identification unit 209 identifies the posture (position and orientation) of the figure in the three-dimensional space of the work area. For example, if the matching figure identified by the recognition unit 208 is an AR marker, the identification unit 209 identifies the posture of the figure in the three-dimensional space of the work area from the matching figure identified by the recognition unit 208. The AR marker includes three-dimensional information on the x, y, and z directions in the three-dimensional space of the work area, as well as orientation information corresponding to three pieces of information: roll, pitch, and yaw. The x direction is, for example, one direction contained in the floor plane of the work area. The y direction is, for example, a direction contained in the floor plane and perpendicular to the x direction. The z direction is, for example, a direction perpendicular to the floor plane. In other words, the AR marker includes six-dimensional information that can identify the posture. Therefore, if the matching figure identified by the recognition unit 208 is an AR marker, the identification unit 209 can identify the posture of the figure in the three-dimensional space of the work area using only the matching figure identified by the recognition unit 208.

[0045] Furthermore, for example, if the matching figure identified by the recognition unit 208 is a specific figure other than an AR marker, the identification unit 209 identifies the position of the matching figure in the information acquired by the camera 203. Then, the identification unit 209 may identify the posture (position and orientation) of the specific figure in the three-dimensional space of the workplace from information acquired by the camera 203 that corresponds to the information acquired by the camera 203 with respect to the position of the identified figure (i.e., the three-dimensional information of the display unit 104).

[0046] Note that the processing performed by the processing system 1 in each embodiment of the present disclosure is not limited to the above-described processing. For example, the processing system 1 may perform the processing described below.

[0047] FIG. 10 is a diagram illustrating an example of a processing flow of the processing system 1 according to an embodiment of the present disclosure. FIG. 11 is an image diagram illustrating the surrounding environment of the processing system 1 according to an embodiment of the present disclosure. Next, processing by the processing system 1, including processing for displaying a specific graphic, will be described with reference to FIG. 10 . It is assumed that the task instruction device 201 instructs the controller 204 to perform a sorting task, which is a desired task for the robot 205, in which the robot 205 transfers products M in a cardboard box C carried by a belt conveyor shown in FIG. 11 to a tray T, places the tray T on another belt conveyor, and moves it to a predetermined position. It is also assumed that the two-dimensional camera 2031 acquires two-dimensional information (i.e., a planar image) of the object being photographed. It is also assumed that the depth camera 2032 acquires three-dimensional information of the object being photographed, including depth information. As specific examples of processing performed by the processing system 1, the processing performed by the processing system 1 will be described using the fifth and sixth sequences corresponding to the third task described above. It is also assumed that the plan (i.e., the sequence) generated by the generation unit 202 is notified to the management system 10. Furthermore, when the generation unit 202 notifies the management system 10 of the plan, the control unit 103 causes the display unit 104 to display a notice inquiring about the completion of the work in response to the notification.

[0048] The task instruction device 201 instructs the controller 204 to execute a task as a desired task for the robot 205, which is to transfer products M in a cardboard box C carried by a belt conveyor onto a tray T, place the tray T on another belt conveyor, and move the tray T to a predetermined position (step S1). For example, the task instruction device 201 outputs, as task targets, information indicating the type of products M, the number of products M to be moved, the origin of the products M, and the destination of the products M, as well as information indicating a predetermined task unit for confirming completion of the task, to the generation unit 202. In addition, for example, the task instruction device 201 outputs, as constraint conditions, no-entry areas when moving the products M, areas that deviate from the range of motion of the robot 205, and further conditions for the surface of the products M regarding gripping the products M, releasing gripping of the products M, or changing the products M, to the generation unit 202.

[0049] The generation unit 202 generates a plan indicating the flow of operations of the robot 205 based on the task goal and constraints input by the task instruction device 201 .

[0050] For example, when the robot 205 has completed the task of moving a cardboard box C from the stop position of the belt conveyor onto the work table F, the generation unit 202 generates a fifth sequence for moving the product M in the cardboard box C on the work table F to the tray T (step S2). The generation unit 202 also generates a sixth sequence for confirming whether the task according to the fifth sequence has been completed. The generation unit 202 notifies the management system 10 that the fifth sequence has been generated (step S3). In response to the notification, the control unit 103 causes the display unit 104 to display a notification inquiring about the completion of the task (for example, the notification shown in FIG. 4 ) (step S4).

[0051] The generation unit 202 outputs the generated fifth sequence to the controller 204. Based on the fifth sequence output by the generation unit 202, the controller 204 generates a control signal for controlling the robot 205 to move the product M in the cardboard box C on the work table F to the tray T (step S5).

[0052] The controller 204 outputs the generated control signal to the robot 205. In response to the control signal output by the controller 204, the robot 205 moves the product M in the cardboard box C on the work table F to the tray T (step S6). The generation unit 202 generates a sixth sequence for checking whether the work according to the fifth sequence has been completed (step S7). The determination unit 206 determines whether the work according to the fifth sequence has been completed in accordance with the sixth sequence generated by the generation unit 202 (step S8).

[0053] If the determination unit 206 determines that the work according to the fifth sequence has not been completed (NO in step S8), the controller 204 generates a control signal for moving the product M again so that it fits within a predetermined range, for example, based on information about the product M acquired at the destination by the camera 203 (step S9). The controller 204 then outputs the generated control signal to the robot 205. The robot 205 then moves the product M so that it fits within the predetermined range (step S10). The robot 205 then returns to the processing of step S8.

[0054] Furthermore, if the determination unit 206 determines that the work according to the fifth sequence has been completed (YES in step S8), the recognition unit 208 identifies a specific figure associated with the notification of the work determined by the determination unit 206 to have been completed. Then, the recognition unit 208 identifies a figure that matches the identified specific figure from among the specific figures displayed on the display unit 104 in the information (i.e., the planar image) acquired by the camera 203 (step S11). The identification unit 209 identifies a specific figure that matches the specific figure identified by the recognition unit 208 in the database DB1 stored in the storage unit 207 (step S12). Then, the identification unit 209 identifies the posture (position and orientation) of the identified specific figure in the three-dimensional space of the work location (step S13).

[0055] The controller 204 generates a control signal for touching a specific figure with the stylus 205a according to the posture identified by the identification unit 209 (step S14). The controller 204 then outputs the generated control signal to the robot 205. The robot 205 touches a specific figure corresponding to the completion of the task with the stylus 205a (step S15). Because the robot 205 operates the location where the specific figure is displayed, the process of searching for the location to operate can be reduced. The process of step S15 notifies the management system 10 that the task of moving the product M from the cardboard box C to the tray T has been completed. The management unit 101 rewrites the flag for the task of moving the product M from the cardboard box C to the tray T stored in the memory unit 102 from 0 (unprocessed) to 1 (processed) (step S16). The process of step S16 enables the management system 10 to manage the progress of each task.

[0056] (Advantages) The processing system 1 according to the first embodiment of the present disclosure has been described above. In the robot system 20 of the processing system 1, the touch pen 205a (an example of an operation means) operates the display unit 104 (an example of an interface) that displays information about the system in which the robot 205 operates. The controller 204 (an example of a control means) identifies a specific graphic (an example of information about the operation of the robot 205) on the display unit 104 and controls the operation of the robot 205.

[0057] This allows the robot to operate the interface and act according to the information about the actions displayed in the interface, resulting in proper interaction between the robot and the software that causes the interface to display information about the system in which the robot operates.

[0058] Second Embodiment A processing system 1 according to a second embodiment of the present disclosure will be described. The processing system 1 is a system in which, instead of the robot 205 according to the first embodiment of the present disclosure touching a specific figure on the display unit 104 with the stylus 205 a, the robot 205 performs an operation of emitting a sound, thereby performing a predetermined task similar to the case where the robot 205 performs a touching operation.

[0059] The processing system 1 includes a management system 10 and a robot system 20. Fig. 12 is a diagram showing an example of the configuration of the management system 10 according to an embodiment of the present disclosure. As shown in Fig. 12, the management system 10 includes a management unit 101, a storage unit 102, a control unit 103, and a display unit 104. Furthermore, as shown in Fig. 12, the management system 10 includes a sound collector 105.

[0060] The management unit 101 writes the status of each task in the storage unit 102. Specifically, for example, the management unit 101 determines whether the sound collected by the sound collector 105 is a 440 Hz sound, which indicates that the task has been completed, or a 460 Hz sound, which indicates that the task has not been completed. If the management unit 101 determines that the sound is a 440 Hz sound, it determines that the task in question has been completed. Then, the management unit 101 writes a completion flag, which indicates that the task has been completed, in association with the task determined to have been completed in the storage unit 102.

[0061] The memory unit 102 stores various information necessary for the processing performed by the management system 10. For example, the memory unit 102 stores the status of each task. Specifically, for example, the memory unit 102 stores a flag of 0 (incomplete) or 1 (completed) associated with the task. Furthermore, for example, the memory unit 102 stores information that the control unit 103 causes the display unit 104 to display according to the status of each task. FIG. 13 is a diagram illustrating an example of a database DB2 stored in the memory unit 102 according to an embodiment of the present disclosure. The information that the control unit 103 causes the display unit 104 to display according to the status of each task is a sound frequency, which is information according to the status of each task, information to be notified to the worker that corresponds one-to-one to the sound frequency, and a specific graphic that corresponds one-to-one to the sound frequency. As shown in FIG. 13 , the memory unit 102 stores the sound frequency, which is information according to the status of each task, the information to be notified to the worker, and the specific graphic in a one-to-one association as the database DB2.

[0062] The control unit 103 controls the display of the display unit 104. For example, the control unit 103 causes the display unit 104 to display the sound frequency, which is information corresponding to the status of each task stored in the memory unit 102, the information to be notified to the worker that is stored in association with the sound frequency, and a specific figure that is stored in association with the sound frequency.

[0063] 14 is a diagram illustrating an example of a display displayed by the display unit 104 according to an embodiment of the present disclosure. As shown in FIG. 14 , under the control of the control unit 103, the display unit 104 displays the frequency of a sound, which is information according to the status of each task, information to be notified to the worker, and a specific graphic.

[0064] The sound collector 105 collects sounds output by a speaker 210 (described later) provided in the robot system 20. The sound collector 105 converts the collected sounds into electrical signals that indicate the frequency of the sounds. The sound collector 105 outputs the electrical signals to the management unit 101.

[0065] Fig. 15 is a diagram illustrating an example of the configuration of a robot system 20 according to an embodiment of the present disclosure. As shown in Fig. 15, the robot system 20 includes a task instruction device 201, a generation unit 202, a camera 203, a controller 204, a robot 205, a determination unit 206, a storage unit 207, and a recognition unit 208. The robot system 20 also includes a speaker 210 as shown in Fig. 15.

[0066] The task instruction device 201 instructs the controller 204 on a task for causing the robot 205 to perform a desired task. For example, the task instruction device 201 instructs the controller 204 on a task via a network (or, in some cases, by using a cloud service).

[0067] For example, if the desired task is to move product M in a cardboard box C to the destination tray T, determine whether the task of moving product M to tray T is completed, and if it is determined that the task is completed, perform a sorting task of moving the tray T to a predetermined position on a conveyor belt, the task instruction device 201 inputs the task goal and constraint conditions to the generation unit 202 as a procedure for instructing the task.

[0068] The generation unit 202 generates a plan indicating the flow of operations of the robot 205 based on the task goal and constraint conditions input by the task instruction device 201. Then, every time the generation unit 202 generates each plan (i.e., sequence), it outputs the generated sequence to the controller 204.

[0069] The camera 203 is provided at a position (for example, above the robot 205) that allows a bird's-eye view of the work performed by the robot system 20. The camera 203 includes a two-dimensional camera 2031 and a depth camera 2032.

[0070] The two-dimensional camera 2031 acquires two-dimensional information (i.e., a planar image) of the object being photographed, and the depth camera 2032 acquires three-dimensional information of the object being photographed, including depth information.

[0071] The controller 204 generates a control signal for controlling the robot 205 based on the sequence output by the generation unit 202. Note that the controller 204 may generate a control signal that optimizes an evaluation function when generating the control signal. Examples of the evaluation function include a function that represents the amount of energy consumed by the robot 205 when moving the product M, and a function that represents the distance along the path along which the product M is moved. The controller 204 outputs the generated control signal to the robot 205.

[0072] Furthermore, when the determination unit 206 determines that the work is completed in determining whether the work to be performed for each predetermined unit of work in the plan has been completed, the controller 204 generates a sound signal that causes the speaker 210 to output a sound of a frequency that corresponds to the specific figure identified by the recognition unit 208. Then, the controller 204 causes the speaker 210 to output a sound that corresponds to the generated sound signal.

[0073] The robot 205 operates in response to a control signal generated by the controller 204. For example, if the control signal is a control signal for moving the product M from the origin to the destination, the robot 205 moves the product M that it is holding from the origin to the destination in response to the control signal.

[0074] The determination unit 206 determines whether or not each predetermined unit of work in the plan has been completed. For example, when the determination unit 206 makes a determination regarding the third work described above, in accordance with the sixth sequence, the determination unit 206 determines whether or not the product M is within a predetermined range based on the information about the product M acquired by the camera 203 at the destination. Then, for example, if the determination unit 206 determines that the product M is within the predetermined range based on the information about the product M acquired by the camera 203 at the destination, the determination unit 206 determines that the third work has been completed. Furthermore, for example, if the determination unit 206 determines that the product M is not within the predetermined range based on the information about the product M acquired by the camera 203 at the destination, the determination unit 206 determines that the work controlled by the plan has not been completed.

[0075] When the determination unit 206 determines that the work controlled by the plan has been completed, the controller 204 generates, for example, a sound signal that causes the speaker 210 to output a sound of a frequency corresponding to the completion of the work. Then, the controller 204 causes the speaker 210 to output a sound corresponding to the generated sound signal.

[0076] Furthermore, when the determination unit 206 determines that the work under the control of the plan has not been completed, the controller 204 may generate a control signal to move the product M again so that it falls within a predetermined range, for example, based on information about the product M acquired at the destination by the camera 203. Then, the controller 204 may output the generated control signal to the robot 205.

[0077] If the determination unit 206 determines that the work under the control of the plan has not been completed, the controller 204 may, instead of the above-described processing, notify the worker that the work has not been completed, for example, by notifying the task instruction device 201, etc. Then, if the worker confirms that the work has not been completed, the worker may instruct the processing system 1 to move the product M so that it falls within a predetermined range, or the worker may move the product M into the predetermined range.

[0078] The storage unit 207 stores various information necessary for processing performed by the robot system 20. For example, the storage unit 207 stores files indicating constraint conditions, reference figures used by the recognition unit 208 to identify specific figures, etc. Furthermore, for example, the storage unit 207 stores a database DB2 similar to the database DB2 stored in the storage unit 102.

[0079] The recognition unit 208 identifies a specific graphic to be displayed on the display unit 104 in the information (i.e., the planar image) acquired by the camera 203. For example, the recognition unit 208 compares the information of each reference graphic stored in the storage unit 207 with the information acquired by the camera 203. Then, when the degree of match in the comparison result is equal to or greater than a predetermined degree, the recognition unit 208 identifies the matching graphic and the position of the matching graphic in the information acquired by the camera 203. The speaker 210 outputs a sound corresponding to the sound signal generated by the controller 204.

[0080] Note that the processing performed by the processing system 1 in each embodiment of the present disclosure is not limited to the above-described processing. For example, the processing system 1 may perform the processing described below.

[0081] FIG. 16 is a diagram illustrating an example of a processing flow of the processing system 1 according to an embodiment of the present disclosure. Next, processing by the processing system 1, including processing for outputting sound, will be described with reference to FIG. 16 . Similar to the first embodiment of the present disclosure, the task instruction device 201 instructs the controller 204 to perform a sorting task, which involves transferring products M in a cardboard box C carried by a conveyor belt onto a tray T, placing the tray T on another conveyor belt, and moving it to a predetermined position. The two-dimensional camera 2031 is assumed to acquire two-dimensional information (i.e., a planar image) of the object being photographed. The depth camera 2032 is assumed to acquire three-dimensional information of the object being photographed, including depth information. Similar to the first embodiment of the present disclosure, specific examples of processing performed by the processing system 1 will be described using the fifth and sixth sequences corresponding to the third task. The plan (i.e., the sequence) generated by the generation unit 202 is assumed to be notified to the management system 10. Furthermore, when the generation unit 202 notifies the management system 10 of the plan, the control unit 103 causes the display unit 104 to display a notice inquiring about the completion of the work in response to the notification.

[0082] The task instruction device 201 instructs the controller 204 to execute a task as a desired task for the robot 205, which is to transfer products M in a cardboard box C carried by a belt conveyor onto a tray T, place the tray T on another belt conveyor, and move the tray T to a predetermined position (step S1). For example, the task instruction device 201 outputs, as task targets, information indicating the type of products M, the number of products M to be moved, the origin of the products M, and the destination of the products M, as well as information indicating a predetermined task unit for confirming completion of the task, to the generation unit 202. In addition, for example, the task instruction device 201 outputs, as constraint conditions, no-entry areas when moving the products M, areas that deviate from the range of motion of the robot 205, and further conditions for the surface of the products M regarding gripping the products M, releasing gripping of the products M, or changing the products M, to the generation unit 202.

[0083] The generation unit 202 generates a plan indicating the flow of operations of the robot 205 based on the task goal and constraints input by the task instruction device 201 .

[0084] For example, when the robot 205 has completed the task of moving a cardboard box C from the stop position of the belt conveyor onto the work table F, the generation unit 202 generates a fifth sequence for moving the product M in the cardboard box C on the work table F to the tray T (step S2). The generation unit 202 also generates a sixth sequence for confirming whether the task according to the fifth sequence has been completed. The generation unit 202 notifies the management system 10 that the fifth sequence has been generated (step S3). In response to the notification, the control unit 103 causes the display unit 104 to display a notification inquiring about the completion of the task (for example, the notification shown in FIG. 4 ) (step S4).

[0085] The generation unit 202 outputs the generated fifth sequence to the controller 204. Based on the fifth sequence output by the generation unit 202, the controller 204 generates a control signal for controlling the robot 205 to move the product M in the cardboard box C on the work table F to the tray T (step S5).

[0086] The controller 204 outputs the generated control signal to the robot 205. In response to the control signal output by the controller 204, the robot 205 moves the product M in the cardboard box C on the work table F to the tray T (step S6). The generation unit 202 generates a sixth sequence for checking whether the work according to the fifth sequence has been completed (step S7). The determination unit 206 determines whether the work according to the fifth sequence has been completed in accordance with the sixth sequence generated by the generation unit 202 (step S8).

[0087] If the determination unit 206 determines that the work according to the fifth sequence has not been completed (NO in step S8), the controller 204 generates a control signal for moving the product M again so that it fits within a predetermined range, for example, based on information about the product M acquired at the destination by the camera 203 (step S9). The controller 204 then outputs the generated control signal to the robot 205. The robot 205 then moves the product M so that it fits within the predetermined range (step S10). The robot 205 then returns to the processing of step S8.

[0088] Furthermore, when the determination unit 206 determines that the work according to the fifth sequence has been completed (YES in step S8), the recognition unit 208 identifies a specific graphic associated with the notification of the work determined to have been completed by the determination unit 206. Then, the recognition unit 208 identifies a graphic that matches the identified specific graphic from among the specific graphics displayed on the display unit 104 in the information acquired by the camera 203 (i.e., the planar image) (step S11).

[0089] The controller 204 generates a sound signal that causes the speaker 210 to output a sound of a frequency that corresponds to the specific figure identified by the recognition unit 208 (step S21). The controller 204 then causes the speaker 210 to output a sound that corresponds to the generated sound signal. The speaker 210 outputs a sound that corresponds to the sound signal (step S22). By processing this step S22, the management system 10 is notified that the task of moving the products M in the bowl C to the tray T has been completed.

[0090] The sound collector 105 collects sound output by the speaker 210 provided in the robot system 20 (step S23). The sound collector 105 converts the collected sound into an electrical signal indicating the frequency of the sound (step S24). The sound collector 105 outputs the electrical signal to the management unit 101. The management unit 101 rewrites the flag for the task of moving the product M in the bowl C to the tray T, which is stored in the memory unit 102, from 0 (unprocessed) to 1 (processed) in accordance with the frequency indicated by the electrical signal output by the sound collector 105 (step S25). The processing of step S25 enables the management system 10 to manage the progress of each task.

[0091] (Advantages) The processing system 1 according to the second embodiment of the present disclosure has been described above. In the robot 205 of the processing system 1, the speaker 210 (an example of a sound source) operates (indirectly via the sound collector 105) the display unit 104 (an example of an interface) that displays information about the management system 10 in which the robot 205 operates. The controller 204 (an example of a control means) identifies information about the operation of the robot 205 on the display unit 104 and controls the operation of the robot 205.

[0092] This allows the robot to operate the interface and act according to the operational information displayed in the interface. The robot can then interact appropriately with the software that causes the interface to display information about the system in which the robot operates.

[0093] Third Embodiment A processing system 1 according to a third embodiment of the present disclosure will be described. The processing system 1 is a system in which, instead of performing a predetermined task similar to that performed when a touch operation is performed by the robot 205 according to the second embodiment of the present disclosure by performing an operation to emit sound, the robot 205 performs a predetermined task similar to that performed when a sound operation is performed by performing an operation to emit light.

[0094] The processing system 1 includes a management system 10 and a robot system 20. Fig. 17 is a diagram showing an example of the configuration of the management system 10 according to an embodiment of the present disclosure. As shown in Fig. 17, the management system 10 includes a management unit 101, a storage unit 102, a control unit 103, and a display unit 104. Furthermore, as shown in Fig. 17, the management system 10 includes a condenser 106.

[0095] The management unit 101 writes the status of each task in the storage unit 102. Specifically, for example, the management unit 101 determines whether the light collected by the concentrator 106 is light with a wavelength of 440 nm, which indicates that the task has been completed, or light with a wavelength of 460 nm, which indicates that the task has not been completed. If the management unit 101 determines that the light has a wavelength of 440 nm, it determines that the task in question has been completed. Then, the management unit 101 writes a completion flag, which indicates that the task has been completed, into the storage unit 102, in association with the task that has been determined to have been completed.

[0096] The memory unit 102 stores various information necessary for the processing performed by the management system 10. For example, the memory unit 102 stores the status of each task. Specifically, for example, the memory unit 102 stores a flag of 0 (incomplete) or 1 (completed) associated with each task. Furthermore, for example, the memory unit 102 stores information that the control unit 103 causes the display unit 104 to display according to the status of each task. FIG. 18 is a diagram illustrating an example of a database DB3 stored in the memory unit 102 according to an embodiment of the present disclosure. The information that the control unit 103 causes the display unit 104 to display according to the status of each task includes wavelengths of light corresponding to the status of each task, information to be notified to the worker that corresponds one-to-one to the wavelengths of light, and a specific graphic that corresponds one-to-one to the wavelengths of light. As shown in FIG. 18 , the memory unit 102 stores the wavelengths of light corresponding to the status of each task, the information to be notified to the worker, and the specific graphic in a one-to-one association as the database DB3.

[0097] The control unit 103 controls the display of the display unit 104. For example, the control unit 103 causes the display unit 104 to display the wavelength of light, which is information corresponding to the status of each task stored in the storage unit 102, the information to be notified to the worker that is stored in association with the wavelength of light, and a specific figure that is stored in association with the wavelength of light.

[0098] 19 is a diagram illustrating an example of a display displayed by the display unit 104 according to an embodiment of the present disclosure. As shown in FIG. 19 , under the control of the control unit 103, the display unit 104 displays the wavelength of light, which is information according to the status of each task, information to be notified to the worker, and a specific graphic.

[0099] The condenser 106 condenses light output from a light source 211 (described later) included in the robot system 20. The condenser 106 converts the condensed light into an electrical signal indicating the wavelength of the light. The condenser 106 outputs the electrical signal to the management unit 101.

[0100] Fig. 20 is a diagram illustrating an example of the configuration of a robot system 20 according to an embodiment of the present disclosure. As shown in Fig. 20, the robot system 20 includes a task instruction device 201, a generation unit 202, a camera 203, a controller 204, a robot 205, a determination unit 206, a storage unit 207, and a recognition unit 208. The robot system 20 also includes a light source 211 as shown in Fig. 20.

[0101] The task instruction device 201 instructs the controller 204 on a task for causing the robot 205 to perform a desired task. For example, the task instruction device 201 instructs the controller 204 on a task via a network (or, in some cases, by using a cloud service).

[0102] For example, if the desired task is to move product M in a cardboard box C to the destination tray T, determine whether the task of moving product M to tray T is completed, and if it is determined that the task is completed, perform a sorting task of moving the tray T to a predetermined position on a conveyor belt, the task instruction device 201 inputs the task goal and constraint conditions to the generation unit 202 as a procedure for instructing the task.

[0103] The generation unit 202 generates a plan indicating the flow of operations of the robot 205 based on the task goal and constraint conditions input by the task instruction device 201. Then, every time the generation unit 202 generates each plan (i.e., sequence), it outputs the generated sequence to the controller 204.

[0104] The camera 203 is provided at a position (for example, above the robot 205) that allows a bird's-eye view of the work performed by the robot system 20. The camera 203 includes a two-dimensional camera 2031 and a depth camera 2032.

[0105] The two-dimensional camera 2031 acquires two-dimensional information (i.e., a planar image) of the object being photographed, and the depth camera 2032 acquires three-dimensional information of the object being photographed, including depth information.

[0106] The controller 204 generates a control signal for controlling the robot 205 based on the sequence output by the generation unit 202. Note that the controller 204 may generate a control signal that optimizes an evaluation function when generating the control signal. Examples of the evaluation function include a function that represents the amount of energy consumed by the robot 205 when moving the product M, and a function that represents the distance along the path along which the product M is moved. The controller 204 outputs the generated control signal to the robot 205.

[0107] Furthermore, when the determination unit 206 determines that the work is completed in determining whether the work performed for each predetermined unit of work in the plan is completed, the controller 204 generates a light source signal that causes the light source 211 to output light of a wavelength that corresponds to the specific figure identified by the recognition unit 208. Then, the controller 204 causes the light source 211 to output light that corresponds to the generated light source signal.

[0108] The robot 205 operates in response to a control signal generated by the controller 204. For example, if the control signal is a control signal for moving the product M from the origin to the destination, the robot 205 moves the product M that it is holding from the origin to the destination in response to the control signal.

[0109] The determination unit 206 determines whether or not each predetermined unit of work in the plan has been completed. For example, when the determination unit 206 makes a determination regarding the third work described above, in accordance with the sixth sequence, the determination unit 206 determines whether or not the product M is within a predetermined range based on the information about the product M acquired by the camera 203 at the destination. Then, for example, if the determination unit 206 determines that the product M is within the predetermined range based on the information about the product M acquired by the camera 203 at the destination, the determination unit 206 determines that the third work has been completed. Furthermore, for example, if the determination unit 206 determines that the product M is not within the predetermined range based on the information about the product M acquired by the camera 203 at the destination, the determination unit 206 determines that the work controlled by the plan has not been completed.

[0110] When the determination unit 206 determines that the work under the control of the plan has been completed, the controller 204 generates, for example, a light source signal that causes the light source 211 to output light of a wavelength corresponding to the completion of the work. Then, the controller 204 causes the light source 211 to output light corresponding to the generated light source signal.

[0111] Furthermore, when the determination unit 206 determines that the work under the control of the plan has not been completed, the controller 204 may generate a control signal to move the product M again so that it falls within a predetermined range, for example, based on information about the product M acquired at the destination by the camera 203. Then, the controller 204 may output the generated control signal to the robot 205.

[0112] If the determination unit 206 determines that the work under the control of the plan has not been completed, the controller 204 may, instead of the above-described processing, notify the worker that the work has not been completed, for example, by notifying the task instruction device 201, etc. Then, if the worker confirms that the work has not been completed, the worker may instruct the processing system 1 to move the product M so that it falls within a predetermined range, or the worker may move the product M into the predetermined range.

[0113] The storage unit 207 stores various information necessary for processing performed by the robot system 20. For example, the storage unit 207 stores files indicating constraint conditions, reference figures used by the recognition unit 208 to identify specific figures, etc. Furthermore, for example, the storage unit 207 stores a database DB3 similar to the database DB3 stored in the storage unit 102.

[0114] The recognition unit 208 identifies a specific graphic to be displayed on the display unit 104 in the information (i.e., the planar image) acquired by the camera 203. For example, the recognition unit 208 compares the information of each reference graphic stored in the storage unit 207 with the information acquired by the camera 203. Then, when the degree of match in the comparison result is equal to or greater than a predetermined degree, the recognition unit 208 identifies the matching graphic and the position of the matching graphic in the information acquired by the camera 203. The light source 211 outputs light corresponding to the light source signal generated by the controller 204.

[0115] Note that the processing performed by the processing system 1 in each embodiment of the present disclosure is not limited to the above-described processing. For example, the processing system 1 may perform the processing described below.

[0116] FIG. 21 is a diagram illustrating an example of a processing flow of the processing system 1 according to an embodiment of the present disclosure. Next, processing by the processing system 1, including processing for outputting light, will be described with reference to FIG. 21 . Similar to the first embodiment of the present disclosure, the task instruction device 201 instructs the controller 204 to perform a sorting task, which involves transferring products M in a cardboard box C carried by a conveyor belt onto a tray T, placing the tray T on another conveyor belt, and moving it to a predetermined position. The two-dimensional camera 2031 is also assumed to acquire two-dimensional information (i.e., a planar image) of the object being photographed. The depth camera 2032 is also assumed to acquire three-dimensional information of the object being photographed, including depth information. Similar to the first embodiment of the present disclosure, specific examples of processing performed by the processing system 1 will be described using the fifth and sixth sequences corresponding to the third task. The plan (i.e., the sequence) generated by the generation unit 202 is also assumed to be notified to the management system 10. Furthermore, when the generation unit 202 notifies the management system 10 of the plan, the control unit 103 causes the display unit 104 to display a notice inquiring about the completion of the work in response to the notification.

[0117] The task instruction device 201 instructs the controller 204 to execute a task as a desired task for the robot 205, which is to transfer products M in a cardboard box C carried by a belt conveyor onto a tray T, place the tray T on another belt conveyor, and move the tray T to a predetermined position (step S1). For example, the task instruction device 201 outputs, as task targets, information indicating the type of products M, the number of products M to be moved, the origin of the products M, and the destination of the products M, as well as information indicating a predetermined task unit for confirming completion of the task, to the generation unit 202. In addition, for example, the task instruction device 201 outputs, as constraint conditions, no-entry areas when moving the products M, areas that deviate from the range of motion of the robot 205, and further conditions for the surface of the products M regarding gripping the products M, releasing gripping of the products M, or changing the products M, to the generation unit 202.

[0118] The generation unit 202 generates a plan indicating the flow of operations of the robot 205 based on the task goal and constraints input by the task instruction device 201 .

[0119] For example, when the robot 205 has completed the task of moving a cardboard box C from the stop position of the belt conveyor onto the work table F, the generation unit 202 generates a fifth sequence for moving the product M in the cardboard box C on the work table F to the tray T (step S2). The generation unit 202 also generates a sixth sequence for confirming whether the task according to the fifth sequence has been completed. The generation unit 202 notifies the management system 10 that the fifth sequence has been generated (step S3). In response to the notification, the control unit 103 causes the display unit 104 to display a notification inquiring about the completion of the task (for example, the notification shown in FIG. 4 ) (step S4).

[0120] The generation unit 202 outputs the generated fifth sequence to the controller 204. Based on the fifth sequence output by the generation unit 202, the controller 204 generates a control signal for controlling the robot 205 to move the product M in the cardboard box C on the work table F to the tray T (step S5).

[0121] The controller 204 outputs the generated control signal to the robot 205. In response to the control signal output by the controller 204, the robot 205 moves the product M in the cardboard box C on the work table F to the tray T (step S6). The generation unit 202 generates a sixth sequence for checking whether the work according to the fifth sequence has been completed (step S7). The determination unit 206 determines whether the work according to the fifth sequence has been completed in accordance with the sixth sequence generated by the generation unit 202 (step S8).

[0122] If the determination unit 206 determines that the work according to the fifth sequence has not been completed (NO in step S8), the controller 204 generates a control signal for moving the product M again so that it fits within a predetermined range, for example, based on information about the product M acquired at the destination by the camera 203 (step S9). The controller 204 then outputs the generated control signal to the robot 205. The robot 205 then moves the product M so that it fits within the predetermined range (step S10). The robot 205 then returns to the processing of step S8.

[0123] Furthermore, when the determination unit 206 determines that the work according to the fifth sequence has been completed (YES in step S8), the recognition unit 208 identifies a specific graphic associated with the notification of the work determined to have been completed by the determination unit 206. Then, the recognition unit 208 identifies a graphic that matches the identified specific graphic from among the specific graphics displayed on the display unit 104 in the information acquired by the camera 203 (i.e., the planar image) (step S11).

[0124] The controller 204 generates a light source signal that causes the light source 211 to output light of a wavelength that corresponds to the specific figure identified by the recognition unit 208 (step S31). Then, the controller 204 causes the light source 211 to output light that corresponds to the generated light source signal. The light source 211 outputs light that corresponds to the light source signal (step S32). By processing this step S32, the management system 10 is notified that the task of moving the products M in the bowl C to the tray T has been completed.

[0125] The condenser 106 condenses the light output by the light source 211 included in the robot system 20 (step S33). The condenser 106 converts the condensed light into an electrical signal indicating the wavelength of the light (step S34). The condenser 106 outputs the electrical signal to the management unit 101. In response to the electrical signal output by the condenser 106, the management unit 101 rewrites the flag stored in the memory unit 102 for the task of moving the product M in the bowl C to the tray T from 0 (unprocessed) to 1 (processed) (step S35). The processing of step S35 enables the management system 10 to manage the progress of each task.

[0126] (Advantages) The processing system 1 according to the third embodiment of the present disclosure has been described above. In the robot 205 of the processing system 1, the light source 211 (an example of a light source) operates (indirectly via the condenser 106) the display unit 104 (an example of an interface) that displays information about the management system 10 in which the robot 205 operates. The controller 204 (an example of a control means) identifies information about the operation of the robot 205 on the display unit 104 and controls the operation of the robot 205.

[0127] This allows the robot to operate the interface and act according to the operational information displayed in the interface. The robot can then interact appropriately with the software that causes the interface to display information about the system in which the robot operates.

[0128] In the first to third embodiments of the present disclosure, the robot system 20 has been described as being configured such that the task instruction device 201 simply instructs a task to be performed, the generation unit 202 generates a sequence, the controller 204 generates a control signal according to the sequence, and the robot 205 is controlled by the control signal. However, in another embodiment of the present disclosure, the task instruction device 201 may instruct a task to be performed for each arbitrary task unit, for example, for each predetermined task unit described in the first embodiment of the present disclosure. In other words, in the robot system 20 according to another embodiment of the present disclosure, a task may be instructed for each arbitrary task unit, the generation unit 202 may generate a sequence for each instruction, the controller 204 may generate a control signal according to the sequence, and the robot 205 may be controlled by the control signal. The arbitrary task unit may also include stopping.

[0129] In the first embodiment of the present disclosure, the display unit 104 has been described as a display with a touch panel function. However, in another embodiment of the present disclosure, the display unit 104 is not limited to a display with a touch panel function. For example, the display unit 104 may be a non-contact display that responds to proximity, and may respond when a touch pen 205 a or the like is brought close to the display.

[0130] In the first embodiment of the present disclosure, the robot 205 is described as responding with a specific graphic using the touch pen 205a. However, in another embodiment of the present disclosure, the robot 205 may respond with a specific graphic using an object other than the touch pen 205a, within the range in which the display unit 104 responds. Examples of the object other than the touch pen 205a include an object made of a conductive material and a rod-shaped object 205b (shown in FIG. 7 ).

[0131] Furthermore, in the first to third embodiments of the present disclosure, the orientation of a specific figure displayed on the display unit 104 has been described as being identified based on information acquired by the camera 203 including a two-dimensional camera 2031 and a depth camera 2032. However, depending on the installation location of the camera 203, it may be impossible to identify the orientation of a specific figure displayed on the display unit 104 based solely on the information acquired by the camera 203. Therefore, the camera 203 according to another embodiment of the present disclosure is not limited to a camera including a two-dimensional camera 2031 and a depth camera 2032. FIG. 22 is an image diagram of the surrounding environment of the processing system 1 according to an embodiment of the present disclosure. The camera 203 according to another embodiment of the present disclosure may include a two-dimensional camera 2031 and a depth camera 2032, and may further include a two-dimensional camera 2033 located in a position that makes it easy to photograph the display unit 104, as shown in FIG. 22.

[0132] FIG. 23 is a diagram illustrating an example of a configuration of a management system 10 according to an embodiment of the present disclosure. FIG. 24 is a diagram illustrating an example of a configuration of a robot system 20 according to an embodiment of the present disclosure. In an embodiment of the present disclosure, the management system 10 may include a display unit 104, a sound collector 105, and a light collector 106 as shown in FIG. 23 . Furthermore, the robot system 20 may include an identification unit 209, a speaker 210, and a light source 211 as shown in FIG. 24 . The processing system 1 may select and implement a response by having the robot 205 touch a specific figure displayed on the display unit 104 with a touch pen 205 a or a rod-shaped object 205 b as described in the first embodiment, a response by outputting a sound from the speaker 210 to the robot 205 as described in the second embodiment, or a response by outputting light from the light source 211 to the robot 205 as described in the third embodiment.

[0133] FIG. 25 is a diagram illustrating an example of a sound source 501 and a light source 502 that can be used by a worker in an embodiment of the present disclosure. In the first to third embodiments of the present disclosure, the worker responds by touching a button displayed on the display unit 104. However, the worker may respond with the sound source 501 or the light source 502 depending on the configuration of the management system 10. For example, the sound source 501 has a function of setting the frequency of a sound and outputs a sound of the set frequency when a predetermined button is pressed. Furthermore, for example, the light source 502 has a function of setting the wavelength of light and outputs light of the set wavelength when a predetermined button is pressed.

[0134] Next, a robot 300 with a minimum configuration according to an embodiment of the present disclosure will be described. Fig. 26 is a diagram showing an example of a robot 300 with a minimum configuration according to an embodiment of the present disclosure. The robot 300 with a minimum configuration includes an operation means 301 and a control means 302. The operation means 301 operates an interface on which information about a system in which the robot 300 operates is displayed. The control means 302 identifies information related to the operation of the robot 300 in the interface and controls the operation of the robot 300.

[0135] The operation means 301 can be realized, for example, by using the functions of the touch pen 205a and rod-shaped object 205b illustrated in Fig. 7, the speaker 210 illustrated in Fig. 15, and the light source 211 illustrated in Fig. 20. The control means 302 can be realized, for example, by using the functions of the controller 204 illustrated in Fig. 7, 15, and 20.

[0136] Next, a description will be given of processing by the robot 300 with the minimum configuration. Fig. 27 is a diagram showing an example of a processing flow of the robot 300 with the minimum configuration according to an embodiment of the present disclosure. Here, processing by the robot 300 with the minimum configuration will be described with reference to Fig. 27.

[0137] In the robot 300, the operation means 301 operates an interface that displays information about the system in which the robot 300 operates (step S101). The control means 302 identifies information about the operation of the robot 300 in the interface and controls the operation of the robot 300 (step S102).

[0138] The foregoing describes a minimal configuration of the robot 300 according to one embodiment of the present disclosure, which allows the robot 300 to properly interact with software that displays information about the system in which the robot operates in an interface when the robot and a worker collaborate and perform tasks on behalf of the worker.

[0139] In each of the above-described embodiments of the present disclosure, "operating an interface" is not limited to directly operating the touch panel that is the interface, for example, if the interface is a touch panel that displays information about the system in which the robot operates. For example, in the above-described exemplary embodiments of the present disclosure, "operating an interface" also includes performing an operation that produces the same results as when the touch panel is directly operated, such as outputting sound of a predetermined frequency from a sound source to a sound collector or outputting light of a predetermined wavelength from a light source to a light collector.

[0140] Although the embodiments of the present disclosure have been described, the processing system 1, management system 10, robot system 20, and other control devices may have a computer device inside. The above-described processing steps are stored in the form of a program on a computer-readable recording medium, and the computer reads and executes this program to perform the above processing. Specific examples of computers are shown below.

[0141] FIG. 28 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. As shown in FIG. 28 , the computer 5 includes a CPU (Central Processing Unit) 6, a main memory 7, a storage 8, and an interface 9. For example, the processing system 1, management system 10, robot system 20, and other control devices described above are each implemented in the computer 5. The operation of each of the processing units described above is stored in the storage 8 in the form of a program. The CPU 6 reads the program from the storage 8, loads it into the main memory 7, and executes the above-described processing in accordance with the program. The CPU 6 also allocates storage areas in the main memory 7 corresponding to each of the storage units described above in accordance with the program.

[0142] Examples of storage 8 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a digital versatile disc read-only memory (DVD-ROM), and a semiconductor memory. Storage 8 may be an internal medium directly connected to the bus of computer 5, or an external medium connected to computer 5 via interface 9 or a communication line. Furthermore, when this program is distributed to computer 5 via a communication line, computer 5 that receives the program may load the program into main memory 7 and execute the above-described processing. In at least one embodiment, storage 8 is a non-transitory tangible recording medium.

[0143] The program may also implement some of the functions described above. Furthermore, the program may be a file that can implement the functions described above in combination with a program already stored in the computer device, a so-called differential file (differential program).

[0144] The order of the processes in each embodiment of the present disclosure may be changed as long as the processes are performed appropriately.

[0145] Although several embodiments of the present disclosure have been described, these embodiments are merely examples and do not limit the scope of the disclosure. Various additions, omissions, substitutions, and modifications may be made to these embodiments without departing from the spirit of the disclosure.

[0146] Note that part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0147] (Supplementary Note 1) A robot comprising: an operation means for operating an interface that displays information about a system in which the robot operates; and a control means for identifying information about the operation of the robot in the interface and controlling the operation of the robot.

[0148] (Supplementary Note 2) The robot according to Supplementary Note 1, wherein the operation means operates a portion of the interface where information relating to the operation of the robot is displayed.

[0149] (Supplementary Note 3) The robot according to Supplementary Note 1 or Supplementary Note 2, wherein the operation means is a touch pen, a sound source, a light source, or a rod-shaped object.

[0150] (Supplementary Note 4) The robot described in any one of Supplementary Note 1 to Supplementary Note 3, wherein when the operation means is a touch pen or a rod-shaped object, the control means controls the operation of the robot so that a specific figure displayed on the interface is touched with the touch pen or the rod-shaped object.

[0151] (Supplementary Note 5) The robot according to Supplementary Note 4, wherein the specific graphic is an AR (Augmented Reality) marker, a QR (Quick Response) code, or a geometric figure.

[0152] (Supplementary Note 6) The robot according to any one of Supplementary Note 1 to Supplementary Note 5, wherein, when the operation means is a sound source, the control means controls the operation of the robot so as to cause the sound source to output sound of a predetermined frequency to a sound collector.

[0153] (Supplementary Note 7) The robot described in any one of Supplementary Note 1 to Supplementary Note 6, wherein when the operating means is a light source, the control means controls the operation of the robot so as to cause the light source to output light of a predetermined wavelength to a collector.

[0154] (Supplementary Note 8) A processing system comprising: a robot according to any one of Supplementary Notes 1 to 7; and a management system that causes the interface to display information about a system in which the robot operates.

[0155] (Supplementary Note 9) A processing method comprising: operating an interface that displays information about a system in which a robot operates; identifying information about the operation of the robot in the interface; and controlling the operation of the robot.

[0156] (Supplementary Note 10) A recording medium storing a program that causes a computer to operate an interface that displays information about a system in which a robot operates, identify information about the operation of the robot in the interface, and control the operation of the robot.

[0157] Aspects of the present disclosure allow for appropriate interaction between a robot and software that causes an interface to display information about the system in which the robot operates.

[0158] REFERENCE SIGNS LIST 1 Processing system 5 Computer 6 CPU 7 Main memory 8 Storage 9 Interface 10 Management system 20 Robot system 101 Management unit 102, 207 Storage unit 103 Control unit 104 Display unit 201 Task instruction device 202 Generation unit 203 Camera 204 Controller 205 Robot 206 Determination unit 208 Recognition unit 209 Identification unit C Cardboard box M Product T Tray

Claims

1. An operating means for operating an interface that displays information about the system in which the robot operates, The interface includes a control means for identifying information regarding the robot's operation and controlling the robot's operation, A robot equipped with [the following features].

2. The aforementioned operating means is The part of the interface that displays information about the robot's operation is operated. The robot according to claim 1.

3. The aforementioned operating means is A stylus, sound source, light source, or an object with a rod-like shape. The robot according to claim 1 or claim 2.

4. If the operating means is a stylus or a rod-shaped object, The control means is The robot's movements are controlled to cause it to touch a specific shape displayed on the interface with the stylus or the rod-shaped object. The robot according to claim 1.

5. The aforementioned specific figure is AR (Augmented Reality) markers, QR (Quick Response) codes, or geometric shapes. The robot according to claim 4.

6. If the aforementioned operating means is a sound source, The control means is The robot's movements are controlled to cause the sound collector to output a sound of a predetermined frequency from the sound source. The robot according to claim 1.

7. When the operating means is a light source, The control means is The robot's movement is controlled to cause the light source to output light of a predetermined wavelength to the light collector. The robot according to claim 1.

8. The robot according to claim 1, A management system that displays information about the system in which the robot operates on the interface, A processing system equipped with the following features.

9. Operate the interface that displays information about the system in which the robot operates. In the interface, information regarding the operation of the robot is identified, Controlling the operation of the aforementioned robot, Processing method.

10. This involves operating an interface that displays information about the system in which the robot operates, The interface identifies information regarding the operation of the robot, Controlling the movement of the aforementioned robot, A program that causes a computer to execute something.