Control device and control method

Abstract

This control device is provided with a control means for creating display information including an input form for receiving an input from the outside, and information relating to the operation of a robot, the information being displayed so as to be superimposed on the input form.

Description

Control device, control method, and recording medium

[0001] The present disclosure relates to a control device, a control method, and a recording medium.

[0002] Robots are used in various fields such as logistics. Japanese Patent Application Laid-Open No. 2006-144992 discloses a related technology relating to a robot system having a configuration capable of efficiently accessing desired information.

[0003] Japanese Patent Application Laid-Open No. 2020-121351

[0004] In the robot system described in Patent Document 1 and elsewhere, the robot and a worker may collaborate to perform a task on behalf of the worker. In such cases, the robot is required to perform the same operations as the worker. Therefore, there is a demand for technology that can control task-related software regardless of the type of operator.

[0005] One of the objectives of each aspect of the present disclosure is to provide a control device, a control method, and a recording medium that can solve the above-mentioned problems.

[0006] According to one aspect of the present disclosure, the control device includes a control means for creating display information including an input form that receives input from outside and information related to the robot's operation that is displayed superimposed on the input form.

[0007] According to another aspect of the present disclosure, a control method creates display information including an input form for receiving input from an external source and information related to the robot's operation that is displayed superimposed on the input form.

[0008] According to each aspect of the present disclosure, it is possible to control software related to a task regardless of the type of the operating entity.

[0009] FIG. 1 is a diagram illustrating an example of a configuration of a processing system according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a configuration of a management system according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating a first example of a display of a display unit according to an embodiment of the present disclosure. FIG. 4 is a diagram illustrating a second example of a display of a display unit according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating a third example of a display of a display unit according to an embodiment of the present disclosure. FIG. 6 is a diagram illustrating a sixth example of a display of a display unit according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating a seventh example of a display of a display unit according to an embodiment of the present disclosure. FIG. 8 is a diagram illustrating an example of a sequence of a plan generated by a generation unit according to an embodiment of the present disclosure. FIG. 9 is a diagram illustrating an example of a control signal of a plan generated by a controller according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating an example of a processing flow of a processing system according to an embodiment of the present disclosure. FIG. 11 is an image of the surrounding environment of a processing system according to an embodiment of the present disclosure. FIG. 12 is an image of the surrounding environment of a processing system according to an embodiment of the present disclosure. FIG. 13 is a diagram illustrating an example of a control device with a minimum configuration according to an embodiment of the present disclosure. FIG. 14 is a diagram illustrating an example of a processing flow of a control device with a minimum configuration according to an embodiment of the present disclosure. FIG. 15 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment.

[0010] Hereinafter, embodiments will be described in detail with reference to the drawings. <Embodiment> A processing system 1 according to an embodiment of the present disclosure will be described. The processing system 1 includes at least a management system 10. A robot system 20 that performs operations in place of workers can be introduced into the processing system 1. FIG. 1 is a diagram illustrating an example of the configuration of the processing system 1 according to an embodiment of the present disclosure. The processing system 1 illustrated in FIG. 1 illustrates an example of the configuration after the robot system 20 is introduced. Below, the processing system 1 incorporating the robot system 20 will be described using a sorting task as a specific example of the work, in which products M in cardboard boxes C carried by a belt conveyor are transferred to trays T, and the trays T are placed on another belt conveyor and moved to a predetermined position.

[0011] As shown in FIG. 1 , the processing system 1 includes a management system 10 (an example of a control device) and a robot system 20 .

[0012] 2 is a diagram illustrating an example of the configuration of the management system 10 according to an embodiment of the present disclosure. As shown in FIG. 2, the management system 10 includes a management unit 101, a storage unit 102, a control unit 103, and a display unit 104.

[0013] The management unit 101 writes the status of each task in the storage unit 102. For example, the management unit 101 writes the status of each task in the storage unit 102 based on external input received via an input form (described later) displayed on the display unit 104. The input form is a form for receiving external input. Examples of input forms include radio buttons, check boxes, and menus (including drop-down menus). Specifically, for example, when the display unit 104 displays a Complete button as an input form, the management unit 101 determines whether the Complete button has been touched. If the management unit 101 determines that the Complete button has been touched, it determines that the task corresponding to the Complete button has been completed. Then, the management unit 101 writes a completion flag indicating that the task has been completed, associated with the task determined to be completed, into the storage unit 102.

[0014] The memory unit 102 stores various information necessary for the processing performed by the management system 10. For example, the memory unit 102 stores the status of each task. Specifically, for example, the memory unit 102 stores a flag of 0 (incomplete) or 1 (completed) associated with the task. Furthermore, for example, the memory unit 102 stores information that the control unit 103 causes the display unit 104 to display according to the status of each task. The information that the control unit 103 causes the display unit 104 to display according to the status of each task includes the above-mentioned input form and information related to the operation of the robot 205, which is superimposed on the input form and is described below. Examples of the information related to the operation of the robot 205 include yes (YES), no (NO), and commands to the robot 205. Examples of commands to the robot 205 include a command to touch a predetermined location when a certain task is completed, and a command to touch a predetermined location before starting a certain task. Among the information that the control unit 103 causes to be displayed on the display unit 104 according to the status of each task, a specific graphic is associated with information that the robot 205 directly operates on the display unit 104. Examples of the specific graphic include an AR (Augmented Reality) marker, a QR (Quick Response) code, and geometric shapes such as a circle (○) or a triangle (△).

[0015] The control unit 103 generates display information including an input form and information related to the operation of the robot 205, which is displayed superimposed on the input form. The control unit 103 controls the display of the generated display information on the display unit 104. For example, the control unit 103 causes the generated display information to be displayed on the display unit 104. Also, for example, the control unit 103 changes the display content of the display information displayed on the display unit 104 after a predetermined operation of the robot 205 is completed. Also, for example, the control unit 103 hides the display information displayed on the display unit 104 after the predetermined operation of the robot 205 is completed. Also, for example, the control unit 103 causes the display unit 104 to display the display information at the timing when the operation of the robot in a predetermined process is completed or at the estimated timing when the operation of the robot in the predetermined process is likely to be completed. Note that the display information may be information representing control content for the robot 205. Also, the display information may be information representing a desired state to be achieved by the robot 205.

[0016] The display unit 104 displays display information under the control of the control unit 103. For example, under the control of the control unit 103, the display unit 104 displays an input form and information related to the operation of the robot 205 superimposed on the input form. Furthermore, for example, under the control of the control unit 103, the display unit 104 also displays a specific graphic associated with information that the robot 205 directly operates on the display unit 104, among the information displayed according to the status of each task. For example, the display unit 104 is a display with a touch panel function. For example, the display unit 104 is installed in a work location.

[0017] FIG. 3 is a diagram illustrating a first example of a display on the display unit 104 according to an embodiment of the present disclosure. FIG. 4 is a diagram illustrating a second example of a display on the display unit 104 according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating a third example of a display on the display unit 104 according to an embodiment of the present disclosure. FIG. 6 is a diagram illustrating a fourth example of a display on the display unit 104 according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating a fifth example of a display on the display unit 104 according to an embodiment of the present disclosure. FIG. 8 is a diagram illustrating a sixth example of a display on the display unit 104 according to an embodiment of the present disclosure. For example, under the control of the control unit 103, the display unit 104 displays, at a predetermined timing, a button showing, within a frame, an AR marker, which is a specific graphic corresponding to the selection content for selecting "yes" or "no" as to whether the work is completed. Furthermore, for example, under the control of the control unit 103, the display unit 104 displays, within a frame, the work content of the next process (an example of a predetermined process) and an AR marker, which is a specific graphic corresponding to the content, at a predetermined timing, as shown in FIG. 4. Further, for example, under the control of the control unit 103, the display unit 104 displays, at a predetermined timing, a button showing an AR marker, which is a specific graphic corresponding to the selection content for selecting "yes" or "no" as to whether the work is completed, along with the work content of the next process (an example of a predetermined process), within a frame, as shown in FIG. 5 . Further, for example, under the control of the control unit 103, the display unit 104 hides the displayed information at a predetermined timing, as shown in FIG. 6 . Further, for example, under the control of the control unit 103, the display unit 104 displays, at a predetermined timing, a planned completion date (an example of an estimated timing) of the operation of the robot 205 in the current process (an example of a predetermined process), as shown in FIGS. 7 and 8 . Examples of the predetermined timing include after the operation of the robot 205 is completed, the estimated completion timing of the operation of the robot 205, the start of the predetermined process, and the end of the predetermined process.

[0018] 9 is a diagram illustrating an example of the configuration of a robot system 20 according to an embodiment of the present disclosure. As illustrated in FIG. 9 , the robot system 20 includes a task instruction device 201, a generation unit 202, a camera 203, a controller 204, a robot 205, a determination unit 206, a storage unit 207, a recognition unit 208, and an identification unit 209.

[0019] The task instruction device 201 instructs the controller 204 on a task for causing the robot 205 to perform a desired task. For example, the task instruction device 201 instructs the controller 204 on a task via a network (or, in some cases, by using a cloud service).

[0020] For example, if the desired task is to move products M in a cardboard box C to a destination tray T, determine whether the task of moving the products M to the tray T is complete, and, if it is determined that the task is complete, perform a sorting task of moving the tray T to a predetermined position on a conveyor belt, the task instruction device 201 inputs task goals and constraint conditions to the generation unit 202 as a procedure for instructing the task. Examples of task goals include information indicating the type of products M, the number of products M to be moved, the source of the products M, and the destination of the products M, as well as information indicating a predetermined task unit for confirming the completion of the task. Examples of constraint conditions include no-entry areas when moving the products M, areas that deviate from the range of motion of the robot 205, and further conditions on the surface of the products M regarding gripping the products M, releasing gripping of the products M, or changing the products M. The task instruction device 201 may receive an input from the worker as a task goal, such as "move six parts A from cardboard box C to tray T," and specify that the type of product M to be moved is part A, the quantity of product M to be moved is six, the source of product M is cardboard box C, and the destination of product M is tray T. The task instruction device 201 may then input the specified information to the generation unit 202. Furthermore, the task instruction device 201 may receive, as a task goal, an input from the worker, for example, that "the units of predetermined tasks for confirming completion of the tasks are a first task of stopping the belt conveyor for moving the cardboard box C at the position of the robot 205, a second task of moving the cardboard box C from the stop position of the belt conveyor onto the work table F, a third task of moving the product M in the cardboard box C on the work table F to the tray T, a fourth task of stopping the belt conveyor for moving the tray T, a fifth task of moving the tray T onto the belt conveyor for moving the tray T, and a sixth task of moving the belt conveyor for moving the tray T," and cause the generation unit 202 to generate a process for confirming completion of the task after each of the first to sixth tasks. Furthermore, the task instruction device 201 may set the position of the product M identified in information acquired by the camera 203, which will be described later, as the source of movement of the product M.Furthermore, the task instruction device 201 may receive, for example, from a worker, the positions of obstacles along the path of moving the product M from the start point to the destination point as constraint conditions indicating no-entry areas, and input the information to the generation unit 202. Alternatively, a file indicating the constraint conditions may be stored in, for example, the storage unit 207, and the task instruction device 201 may input the constraint conditions indicated in the file to the generation unit 202, or the generation unit 202 may read the constraint conditions directly from the file, or both. In other words, any method of acquisition may be used as long as the generation unit 202 can acquire the necessary task goals and necessary constraint conditions.

[0021] The generation unit 202 generates a plan indicating the flow of operations of the robot 205 based on the task goal and constraints input by the task instruction device 201 .

[0022] For example, when the task goal and constraint conditions are input by the task instruction device 201, the generation unit 202 generates a first sequence that determines whether the cardboard box C has moved to the position of the robot 205, continues executing the determination until it is determined that the cardboard box C has moved, and stops the belt conveyor for moving the cardboard box C when it is determined that the cardboard box C has moved. The generation unit 202 also generates a second sequence that confirms whether the work performed by the first sequence has been completed. The first sequence corresponds to the first work described above. The second sequence is a sequence generated by the generation unit 202 in accordance with the first work, which is a predetermined unit of work for confirming completion of the work.

[0023] FIG. 10 is a diagram illustrating a seventh example of a display on the display unit 104 according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating an example of a display displayed by the display unit 104 when the robot 205 stops the belt conveyor. When the robot 205 stops the belt conveyor, the display unit 104 displays a button BT1 including a specific graphic for stopping the belt conveyor, as shown in FIG. 10 . For example, the robot 205 stops the belt conveyor by touching the button BT1 displayed on the display unit 104. Note that when the robot 205 stops the belt conveyor, the display unit 104 may or may not display a command to stop the belt conveyor along with the button BT1. For example, when the robot 205 stops the belt conveyor, the display unit 104 may display only the button BT1 and information indicating a command to be executed after the robot 205 stops the belt conveyor. That is, under the control of the control unit 103, the display unit 104 displays buttons for causing the robot 205 to perform a predetermined process at a predetermined timing, and also displays instructions to be performed as necessary.

[0024] Furthermore, for example, the generation unit 202 generates a third sequence for moving the cardboard box C from the stop position of the belt conveyor onto the work table F. The generation unit 202 also generates a fourth sequence for confirming whether or not the work performed by the third sequence has been completed. The third sequence is a sequence corresponding to the second work described above. The fourth sequence is a sequence generated by the generation unit 202 in accordance with the second work, which is a predetermined unit of work for confirming the completion of the work.

[0025] Furthermore, for example, the generation unit 202 generates a fifth sequence for moving the product M in the cardboard box C on the work table F to the tray T. The generation unit 202 also generates a sixth sequence for confirming whether the work according to the fifth sequence has been completed. The fifth sequence is a sequence corresponding to the third work described above. The sixth sequence is also a sequence generated by the generation unit 202 in accordance with the third work, which is a predetermined unit of work for confirming the completion of the work.

[0026] More specifically, the generation unit 202 acquires, from the camera 203, two-dimensional and three-dimensional information (described later) of the origin of the product M indicated by the task target. The generation unit 202 can recognize the state (i.e., position and posture) of the product M at the origin of the movement from the two-dimensional and three-dimensional information acquired from the camera 203. The generation unit 202 generates, for example, by simulation, a movement path including the state of the product M from the state of the product M at the origin of the movement to the state of the product M at the destination of the movement. Information representing the movement path is information necessary for the controller 204 to generate a control signal for controlling the robot 205. The generation unit 202 then generates, for example, by simulation, information (i.e., a fifth sequence) representing each state of the robot 205 at each time step during the movement (such as the type (including the shape) of the product M, the position and posture of the robot 205, and the behavior of the robot 205 (such as the grip strength of the product M)).

[0027] Furthermore, for example, the generation unit 202 generates a seventh sequence that stops the belt conveyor for moving the tray T. The generation unit 202 also generates an eighth sequence that checks whether the work according to the seventh sequence has been completed. The seventh sequence is a sequence that corresponds to the above-mentioned fourth work. The eighth sequence is a sequence generated by the generation unit 202 in accordance with the fourth work, which is a predetermined work unit that checks whether the work has been completed.

[0028] Furthermore, for example, the generation unit 202 generates a ninth sequence for moving the tray T on a belt conveyor for moving the tray T. The generation unit 202 also generates a tenth sequence for confirming whether or not the work according to the ninth sequence has been completed. The ninth sequence is a sequence corresponding to the fifth work described above. The tenth sequence is a sequence generated by the generation unit 202 in accordance with the fifth work, which is a predetermined work unit for confirming the completion of the work.

[0029] Furthermore, for example, the generation unit 202 generates an eleventh sequence for moving a belt conveyor for moving trays T. The generation unit 202 also generates a twelfth sequence for confirming whether or not the work performed by the eleventh sequence has been completed. The eleventh sequence corresponds to the sixth work described above. The twelfth sequence is a sequence generated by the generation unit 202 in accordance with the sixth work, which is a predetermined unit of work for confirming the completion of the work. The generation unit 202 outputs the generated sequence to the controller 204 each time it generates each of the first to twelfth sequences.

[0030] 11 is a diagram illustrating an example of a sequence TBL1 of a plan generated by the generation unit 202 according to an embodiment of the present disclosure. For example, the sequence TBL1 of a plan generated by the generation unit 202 is a sequence indicating each state of the robot 205 for each n time step from the movement origin of the product M to the destination, as shown in FIG.

[0031] The camera 203 is provided at a position (for example, above the robot 205) that allows a bird's-eye view of the work performed by the robot system 20. The camera 203 includes a two-dimensional camera 2031 and a depth camera 2032.

[0032] The two-dimensional camera 2031 acquires two-dimensional information (i.e., a planar image) of the object being photographed, and the depth camera 2032 acquires three-dimensional information of the object being photographed, including depth information.

[0033] The controller 204 generates a control signal for controlling the robot 205 based on the sequence output by the generation unit 202. Note that the controller 204 may generate a control signal that optimizes an evaluation function when generating the control signal. Examples of the evaluation function include a function that represents the amount of energy consumed by the robot 205 when moving the product M, and a function that represents the distance along the path along which the product M is moved. The controller 204 outputs the generated control signal to the robot 205.

[0034] Furthermore, when the determination unit 206 determines that the work has been completed in determining whether the work to be performed for each predetermined unit of work in the plan has been completed, the controller 204 generates a control signal that causes the robot 205 to touch, for example, with a touch pen, a specific figure in the three-dimensional space of the work location identified by the identification unit 209. Then, the controller 204 outputs the generated control signal to the robot 205.

[0035] 12 is a diagram illustrating an example of the control signal Cnt of the plan generated by the controller 204 according to an embodiment of the present disclosure. For example, as shown in FIG. 12 , the control signal Cnt of the plan generated by the controller 204 is, for example, each control signal for n time steps from the movement origin of the product M to the destination.

[0036] The robot 205 operates in response to a control signal generated by the controller 204. For example, if the control signal is a control signal for moving the product M from the origin to the destination, the robot 205 moves the grasped product M from the origin to the destination in response to the control signal. In each embodiment of the present disclosure, "grasping" includes "suction" in which the target object M is sucked by a vacuum or the like, and "clamping" in which an object is pinched between two or more pseudo-fingers that mimic the fingers of a human or animal. For example, if the control signal is a control signal for touching a specific figure in the three-dimensional space of the work area with a stylus, the robot 205 touches the specific figure with the stylus in response to the control signal. The stylus may be retractable within the robot 205 and may appear from its storage location when the specific figure is touched with the stylus.

[0037] The determination unit 206 determines whether or not each predetermined unit of work in the plan has been completed. For example, when the determination unit 206 makes a determination regarding the third work described above, in accordance with the sixth sequence, the determination unit 206 determines whether or not the product M is within a predetermined range based on the information about the product M acquired by the camera 203 at the destination. Then, for example, if the determination unit 206 determines that the product M is within the predetermined range based on the information about the product M acquired by the camera 203 at the destination, the determination unit 206 determines that the third work has been completed. Furthermore, for example, if the determination unit 206 determines that the product M is not within the predetermined range based on the information about the product M acquired by the camera 203 at the destination, the determination unit 206 determines that the work controlled by the plan has not been completed.

[0038] When the determination unit 206 determines that the work under the control of the plan has been completed, the controller 204 generates, for example, a control signal that causes the robot 205 to touch a specific figure corresponding to the completion of the work with a touch pen. Then, the controller 204 outputs the generated control signal to the robot 205.

[0039] Furthermore, when the determination unit 206 determines that the work under the control of the plan has not been completed, the controller 204 may generate a control signal to move the product M again so that it falls within a predetermined range, for example, based on information about the product M acquired at the destination by the camera 203. Then, the controller 204 may output the generated control signal to the robot 205.

[0040] Note that, when the determination unit 206 determines that the work under the control of the plan has not been completed, the controller 204 may, instead of the above-described processing, notify the worker that the work has not been completed, for example, by notifying the task instruction device 201, etc. Then, when the worker confirms that the work has not been completed, the worker may instruct the processing system 1 to move the product M so that it falls within a predetermined range, or the worker may move the product M into the predetermined range.

[0041] The storage unit 207 stores various information necessary for the processing performed by the robot system 20. For example, the storage unit 207 stores a file indicating constraint conditions, a reference figure for the recognition unit 208 to identify a specific figure, and the like.

[0042] Recognition unit 208 identifies a specific graphic displayed on display unit 104 in the information (i.e., the planar image) acquired by camera 203. For example, recognition unit 208 compares information on each reference graphic stored in storage unit 207 with the information acquired by camera 203. Then, when the degree of match in the comparison result is equal to or greater than a predetermined degree, recognition unit 208 identifies the matching graphic and the position of the matching graphic in the information acquired by camera 203.

[0043] The identification unit 209 identifies the posture (position and orientation) of the figure in the three-dimensional space of the work area. For example, if the matching figure identified by the recognition unit 208 is an AR marker, the identification unit 209 identifies the posture of the figure in the three-dimensional space of the work area from the matching figure identified by the recognition unit 208. The AR marker includes three-dimensional information on the x, y, and z directions in the three-dimensional space of the work area, as well as orientation information corresponding to three pieces of information: roll, pitch, and yaw. The x direction is, for example, one direction contained in the floor plane of the work area. The y direction is, for example, a direction contained in the floor plane and perpendicular to the x direction. The z direction is, for example, a direction perpendicular to the floor plane. In other words, the AR marker includes six-dimensional information that can identify the posture. Therefore, if the matching figure identified by the recognition unit 208 is an AR marker, the identification unit 209 can identify the posture of the figure in the three-dimensional space of the work area using only the matching figure identified by the recognition unit 208.

[0044] Furthermore, for example, if the matching figure identified by the recognition unit 208 is a specific figure other than an AR marker, the identification unit 209 identifies the position of the matching figure in the information acquired by the camera 203. Then, the identification unit 209 may identify the posture (position and orientation) of the specific figure in the three-dimensional space of the workplace from information acquired by the camera 203 that corresponds to the information acquired by the camera 203 with respect to the position of the identified figure (i.e., the three-dimensional information of the display unit 104).

[0045] Note that the processing performed by the processing system 1 in each embodiment of the present disclosure is not limited to the above-described processing. For example, the processing system 1 may perform the processing described below.

[0046] FIG. 13 is a diagram illustrating an example of a processing flow of the processing system 1 according to an embodiment of the present disclosure. FIG. 14 is an image diagram illustrating the surrounding environment of the processing system 1 according to an embodiment of the present disclosure. Next, processing by the processing system 1, including processing for displaying a specific graphic, will be described with reference to FIGS. 13 and 14 . It is assumed that the task instruction device 201 instructs the controller 204 to perform a sorting task, which is a desired task for the robot 205, in which the items M in the cardboard boxes C transported by the belt conveyor shown in FIG. 14 are transferred to trays T, and the trays T are placed on another belt conveyor and moved to a predetermined position. It is also assumed that the two-dimensional camera 2031 acquires two-dimensional information (i.e., a planar image) of the object being photographed. It is also assumed that the depth camera 2032 acquires three-dimensional information of the object being photographed, including depth information. As specific examples of processing performed by the processing system 1, the processing according to the fifth and sixth sequences corresponding to the third task described above will be described. The plan (i.e., sequence) generated by the generation unit 202 is notified to the management system 10. When the generation unit 202 notifies the management system 10 of the plan, the control unit 103, in response to the notification, causes the display unit 104 to display a notification inquiring about the completion of the work.

[0047] The task instruction device 201 instructs the controller 204 to execute a task as a desired task for the robot 205, which is to transfer products M in a cardboard box C carried by a belt conveyor onto a tray T, place the tray T on another belt conveyor, and move the tray T to a predetermined position (step S1). For example, the task instruction device 201 outputs, as task targets, information indicating the type of products M, the number of products M to be moved, the origin of the products M, and the destination of the products M, as well as information indicating a predetermined task unit for confirming completion of the task, to the generation unit 202. In addition, for example, the task instruction device 201 outputs, as constraint conditions, no-entry areas when moving the products M, areas that deviate from the range of motion of the robot 205, and further conditions for the surface of the products M regarding gripping the products M, releasing gripping of the products M, or changing the products M, to the generation unit 202.

[0048] The generation unit 202 generates a plan indicating the flow of operations of the robot 205 based on the task goal and constraints input by the task instruction device 201 .

[0049] For example, when the robot 205 has completed the task of moving a cardboard box C from the stop position of the belt conveyor onto the work table F, the generation unit 202 generates a fifth sequence for moving the product M in the cardboard box C on the work table F to the tray T (step S2). The generation unit 202 also generates a sixth sequence for confirming whether the task according to the fifth sequence has been completed. The generation unit 202 notifies the management system 10 that the fifth sequence has been generated (step S3). In response to the notification, the control unit 103 causes the display unit 104 to display a display corresponding to the fifth sequence (for example, the displays shown in FIGS. 3 to 8 ) at a predetermined timing (step S4).

[0050] The generation unit 202 also outputs the generated fifth sequence to the controller 204. Based on the fifth sequence output by the generation unit 202, the controller 204 generates a control signal for controlling the robot 205 to move the product M in the cardboard box C on the work table F to the tray T (step S5).

[0051] The controller 204 outputs the generated control signal to the robot 205. In response to the control signal output by the controller 204, the robot 205 moves the product M in the cardboard box C on the work table F to the tray T (step S6). The generation unit 202 generates a sixth sequence for checking whether the work according to the fifth sequence has been completed (step S7). The determination unit 206 determines whether the work according to the fifth sequence has been completed in accordance with the sixth sequence generated by the generation unit 202 (step S8).

[0052] If the determination unit 206 determines that the work according to the fifth sequence has not been completed (NO in step S8), the controller 204 generates a control signal for moving the product M again so that it fits within a predetermined range, for example, based on information about the product M acquired at the destination by the camera 203 (step S9). The controller 204 then outputs the generated control signal to the robot 205. The robot 205 then moves the product M so that it fits within the predetermined range (step S10). The robot 205 then returns to the processing of step S8.

[0053] Furthermore, if the determination unit 206 determines that the work according to the fifth sequence has been completed (YES in step S8), the recognition unit 208 identifies a specific figure associated with the notification of the work determined by the determination unit 206 to have been completed. Then, the recognition unit 208 identifies a figure that matches the identified specific figure from among the specific figures displayed on the display unit 104 in the information (i.e., the planar image) acquired by the camera 203 (step S11). The identification unit 209 identifies a specific figure that matches the specific figure identified by the recognition unit 208 in the database DB1 stored in the storage unit 207 (step S12). Then, the identification unit 209 identifies the posture (position and orientation) of the identified specific figure in the three-dimensional space of the work location (step S13).

[0054] The controller 204 generates a control signal for touching a specific figure with the stylus according to the posture identified by the identification unit 209 (step S14). The controller 204 then outputs the generated control signal to the robot 205. The robot 205 touches a specific figure with the stylus, indicating that the task has been completed (step S15). The processing of step S15 notifies the management system 10 that the task of moving the product M from the cardboard box C to the tray T has been completed. The management unit 101 rewrites the flag for the task of moving the product M from the cardboard box C to the tray T, which is stored in the memory unit 102, from 0 (unprocessed) to 1 (processed) (step S16). The processing of step S16 enables the management system 10 to manage the progress of each task.

[0055] 13 is an example, and is not limited to the display described in the processing flow. As described using Figures 3 to 8, the display unit 104 displays information in accordance with processing at a predetermined timing under the control of the control unit 103.

[0056] (Advantages) The processing system 1 according to an embodiment of the present disclosure has been described above. The management system 10 (an example of a control device) of the processing system 1 includes an input form that receives input from outside, and a control unit 103 (an example of a control means) that creates display information that includes information related to the operation of the robot and is displayed superimposed on the input form.

[0057] In this way, the management system 10 can display the input form and the information related to the robot's operation superimposed on the input form, thereby enabling the management system 10 to control the software related to the work regardless of the type of entity performing the operation (whether it is a worker or a robot).

[0058] In the embodiment of the present disclosure, the robot system 20 has been described as being able to control the robot 205 by simply instructing the task instruction device 201 to perform a task, causing the generation unit 202 to generate a sequence, and the controller 204 to generate a control signal according to the sequence. However, in another embodiment of the present disclosure, the task instruction device 201 may instruct a task to be performed for each arbitrary task unit, for example, for each predetermined task unit described in the embodiment of the present disclosure. In other words, in the robot system 20 according to the another embodiment of the present disclosure, a task may be instructed for each arbitrary task unit, the generation unit 202 may generate a sequence for each instruction, the controller 204 may generate a control signal according to the sequence, and the robot 205 may be controlled by the control signal. The arbitrary task unit may also include stopping.

[0059] In one embodiment of the present disclosure, the display unit 104 has been described as a display with a touch panel function. However, in another embodiment of the present disclosure, the display unit 104 is not limited to a display with a touch panel function. For example, the display unit 104 may be a non-contact proximity-responsive display that responds when a touch pen or the like is brought close to the display.

[0060] In one embodiment of the present disclosure, the robot 205 has been described as responding with a specific figure using a touch pen. However, in another embodiment of the present disclosure, the robot 205 may respond with a specific figure using an object other than a touch pen, as long as the display unit 104 responds. Examples of the object other than a touch pen include an object made of a conductive material and an object in a rod shape.

[0061] Furthermore, in the embodiment of the present disclosure, the orientation of a specific figure displayed on the display unit 104 is described as being identified based on information acquired by the camera 203 including a two-dimensional camera 2031 and a depth camera 2032. However, depending on the installation location of the camera 203, it may be impossible to identify the orientation of a specific figure displayed on the display unit 104 based solely on the information acquired by the camera 203. Therefore, the camera 203 according to another embodiment of the present disclosure is not limited to a camera including a two-dimensional camera 2031 and a depth camera 2032. FIG. 15 is an image diagram of the surrounding environment of the processing system 1 according to an embodiment of the present disclosure. The camera 203 according to an embodiment of the present disclosure may include a two-dimensional camera 2031 and a depth camera 2032, similar to the camera 203 according to an embodiment of the present disclosure, and may further include a two-dimensional camera 2033 located in a position that makes it easy to photograph the display unit 104, as shown in FIG. 15.

[0062] Next, a control device 300 with a minimum configuration according to an embodiment of the present disclosure will be described. FIG. 16 is a diagram illustrating an example of a control device 300 with a minimum configuration according to an embodiment of the present disclosure. The control device 300 with a minimum configuration includes a control unit 301. The control unit 301 creates display information including an input form for receiving input from outside and information related to the operation of the robot, which is displayed superimposed on the input form. The control unit 301 can be realized, for example, using the functions of the control unit 103 illustrated in FIG. 2.

[0063] Next, a description will be given of processing performed by the control device 300 with the minimum configuration. Fig. 17 is a diagram showing an example of a processing flow of the control device 300 with the minimum configuration according to an embodiment of the present disclosure. Here, the processing of the control device 300 with the minimum configuration will be described with reference to Fig. 17.

[0064] In the control device 300, the control means 301 creates display information including an input form for receiving input from the outside and information relating to the operation of the robot, which is displayed superimposed on the input form (step S101).

[0065] The above describes the minimum configuration of the control device 300 according to the embodiment of the present disclosure. This control device 300 can control software related to work regardless of the type of operating entity.

[0066] Although the embodiments of the present disclosure have been described, the processing system 1, management system 10, robot system 20, and other control devices may have a computer device inside. The above-described processing steps are stored in the form of a program on a computer-readable recording medium, and the computer reads and executes this program to perform the above processing. Specific examples of computers are shown below.

[0067] 18 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. As shown in FIG. 18 , the computer 5 includes a CPU (Central Processing Unit) 6, a main memory 7, a storage 8, and an interface 9. For example, the processing system 1, management system 10, robot system 20, and other control devices described above are each implemented in the computer 5. The operations of each of the processing units described above are stored in the storage 8 in the form of a program. The CPU 6 reads the program from the storage 8, loads it into the main memory 7, and executes the above-described processing in accordance with the program. The CPU 6 also allocates storage areas in the main memory 7 corresponding to each of the storage units described above in accordance with the program.

[0068] Examples of storage 8 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a digital versatile disc read-only memory (DVD-ROM), and a semiconductor memory. Storage 8 may be an internal medium directly connected to the bus of computer 5, or an external medium connected to computer 5 via interface 9 or a communication line. Furthermore, when this program is distributed to computer 5 via a communication line, computer 5 that receives the program may load the program into main memory 7 and execute the above-described processing. In at least one embodiment, storage 8 is a non-transitory tangible recording medium.

[0069] The program may also implement some of the functions described above. Furthermore, the program may be a file that can implement the functions described above in combination with a program already stored in the computer device, a so-called differential file (differential program).

[0070] The order of the processes in each embodiment of the present disclosure may be changed as long as the processes are performed appropriately.

[0071] Although several embodiments of the present disclosure have been described, these embodiments are merely examples and do not limit the scope of the disclosure. Various additions, omissions, substitutions, and modifications may be made to these embodiments without departing from the spirit of the disclosure.

[0072] Note that part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0073] (Supplementary Note 1) A control device comprising: an input form for receiving input from the outside; and a control means for creating display information including information relating to the operation of a robot, which is displayed superimposed on the input form.

[0074] (Supplementary Note 2) The control device according to Supplementary Note 1, wherein the control means changes the display content of the display information.

[0075] (Supplementary Note 3) The control device according to Supplementary Note 2, wherein the control means changes the display content of the display information after a predetermined operation of the robot is completed.

[0076] (Supplementary Note 4) The control device according to any one of Supplementary Note 1 to Supplementary Note 3, wherein the control means makes the display of the display information non-displayable.

[0077] (Supplementary Note 5) The control device according to Supplementary Note 4, wherein the control means makes the display of the display information non-displayable after a predetermined operation of the robot is completed.

[0078] (Supplementary Note 6) The control device according to any one of Supplementary Notes 1 to 5, wherein the control means displays an estimated timing of completion of the operation of the robot in a predetermined process.

[0079] (Supplementary Note 7) The control device according to any one of Supplementary Notes 1 to 6, wherein the control means displays the display information at a timing when the robot's operation in a predetermined process is completed or at an estimated timing when the robot's operation in a predetermined process is likely to be completed.

[0080] (Supplementary Note 8) The control device according to any one of Supplementary Notes 1 to 7, wherein the display information is information that represents control content for the robot.

[0081] (Supplementary Note 9) The control device according to any one of Supplementary Notes 1 to 8, wherein the display information is information representing a desired state to be achieved by the robot.

[0082] (Supplementary Note 10) A control method for creating display information including an input form for receiving an input from an external device and information relating to the operation of a robot, the information being displayed superimposed on the input form.

[0083] (Supplementary Note 11) A recording medium storing a program that causes a computer to create display information including an input form that receives input from an external device and information related to the robot's operations that is displayed superimposed on the input form.

[0084] According to each aspect of the present disclosure, it is possible to control software related to a task regardless of the type of the operating entity.

[0085] REFERENCE SIGNS LIST 1 Processing system 5 Computer 6 CPU 7 Main memory 8 Storage 9 Interface 10 Management system 20 Robot system 101 Management unit 102, 207 Storage unit 103 Control unit 104 Display unit 201 Task instruction device 202 Generation unit 203 Camera 204 Controller 205 Robot 206 Determination unit 208 Recognition unit 209 Identification unit C Cardboard box M Product T Tray

Claims

1. A control means that includes an input form for receiving external input and a display information that includes information related to the robot's operation, which is superimposed on the input form. A control device equipped with the following features.

2. The control means is To change the display content of the aforementioned display information, The control device according to claim 1.

3. The control means is The content of the display information is changed after the robot completes a predetermined operation. The control device according to claim 2.

4. The control means is To hide the display of the aforementioned display information, A control device according to any one of claims 1 to 3.

5. The control means is The display of the aforementioned display information is hidden after the robot completes a predetermined operation. The control device according to claim 4.

6. The control means is To display the estimated completion timing of the robot's operation in a predetermined process. A control device according to any one of claims 1 to 3.

7. The control means is The display information is to be displayed at the time when the robot's operation in a predetermined process is completed, or at the estimated completion time of the robot's operation in a predetermined process. A control device according to any one of claims 1 to 3.

8. The aforementioned display information is, This information represents the control content for the aforementioned robot. A control device according to any one of claims 1 to 3.

9. The aforementioned display information is, This is information representing the desired state achieved by the robot. A control device according to any one of claims 1 to 3.

10. This system creates an input form that receives input from an external source, and display information that includes information related to the robot's movements, which is superimposed on the input form. Control method.