Information processing device, information processing system, and program

Abstract

This information processing device is provided with a display unit, an input unit for receiving user input, and a control unit. The control unit displays, on the display unit, a layout of a candidate set for multiple hardware components required for work of a robot, and evaluation indices of the work according to the layout. When the input unit receives first input involving editing of the layout, the control unit acquires the evaluation indices based on the first input to redisplay the evaluation indices on the display unit.

Description

Information processing device, information processing system, and program

[0001] The present disclosure relates to an information processing device.

[0002] Patent Document 1 discloses a technology for processing information relating to a robot.

[0003] JP 2017-24113 A

[0004] An information processing device, an information processing system, and a program are disclosed. In one embodiment, the information processing device includes a display unit, an input unit that accepts user input, and a control unit. The control unit causes the display unit to display a layout of a candidate set of hardware required for a robot task and an evaluation index for the task corresponding to the layout, and when the input unit accepts a first input related to editing the layout, causes the control unit to calculate an evaluation index based on the first input and display it again on the display unit.

[0005] In one embodiment, an information processing system includes the information processing device described above and a display device communicably connected to the information processing device. The information processing device is capable of outputting information about a layout to the display device. The display device virtually reproduces and displays the layout based on the information.

[0006] In one embodiment, the program is a program for causing a computer device to function as the information processing device or the information processing system.

[0007] FIG. 1 is a schematic diagram showing an example of the configuration of an information processing device. FIG. 2 is a schematic diagram showing an example of a display of the information processing device. FIG. 3 is a schematic diagram showing an example of a display of the information processing device. FIG. 4 is a schematic diagram showing an example of a display of the information processing device. FIG. 5 is a schematic diagram showing an example of a display of the information processing device. FIG. 6 is a schematic diagram showing an example of a display of the information processing device. FIG. 7 is a schematic diagram showing an example of a display of the information processing device. FIG. 8 is a schematic diagram showing an example of a display of the information processing device. FIG. 9 is a schematic diagram showing an example of a display of the information processing device. FIG. 10 is a schematic diagram showing an example of a display of the information processing device. FIG. 11 is a schematic diagram showing an example of a display of the information processing device. FIG. 12 is a schematic diagram showing an example of a display of the information processing device. FIG. 13 is a schematic diagram showing an example of a display of the information processing device. FIG. 14 is a schematic diagram showing an example of a display of the information processing device. FIG. 15 is a schematic diagram showing an example of a display of the information processing device. FIG. 16 is a schematic diagram showing an example of a display of the information processing device. FIG. 17 is a schematic diagram showing an example of a display of the information processing device. FIG. 18 is a schematic diagram showing an example of a display of the information processing device. FIG. 19 is a schematic diagram showing an example of a display of the information processing device. FIG. 20 is a schematic diagram showing an example of a display of the information processing device. FIG. 21 is a schematic diagram showing an example of a display of the information processing device. FIG. 22 is a schematic diagram showing an example of a display on an information processing device. FIG. 23 is a schematic diagram showing an example of a display on an information processing device. FIG. 24 is a schematic diagram showing an example of a display on an information processing device. FIG. 25 is a schematic diagram showing an example of a display on an information processing device. FIG. 26 is a schematic diagram for explaining an example of operation of an information processing device. FIG. 27 is a schematic diagram showing an example of a display on an information processing device. FIG. 28 is a schematic diagram showing an example of a display on an information processing device. FIG. 29 is a schematic diagram showing an example of a display on an information processing device. FIG. 30 is a schematic diagram showing an example of a display on a VR display device. FIG. 31 is a schematic diagram showing an example of a display on an AR display device. FIG. 32 is a schematic diagram showing an example of a display on an information processing device. FIG. 33 is a schematic diagram showing an example of a display on an information processing device. FIG. 34 is a schematic diagram showing an example of a display on an information processing device. FIG. 35 is a schematic diagram showing an example of a display on an information processing device. FIG. 36 is a schematic diagram showing an example of a display on an information processing device. FIG. 37 is a schematic diagram showing an example of a display on an information processing device. FIG. 38 is a schematic diagram showing an example of a display on an information processing device. FIG. 39 is a schematic diagram showing an example of a display on an information processing device. FIG. 40 is a schematic diagram showing an example of a display on an information processing device. FIG. 41 is a schematic diagram showing an example of a display on an information processing device.Fig. 42 is a schematic diagram showing an example of a display on an information processing device. Fig. 43 is a schematic diagram showing an example of a display on an information processing device. Fig. 44 is a schematic diagram showing an example of a display on an information processing device. Fig. 45 is a schematic diagram showing an example of a display on an information processing device. Fig. 46 is a schematic diagram showing an example of a display on an information processing device. Fig. 47 is a schematic diagram showing an example of a display on an information processing device. Fig. 48 is a schematic diagram showing an example of a display on an information processing device. Fig. 49 is a schematic diagram showing an example of a display on an information processing device. Fig. 50 is a schematic diagram showing an example of a display on an information processing device.

[0008] 1 is a schematic diagram showing an example of the configuration of an information processing device 1. The information processing device 1 is capable of receiving input from a user and executing processing in accordance with the received input. The information processing device 1 is, for example, a computer device. The information processing device 1 may be, for example, a desktop or notebook personal computer, a tablet terminal, a mobile phone such as a smartphone, or another device.

[0009] The information processing device 1 can have a dialogue with a user regarding the robot's work. Specifically, the information processing device 1 can listen to requests from the user regarding the robot's work and make suggestions to the user regarding the robot's work based on the requests. The information processing device 1 can also modify the content of the suggestions based on the user's request and present the modified suggestions to the user. Hereinafter, the robot's work, i.e., the work performed by a robot, may be referred to as robot work.

[0010] <Configuration Example of Information Processing Apparatus> As shown in FIG. 1, an information processing apparatus 1 includes, for example, a control unit 2, a storage unit 3, a display unit 4, an input unit 5, and an interface 6.

[0011] The interface 6 can communicate with, for example, a device external to the information processing device 1 (also referred to as an external device). The interface 6 may communicate with the external device via a network including the Internet or the like, or may communicate directly with the external device. The interface 6 may perform wired communication or wireless communication. The interface 6 may also be referred to as, for example, an interface circuit. The interface 6 may also be referred to as, for example, a communication unit or a communication circuit. The interface 6 inputs information received from the external device to the control unit 2. The interface 6 transmits information from the control unit 2 to the external device. The external device may include a cloud server, a mobile phone such as a smartphone, a tablet terminal, a personal computer, a VR display device, or an AR display device. VR is an abbreviation for Virtual Reality, and AR is an abbreviation for Augmented Reality.

[0012] The control unit 2 can control the other components of the information processing device 1 to provide overall management of the operation of the information processing device 1. The control unit 2 can also be referred to as, for example, a control circuit. The control unit 2 includes at least one processor to provide control and processing power for performing various functions, as described in more detail below.

[0013] According to various embodiments, the at least one processor may be implemented as a single integrated circuit (IC) or as multiple communicatively connected integrated circuits ICs and / or discrete circuits. The at least one processor may be implemented according to various known techniques.

[0014] In one embodiment, a processor includes one or more circuits or units configured to perform one or more data computational procedures or processes, for example, by executing instructions stored in associated memory. In other embodiments, a processor may be firmware (e.g., discrete logic components) configured to perform one or more data computational procedures or processes.

[0015] According to various embodiments, the processor may include one or more processors, controllers, microprocessors, microcontrollers, application specific integrated circuits (ASICs), digital signal processors, programmable logic devices, field programmable gate arrays, or any combination of these devices or configurations, or other known devices and configurations, to perform the functions described below.

[0016] The control unit 2 may include, for example, a central processing unit (CPU) and a graphics processing unit (GPU) as processors. The storage unit 3 may include a non-transitory recording medium readable by the CPU of the control unit 2, such as a read-only memory (ROM) and a random access memory (RAM). The storage unit 3 stores, for example, multiple types of programs 3a for controlling the information processing device 1. The various functions of the control unit 2 are realized, for example, by the CPU of the control unit 2 executing the programs 3a in the storage unit 3.

[0017] The configuration of the control unit 2 is not limited to the above example. For example, the control unit 2 may include multiple CPUs. The control unit 2 may also include at least one DSP (Digital Signal Processor). All or some of the functions of the control unit 2 may be realized by a hardware circuit that does not require software to realize the function. The storage unit 3 may also include a computer-readable non-transitory recording medium other than ROM and RAM. The storage unit 3 may also include, for example, a small hard disk drive or SSD (Solid State Drive).

[0018] The input unit 5 is capable of accepting various inputs from the user. The input unit 5 may include, for example, a mouse and a keyboard. The input unit 5 may also include a touch sensor that accepts touch operations by the user. The input unit 5 may also include a microphone that accepts voice input by the user. The input unit 5 may also include a VR controller or a gesture controller. The control unit 2 can recognize the content of the user input accepted by the input unit 5 based on the output signal from the input unit 5.

[0019] The display unit 4 is capable of displaying various types of information under the control of the control unit 2. The display unit 4 has a display surface that displays various types of information. The display unit 4 may be, for example, a liquid crystal display, an organic electroluminescence (EL) display, or a plasma display. Furthermore, if the input unit 5 includes a touch sensor, the touch sensor and the display surface of the display unit 4 may constitute a touch panel display having a display function and a touch detection function. In this case, the input unit 5 can detect a touch operation on the display surface of the display unit 4. The input unit 5 and the display unit 4 constitute a user interface. Hereinafter, the term "display surface" refers to the display surface of the display unit 4. Furthermore, the term "display" refers to a display on the display surface of the display unit 4.

[0020] <Example of operation of information processing device> An icon for executing a predetermined application program 3 a included in multiple types of programs 3 a in the storage unit 3 is displayed on the display surface of the information processing device 1. When the control unit 2 executes the predetermined application program 3 a, the information processing device 1 can have a dialogue with the user regarding robot work.

[0021] When a user selects a predetermined icon displayed on the display surface, that is, when the input unit 5 receives a selection input for selecting a predetermined icon, the control unit 2 starts executing a predetermined application program 3a in the storage unit 3. The selection input for selecting an object displayed on the display surface, such as an icon, may be a touch operation on the object, or a predetermined operation of the mouse with the pointer positioned on the object.

[0022] When the control unit 2 starts executing a predetermined application program 3a, the display unit 4 displays the application screen 10 under the control of the control unit 2. The control unit 2 can cause the display unit 4 to display the application screen 10 based on the information stored in the storage unit 3.

[0023] 2 is a schematic diagram showing an example of an application screen 10. The application screen 10 includes, for example, a main screen 11 used for interaction and a flow screen 15 including a flow diagram 16 showing the stages of an interaction between the information processing device 1 and a user. On the application screen 10, the flow screen 15 is displayed on the left edge, and the main screen 11 is displayed in the remaining portion.

[0024] In the flow diagram 16, for example, multiple stages of a dialogue are represented by circles, squares, and diamonds. In the flow diagram 16, the mark indicating the current stage is highlighted. In this example, the mark indicating the current stage is shaded to indicate the highlighting. In each screen displayed on the display surface, which will be described later, the highlighted area is also shaded.

[0025] The display on the main screen 11 switches sequentially. When execution of a predetermined application program 3a in the storage unit 3 starts, a sign-in screen is first displayed on the main screen 11. When the user signs in using the sign-in screen, the project management screen 20, as shown in FIG. 2, is displayed on the main screen 11.

[0026] <Example of Project Management Screen> The project management screen 20 is a screen that displays a list of projects that have been launched in the past. A project is a task that is carried out between the information processing device 1 and a user, in which the information processing device 1 and the user have a dialogue in stages, the information processing device 1 makes a proposal to the user regarding a robot task based on a request from the user, and the task continues until the user finally agrees to the proposal.

[0027] The project management screen 20 includes a project name 21, a thumbnail 22 related to the project, and a project identification number 23. The project management screen 20 also includes titles 25 of robot operations handled in the project, and progress information 26 showing the progress of the project.

[0028] The types of progress information 26 include, for example, progress information 26a indicating that the project is in progress, progress information 26b indicating that the project has been completed, and progress information 26c indicating that the project is pending. When the user selects progress information 26 displayed on the display surface, that is, when the input unit 5 receives a selection input to select the progress information 26, a confirmation screen 30 for confirming the proposal content for the project having the progress status indicated by the selected progress information 26 is displayed on the main screen 11. Figure 3 is a schematic diagram showing an example display of the application screen 10 including the confirmation screen 30.

[0029] The confirmation screen 30, which is displayed when progress information 26a indicating that the project is in progress is selected, shows the latest proposed content for the project in progress. Furthermore, the confirmation screen 30, which is displayed when progress information 26b indicating that the project is completed is selected, shows the final proposed content for the completed project, i.e., the proposed content to which the user finally agreed. The confirmation screen 30, which is displayed when progress information 26c indicating that the project is pending is selected, shows the final proposed content for the pending project. Details of the confirmation screen 30 will be described later.

[0030] The project management screen 20 includes an instruction area 27 for instructing to return the main screen 11 to the previous display. When the user selects the instruction area 27, the main screen 11 returns to the sign-in screen.

[0031] The project management screen 20 includes an instruction area 28 for instructing the launch of a new project. When the user selects the instruction area 28, the dialogue advances by one stage and a new project is launched. Then, a work outline hearing screen 40 for hearing from the user about an outline of the desired robot work is displayed on the main screen 11. The dialogue stage can also be considered, for example, as a project stage. The work outline hearing screen 40 can also be considered a screen for hearing requests from the user.

[0032] <Example of Work Outline Hearing Screen> FIG. 4 is a schematic diagram showing an example of a work outline hearing screen 40. The work outline hearing screen 40 includes, for example, multiple designation areas 41. Each designation area 41 is an area for the user to specify an outline of the robot work desired by the user. The multiple designation areas 41 include, for example, a designation area 41a corresponding to a first work outline and a designation area 41b corresponding to a second work outline. The first work outline, for example, is a content in which the robot holds workpieces aligned in a tray on the loading side (also referred to as an in-tray) of the work environment, moves them to a tray on the unloading side (also referred to as an out-tray), and aligns the workpieces in the out-tray. The second work outline, for example, is a content in which the robot holds workpieces stacked loosely in the in-tray.

[0033] When the user selects the designation area 41a, the control unit 2 recognizes that the first work summary has been specified, and sets the initial settings of the screen to be displayed after the work summary hearing screen 40 to a setting corresponding to the first work summary. On the other hand, when the user selects the designation area 41b, the control unit 2 recognizes that the second work summary has been specified, and sets the initial settings of the screen to be displayed after the work summary hearing screen 40 to a setting corresponding to the second work summary. When the designation area 41 is selected, the designation area 41 is highlighted. Figure 5 is a schematic diagram showing an example of the state when the designation area 41a is selected.

[0034] The work summary hearing screen 40 includes an instruction area 45 for instructing to return the main screen 11 to the previous display. When the user selects the instruction area 45, the main screen 11 returns to the project management screen 20, and the dialogue returns to the previous stage.

[0035] The work overview hearing screen 40 includes an instruction area 46 for instructing to switch the display of the main screen 11 to the next one. When the user selects the instruction area 46 after selecting the designation area 41, the dialogue advances to the next stage, and a detail hearing screen 50 for hearing the user's request in detail is displayed on the main screen 11.

[0036] 6 is a schematic diagram showing an example of a detailed listening screen 50 that is displayed when the designation area 41a is selected and then the instruction area 46 is selected. The detailed listening screen 50 includes instruction areas 51 and 52. The detailed listening screen 50 also includes the above-mentioned instruction areas 45 and 46.

[0037] The instruction area 51 is an area for instructing the display of an object hearing screen 60 for hearing requests from the user regarding the workspace, in-tray, out-tray, and workpieces. The instruction area 52 is an area for instructing the display of a priority hearing screen 170 for hearing from the user about priority details regarding robot work.

[0038] When the designation area 51 is selected, an object listening screen 60 is displayed on the detailed listening screen 50. When the designation area 52 is selected, a priority listening screen 170 is displayed on the detailed listening screen 50. Figure 6 shows an example of how the object listening screen 60 is displayed on the detailed listening screen 50 when the designation area 51 is selected.

[0039] <Example of Object Listening Screen> The object listening screen 60 includes designation areas 61 to 64. The designation area 61 is an area for instructing the display of a space listening screen 70 for listening to requests regarding the workspace from the user. When the designation area 61 is selected, the space listening screen 70 is displayed on the object listening screen 60. FIG. 6 shows an example of how the designation area 61 is selected and the space listening screen 70 is displayed on the object listening screen 60. When the designation area 61 is selected, for example, the designation area 61 is surrounded by a frame. The same applies when the designation areas 62 to 64 are selected.

[0040] The instruction area 62 is an area for instructing the display of an in-tray listening screen 80 for listening to an in-tray-related request from the user. When the instruction area 62 is selected, the in-tray listening screen 80 is displayed on the object listening screen 60.

[0041] The instruction area 63 is an area for instructing the display of an out-tray listening screen 90 for listening to a request regarding the out-tray from the user. When the instruction area 63 is selected, the out-tray listening screen 90 is displayed on the object listening screen 60.

[0042] The instruction area 64 is an area for instructing the display of a workpiece listening screen 100 for listening to a request regarding a workpiece from the user. When the instruction area 64 is selected, the workpiece listening screen 100 is displayed on the object listening screen 60.

[0043] <Example of Space Listening Screen> The space listening screen 70 includes, for example, a designation area 75 for specifying the size of the workspace. The designation area 75 includes a slider for specifying the size of the workspace. The user can move the slider, for example, left and right, by making a predetermined input to the input unit 5. The user can specify the size of the workspace by moving the slider left and right. When the slider is moved to the right, the size of the specified workspace increases. On the other hand, when the slider is moved to the left, the size of the specified workspace decreases. The input unit 5 accepts the user's operation of the slider and accepts the user's specification of the size of the workspace. Note that the method of operating the slider included in the designation area, which will be described later, is the same as the method of operating the slider included in the designation area 75.

[0044] The space hearing screen 70 includes a work summary image 71 that shows an overview of the desired robot work as heard from the user on the work summary hearing screen 40. The example in FIG. 6 is an example when the designated area 41a included in the work summary hearing screen 40 is selected, and therefore, FIG. 6 shows a work summary image 71 that shows the first work summary specified by the user. The work summary image 71 that shows the first work summary includes an image 71a that shows the work space and an image 71b that shows the robot. Furthermore, since both an in-tray and an out-tray are used in the first work summary, the work summary image 71 includes an image 71c that shows the in-tray and an image 71d that shows the out-tray.

[0045] The work overview image 71 changes depending on the size of the work space specified in the designation area 75. Fig. 7 shows the work overview image 71 when the slider in the designation area 75 is moved to the left, reducing the size of the designated work space. Fig. 8 shows the work overview image 71 when the slider in the designation area 75 is moved to the right, increasing the size of the designated work space.

[0046] <Example of In-Tray Listening Screen> Figure 9 is a schematic diagram showing an example of the display of the In-Tray Listening screen 80. The In-Tray Listening screen 80 includes designation areas 85 to 87. Designation area 85 is an area for designating the degree of dispersion of workpieces within the In-Tray (also referred to as the In-Tray work dispersion degree). The In-Tray work dispersion degree can be referred to as, for example, the degree of dispersion of workpieces within the In-Tray, or the degree of misalignment of workpieces within the In-Tray. Designation area 86 is an area for designating the size of the In-Tray. Designation area 87 is an area for designating the number of workpieces within the In-Tray.

[0047] Designation area 85 includes a slider, similar to the above-described designation area 75. As the slider moves to the right, the specified in-tray work distribution degree increases. On the other hand, as the slider moves to the left, the specified in-tray work distribution degree decreases. When the slider is set to the left end, it is specified that the workpieces are aligned and arranged in the in-tray. On the other hand, when the slider is set to the right of the left end, it is specified that the workpieces are arranged dispersedly in the in-tray. In other words, it is specified that the workpieces are piled loosely in the in-tray. Designation areas 86 and 87 each include a slider, similar to the designation area 85.

[0048] The initial setting of the slider position of the designation area 85 is, for example, a position corresponding to the outline of the desired robot work obtained from the user on the work outline hearing screen 40. Fig. 9 shows the initial setting position of the slider of the designation area 85. The example of Fig. 9 shows the case where the designation area 41a included in the work outline hearing screen 40 is selected. In the first work outline, the robot holds workpieces aligned in an in-tray, and therefore the initial setting position of the slider of the designation area 85 is set to the leftmost position.

[0049] The intray listening screen 80 includes a screen 81 showing an intray and workpieces according to the content specified in the specified areas 85 to 87. The screen 81 includes an intray image 82 showing an intray having a size specified in the specified area 86, and a workpiece image 83 showing a workpiece in the intray according to the content specified in the specified areas 85 and 86.

[0050] The workpiece image 83 changes depending on the in-tray work distribution degree specified in the specification area 85. Figure 10 shows the workpiece image 83 when the slider in the specification area 85 is moved to the right and the specified in-tray work distribution degree increases.

[0051] The intray image 82 changes depending on the size of the intray specified in the specification area 86. Fig. 11 shows the intray image 82 when the slider in the specification area 86 is moved to the right and the specified intray size increases.

[0052] The workpiece image 83 changes depending on the number of workpieces in the in-tray specified in the specification area 87. Figure 12 shows the workpiece image 83 when the slider in the specification area 87 is moved to the right and the number of workpieces in the specified in-tray increases.

[0053] The intray listening screen 80 includes a designation area 89 for instructing the addition of an intray to be listened to. If the user wishes to use multiple intrays in a robot operation, the user selects the designation area 89. When the designation area 89 is selected, a screen 81 relating to the additional intray is displayed on the intray listening screen 80. When the screen 81 relating to the additional intray is displayed, the user can specify the additional intray in designation areas 85 to 87.

[0054] Each time a designation area 89 is selected, a new designation area 89 is added to the in-tray listening screen 80, and by selecting a new designation area 89, it becomes possible to specify the third and subsequent in-trays in the designation areas 85 to 87.

[0055] <Example of Out-Tray Listening Screen> Fig. 13 is a schematic diagram showing a display example of the out-tray listening screen 90. The out-tray listening screen 90 includes designation areas 95 to 97. Designation area 95 is an area for designating the degree of dispersion of workpieces in the out-tray (also referred to as out-tray work dispersion degree). Designation area 96 is an area for designating the size of the out-tray. Designation area 97 is an area for designating the number of workpieces in the out-tray.

[0056] Each of the designation areas 95-97 includes a slider. When the slider in designation area 95 is in the leftmost position, it designates that the workpieces are aligned in the outlay.

[0057] The initial setting of the slider position in the designation area 95 is, for example, a position corresponding to the outline of the desired robot work obtained from the user on the work outline hearing screen 40. Fig. 13 shows the initial setting positions of the slider in the designation area 95. The example in Fig. 13 is an example when the designation area 41a included in the work outline hearing screen 40 is selected. In the first work outline, the robot is to align and place the workpieces on the out-tray, and therefore the initial setting position of the slider in the designation area 95 is set to the leftmost position.

[0058] The out-tray listening screen 90 includes a screen 91 showing the out-tray and workpieces according to the content specified in the designation areas 95 to 97. The screen 91 includes an out-tray image 92 showing the out-tray having a size specified in the designation area 96, and a workpiece image 93 showing the workpieces in the out-tray according to the content specified in the designation areas 95 and 96.

[0059] Like the workpiece image 83 included in the in-tray listening screen 80, the workpiece image 93 changes depending on the out-tray work distribution degree specified in the designation area 95. Also, like the in-tray image 82 included in the in-tray listening screen 80, the out-tray image 92 changes depending on the out-tray size specified in the designation area 96. Also, like the workpiece image 83, the workpiece image 93 changes depending on the number of workpieces in the out-tray specified in the designation area 97.

[0060] The out-tray listening screen 90 includes a designation area 99 for instructing the addition of an out-tray to be listened to. If the user desires to use multiple out-trays in the robot operation, the user selects the designation area 99. When the designation area 99 is selected, a screen 91 relating to the additional out-tray is displayed on the out-tray listening screen 90. When the screen 91 relating to the additional out-tray is displayed, the user can specify the additional out-tray in the designation areas 95 to 97. Note that the out-tray listening screen 90 may also allow the user to assign an alignment order number when aligning workpieces in the out-tray.

[0061] Each time a designation area 99 is selected, a new designation area 99 is added to the out-tray listening screen 90, and by selecting a new designation area 99, it becomes possible to specify the third and subsequent out-trays in the designation areas 95 to 97.

[0062] <Example of Workpiece Listening Screen> Fig. 14 is a schematic diagram showing an example of the display of a workpiece listening screen 100. The workpiece listening screen 100 displays a workpiece processing listening screen 101 for listening to requests from the user regarding processing to be performed on workpieces between the intray and the outtray. The workpiece processing listening screen 101 includes a designation area 102 for instructing the display of a designation screen 103 for designating the type of processing for the workpiece (in other words, the type of processing method). When the designation area 102 is selected, the designation screen 103 is displayed as shown in Fig. 14.

[0063] The designation screen 103 includes a plurality of designation areas 104 corresponding to a plurality of types of processing. The designation areas 104 are areas for designating the type of processing corresponding to the designation area 104 as processing to be performed on the workpiece.

[0064] The multiple designated areas 104 include, for example, designated areas 104a, 104b, 104c, and 104d. Designated area 104a is an area corresponding to a processing process for processing a workpiece. When designated area 104a is selected, a processing process is specified as the process to be performed on the workpiece. Designated area 104b is an area corresponding to an assembly process for assembling the workpiece. When designated area 104b is selected, an assembly process is specified as the process to be performed on the workpiece. Designated area 104c is an area corresponding to an inspection process for inspecting the workpiece. When designated area 104c is selected, an inspection process is specified as the process to be performed on the workpiece. Designated area 104d is an area corresponding to a movement process for moving the workpiece. When designated area 104d is selected, a movement process is specified as the process to be performed on the workpiece.

[0065] When a designated area 104 is selected, the user can specify, via the input unit 5, the content of the type of processing corresponding to the selected designated area 104 to the information processing device 1. A processing content screen 105 showing the content of the processing specified by the user is displayed on the work processing hearing screen 101. Fig. 14 shows the processing content screen 105 (also referred to as processing content screen 105a) that is displayed when a designated area 104b corresponding to an assembly process is selected.

[0066] For example, after designating the designated area 104b, the user can specify the details of assembly via the input unit 5. For example, suppose the user designates that a first object (also referred to as object 1) and a second object (also referred to as object 2) are assembled to manufacture a third object (also referred to as object 3) as a finished product. In this case, as shown in FIG. 14 , the processing content screen 105a displays a figure 106a representing the first object, a figure 106b representing the second object, and a figure 106c representing the third object. Furthermore, the processing content screen 105a also displays an arrow 106d indicating how the first object and the second object are assembled to complete the third object.

[0067] When the user designates a first object and a second object as objects to be assembled, the display unit 4 can display a first object detail listening screen 110 for listening to details about the first object and a second object detail listening screen 120 for listening to details about the second object on the work listening screen 100. Furthermore, when the user designates a third object as a finished product, the display unit 4 can display a third object detail listening screen 130 for listening to details about the third object on the work listening screen 100.

[0068] The first object detail listening screen 110 has an instruction area 110a for instructing display of the entire first object detail listening screen 110 when the entire first object detail listening screen 110 is not displayed. The second object detail listening screen 120 has an instruction area 120a for instructing display of the entire second object detail listening screen 120 when the entire second object detail listening screen 120 is not displayed. The third object detail listening screen 130 has an instruction area 130a for instructing display of the entire third object detail listening screen 130 when the entire third object detail listening screen 130 is not displayed.

[0069] Also, like the first object detail listening screen 110, the work processing listening screen 101 has an instruction area 101a for instructing the display of the entire work processing listening screen 101 when the entire work processing listening screen 101 is not displayed.

[0070] When a first object is designated on the work processing listening screen 101, only the designation area 110a of the first object detail listening screen 110 is displayed on the work listening screen 100, as shown in Fig. 14. Specifically, in Fig. 14, when the designation area 110a is selected when the entire first object detail listening screen 110 is not displayed, the entire first object detail listening screen 110 is displayed on the work listening screen 100, as shown in Fig. 15. At this time, with respect to the work processing listening screen 101, the second object detail listening screen 120, and the third object detail listening screen 130, only the designation area 101a, the designation area 120a, and the designation area 130a are displayed.

[0071] When a second object is designated on the work processing listening screen 101, as shown in Fig. 14, only the designation area 120a of the second object detail listening screen 120 is displayed on the work listening screen 100. Specifically, in Fig. 14, when the designation area 120a is selected when the entire second object detail listening screen 120 is not displayed, the entire second object detail listening screen 120 is displayed on the work listening screen 100, as shown in Fig. 16. At this time, with respect to the work processing listening screen 101, the first object detail listening screen 110, and the third object detail listening screen 130, only the designation area 101a, the designation area 110a, and the designation area 130a are displayed.

[0072] When a third object is designated on the work processing listening screen 101, as shown in Fig. 14, only the designation area 130a of the third object detail listening screen 130 is displayed on the work listening screen 100. Specifically, in Fig. 14, when the designation area 130a is selected when the entire third object detail listening screen 130 is not displayed, the entire third object detail listening screen 130 is displayed on the work listening screen 100 as shown in Fig. 17. At this time, with respect to the work processing listening screen 101, the first object detail listening screen 110, and the second object detail listening screen 120, only the designation area 101a, the designation area 110a, and the designation area 120a are displayed.

[0073] As shown in Figures 15 to 17, when the designation area 101a is selected when the entire work processing hearing screen 101 is not displayed, the entire work processing hearing screen 101 is displayed on the work hearing screen 100, as shown in Figure 14.

[0074] <Example of First Object Details Listening Screen> On the first object details listening screen 110, for example, the shape and material of the first object to be assembled can be specified. As shown in FIG. 15 , the first object details listening screen 110 includes designation areas 113 to 117. Designation area 113 is an area for specifying the number of corners (in other words, the number of angles) of the outer shape of the first object. Designation area 114 is an area for specifying the curvature of the outer shape of the first object. Designation area 115 is an area for specifying the number of holes in the first object. Designation area 116 is an area for specifying the surface roughness of the first object. Designation area 117 is an area for specifying the softness of the first object. Each of designation areas 113 to 117 includes a slider.

[0075] The first object detail hearing screen 110 includes a first object image 111 representing a first object according to the content designated in the designation areas 113 to 117. The first object image 111 changes according to the content designated in the designation areas 113 to 117.

[0076] <Example of Second Object Details Listening Screen> On the second object details listening screen 120, for example, the shape and material of the second object to be assembled can be specified. As shown in FIG. 16 , the second object details listening screen 120 includes designation areas 123 to 127. Designation area 123 is an area for designating the number of corners of the outer shape of the second object. Designation area 124 is an area for designating the curvature of the outer shape of the second object. Designation area 125 is an area for designating the number of holes in the second object. Designation area 126 is an area for designating the surface roughness of the second object. Designation area 127 is an area for designating the softness of the second object. Each of designation areas 123 to 127 includes a slider.

[0077] The second object detail hearing screen 120 includes a second object image 121 representing a second object according to the content designated in the designation areas 123 to 127. The second object image 121 changes according to the content designated in the designation areas 123 to 127.

[0078] <Example of Third Object Detail Interview Screen> The third object detail interview screen 130 allows, for example, the user to specify an assembly method for the third object. As shown in FIG. 17 , the third object detail interview screen 130 includes an object image placement area 150. The object image placement area 150 includes a first object image 151 representing the first object whose shape and material are specified on the first object detail interview screen 110, and a second object image 152 representing the second object whose shape and material are specified on the second object detail interview screen 120. Hereinafter, when there is no need to particularly distinguish between the first object image 151 and the second object image 152, they may each be referred to as an object image 153.

[0079] The third object detail hearing screen 130 includes an assembly method designation screen 140 for designating an assembly method for the third object. The user can move the object image 153 in the object image placement area 150 to the assembly method designation screen 140 by performing a predetermined input on the input unit 5. The object image 153 may be moved to the assembly method designation screen 140 by drag and drop, for example.

[0080] For example, a second object image 152 is placed on the assembly method specification screen 140. Then, as shown in FIG. 17 , when a first object image 151 is overlaid on the second object image 152, it is specified that the second object and the first object are combined to manufacture a third object. Note that the second object image 152 may be overlaid on the first object image 151 after the first object image 151 is placed on the assembly method specification screen 140. Furthermore, if there are areas on the second object where multiple first objects can be overlaid, the order in which the multiple first objects are to be overlaid on the second object may be input in advance to the information processing device 1, and the multiple first object images 151 may be automatically overlaid on the second object image 152 on the assembly method specification screen 140.

[0081] In this way, the user can use the third object details hearing screen 130 to specify the assembly method for the third object.

[0082] The following describes the operation of the information processing device 1 when it is specified on the workpiece hearing screen 100 that a third object is manufactured by assembling a first object and a second object.

[0083] <Example of Priority Content Listening Screen> FIG. 18 is a schematic diagram showing an example of the priority listening screen 170 displayed when the designation area 52 included in the detailed listening screen 50 is selected.

[0084] The priority hearing screen 170 includes, for example, designation areas 171 to 176. Designation area 171 is an area for designating the priority of safety for the robot work. Designation area 172 is an area for designating the priority of productivity for the robot work. Designation area 173 is an area for designating the priority of delivery date for a product (e.g., a third object) produced by the robot work. Designation area 174 is an area for designating the priority of inventory for a product produced by the robot work. Designation area 175 is an area for designating the priority of quality for a product produced by the robot work. Designation area 176 is an area for designating the priority of cost for a product produced by the robot work. Each of designation areas 171 to 176 has a slider.

[0085] For example, if the user wants to prioritize the safety of the robot work, the user moves the slider in designation area 171 to the right. Furthermore, if the user wants to prioritize the productivity of the robot work, the user moves the slider in designation area 172 to the right. Furthermore, if the user wants to prioritize short delivery times for products produced by robot work, the user moves the slider in designation area 173 to the left. Furthermore, if the user wants to prioritize low inventory levels for products produced by robot work, the user moves the slider in designation area 173 to the left. Furthermore, if the user wants to prioritize the quality of products produced by robot work, the user moves the slider in designation area 175 to the right. Furthermore, if the user wants to prioritize low costs for products produced by robot work, the user moves the slider in designation area 176 to the left.

[0086] Furthermore, on the priority hearing screen 170, evaluation indices for robot work based on the priority contents designated in the designation areas 171 to 176 are displayed, for example, in a radar chart 178. Hereinafter, the evaluation indices for robot work will be referred to as work evaluation indices.

[0087] In this example, for example, five task evaluation indexes are used. The five task evaluation indexes include, for example, a success index indicating the degree of success of the robot task; a task time index indicating the task time of the robot task; a range of motion index indicating the range of motion of the robot in the workspace available for the robot task; a psychological safety index indicating the psychological safety of humans during the robot task; and a stability index indicating the stability of the robot's movements during the robot task. Psychological safety of humans during the robot task refers to safety such that the sound of the robot's operation or the movement of the robot arm crossing in front of the worker's eyes does not cause auditory or visual stress to workers working around the robot. The priority listening screen 170 displays a radar chart 178 showing the success index, task time index, range of motion index, psychological safety index, and stability index. Each of the success index, task time index, range of motion index, psychological safety index, and stability index is expressed, for example, as a numerical value between 0 and 1.

[0088] In this example, on the screen displayed on the display unit 4, the success index, work time index, range of motion index, psychological safety index, and stability index are respectively indicated by, for example, "Product Quality," "Cycle Time," "Layout Scale," "Safety," and "Stability." Hereinafter, the success index, work time index, range of motion index, psychological safety index, and stability index will be collectively referred to as robot KPI. KPI is an abbreviation for Key Performance Indicator.

[0089] The control unit 2 of the information processing device 1 determines the robot KPIs to be displayed on the priority content hearing screen 170 based on, for example, the content specified in the designation areas 171 to 176. The control unit 2, for example, quantifies each of the content specified in the six designation areas 171 to 176. The control unit 2 then calculates the success index, the work time index, the range of motion index, the psychological safety index, and the stability index based on the six obtained numerical values. For example, the control unit 2 may determine the success index as a value obtained by weighting and adding the six numerical values. In this case, the weighting coefficients are stored in advance in the storage unit 3. The work time index, the range of motion index, the psychological safety index, and the stability index may also be calculated in a similar manner.

[0090] In this way, the user uses the object hearing screen 60 and the priority hearing screen 170 to communicate requests regarding the intray, outtray, and workpiece, and priorities regarding the robot work, to the information processing device 1, and then selects the instruction area 46 included in the detail hearing screen 50. When the instruction area 46 is selected, a reference hearing screen 180 for hearing from the user about the reference position to be used in the robot work is displayed on the main screen 11. When the instruction area 45 included in the detail hearing screen 50 is selected, the dialogue returns to the previous step, and the work outline hearing screen 40 is displayed on the main screen 11.

[0091] <Example of Reference Listening Screen> Figure 19 is a schematic diagram showing a display example of the application screen 10 including a reference listening screen 180. The reference listening screen 180 includes instruction areas 45 and 46. The reference listening screen 180 also displays a work reference listening screen 181 for listening to the reference position of a workpiece from the user. Figure 19 shows the work reference listening screen 181 when the user desires to perform an assembly process for assembling a first object and a second object to create a third object, as in the examples of Figures 14 to 17.

[0092] The work-based listening screen 181 includes, for example, designation areas 182 to 184. Designation area 182 is an area for instructing the display of a first-object-based listening screen 191 for listening to the reference position to be set for the first object. When designation area 182 is selected, as shown in FIG. 19, the first-object-based listening screen 191 is displayed on the work-based listening screen 181. Designation area 183 is an area for instructing the display of a second-object-based listening screen 192 for listening to the reference position to be set for the second object. When designation area 183 is selected, as shown in FIG. 20, the second-object-based listening screen 192 is displayed on the work-based listening screen 181. Designation area 184 is an area for instructing the display of a third-object-based listening screen 193 for listening to the reference position to be set for the third object. When designation area 184 is selected, as shown in FIG. 21, the third-object-based listening screen 193 is displayed on the work-based listening screen 181.

[0093] <Example of First-Object-Based Listening Screen> On the first-object-based listening screen 191, it is possible to specify the position of the first object held by the robot, the assembly reference position of the first object when assembling the first object and the second object, and the placement reference position of the first object in the tray. Hereinafter, the position of the object held by the robot may be referred to as the holding reference position or the handling reference position.

[0094] As will be described later, the assembly reference position of the second object when assembling the first object and the second object can be specified on the second object-reference listening screen 192. In this example, it is assumed that the first object and the second object are assembled so that the assembly reference position of the first object and the assembly reference position of the second object coincide with each other.

[0095] Furthermore, when the first objects are aligned and arranged in the tray, as described below, the user can specify the arrangement reference position of the tray where the first objects are to be arranged. In this example, when the first objects are aligned and arranged in the tray, it is assumed that the first objects are aligned and arranged in the tray so that the arrangement reference position of the first objects coincides with the arrangement reference position of the tray. Note that the application screen 10 may be configured so that the user can indicate whether they want the objects to be aligned physically or logically with respect to the rule for aligning the objects with respect to the arrangement reference position.

[0096] As shown in FIG. 19 , the first-object-based listening screen 191 includes a first object image 191a representing the first object. The first-object-based listening screen 191 also includes a coordinate system graphic 191c representing a three-dimensional coordinate system for specifying a holding reference position (in other words, a handling reference position). The user can move the coordinate system graphic 191c on the first-object-based listening screen 191 by performing a predetermined input on the input unit 5. The holding reference position of the first object is specified by overlaying an origin portion indicating the origin of the coordinate system graphic 191c on the first object image 191a. The position of the origin portion of the coordinate system graphic 191c on the first object image 191a indicates the holding reference position of the first object (in other words, the handling reference position).

[0097] The orientation of the coordinate system graphic 191c may indicate the orientation of the first object when held by the robot. In this case, the user may be able to specify the orientation of the first object when held by the robot by setting the orientation of the coordinate system graphic 191c via the input unit 5. The orientation of the coordinate system graphic 191c may be used to specify other information.

[0098] The first object-based listening screen 191 includes a coordinate system graphic 191b representing a three-dimensional coordinate system for specifying the assembly reference position of the first object. The user can move the coordinate system graphic 191b on the first object-based listening screen 191 by making a predetermined input to the input unit 5. The assembly reference position of the first object is specified by overlaying an origin portion indicating the origin of the coordinate system graphic 191b on the first object image 191a. The position of the origin portion of the coordinate system graphic 191b on the first object image 191a indicates the assembly reference position of the first object.

[0099] The orientation of the coordinate system graphic 191b may indicate an assembly orientation of the first object when the robot assembles the first object and the second object. In this case, the user may be able to specify the assembly orientation of the first object by setting the orientation of the coordinate system graphic 191b via the input unit 5. The orientation of the coordinate system graphic 191b may be used to specify other information.

[0100] The first object-based listening screen 191 includes a coordinate system graphic 191d representing a three-dimensional coordinate system for specifying the placement reference position of the first object in the tray. The user can move the coordinate system graphic 191d on the first object-based listening screen 191 by making a predetermined input to the input unit 5. The placement reference position of the first object is specified by overlaying an origin portion indicating the origin of the coordinate system graphic 191d on the first object image 191a. The position of the origin portion of the coordinate system graphic 191d on the first object image 191a indicates the placement reference position of the first object.

[0101] The orientation of the coordinate system graphic 191d may indicate the placement orientation of the first object in the tray. In this case, the user may be able to specify the placement orientation of the first object by setting the orientation of the coordinate system graphic 191d via the input unit 5. The orientation of the coordinate system graphic 191d may be used to specify other information.

[0102] <Example of Second-Object-Based Listening Screen> The second-object-based listening screen 192 allows the user to specify a holding reference position of the second object, an assembly reference position of the second object when assembling the first object and the second object, and a placement reference position of the second object in the tray. When the second objects are aligned and arranged in the tray, as described below, the user can specify a placement reference position of the tray in which the second objects are arranged. In this example, when the second objects are aligned and arranged in the tray, it is assumed that the second objects are aligned and arranged in the tray so that the placement reference position of the second objects coincides with the placement reference position of the tray.

[0103] As shown in FIG. 20 , the second-object-based listening screen 192 includes a second object image 192a representing the second object. The second-object-based listening screen 192 also includes a coordinate system graphic 192c representing a three-dimensional coordinate system for specifying a holding reference position (in other words, a handling reference position). The user can move the coordinate system graphic 192c on the second-object-based listening screen 192 by performing a predetermined input on the input unit 5. The holding reference position of the second object is specified by overlaying an origin portion indicating the origin of the coordinate system graphic 192c on the second object image 192a. The position of the origin portion of the coordinate system graphic 192c on the second object image 192a indicates the holding reference position of the second object (in other words, the handling reference position).

[0104] The orientation of the coordinate system graphic 192c may indicate the orientation of the second object when held by the robot. In this case, the user may be able to specify the orientation of the second object when held by the robot by setting the orientation of the coordinate system graphic 192c via the input unit 5. The orientation of the coordinate system graphic 192c may be used to specify other information.

[0105] The second object-based listening screen 192 includes a coordinate system graphic 192b representing a three-dimensional coordinate system for specifying the assembly reference position of the second object. The user can move the coordinate system graphic 192b on the second object-based listening screen 191 by making a predetermined input to the input unit 5. The assembly reference position of the second object is specified by overlaying an origin portion indicating the origin of the coordinate system graphic 192b on the second object image 192a. The position of the origin portion of the coordinate system graphic 192b on the second object image 192a indicates the assembly reference position of the second object.

[0106] The orientation of the coordinate system graphic 192b may indicate an assembly orientation of the second object when the robot assembles the first object and the second object. In this case, the user may be able to specify the assembly orientation of the second object by setting the orientation of the coordinate system graphic 192b via the input unit 5. The orientation of the coordinate system graphic 192b may be used to specify other information.

[0107] The second-object-based listening screen 192 includes a coordinate system graphic 192d representing a three-dimensional coordinate system for specifying the placement reference position of the second object in the tray. The user can move the coordinate system graphic 192d on the second-object-based listening screen 192 by making a predetermined input to the input unit 5. The placement reference position of the second object is specified by overlaying an origin portion indicating the origin of the coordinate system graphic 192d on the second object image 192a. The position of the origin portion of the coordinate system graphic 192d on the second object image 192a indicates the placement reference position of the second object.

[0108] The orientation of the coordinate system graphic 192d may indicate the arrangement orientation of the second object in the tray. In this case, the user may be able to specify the arrangement orientation of the second object by setting the orientation of the coordinate system graphic 192d via the input unit 5. The orientation of the coordinate system graphic 192d may be used to specify other information.

[0109] <Example of Third-Object-Based Listening Screen> The third-object-based listening screen 193 allows the user to specify a holding reference position of a third object manufactured by assembling a first object and a second object, and a placement reference position of the third object when placing the third object on a tray. As will be described later, the user can specify a placement reference position of the tray on which the third object will be placed. In this example, it is assumed that the third object is placed on the tray so that the placement reference position of the third object coincides with the placement reference position of the tray.

[0110] As shown in FIG. 21 , the third-object-based listening screen 193 includes a third object image 193a representing the assembled first and second objects, i.e., the third object. The third-object-based listening screen 193 also includes a coordinate system graphic 192c representing a three-dimensional coordinate system for specifying the holding reference position of the third object. The user can move the coordinate system graphic 193c on the third-object-based listening screen 193 by performing a predetermined input on the input unit 5. The holding reference position of the third object is specified by overlaying the origin portion of the coordinate system graphic 193c on the third object image 193a. The position of the origin portion of the coordinate system graphic 193c on the third object image 193a indicates the holding reference position (in other words, the handling reference position) of the third object.

[0111] The orientation of the coordinate system graphic 193c may indicate the orientation of the third object when held by the robot. In this case, the user may be able to specify the orientation of the third object when held by the robot by setting the orientation of the coordinate system graphic 193c via the input unit 5. The orientation of the coordinate system graphic 193c may be used to specify other information.

[0112] The third-object-based listening screen 193 includes a coordinate system graphic 193b representing a three-dimensional coordinate system for specifying the placement reference position of the third object. The user can move the coordinate system graphic 193b on the third-object-based listening screen 193 by making a predetermined input to the input unit 5. The assembly reference position of the third object is specified by overlaying an origin portion indicating the origin of the coordinate system graphic 193b on the third object image 193a. The position of the origin portion of the coordinate system graphic 193b on the third object image 193a indicates the placement reference position of the third object.

[0113] The orientation of the coordinate system graphic 193b may indicate the placement orientation of the third object in the tray. In this case, the user may be able to specify the placement orientation of the third object by setting the orientation of the coordinate system graphic 193b via the input unit 5. The orientation of the coordinate system graphic 193b may be used to specify other information.

[0114] In addition to the work reference listening screen 181, the display unit 4 can also display a tray reference listening screen 200 on the reference listening screen 180 for listening to the reference positions of the in-tray and out-tray from the user.

[0115] The tray-based listening screen 200 has an instruction area 200a for instructing the display of the entire tray-based listening screen 200 when the entire tray-based listening screen 200 is not displayed.

[0116] The work-based listening screen 181 also has an instruction area 181a for instructing the display of the entire work-based listening screen 181 when the entire work-based listening screen 181 is not displayed.

[0117] As shown in Fig. 19 and other figures, when the entire work-based listening screen 181 is displayed, a designation area 182a is displayed on the tray-based listening screen 200. When the designation area 200a is selected in this state, the entire tray-based listening screen 200 is displayed, as shown in Fig. 22, and only the designation area 181a is displayed on the work-based listening screen 181. When the designation area 181a is selected in this state, the entire work-based listening screen 181 is displayed, as shown in Fig. 19 and other figures, and only the designation area 200a is displayed on the tray-based listening screen 200.

[0118] <Example of Tray-Based Listening Screen> Fig. 22 shows an example of a tray-based listening screen 200 when a request regarding two intrays, i.e., the first intray and the second intray, is heard on the intray listening screen 80. That is, Fig. 22 shows an example of the tray-based listening screen 200 when a desire to use two intrays is heard from the user. The following describes the operation of the information processing device 1 when a request regarding the first intray and the second intray is heard on the intray listening screen 80.

[0119] The tray-reference listening screen 200 includes, for example, instruction areas 201a, 201b, and 201c. The instruction area 201a is an area for instructing the display of a first in-tray reference listening screen 210 for asking the user about the reference position of the first in-tray. When the instruction area 201a is selected, the tray-reference listening screen 200 displays the first in-tray reference listening screen 210, as shown in FIG.

[0120] The instruction area 201b is an area for instructing the display of a second in-tray reference listening screen 220 for asking the user about the reference position of the second in-tray. When the instruction area 201b is selected, the second in-tray reference listening screen 220 is displayed on the tray reference listening screen 200, as shown in FIG.

[0121] The instruction area 201c is an area for instructing the display of an out-tray reference listening screen 230 for asking the user about the reference position of the out-tray. When the instruction area 201c is selected, the out-tray reference listening screen 230 is displayed on the tray reference listening screen 200, as shown in FIG.

[0122] The tray-based listening screen 200 also includes a plurality of designated areas 202. A workpiece is associated with each designated area 202. The designated area 202 is an area for designating the corresponding workpiece as a workpiece to be placed on the tray.

[0123] The multiple designated areas 202 include designated area 202a corresponding to a first object, designated area 202b corresponding to a second object, and designated area 202c corresponding to a third object. When designated area 202a is selected, the first object is designated as a workpiece to be placed on the tray. When designated area 202b is selected, the second object is designated as a workpiece to be placed on the tray. When designated area 202c is selected, the third object is designated as a workpiece to be placed on the tray.

[0124] When multiple designated areas 202 are repeatedly selected, the multiple designated areas 202 may be automatically selected based on structural logical information, such as sorting the multiple designated areas 202 in order of object number, automatically arranging them into a 3x2 or 3x3 grid, and filling the grid by moving from the upper left to the right. Alternatively, the designated area 202 may be automatically selected based on table data indicating the relationship between each designated area 202 and the zth object (z is an integer equal to or greater than 1).

[0125] When the first intray reference listening screen 210 is displayed and a designated area 202 is selected, the object corresponding to the selected designated area 202 is designated as the workpiece to be placed in the first intray. For example, when the first intray reference listening screen 210 is displayed and a designated area 202a is selected, the first object corresponding to the selected designated area 202a is designated as the workpiece to be placed in the first intray.

[0126] When the second in-tray reference listening screen 220 is displayed and a designated area 202 is selected, the object corresponding to the selected designated area 202 is designated as the workpiece to be placed in the second in-tray. For example, when the second in-tray reference listening screen 220 is displayed and a designated area 202b is selected, the second object corresponding to the selected designated area 202b is designated as the workpiece to be placed in the second in-tray.

[0127] When the out-tray reference listening screen 230 is displayed and a designated area 202 is selected, the object corresponding to the selected designated area 202 is designated as the workpiece to be placed on the out-tray. For example, when the out-tray reference listening screen 230 is displayed and a designated area 202c is selected, the third object corresponding to the selected designated area 202c is designated as the workpiece to be placed on the out-tray.

[0128] 22 shows an example of a state in which a designated area 202a corresponding to a first object is selected when a first intray reference listening screen 210 is displayed. The first intray reference listening screen 210 includes a first intray image 211 representing the first intray.

[0129] Now, consider a case where the slider position included in the designation area 85 on the in-tray listening screen 80 described above is set to the right of the left edge, specifying that the workpieces are to be dispersed within the first in-tray. In other words, consider a case where the workpieces are to be stacked loosely within the first in-tray. In this case, the first in-tray reference listening screen 210 includes a coordinate system graphic 212 representing a three-dimensional coordinate system for specifying the dispersion reference position of the workpieces in the first in-tray. In this example, it is assumed that the workpieces in the first in-tray are dispersed outward from the dispersion reference position as the base point.

[0130] The user can move the coordinate system graphic 212 on the first intray reference listening screen 210 by making a predetermined input to the input unit 5. The dispersion reference position of the first intray is specified by superimposing an origin portion indicating the origin of the coordinate system graphic 212 on the first intray image 211. The position of the origin portion of the coordinate system graphic 212 on the first intray image 211 indicates the dispersion reference position of the first intray.

[0131] In addition to the coordinate system figure 212, the first in-tray reference listening screen 210 also displays multiple coordinate system figures 213 representing three-dimensional coordinate systems with different orientations from the coordinate system figure 212, in order to show how the workpieces are dispersed and arranged within the first in-tray.

[0132] Furthermore, when it is specified that the workpieces are to be dispersed within the first intray, the first intray reference listening screen 210 includes an input area 215 for inputting the number of workpieces in the first intray. In the example of FIG. 22 , the specified area 202a is selected, and therefore the number of first objects in the first intray can be input in the input area 215. When the user selects the input area 215, the number of workpieces in the first intray can be input to the information processing device 1. When the user inputs the number of workpieces in the first intray into the input unit 5, the input number is displayed in the input area 215.

[0133] 23 shows an example of a state in which a designated area 202b corresponding to a second object is selected when a second intray reference listening screen 220 is displayed. The second intray reference listening screen 220 includes a second intray image 221 representing the second intray.

[0134] Now, consider a case where the slider position included in the designation area 85 on the in-tray listening screen 80 described above is set to the left edge, specifying that the workpieces are to be aligned in the second in-tray. In this case, the second in-tray reference listening screen 220 includes a coordinate system graphic 222 representing a three-dimensional coordinate system for specifying the placement reference position of the second in-tray. In this example, when the second objects are aligned in the second tray, it is assumed that the second objects are aligned in the second tray so that the placement reference position of the second objects coincides with the placement reference position of the second tray.

[0135] The user can move the coordinate system graphic 222 on the second intray reference listening screen 220 by making a predetermined input to the input unit 5. The placement reference position of the second intray is specified by superimposing an origin portion indicating the origin of the coordinate system graphic 222 on the second intray image 221. The position of the origin portion of the coordinate system graphic 222 on the second intray image 221 indicates the placement reference position of the second intray.

[0136] In addition to the coordinate system figure 222, the second in-tray reference listening screen 220 also displays multiple coordinate system figures 223 representing three-dimensional coordinate systems that have the same orientation as the coordinate system figure 222, in order to show how objects are aligned and arranged in the second in-tray.

[0137] Furthermore, when it is specified that the workpieces are to be arranged in a line in the second intray, the second intray reference listening screen 220 includes input areas 225a and 225b. Input area 225a is an area for inputting the number of workpieces in the vertical direction (i.e., the column direction) in the second intray. Input area 225b is an area for inputting the number of workpieces in the horizontal direction (i.e., the row direction) in the second intray. In the example of FIG. 23 , since designation area 202b is selected, input area 225a allows the number of second objects in the vertical direction to be input, and input area 225b allows the number of second objects in the horizontal direction to be input.

[0138] When the user selects input area 225a, the user can input the numerical value of the vertical number of workpieces in the second intray into the information processing device 1. When the user inputs the numerical value of the vertical number of workpieces in the second intray into the input unit 5, the input numerical value is displayed in input area 225a. Similarly, when the user selects input area 225b, the user can input the numerical value of the horizontal number of workpieces in the second intray into the information processing device 1. When the user inputs the numerical value of the horizontal number of workpieces in the second intray into the input unit 5, the input numerical value is displayed in input area 225b.

[0139] In addition, if it is specified in the in-tray listening screen 80 that the workpieces are to be dispersed within the second in-tray, the display of the second in-tray reference listening screen 220 will be the same as the display of the first in-tray reference listening screen 210 shown in Figure 22.

[0140] Also, if the in-tray listening screen 80 specifies that the workpieces are to be aligned and placed in the first in-tray, the display of the first in-tray reference listening screen 210 will be the same as the display of the second in-tray reference listening screen 220 shown in Figure 23.

[0141] 24 shows an example of a state in which the designated area 202c corresponding to the third object is selected when the out-tray-reference listening screen 230 is displayed. The out-tray-reference listening screen 230 includes an out-tray image 231 representing the out-tray.

[0142] Now, consider a case where the slider position included in the designation area 95 is set to the left edge on the above-described out-tray listening screen 90, specifying that the workpieces are to be aligned in the out-tray. In this case, the out-tray reference listening screen 230 includes a coordinate system graphic 232 representing a three-dimensional coordinate system for specifying the placement reference position of the out-tray. In this example, when a third object is to be aligned in the out-tray, it is assumed that the third object is aligned in the out-tray so that the placement reference position of the third object coincides with the placement reference position of the out-tray.

[0143] The user can move the coordinate system graphic 232 on the out-tray reference listening screen 230 by making a predetermined input to the input unit 5. The placement reference position of the out-tray is specified by superimposing the origin portion indicating the origin of the coordinate system graphic 232 on the out-tray image 231. The position of the origin portion of the coordinate system graphic 232 on the out-tray image 231 indicates the placement reference position of the out-tray.

[0144] In addition to the coordinate system figure 232, the out-tray reference listening screen 230 also displays multiple coordinate system figures 233 representing three-dimensional coordinate systems that have the same orientation as the coordinate system figure 232, in order to show how the workpieces are aligned and arranged within the out-tray.

[0145] Furthermore, when it is specified that the workpieces are to be aligned in the out-tray, the out-tray reference listening screen 230 includes input areas 235a and 235b. Input area 235a is an area for inputting the numerical value of the vertical number of workpieces in the out-tray. Input area 235b is an area for inputting the numerical value of the horizontal number of workpieces in the second in-tray. In the example of FIG. 24 , since designation area 202c is selected, the numerical value of the vertical number of the second object in the out-tray can be input in input area 235a, and the numerical value of the horizontal number of the second object in the out-tray can be input in input area 235b.

[0146] When the user selects input area 235a, the user can input the numerical value of the vertical number of workpieces in the out-tray into the information processing device 1. When the user inputs the numerical value of the vertical number of workpieces in the out-tray into the input unit 5, the input numerical value is displayed in input area 235a. Similarly, when the user selects input area 235b, the user can input the numerical value of the horizontal number of workpieces in the out-tray into the information processing device 1. When the user inputs the numerical value of the horizontal number of workpieces in the out-in-tray into the input unit 5, the input numerical value is displayed in input area 235b.

[0147] In addition, if it is specified on the out-tray listening screen 90 that the workpieces are to be dispersed and placed within the out-tray, the display of the out-tray reference listening screen 230 will be the same as the display of the first in-tray reference listening screen 210 shown in Figure 22.

[0148] In this way, by listening to the user's desired reference positions for the workpieces and trays using the reference listening screen 180, listening to the user's requests regarding the robot operation is completed. When the instruction area 46 included in the reference listening screen 180 is selected, the dialogue advances one step, and an asset proposal screen 250 is displayed on the main screen 11 to propose to the user a candidate set of multiple assets required for the robot operation desired by the user. Based on the requests heard from the user so far, the control unit 2 determines a candidate set of multiple assets required for the robot operation to be presented to the user. Hereinafter, a candidate set of multiple assets may be referred to as a candidate asset set.

[0149] When the instruction area 45 included in the reference listening screen 180 is selected, the main screen 11 returns to the detailed listening screen 50 .

[0150] <Example of Asset Proposal Screen> Fig. 25 is a schematic diagram showing a display example of the application screen 10 including an asset proposal screen 250. As shown in Fig. 25, the asset proposal screen 250 includes a designation area 45. When the designation area 45 is selected, the dialogue returns to the previous step, and the main screen 11 returns to the reference listening screen 180.

[0151] Furthermore, the asset proposal screen 250 includes, for example, candidate set information 251 indicating an asset candidate set to be proposed to the user. For example, the asset proposal screen 250 includes multiple pieces of candidate set information 251 indicating multiple different asset candidate sets, respectively. The candidate set information 251 can also be said to be information or a screen for specifying an asset candidate set to be proposed to the user. The asset proposal screen 250 can also be said to be a screen showing at least one asset candidate set to be proposed to the user. Note that, here, an example will be described in which at least one asset candidate set has multiple asset candidate sets.

[0152] The control unit 2 determines a plurality of asset candidate sets required for a robot operation desired by the user based on requests heard up to now from the user by the information processing device 1. Then, the control unit 2 causes the display unit 4 to display an asset proposal screen 250 including a plurality of candidate set information 251 indicating each of the determined plurality of asset candidate sets.

[0153] 25 , the asset proposal screen 250 proposes four asset candidate sets to the user. The plurality of candidate set information 251 shown on the asset proposal screen 250 includes, for example, first candidate set information 251 a indicating a first asset candidate set, second candidate set information 251 b indicating a second asset candidate set, third candidate set information 251 c indicating a third asset candidate set, and fourth candidate set information 251 d indicating a fourth asset candidate set.

[0154] The asset may be, for example, hardware, such as an arm, an end effector, a work table, an overhead camera for photographing the work table, a safety camera for detecting people, an in-tray, and an out-tray.

[0155] The candidate set information 251 includes, for example, a candidate set image 253 that represents an overall image of the asset candidate set. The candidate set information 251 also includes a plurality of pieces of asset specification information 252 for specifying each of the plurality of assets that make up the asset candidate set.

[0156] The first asset candidate set includes, for example, a medium-sized arm, a gripper as an effector, an overhead camera, a small-sized workbench, a small-sized first intray, a small-sized second intray, and a large-sized outtray. The first candidate set information 251a includes, for example, asset identification information 252a, 252b, 252c, 252d, 252e, 252f, and 252g.

[0157] Asset specific information 252a indicates that the asset candidate set includes a medium-sized arm. Asset specific information 252b indicates that the asset candidate set includes a gripper. Asset specific information 252c indicates that the asset candidate set includes an overhead camera. Asset specific information 252d indicates that the asset candidate set includes a small-sized workbench. Asset specific information 252e indicates that the asset candidate set includes a small-sized first in-tray. Asset specific information 252f indicates that the asset candidate set includes a small-sized second in-tray. Asset specific information 252g indicates that the asset candidate set includes a large-sized out-tray.

[0158] The second asset candidate set includes, for example, a large-sized arm, a vacuum effector, an overhead camera, a safety camera, a large-sized workbench, a small-sized first intray, a small-sized second intray, and a small-sized outtray. The second candidate set information 251b includes, for example, asset identification information 252h, 252i, 252c, 252j, 252k, 252e, 252f, and 252l.

[0159] Asset specific information 252h indicates that the asset candidate set includes a large-sized arm. Asset specific information 252i indicates that the asset candidate set includes a vacuum. Asset specific information 252k indicates that the asset candidate set includes a safety camera. Asset specific information 252l indicates that the asset candidate set includes a small-sized out-tray.

[0160] The third asset candidate set includes, for example, a medium-sized arm, a vacuum, an overhead camera, a safety camera, a medium-sized workbench, a small-sized first intray, a small-sized second intray, and a small-sized outtray. The third candidate set information 251c includes, for example, asset identification information 252a, 252i, 252c, 252j, 252m, 252e, 252f, and 252l. The asset identification information 252m indicates that the asset candidate set includes a medium-sized workbench.

[0161] The fourth asset candidate set includes, for example, a medium-sized arm, a gripper, an overhead camera, a medium-sized workbench, a small-sized first intray, a small-sized second intray, and a large-sized outtray. The fourth candidate set information 251d includes, for example, asset identification information 252a, 252b, 252c, 252m, 252e, 252f, and 252g.

[0162] The asset proposal screen 250 includes, for each asset candidate set, recommendation level information 256 that indicates the recommendation level of the asset candidate set. The recommendation level information 256 may indicate the recommendation level using, for example, the number of stars. In the recommendation level information 256 shown in FIG. 25 , the more filled-in stars there are, the higher the recommendation level. In the asset proposal screen 250 of the example in FIG. 25 , multiple asset candidate sets are displayed from the left in descending order of recommendation level.

[0163] In this way, by displaying multiple pieces of candidate set information 251 indicating multiple asset candidate sets together with the recommendation levels of the multiple asset candidate sets, the user can more easily select an asset candidate set. Note that, to make it easier for the user to select an asset candidate set, for example, when each piece of asset specification information 252 is selected, detailed information including the manufacturer, model number, size, etc. of each asset may be displayed. This allows the user to select an asset candidate set after checking the detailed asset information.

[0164] Furthermore, the asset proposal screen 250 includes a selection area 258 for selecting an asset candidate set for each asset candidate set. When a selection area 258 is selected, the information processing device 1 is notified of the selection of the asset candidate set corresponding to the selected selection area 258.

[0165] When the selection area 258 is selected, the dialogue advances to the next stage, and a first layout confirmation screen 270 is displayed on the display unit 4 to confirm the layout of the selected asset candidate set (also called the selected asset candidate set).

[0166] <Example of a method for determining an asset candidate set to be presented to a user> Fig. 26 is a schematic diagram for explaining an example of a method for determining an asset candidate set to be presented to a user. When the control unit 2 displays the asset proposal screen 250 on the display unit 4, the control unit 2 represents a large number of requests (here, i requests) related to robot work that the information processing device 1 has received from users up to that point as an i-dimensional vector. i is an integer equal to or greater than 2. The i requests include, for example, requests related to workpieces, requests related to trays, and priority details related to robot work.

[0167] Next, the control unit 2 performs a first mapping to convert the i-dimensional vector into a d-dimensional vector having a smaller degree, where d is an integer equal to or greater than 1 and smaller than i.

[0168] In this example, for example, d=3, and the d variables of the d-dimensional vector are three variables: energy usage, area volume, and process work time. Energy usage is the power used in robot work. Area volume is the volume of space required for robot work. Process work time is the time required for robot work. It can also be said that the control unit 2 performs a first mapping that converts n requests into energy usage, area volume, and process work time. However, the number of d variables of the d-dimensional vector is not limited to this. Hereinafter, the d-dimensional vector obtained by the first mapping will be referred to as the first d-dimensional vector.

[0169] In this example, p asset candidate sets are prepared in advance. p is an integer equal to or greater than 2. The p asset candidate sets include, for example, the first asset candidate set, the second asset candidate set, the third asset candidate set, and the fourth asset candidate set described above, as well as at least one other asset candidate set. Information regarding the p asset candidate sets is stored in the storage unit 3, and the control unit 2 can identify the p asset candidate sets based on the information in the storage unit 3.

[0170] For each of p asset candidate sets prepared in advance, the control unit 2 represents the multiple assets that make up the asset candidate set with a j-dimensional vector. Then, the control unit 2 performs a second mapping to convert the j-dimensional vector for each of the p asset candidate sets into a d-dimensional vector with energy usage, area volume, and process work time as variables. Hereinafter, the d-dimensional vector obtained by the second mapping will be referred to as the second d-dimensional vector. Here, a second d-dimensional vector is obtained for each of the p asset candidate sets.

[0171] The first mapping and the second mapping may be realized by, for example, AI technology such as machine learning. AI is an abbreviation for Artificial Intelligence. For example, the control unit 2 may include a first trained neural network that receives an i-dimensional vector and outputs a first d-dimensional vector. The control unit 2 may also include a second trained neural network that receives a j-dimensional vector and outputs a second d-dimensional vector.

[0172] The control unit 2 calculates the similarity (e.g., cosine similarity) between the second d-dimensional vector and the first d-dimensional vector for each of the p asset candidate sets. Then, the control unit 2 determines the top q asset candidate sets having the highest calculated similarity from among the p asset candidate sets as the asset candidate sets to be presented to the user. In other words, the control unit 2 determines the top q asset candidate sets having the highest calculated similarity from among the p asset candidate sets as the asset candidate sets to be displayed on the asset proposal screen 250. q is an integer greater than or equal to 1 and less than or equal to p. FIG. 25 shows the asset proposal screen 250 when q=4.

[0173] Furthermore, the control unit 2 causes the display unit 4 to display recommendation level information 256, which is the degree of similarity calculated for the top q asset candidate sets, as the degree of recommendation.

[0174] In addition, when q=1, the control unit 2 determines only the asset candidate set having the highest calculated similarity among the p asset candidate sets as the asset candidate set to be presented to the user. In this case, only one piece of candidate set information 251 is displayed on the asset proposal screen 250.

[0175] <Example of First Layout Confirmation Screen> Fig. 27 is a schematic diagram showing a display example of a first layout confirmation screen 270. The first layout confirmation screen 270 includes a designation area 27. When the designation area 27 is selected, the dialogue returns to the asset proposal screen 250. The first layout confirmation screen 270 also includes a designation area 46. When the designation area 46 is selected, the dialogue advances to the next step, and a layout editing screen 300 for editing the layout of the multiple assets (in other words, multiple pieces of hardware) that make up the selected asset candidate set is displayed on the main screen 11.

[0176] The first layout confirmation screen 270 includes a layout image 275 showing the layout of the selected asset candidate set, that is, the layout of the multiple assets that make up the selected asset candidate set. The layout image 275 is, for example, a three-dimensional image. The layout image 275 may be, for example, a three-dimensional computer image (in other words, an animated image). Here, the layout image 275 shows the initial setting of the layout of the selected asset candidate set. The layout of the selected asset candidate set can be changed from the initial setting on the layout editing screen 300. Hereinafter, the layout of the selected asset candidate set, that is, the layout of the multiple assets that make up the selected asset candidate set, may be referred to as the selected set layout.

[0177] In the example of Figure 27, the selected asset candidate set includes an arm, end effector, workbench, first intray, second intray, outtray, overhead camera, and safety camera. Layout image 275 shows the layout of the arm, end effector, workbench, first intray, second intray, outtray, overhead camera, and safety camera. Layout image 275 includes an image representing the arm, an image representing the end effector, an image representing the workbench, an image representing the first intray, an image representing the second intray, an image representing the outtray, an image representing the overhead camera, and an image representing the safety camera. The relative positions of these images represent the selected set layout. Layout image 275 also includes a background image representing the background.

[0178] The first layout confirmation screen 270 includes an instruction area 276 for instructing that the display viewpoint of the layout can be changed. When the instruction area 276 is selected, the user can change the display viewpoint of the selected set layout shown in the three-dimensional layout image 275 by making a predetermined input to the input unit 5. The control unit 2 causes the display unit 4 to display the selected set layout in a manner that allows the display viewpoint to be changed in response to an input received by the input unit 5.

[0179] The control unit 2 can display the selected set layout on the display unit 4 from the viewpoint of a person standing at the point accepted by the input unit 5. This point is called the reference point. The first layout confirmation screen 270 includes a human image 277 representing a human figure for specifying the reference point. The user can move the human image 277 on the layout image 275 by making a predetermined input to the input unit 5. The position of the human image 277 on the layout image 275 represents the reference point. For example, when the position of the human image 277 on the layout image 275 is set as shown in FIG. 28 , the selected set layout is displayed on the layout image 275 as shown in FIG. 29 . The display viewpoint of the selected set layout shown in FIG. 29 can be said to have been changed from that of FIG. 27 . Thereafter, when the designation area 276 is selected, the display of the first layout confirmation screen 270 returns to the display of FIG. 27 .

[0180] In this way, the control unit 2 displays the selected set layout on the display unit 4 in a manner that allows the display viewpoint to be changed according to input received by the input unit 5, and therefore the user can change the display viewpoint of the layout by inputting to the input unit 5. This improves the convenience of the information processing device 1.

[0181] Furthermore, the control unit 2 can display the selected set layout on the display unit 4 from the viewpoint of a person standing at the point accepted by the input unit 5, so that the user can check the selected set layout as seen from the person's viewpoint.

[0182] The first layout confirmation screen 270 includes an instruction area 278 for instructing that the selected set layout be displayed on a second display device that can communicate with the information processing device 1. The second display device is a device that virtually reproduces and displays the selected set layout, and may be, for example, the VR display device 1000 or the AR display device 1200.

[0183] 30 is a schematic diagram showing an example of a VR display device 1000 that can communicate with the information processing device 1. The VR display device 1000 is, for example, a head-mounted display. The VR display device 1000 may be a goggle type or may be in another form.

[0184] When the indication area 278 is selected, the control unit 2 transmits information about the selected set layout, for example, image data 1100 indicating a three-dimensional layout image 275 showing the selected set layout, to the VR display device 1000 via the interface 6. The interface 6 may communicate directly with the VR display device 1000, or may communicate with a VR display device 1000 in a remote location via a network including the Internet or the like. The VR display device 1000 displays the three-dimensional layout image 275 based on the received image data 1100.

[0185] The instruction area 278 may be an area for instructing that the selected set layout be displayed on the AR display device 1200 connected to the information processing device 1. Fig. 31 is a schematic diagram showing an example of the AR display device 1200 that can communicate with the information processing device 1. The AR display device 1200 is, for example, a head-mounted display. The AR display device 1200 may be a goggle type or may be in another form.

[0186] When the designation area 278 is selected, the control unit 2 transmits image data 1300 indicating the portion of the layout image 275 other than the background image, i.e., an image 1310 indicating only the selected asset candidate set, to the AR display device 1200 via the interface 6. The interface 6 may communicate directly with the AR display device 1200, or may communicate with the AR display device 1200 in a remote location (for example, the actual work environment of the robot) via a network including the Internet or the like. The AR display device 1200 displays the image 1310 based on the received image data 1300. At this time, the AR display device 1200 projects an actual background 1320 on its display surface and displays the image 1310 superimposed on the actual background 1320. By displaying the image 1310, the layout of the selected asset candidate set is displayed.

[0187] In addition, the first layout confirmation screen 270 may include both an instruction area 278 for instructing the VR display device 1000 to display the selected set layout, and an instruction area for instructing the AR display device 1200 to display the selected set layout.

[0188] The first layout confirmation screen 270 includes a robot KPI screen 280 showing robot KPIs according to the selected asset candidate set. The robot KPI screen 280 includes an instruction area 281 for instructing to display the robot KPIs in a radar chart, an instruction area 282 for instructing to display the robot KPIs in a doughnut graph, and an instruction area 282 for instructing to display the robot KPIs in a horizontal bar graph.

[0189] When the instruction area 281 is selected, the robot KPI corresponding to the selected asset candidate set is displayed in a radar chart 286 on the robot KPI screen 280, as shown in Figures 27 to 29. "P", "C", "L", "Sa", and "St" shown on the robot KPI screen 280 indicate the success index, the work time index, the range of motion index, the psychological safety index, and the stability index, respectively.

[0190] When the instruction area 282 is selected, the success index, work time index, range of motion index, psychological safety index, and stability index of the robot KPI corresponding to the selected asset candidate set are each displayed individually in a doughnut chart 287 on the robot KPI screen 280, as shown in Fig. 32. When the instruction area 283 is selected, the robot KPI corresponding to the selected asset candidate set is displayed in a horizontal bar graph 288 on the robot KPI screen 280, as shown in Fig. 33.

[0191] The robot KPI screen 280 displays a composite index of the success index, work time index, range of motion index, psychological safety index, and stability index that constitute the robot KPI corresponding to the selected asset candidate set, for example, in the form of a doughnut graph 285. The composite index is, for example, a value obtained by multiplying the success index, work time index, range of motion index, psychological safety index, and stability index by each other. In the center of the doughnut graph 285, the composite index is displayed as a percentage value.

[0192] The control unit 2 obtains the robot KPIs shown on the robot KPI screen 280 using a method different from the method for obtaining the robot KPIs shown on the priority hearing screen 170. Hereinafter, simply referring to the robot KPIs means the robot KPIs corresponding to the selected asset candidate set shown on the robot KPI screen 280. The control unit 2 obtains the robot KPIs based on the requests obtained from the user on the application screen 10. An example of how to obtain the robot KPIs will be described below.

[0193] <Example of how to calculate robot KPI> Here, simply referring to an in-tray means an in-tray included in the selected asset candidate set, simply referring to an out-tray means an out-tray included in the selected asset candidate set, simply referring to an arm means an arm included in the selected asset candidate set, and simply referring to an effector means an end effector included in the selected asset candidate set.

[0194] <Success Index> For example, the control unit 2 calculates the holding success rate when the robot holds the workpieces in the in-tray. Then, the control unit 2 multiplies the holding success rate by the number of products manufactured by the robot operation, such as the third object described above, and sets the result as the success index for the robot KPI. In this case, the success index can also be said to be the number of non-defective products.

[0195] The production quantity (in other words, the planned production quantity) is obtained in advance from the user and stored in the storage unit 3. The control unit 2 estimates and acquires the holding success rate based on the request obtained from the user on the application screen 10. For example, the control unit 2 estimates the holding success rate based on the shape of the intray (also referred to as the intray shape), the intray work distribution degree for the intray, the type of end effector (also referred to as the effector type), and the posture (also referred to as the approach posture) at which the end effector approaches the workpieces in the intray to hold them. If the selection asset candidate set includes multiple intrays, the intray shape, intray work distribution degree, and approach posture for any one intray (e.g., the first intray) of the multiple intrays are used.

[0196] The approach posture of the end effector used in calculating the holding success rate may be, for example, the posture of the end effector approaching a representative workpiece in the intray (e.g., a workpiece at the center of the intray) in order to hold that workpiece. The control unit 2 can estimate and acquire the approach posture based on, for example, the position of the intray, the position of the representative workpiece in the intray, and the posture of the representative workpiece in the intray. Furthermore, the control unit 2 can estimate and acquire the posture of the representative workpiece in the intray based on the intray shape, the intray workpiece distribution degree, and the distribution reference position of the workpieces in the intray.

[0197] When determining the holding success rate, for example, a specific four-dimensional coordinate space (which may also be called a specific four-dimensional coordinate system) having four axes each representing four parameters, i.e., in-tray shape, in-tray work distribution, effector type, and approach posture, is used. Each position in the specific four-dimensional space is associated with a holding success rate corresponding to the combination of in-tray shape, in-tray work distribution, effector type, and approach posture that constitute the coordinates of that position. The holding success rate associated with each position in the specific four-dimensional space is determined in advance by experiment or simulation.

[0198] The control unit 2 identifies a position in a specific four-dimensional space of coordinates consisting of a set of in-tray shape, in-tray work dispersion, effector type, and approach posture based on the user's requests heard on the application screen 10, and determines the success index as the value obtained by multiplying the holding success rate corresponding to the identified position by the number of products produced.

[0199] The control unit 2 may estimate the holding success rate using AI technology such as machine learning. For example, the control unit 2 may include a trained neural network that receives input of the in-tray shape, the in-tray work dispersion degree, the effector type, and the approach posture and outputs the holding success rate.

[0200] For example, when the success index is displayed, the control unit 2 recalculates the success index every time information required to calculate the success index changes, and displays the recalculated success index on the display unit 4. This allows, for example, the display of the success index to be updated in real time.

[0201] <Regarding the Work Time Index> For example, the control unit 2 determines the actual work time of the robot work. Then, the control unit 2 determines the value obtained by dividing the target work time of the robot work by the actual work time of the robot work as the work time index of the robot KPI. The target work time is obtained in advance from the user and stored in the storage unit 3. The control unit 2 estimates and obtains the actual work time based on the request obtained from the user on the application screen 10.

[0202] For example, the control unit 2 estimates the actual operation time based on the in-tray shape, the in-tray workpiece distribution degree, the posture of each workpiece in the in-tray, and the approach posture of the end effector. The control unit 2 also estimates and acquires the posture of each workpiece in the in-tray based on the in-tray workpiece distribution degree, the in-tray shape, and the workpiece distribution reference position in the in-tray. The control unit 2 may estimate the actual operation time in the same way as when estimating the holding success rate.

[0203] For example, when the task time index is being displayed, the control unit 2 recalculates the task time index and displays it on the display unit 4 every time the information required to calculate the task time index changes.

[0204] <Regarding the Movable Range Index> For example, the control unit 2 calculates the volume of the workspace available for robot operation (also referred to as workspace volume). The control unit 2 also calculates the volume of the space occupied by obstacles that impede the robot's movement (obstacle space volume). The control unit 2 then determines the value obtained by subtracting the obstacle space volume from the workspace volume, dividing the result by the workspace volume, as the movable range index. The value obtained by subtracting the obstacle space volume from the workspace volume can be said to be the volume of the range in which the arm can move without interfering with an obstacle.

[0205] The control unit 2 can calculate the workspace volume from the range occupied by multiple assets in the selected set layout. The control unit 2 can also calculate the obstacle space volume from the range occupied by obstacles included in the selected asset candidate set in the selected set layout. In the selected asset candidate set, assets other than the arm and the end effector are obstacles.

[0206] For example, when the movable range indicator is displayed, the control unit 2 recalculates the movable range indicator and displays it on the display unit 4 every time information required to calculate the movable range indicator changes.

[0207] <Regarding the psychological safety index> For example, the control unit 2 calculates a difference value by subtracting the maximum reachable distance of the arm from the shortest distance between the aisle through which people pass and the arm. The control unit 2 then divides the calculated difference value by the shortest distance between the aisle through which people pass and the arm, and sets the result as the psychological safety index. The shortest distance between the base of the arm and the aisle is used as the shortest distance between the aisle through which people pass and the arm. The aisle for people is specified by the user on the layout editing screen 300, which will be described later. Here, the initial setting for the aisle for people is used.

[0208] For example, when the psychological safety index is displayed, the control unit 2 recalculates the psychological safety index and displays it on the display unit 4 every time the information required to calculate the psychological safety index changes.

[0209] <Regarding the Stability Index> For example, the control unit 2 uses the selected asset candidate set to simulate the movement of a robot during robotic work. At this time, the control unit 2 determines a trajectory of the arm that will prevent the arm and end effector from interfering with obstacles (such as a workbench) included in the selected asset candidate set during the robotic work. The trajectory of the arm can also be considered to be the trajectory of the end effector. The control unit 2 then sets the distance of the determined trajectory of the arm (in other words, the path) as the first distance. The control unit 2 may control the display unit 4 so that the simulated movement of the robot is displayed on the layout image 275. In other words, the control unit 2 may control the display unit 4 so that an image of the robot included in the layout image 275 changes in accordance with the simulated movement of the robot.

[0210] On the other hand, the control unit 2 uses the selected asset candidate set to simulate the movement of the robot when no obstacle is present in the robot operation, and calculates the distance of the arm's trajectory when no obstacle is present as the second distance. The second distance can also be said to be the shortest distance of the arm's trajectory.

[0211] For example, the control unit 2 determines the stability index of the robot KPI by dividing the value obtained by subtracting the second distance from the first distance by the first distance. For example, when the stability index is displayed, the control unit 2 recalculates the stability index and displays it on the display unit 4 every time information required to calculate the stability index changes.

[0212] As can be understood from the above description, among the robot KPIs, the success indicator, the work time indicator, and the stability indicator may change depending on changes in the selection set layout. For example, if the position of the in-tray in the selection set layout changes, the success indicator, the work time indicator, and the stability indicator may change. Also, if the position of the out-tray in the selection set layout changes, the stability indicator may change.

[0213] The first layout confirmation screen 270 includes a priority hearing screen 290 for hearing priority details regarding the robot work from the user from a different perspective than the priority hearing screen 170 described above.

[0214] The priority listening screen 290 includes, for example, designation areas 291 to 293. Designation area 291 is an area for designating the priority of energy usage. The higher the priority of energy usage, the smaller the energy usage is desired. Designation area 292 is an area for designating the priority of area volume. The higher the priority of area volume, the smaller the area volume is desired. Designation area 293 is an area for designating the priority of process work time. The higher the priority of process work time, the smaller the process work time is desired. Each of designation areas 291 to 293 has a slider.

[0215] The priority hearing screen 290 includes a notification execution area 295 for notifying the user of the completion of the designation of the priorities of energy use, area volume, and process work time. When the notification execution area 295 is selected, the dialogue returns to the previous step, and the control unit 2 adjusts the first d-dimensional vector determined based on the user's request, based on the priorities of energy use, area volume, and process work time designated on the priority hearing screen 290. Then, the control unit 2 redetermines, as described above, the q asset candidate sets to be proposed to the user based on the adjusted first d-dimensional vector and the second d-dimensional vectors for each of the p asset candidate sets. Then, the control unit 2 causes the display unit 4 to display an asset proposal screen 250 showing the redetermined q asset candidate sets, as shown in FIG. 25 . Thereafter, the information processing device 1 operates in the same manner.

[0216] When the first d-dimensional vector is adjusted in response to a user request, for example, consider a case where the priority of the energy usage specified on the priority hearing screen 290 is higher than the initial value. In this case, the control unit 2 decreases the value of the energy usage of the first d-dimensional vector by an amount corresponding to the absolute value of the difference between the priority specified by the user and the initial value. On the other hand, when the priority of the energy usage specified on the priority content hearing screen 290 is lower than the initial value, the control unit 2 increases the value of the energy usage of the first d-dimensional vector by an amount corresponding to the absolute value of the difference between the priority specified by the user and the initial value.

[0217] Also, consider a case where the priority of the area volume specified on the priority content hearing screen 290 is higher than the initial value. In this case, the control unit 2 reduces the value of the area volume of the first d-dimensional vector by an amount corresponding to the absolute value of the difference between the priority specified by the user and the initial value. On the other hand, if the priority of the area volume specified on the priority content hearing screen 290 is lower than the initial value, the control unit 2 increases the value of the area volume of the first d-dimensional vector by an amount corresponding to the absolute value of the difference between the priority specified by the user and the initial value. The control unit 2 performs similar processing on the process operation time.

[0218] When the value of the usage energy of the first d-dimensional vector changes, the similarity (e.g., cosine similarity) between the first d-dimensional vector and the second d-dimensional vector changes. Therefore, when the value of the usage energy of the first d-dimensional vector changes, the set of q asset candidates proposed to the user may change. Similarly, when the value of the area volume of the first d-dimensional vector changes, the set of q asset candidates proposed to the user may change. Similarly, when the value of the first d-dimensional vector changes, the set of q asset candidates proposed to the user may change.

[0219] In this way, when the priority of the energy usage, etc. is changed on the priority hearing screen 290 included in the first layout confirmation screen 270, the asset candidate set proposed to the user is re-determined based on the priority of the energy usage, etc. Therefore, when the user cannot adopt the selected asset candidate set confirmed on the first layout confirmation screen 270, the user can change the priority of the energy usage, etc., to cause the information processing device 1 to propose an asset candidate set according to the changed priority.

[0220] Furthermore, when the instruction area 27 is selected, the asset proposal screen 250 shown in FIG. 25 is displayed again. Therefore, if the user cannot adopt the selected asset candidate set confirmed on the first layout confirmation screen 270, the user can reselect the asset candidate set by selecting the instruction area 27.

[0221] When the designation area 46 included in the first layout confirmation screen 270 is selected, the dialogue advances to the next step, and a layout editing screen 300 is displayed on the main screen 11 .

[0222] <Example of Layout Editing Screen> Fig. 34 is a schematic diagram showing a display example of the application screen 10 including a layout editing screen 300. The layout editing screen 300 includes designation areas 27 and 46. When the designation area 27 is selected, the dialogue returns two steps, and an asset proposal screen 250 is displayed on the main screen 11. When the designation area 46 is selected, the dialogue advances one step, and a second layout confirmation screen 350 for confirming the selected set layout edited on the layout editing screen 300 is displayed on the main screen 11.

[0223] The layout editing screen 300 includes a layout image 310 showing a selected set layout similar to the above-described layout image 275. The layout editing screen 300 also includes the above-described robot KPI screen 280 showing robot KPIs according to the selected asset candidate set.

[0224] On the layout editing screen 300, for example, the position of a tray in the selected set layout can be changed. The user can change the position of the tray shown in the layout image 310 by making a predetermined input to the input unit 5. In other words, the user can change the position of the image representing the tray included in the layout image 310 on the layout image by making a predetermined input to the input unit 5. Changing the position of the tray shown in the layout image 310 changes the position of the tray in the selected set layout.

[0225] 34, the user can independently move each of the first intray, second intray, and outtray shown in layout image 310. In layout image 310, a two-dimensional coordinate figure representing a two-dimensional coordinate system is superimposed on the image representing the first intray, indicating that the position of the first intray shown in layout image 310 can be changed. The same applies to the second intray and outtray.

[0226] FIG. 35 is a schematic diagram showing an example of a state in which the user has changed the position of the out-tray indicated in the layout image 310 by performing a predetermined input on the input unit 5. In the example of FIG. 35, the position of the image representing the out-tray included in the layout image 310 has been changed from the example of FIG. 34. Changing the position of the out-tray indicated in the layout image 310 changes the position of the out-tray in the selected set layout. In the example of FIG. 35, a dashed arrow indicating that the out-tray indicated in the layout image 310 has moved is superimposed on the layout image 310. The unit of change of the layout position may be a length unit such as the metric system, or may be a grid unit discretized by the size of the bottom surface of the tray. On the layout editing screen 300, the user can select a change unit that is easy to operate from multiple types of change units.

[0227] Fig. 36 is a schematic diagram showing an example of a state in which a user has made a predetermined input to the input unit 5 to change the position of the first intray indicated in the layout image 310. In the example of Fig. 36, the position of the image representing the first intray included in the layout image 310 has been changed from the example of Fig. 34. By changing the position of the first intray indicated in the layout image 310, the position of the first intray in the selected set layout is changed. In the example of Fig. 36, a dashed arrow indicating that the first intray indicated in the layout image 310 has moved is superimposed on the layout image 310.

[0228] 35 and 36, the control unit 2 determines the robot KPIs according to the edited selection set layout. Then, the control unit 2 controls the display unit 4 so that the determined robot KPIs are displayed on the robot KPI screen 280.

[0229] For example, consider a case where the position of the out-tray in the selected set layout is changed, as in the example of FIG. 35 . In this case, the control unit 2 simulates the robot's movement during robot work based on the selected set layout in which the out-tray position has been changed. Based on the results of the simulation, the control unit 2 then recalculates the stability index as described above. The control unit 2 also recalculates the composite index based on the robot KPI including the recalculated stability index. The control unit 2 then displays the robot KPI including the recalculated stability index and the recalculated composite index on the robot KPI screen 280. Note that even if the position of the out-tray in the selected set layout is changed, the success index, task time index, range of motion index, and psychological safety index are not affected. In the example of FIG. 35 , the stability index and composite index have changed from the example of FIG. 34 .

[0230] As another example, consider a case where the position of the first intray in the selected set layout is changed, as shown in the example of FIG. 36 . In this case, the control unit 2 recalculates the success index and the task time index as described above using the changed position of the first intray. The control unit 2 also simulates the robot's movement during the robot task based on the selected set layout in which the position of the first intray has been changed. The control unit 2 then recalculates the stability index as described above based on the results of the simulation. The control unit 2 also recalculates the composite index based on the robot KPIs including the recalculated success index, task time index, and stability index. The control unit 2 then displays the robot KPIs including the recalculated success index, task time index, and stability index, as well as the recalculated composite index, on the robot KPI screen 280. Note that even if the position of the intray in the selected set layout is changed, the movement range index and psychological safety index are not affected. In the example of FIG. 36 , the success index, task time index, stability index, and composite index have changed from the example of FIG. 34 .

[0231] As with the tray, the position of the arm in the selected set layout may be changeable on the layout editing screen 300. Also, the position of the work table in the selected set layout may be changeable on the layout editing screen 300. Also, the position of the overhead camera in the selected set layout may be changeable on the layout editing screen 300, and the position of the safety camera in the selected set layout may be changeable.

[0232] In this way, the control unit 2 can cause the display unit 4 to display the selected set layout so that it can be edited by input received by the input unit 5. This allows the user to edit the selected set layout while checking it by making a predetermined input to the input unit 5. This improves the convenience of the information processing device 1.

[0233] In addition, the control unit 2 calculates a work evaluation index corresponding to the edited selection set layout and displays it on the display unit 4, allowing the user to edit the selection set layout while checking work evaluation indexes such as success indexes.

[0234] The layout editing screen 300 allows a user to specify a pedestrian passage in the selection set layout. The layout editing screen 300 includes a designation area 305 for indicating that a pedestrian passage can be specified. When the designation area 305 is selected, the user can specify the position and range of the pedestrian passage in the selection set layout by making a predetermined input to the input unit 5.

[0235] The position and range of the person's passage are specified, for example, using a 3D object (also referred to as a 3D passage object) representing the person's passage. 3D means three dimensions. In this example, the 3D object is composed of at least one rectangular parallelepiped. The 3D passage object is composed of, for example, at least one rectangular parallelepiped. The user can place the 3D passage object on the layout image 310 by making a predetermined input to the input unit 5. The position and range of the person's passage are specified by placing the 3D passage object on the layout image 310.

[0236] Furthermore, when assembly processing is designated on the above-described work processing hearing screen 101 (see FIG. 14 ), the location where assembly processing is to be performed (also referred to as the assembly location) can be designated on the layout editing screen 300. The layout editing screen 300 includes a designation area 306 for instructing that the assembly location can be designated. When the designation area 306 is selected, the user can designate the position and range of the assembly location in the selected set layout by making a predetermined input to the input unit 5.

[0237] The position and range of the assembly location are specified, for example, using a 3D object (also referred to as a 3D assembly location object) representing the assembly location. The 3D assembly location object is configured, for example, by at least one rectangular parallelepiped. The user can place the 3D assembly location object on the layout image 310 by making a predetermined input to the input unit 5. By placing the 3D assembly location object on the layout image 310, the position and range of the assembly location are specified.

[0238] Furthermore, layout editing screen 300 includes designation area 301 for instructing the display of 3D objects such as 3D passage objects, and designation area 302 for instructing the non-display of the 3D objects. When designation area 301 is selected in a state where no 3D objects are displayed as in Figures 34 to 36, the 3D objects are displayed in layout image 310 as shown in Figure 37. On the other hand, when designation area 302 is selected in a state where 3D objects are displayed as in Figure 37, the 3D objects are not displayed as shown in Figures 34 to 36.

[0239] In the example of FIG. 37 , a 3D passage object 315 is indicated by a two-dot chain line in the layout image 310. Furthermore, a 3D assembly location object 316 is indicated by a single-dot chain line in the layout image 310. Furthermore, 3D objects (also referred to as 3D obstacle objects) 317 representing obstacles included in the selected asset candidate set are indicated by thin solid lines. In the example of FIG. 37 , the 3D obstacle objects 317 include a 3D object representing a first in-tray, a 3D object representing a second in-tray, a 3D object representing an out-in-tray, a 3D object representing a workbench, a 3D object representing an overhead camera, and a 3D object representing a safety camera. These 3D objects are pre-stored in the storage unit 3.

[0240] When a path for people is specified on the layout editing screen 300, the control unit 2 recalculates the psychological safety index according to the specified path. The control unit 2 also recalculates the composite index based on the robot KPI including the recalculated psychological safety index. The control unit 2 then controls the display unit 4 so that the recalculated psychological safety index and composite index are displayed on the robot KPI screen 280.

[0241] Furthermore, when an assembly location is specified on the layout editing screen 300, the control unit 2 re-simulates the robot's movement in the robot operation based on the specified assembly location. At this time, the control unit 2 may control the display unit 4 so that the simulated robot movement is displayed on the layout image 310. The control unit 2 then re-calculates the stability index as described above based on the results of the simulation. The control unit 2 also re-calculates the composite index based on the robot KPI including the re-calculated stability index. The control unit 2 then controls the display unit 4 so that the re-calculated stability index and composite index are displayed on the robot KPI screen 280. Note that before an assembly location is specified, the stability index is calculated based on the initial setting of the assembly location.

[0242] When the designation area 46 included in the layout editing screen 300 is selected, the dialogue advances to the next stage, and a second layout confirmation screen 350 for confirming the latest selected set layout is displayed on the main screen 11 .

[0243] <Example of Second Layout Confirmation Screen> Fig. 38 is a schematic diagram showing a display example of the application screen 10 including a second layout confirmation screen 350. As shown in Fig. 38, the second layout confirmation screen 350 has the same configuration as the above-mentioned first layout confirmation screen 270 (see Fig. 27, etc.). A layout image 275 included in the second layout confirmation screen 350 shows the most recently selected set layout.

[0244] When instruction area 27 included in second layout confirmation screen 350 is selected, the dialogue returns to the previous step, and layout editing screen 300 is displayed on main screen 11. When instruction area 46 included in second layout confirmation screen 350 is selected, the dialogue advances to the next step, and first detailed action listening screen 360 for listening to detailed robot actions during robot work is displayed on main screen 11.

[0245] 39 is a schematic diagram showing a display example of the application screen 10 including a first detailed action listening screen 360. The user can specify, for example, the order of work performed by the robot on the first detailed action listening screen 360. For example, the user can specify the content and order of work steps for the robot work on the first detailed action listening screen 360.

[0246] 39, a layout image 361 showing a selected set layout is displayed on a first detailed action hearing screen 360. The layout image 361 is, for example, a two-dimensional image.

[0247] The layout image 361 shows, for example, an enlarged area in which the in-tray and out-tray exist in the selected set layout. In the example of Fig. 39, the layout image 361 shows the first in-tray, the second in-tray, and the out-tray.

[0248] The layout image 361 includes a point graphic indicating the position of the first intray (referred to as a first intray point graphic 362b) and an identification number 363b for identifying the position of the first intray. In the example of Fig. 39, the identification number 363b is set to "2".

[0249] The layout image 361 includes a point graphic indicating the position of the second intray (referred to as a second intray point graphic 362a) and an identification number 363a for identifying the position of the second intray. In the example of Fig. 39, the identification number 363a is set to "1".

[0250] The layout image 361 includes a point graphic indicating the position of the out-tray (referred to as an out-tray point graphic 362d) and an identification number 363d for identifying the position of the out-tray. In the example of Fig. 39, the identification number 363d is set to "4".

[0251] The layout image 361 includes a point graphic (referred to as an assembly location point graphic 362c) indicating the assembly location (also referred to as the assembly position) specified on the layout editing screen 300, and an identification number 363c for identifying the assembly location. In the example of Fig. 39, the identification number 363c is set to "3." As shown in Fig. 39, the layout image 361 may include an image representing a jig to be used at the assembly location.

[0252] In this example, the content of one work process in the robot work is specified by drawing an arrow between point graphics on the layout image 361. The user can draw an arrow between point graphics by making a predetermined input to the input unit 5. For example, when the user moves his / her finger on the display surface from one point graphic to another, an arrow from one point graphic to another is displayed on the display unit 4. The work sequence is then specified in the order in which the arrows are drawn between the point graphics on the layout image 361. Note that the user may specify the process content by inputting the start and end points of the work without comparing the arrows. Furthermore, when the user selects a start point, candidate points for the end point may be highlighted, for example.

[0253] 39 , when an arrow 365a is first drawn from a second intray point graphic 362a to an assembly location point graphic 362c on a layout image 361, it is specified that the end effector holds a workpiece (e.g., a second object) in the second intray and moves to the assembly location as the first work process (also referred to as the first work process) of the robot operation. Next, when an arrow 365b is drawn from a first intray point graphic 362b to an assembly location point graphic 362c on the layout image 361, it is specified that the end effector holds a workpiece (e.g., a first object) in the first intray and moves to the assembly location as the next work process (also referred to as the second work process) of the robot operation. Next, when an arrow 365c is drawn on the layout image 361 from the assembly location point graphic 362c to the out-tray point graphic 362d, the next work process (also called the third work process) of the robot operation is specified, in which the end effector holds the workpiece (e.g., the third object) assembled at the assembly location and moves it to the out-tray.

[0254] Note that an animation may be displayed in which a 3D object representing the second object (also referred to as a second 3D object) moves along the arrow 365a as the arrow 365a is drawn. In this case, the state of the second 3D object after it has moved to the assembly location point graphic 362c may be displayed. When the state of the second 3D object is displayed, the second 3D object may snap to the assembly location point graphic 362c.

[0255] Similarly, an animation may be displayed in which a 3D object representing the first object (also referred to as a first 3D object) moves along the arrow 365b as the arrow 365b is drawn. At this time, a state of the first 3D object after it has moved to the assembly location point graphic 362c may be displayed. When the state of the first 3D object is displayed, the first 3D object may snap to the assembly location point graphic 362c.

[0256] Similarly, an animation may be displayed in which a 3D object representing a third object (also referred to as a third 3D object) moves along the arrow 365c as the arrow 365c is drawn. At this time, the state of the third 3D object after it has moved to the out-tray point graphic 362d may be displayed. When the state of the third 3D object is displayed, the third out-tray 3D object may snap to the point graphic 362d.

[0257] When an arrow 365a is drawn from the second intray point graphic 362a to the assembly location point graphic 362c on the layout image 361, start point information 364a indicating that the position of the second intray is the start point of the work process is displayed near the second intray point graphic 362a on the layout image 361. Furthermore, when an arrow 365b is drawn from the first intray point graphic 362b to the assembly location point graphic 362c, start point information 364b indicating that the position of the first intray is the start point of the work process is displayed near the first intray point graphic 362b on the layout image 361.

[0258] The assembly location is the finish point of the first and second work processes, and is also the start point of the third work process. Therefore, start / finish point information 364c indicating that the assembly location is the start and finish point is displayed near the assembly location point graphic 362c. The position of the out-tray is the finish point of the third work process. Therefore, finish point information 364d indicating that the position of the out-tray is the finish point is displayed near the out-tray point graphic 362d.

[0259] In the first work process, the end effector moves from the second in-tray position, which is the start point, to the assembly location, which is the goal point. In the second work process, the end effector moves from the first in-tray position, which is the start point, to the assembly location, which is the goal point. And in the third work process, the end effector moves from the assembly location, which is the start point, to the out-tray position, which is the goal point.

[0260] A work detail screen 370 showing the work process content and work process sequence specified by the user is displayed on the first detailed action listening screen 360. The work detail screen 370 includes an instruction area 371 for instructing that the work process content and work process sequence be displayed in a node graph, and an instruction area 372 for instructing that the work process content and work process sequence be displayed in a flow diagram.

[0261] When the instruction area 371 is selected, a node graph 375 showing the work process content and work process sequence is displayed on the work details screen 370, as shown in Fig. 39. When the instruction area 372 is selected, a flow diagram 380 showing the work process content and work process sequence is displayed on the work details screen 370 (see Fig. 41 described below).

[0262] The work detail screen 370 displays identification information 376 for identifying work processes. In this example, "Node" refers to a work process. The identification information 376 of the work process that is to be executed xth (x is an integer of 1 or greater) is represented by "Nodex."

[0263] The work details screen 370 also displays process content information 377 that indicates the details of the work process. On the work details screen 370, the process content information 377 of the xth work process is displayed, for example, connected by a line to the identification information 376 of the xth work process. The process content information 377 includes, for example, information that indicates the start point and finish point of the work process.

[0264] The process content information 377 of the first work process represented by "Node1" includes an identification number 363a (i.e., "1") of the position of the second in-tray, which is the start point of the first work process, and an identification number 363c (i.e., "3") of the assembly location, which is the finish point of the first work process. The process content information 377 of the first work process indicates that in the first work process, the end effector moves from the second in-tray to the work location.

[0265] The process content information 377 of the second work process represented by "Node 2" includes an identification number 363b (i.e., "2") of the position of the first in-tray, which is the start point of the second work process, and an identification number 363c (i.e., "3") of the assembly location, which is the finish point of the second work process. The process content information 377 of the second work process indicates that in the second work process, the end effector moves from the first in-tray to the work location.

[0266] The process content information 377 of the third work process represented by "Node 3" includes an identification number 363c (i.e., "3") of the assembly location which is the start point of the third work process, and an identification number 363d (i.e., "4") of the out-tray position which is the finish point of the third work process. The process content information 377 of the third work process indicates that in the third work process, the end effector moves from the assembly location to the out-tray.

[0267] The process content information 377 may also include information representing the behavior of the robot between the start point and the goal point. For example, the process content information 377 may include a speed profile between the start point and the goal point, or an acceleration profile between the start point and the goal point. The process content information 377 may also include at least one of an upper limit and a lower limit of the speed between the start point and the goal point, or at least one of an upper limit and a lower limit of the acceleration between the start point and the goal point. The information representing the behavior of the robot between the start point and the goal point, which is included in the process content information 377, may be changeable in response to instructions from a user. The process content information 377 may also include information representing instructions for subordinate devices, such as an end effector, at the start point, or information representing instructions for subordinate devices, such as an end effector, at the goal point. The instructions for the devices may include holding on / off or holding force.

[0268] The first detailed action listening screen 360 includes instruction areas 27 and 46. When instruction area 27 is selected, the dialogue returns to the previous step by two, and the layout editing screen 300 is displayed on the main screen 11. When instruction area 46 is selected, the main screen 11 displays a second detailed action listening screen 390 for listening to the detailed actions of the robot in the robot work from a different perspective from that of the first detailed action listening screen 360.

[0269] <Example of Second Detailed Action Listening Screen> Fig. 40 is a schematic diagram showing a display example of the application screen 10 including a second detailed action listening screen 390. As shown in Fig. 40, the second detailed action listening screen 390 includes the above-described layout image 361 and work detail screen 370. The second detailed action listening screen 390 also includes designation areas 27 and 46.

[0270] On the second detailed operation listening screen 390, the user can modify the arm trajectory (in other words, the end effector trajectory) when the robot operates according to the work process content and work process sequence specified on the first detailed operation listening screen 360.

[0271] When the instruction area 46 included in the first detailed action hearing screen 360 is selected, the control unit 2 simulates the movement of the robot when the robot performs the work according to the work process content and work process sequence specified on the first detailed action hearing screen 360. At this time, the control unit 2 calculates an arm trajectory during the robot work such that the arm and end effector do not interfere with obstacles included in the selected asset candidate set. In the example of Figures 39 and 40, the control unit 2 calculates an arm trajectory 367a (also referred to as a first arm trajectory 367a) in the first work process, an arm trajectory 367b (also referred to as a second arm trajectory 367b) in the second work process, and an arm trajectory 367c (also referred to as a third arm trajectory 367c) in the third work process. Then, the control unit 2 controls the display unit 4 so that a first arm trajectory 367a, a second arm trajectory 367b, and a third arm trajectory 367c are displayed on the layout image 361 as shown in FIG.

[0272] On the layout image 361, the first arm trajectory 367a is indicated by an arrow from the second in-tray point graphic 362a to the assembly location point graphic 362c. On the layout image 361, the second arm trajectory 367b is indicated by an arrow from the first in-tray point graphic 362b to the assembly location point graphic 362c. On the layout image 361, the third arm trajectory 367c is indicated by an arrow from the assembly location point graphic 362c to the out-tray point graphic 362d. Hereinafter, when there is no need to particularly distinguish between the first arm trajectory 367a, the second arm trajectory 367b, and the third arm trajectory 367c, they will each be referred to as a process arm trajectory.

[0273] The user can correct the process arm trajectory on the layout image 361 by making a predetermined input to the input unit 5. When the user makes a predetermined input to the input unit 5, a coordinate system graphic 368 indicating a three-dimensional coordinate system for correcting the process arm trajectory is displayed above the process arm trajectory. For example, when the user presses a position on the display surface of the process arm trajectory with his / her finger, the coordinate system graphic 368 for correcting the trajectory of the process arm is displayed at the pressed position. The coordinate system graphic 368 is displayed so that its origin is located on the process arm trajectory. In the example of FIG. 40 , the coordinate system graphic 368 is displayed on the second arm trajectory 367b.

[0274] The user can move the coordinate system graphic 368 on the layout image 361 by making a predetermined input to the input unit 5. For example, when the user touches the coordinate system graphic 368 with his / her finger on the display surface and moves the finger, the display position of the coordinate system graphic 368 moves in accordance with the movement of the finger. The process arm trajectory changes in accordance with the movement of the coordinate system graphic 368 on the layout image 361. The process arm trajectory is changed and corrected so that it passes through the origin part of the moved coordinate system graphic 368. The user can set multiple coordinate system graphics 368 on the process arm trajectory.

[0275] Here, the position on the process arm trajectory where the origin of the coordinate system graphic 368 exists is called a waypoint. The orientation of the coordinate system graphic 368 represents the orientation of the end effector at the waypoint. The user can set the orientation of the coordinate system graphic 368 by making a predetermined input to the input unit 5. By setting the orientation of the coordinate system graphic 368, the user can specify the orientation of the end effector at the waypoint.

[0276] Furthermore, when the user makes a predetermined input to the input unit 5, a coordinate system graphic 369 indicating a three-dimensional coordinate system for correcting the process arm trajectory is displayed on the layout image 361, in addition to the coordinate system graphic 368. For example, when the user presses a position on the display surface of the process arm trajectory with his / her finger, the coordinate system graphic 369 for correcting the process arm trajectory is displayed at the pressed position.

[0277] The user can move the coordinate system graphic 369 on the layout image 361 by making a predetermined input to the input unit 5. For example, when the user touches the coordinate system graphic 369 with his / her finger on the display surface and then moves the finger, the coordinate system graphic 369 moves to the destination of the finger. When the coordinate system graphic 369 moves, the process arm trajectory changes so as to expand toward the destination of the coordinate system graphic 369. For example, when the process arm trajectory is represented by a spline curve, the process arm trajectory may expand toward the destination of the coordinate system graphic 369 so as to maintain the spline curve. The example of FIG. 40 shows a coordinate system graphic 369 for correcting the third arm trajectory 367c.

[0278] Here, the position on the process arm trajectory that is closest to the origin of the coordinate system graphic 369 is called a control point. The orientation of the coordinate system graphic 369 represents the orientation of the end effector at the control point. The user can set the orientation of the coordinate system graphic 369 by performing a predetermined input on the input unit 5. By setting the orientation of the coordinate system graphic 369, the user can specify the orientation of the end effector at the control point.

[0279] When the coordinate system graphic 368 is displayed on the layout image 361, the display unit 4 can display the positions of waypoints and the orientation of the end effector at the waypoints on the work details screen 370. Furthermore, when the coordinate system graphic 369 is displayed on the layout image 361, the display unit 4 can display the positions of control points and the orientation of the end effector at the control points on the work details screen 370.

[0280] When a coordinate system graphic 368 for correcting the process arm trajectory in a certain work process is displayed, if process content information 377 of the certain work process is selected, waypoint information 378 indicating the position of the waypoint and the attitude of the end effector at the waypoint is displayed. The waypoint information 378 is displayed by connecting it to the selected process content information 377 with a line.

[0281] Furthermore, when a coordinate system graphic 369 for correcting the process arm trajectory in a certain work process is displayed, if process content information 377 of the certain work process is selected, control point information 379 indicating the position of a control point and the attitude of the end effector at the control point is displayed. The control point information 379 is displayed by connecting it to the selected process content information 377 with a line.

[0282] In the example of Figure 40, a coordinate system figure 368 is shown for correcting the process arm trajectory (i.e., second arm trajectory 367b) in the second work process, so when process content information 377 for the second work process is selected, waypoint information 378 is shown that indicates the position of the waypoint and the posture of the end effector at that waypoint.

[0283] In the example of Figure 40, a coordinate system figure 369 for correcting the process arm trajectory in the third work process (i.e., the third arm trajectory 367c) is shown, so when the process content information 377 for the third work process is selected, control point information 379 indicating the position of the control point and the posture of the end effector at that control point is displayed.

[0284] In the example of FIG. 40, when the instruction area 372 included in the work details screen 370 is selected, a flow diagram 380 showing the work process content and work process sequence is displayed on the work details screen 370, as shown in FIG.

[0285] When the instruction area 27 included in the second detailed action listening screen 390 is selected, the dialogue returns to the previous step by two, and the layout editing screen 300 is displayed on the main screen 11. When the instruction area 46 included in the second detailed action listening screen 390 is selected, the dialogue advances to the next step, and the third layout confirmation screen 400 for confirming the selected set layout again is displayed on the main screen 11.

[0286] <Example of Third Layout Confirmation Screen> Fig. 42 is a schematic diagram showing a display example of the application screen 10 including the third layout confirmation screen 400. As shown in Fig. 42, the third layout confirmation screen 400 has the same configuration as the first layout confirmation screen 270 and the second layout confirmation screen 350. The layout image 275 included in the third layout confirmation screen 400 displays the latest selected set layout. In addition, the robot KPI screen 280 included in the third layout confirmation screen 400 displays the latest robot KPI. The latest robot KPI includes a stability indicator corresponding to the arm trajectory corrected on the second detailed motion hearing screen 390.

[0287] When the instruction area 27 included in the third layout confirmation screen 400 is selected, the dialogue returns to the previous step, and the first detailed action hearing screen 360 is displayed on the main screen 11 .

[0288] When the user finally agrees to the proposal regarding the robot work from the information processing device 1, the user selects the instruction area 46 included in the third layout confirmation screen 400. When the instruction area 46 is selected, the dialogue proceeds to the final stage, and a final confirmation screen 410 for the user to confirm the content of their final agreement is displayed on the main screen 11. The control unit 2 recognizes that the user has finally agreed to the proposal regarding the robot work by selecting the instruction area 46.

[0289] 43 is a schematic diagram showing a display example of the application screen 10 including a final confirmation screen 410. The final confirmation screen 410 includes final asset set information 411 indicating a final selected asset candidate set. The final selected asset candidate set can be said to be a plurality of assets (also referred to as a final asset set) required for robot operation that are ultimately desired by the user.

[0290] The final confirmation screen 410 includes a final layout image 412 showing the final layout of the final selected asset candidate set. The final layout of the final selected asset candidate set can be said to be the layout (also referred to as the final layout) that the user ultimately desires for the final asset set.

[0291] The final confirmation screen 410 also includes final robot KPI information 413 indicating the final robot KPI. The final robot KPI is a robot KPI corresponding to the final layout of the final asset set and the final arm trajectory. The final arm trajectory is the arm trajectory corrected on the second detailed action hearing screen 390. In FIG. 43 , the final robot KPI information 413 is shown as a vertical bar graph.

[0292] The final confirmation screen 410 includes an instruction area 421 for instructing the information processing device 1 to transmit final layout data indicating the final layout to an external device. When the instruction area 421 is selected, the control unit 2 causes the interface 6 to transmit the final layout data to the external device. The external device of the transmission destination may be a user's computer device, a cloud server, or another device.

[0293] The final confirmation screen 410 includes an instruction area 422 for instructing the information processing device 1 to transmit final arm trajectory data representing the final arm trajectory to an external device. When the instruction area 423 is selected, the control unit 2 causes the interface 6 to transmit the final arm trajectory data to the external device.

[0294] The final confirmation screen 410 includes an instruction area 423 for instructing the information processing device 1 to transmit, to an external device, 3D object data indicating 3D objects representing multiple assets that make up the final asset set. When the instruction area 423 is selected, the control unit 2 causes the interface 6 to transmit the 3D object data to the external device.

[0295] The final confirmation screen 410 includes an instruction area 416 for instructing the information processing device 1 to transmit the final layout data, final arm trajectory data, and 3D object data together to an external device. When the instruction area 416 is selected, the control unit 2 causes the interface 6 to transmit the final layout data, final arm trajectory data, and 3D object data to the external device.

[0296] The final confirmation screen 410 includes an instruction area 415 for instructing the information processing device 1 to transmit the message entered by the user to an external device. When the instruction area 415 is selected, the user can input a predetermined message into the input unit 5 and cause the information processing device 1 to transmit the input message to the external device.

[0297] The final confirmation screen 410 includes backtracking number information 417 indicating the number of times the user has returned to a previous stage in the dialogue between the information processing device 1 and the user using the application screen 10 .

[0298] When any of the designation areas 416, 421, 422, and 423 is selected, the execution of the predetermined application program 3a in the control unit 2 is terminated, and the application screen 10 is no longer displayed on the display surface.

[0299] The confirmation screen 30 (see FIG. 3 ) displayed when progress information 26 a, 26 b, or 26 c included in the project management screen 20 shown in FIG. 2 above is selected is a screen similar to the final confirmation screen 410. As shown in FIG. 3 , the confirmation screen 30 includes asset set information 31 indicating a plurality of assets, a layout image 32 indicating the layout of the plurality of assets, and robot KPI information 33 indicating robot KPIs corresponding to the layout. On the confirmation screen 30, the robot KPI information 33 is displayed, for example, as a vertical bar graph.

[0300] The asset set information 31 on the confirmation screen 30 (referred to as the in-progress project confirmation screen 30) displayed when the progress information 26a included in the project management screen 20 is selected indicates the latest selected asset candidate set for the in-progress project. Also, on the in-progress project confirmation screen 30, the layout image 32 indicates the latest layout of the latest selected asset candidate set for the in-progress project. Also, the robot KPI information 33 on the in-progress project confirmation screen 30 indicates the latest robot KPI for the in-progress project.

[0301] The asset set information 31 on the confirmation screen 30 (also referred to as the completed project confirmation screen 30) that is displayed when the progress information 26b included in the project management screen 20 is selected indicates the final selected asset candidate set for the completed project, similar to the final asset set information 411 on the final confirmation screen 410. Furthermore, the layout image 32 on the completed project confirmation screen 30 indicates the final layout of the final selected asset candidate set for the completed project, similar to the final layout image 412 on the final confirmation screen 410. Furthermore, the robot KPI information 33 on the completed project confirmation screen 30 indicates the final robot KPI for the completed project, similar to the final robot KPI information 413 on the final confirmation screen 410.

[0302] The asset set information 31 on the confirmation screen 30 (also referred to as the pending project confirmation screen 30) displayed when the progress information 26c included in the project management screen 20 is selected indicates the latest selected asset candidate set for the pending project. Also, on the pending project confirmation screen 30, the layout image 32 indicates the latest layout of the latest selected asset candidate set for the pending project. Also, the robot KPI information 33 on the pending project confirmation screen 30 indicates the latest robot KPI for the pending project.

[0303] The confirmation screen 30 includes an instruction area 35 for instructing the information processing device 1 to transmit data to an external device. The first data transmitted to the external device when the instruction area 36 on the confirmation screen 30 for the ongoing project is selected includes layout data indicating the latest layout of the latest selected asset candidate set for the ongoing project. The first data also includes arm trajectory data indicating the trajectory of the arm in the latest layout of the latest selected asset candidate set for the ongoing project. The first data also includes 3D object data indicating 3D objects of multiple assets constituting the latest selected asset candidate set for the ongoing project.

[0304] The second data transmitted to the external device when the instruction area 36 on the completed project confirmation screen 30 is selected includes layout data indicating the final layout of the final selected asset candidate set for the completed project. The second data also includes arm trajectory data indicating the final arm trajectory in the final layout of the final selected asset candidate set for the completed project. The second data also includes 3D object data indicating 3D objects of the multiple assets that make up the final selected asset candidate set for the completed project.

[0305] The third data transmitted to the external device when the instruction area 36 on the pending project confirmation screen 30 is selected includes layout data indicating the latest layout of the latest selected asset candidate set for the pending project. The third data also includes arm trajectory data indicating the trajectory of the arm in the latest layout of the latest selected asset candidate set for the pending project. The third data also includes 3D object data indicating 3D objects of the multiple assets that make up the latest selected asset candidate set for the pending project.

[0306] The confirmation screen 30 includes an instruction area 35 for instructing the information processing device 1 to transmit a message input by the user to an external device. When the instruction area 36 is selected, the user can input a message by performing a predetermined input on the input unit 5 and cause the information processing device 1 to transmit the input message to the external device.

[0307] The confirmation screen 30 includes backtracking information 37 indicating the number of times a user has returned to a previous stage in a dialogue between the information processing device 1 and the user using the application screen 10. The backtracking information 37 on the confirmation screen 30 for an ongoing project indicates the number of times a user has returned to a previous stage in a dialogue for a project that is currently in progress. The backtracking information 37 on the confirmation screen 30 for a completed project indicates the number of times a user has returned to a previous stage in a dialogue for a completed project. The backtracking information 37 on the confirmation screen 30 for a pending project indicates the number of times a user has returned to a previous stage in a dialogue for a pending project.

[0308] The confirmation screen 30 includes an instruction area 27. When the instruction area 27 is selected, the main screen 11 returns to the project management screen 20. The confirmation screen 30 includes an instruction area 39 for instructing the resumption of the project. When the instruction area 39 on the confirmation screen 30 for an ongoing project is selected, the application screen 10 for the dialogue stage of the ongoing project is displayed on the display surface. When the instruction area 39 on the confirmation screen 30 for a pending project is selected, the application screen 10 for the dialogue stage of the pending project is displayed on the display surface. Note that the instruction area 39 is not shown on the confirmation screen 30 for a completed project.

[0309] The user can halt an ongoing project by making a predetermined input to the input unit 5 while the project is in progress. In this case, the halted project becomes an ongoing project. The user can also put a project on hold by making a predetermined input to the input unit 5 while the project is in progress. The control unit 2 may also put a project on hold, for example, when the number of times a dialogue has returned to a previous stage in an ongoing project is large.

[0310] So far, we have described a display example of the application screen 10 when the designated area 41a included in the work-outline-hearing screen 40 shown in Fig. 4 is selected. Next, we will describe a display example of the application screen 10 when the designated area 41b included in the work-outline-hearing screen 40 is selected.

[0311] When the designation area 41b corresponding to the second work outline is selected, the designation area 41b is highlighted as shown in Fig. 44. After the designation area 41b is selected, when the user selects the instruction area 46 on the work outline hearing screen 40, a detailed hearing screen 50 is displayed on the main screen 11. Below, the detailed hearing screen 50 displayed when the designation area 41b is selected will be described, focusing on the differences from the detailed hearing screen 50 displayed when the designation area 41a is selected.

[0312] 45 is a schematic diagram showing an example of the display of the application screen 10 including the detailed listening screen 50 (also referred to as the second detailed listening screen 50B) that is displayed when the designated area 41b is selected. Fig. 45 shows an example of how the object listening screen 60 including the space listening screen 70 is displayed on the second detailed listening screen 50B. Hereinafter, the object listening screen 60 displayed on the second detailed listening screen 50B will be referred to as the second object listening screen 60B. Furthermore, the space listening screen 70 displayed on the second object listening screen 60B will be referred to as the second space listening screen 70B.

[0313] The work summary image 71 shown on the second pace listening screen 70B represents the second work summary. As described above, the second work summary does not specify an out-tray. Therefore, unlike the work summary image 71 on the first pace listening screen 70, the work summary image 71 on the second pace listening screen 70B includes an image 71d that shows the outline of the out-tray in dashed lines.

[0314] 46 is a schematic diagram showing an example of how the second detailed listening screen 50B displays a second object listening screen 60B including an in-tray listening screen 80. Hereinafter, the in-tray listening screen 80 displayed on the second object listening screen 60B will be referred to as the second in-tray listening screen 80B.

[0315] The initial setting of the slider position of the designated area 85 shown on the second in-tray listening screen 80B is a position corresponding to the second work overview. FIG. 46 shows the initial setting of the slider of the designated area 85. As described above, in the second work overview, the robot holds workpieces that are stacked loosely in the inlay. Therefore, as shown in FIG. 46, the initial setting of the slider of the designated area 85 is set to the far right. Therefore, the workpiece image 83 shown on the second in-tray listening screen 80B shows workpieces that are stacked loosely in the in-tray.

[0316] 47 is a schematic diagram showing an example of how the second object listening screen 60B including the outlay listening screen 90 is displayed on the second detailed listening screen 50B. Hereinafter, the outtray listening screen 90 displayed on the second object listening screen 60B will be referred to as the second outtray listening screen 90B.

[0317] The second out-tray listening screen 90B includes a designation area 98 for instructing that listening to requests regarding the out-tray from the user be enabled. When the designation area 98 is selected, the user can operate the sliders in the designation areas 95, 96, and 97 to input requests regarding the out-tray into the information processing device 1. When the second space listening screen 70B is displayed after the designation area 98 is selected, the work overview image 71 on the second space listening screen 70B includes an image 71d showing the outline of the out-tray in solid lines, as shown in FIG. 6 and the like.

[0318] Other configurations of the second detailed hearing screen 50B are the same as those of the detailed hearing screen 50 that is displayed when the designated area 41a is selected.

[0319] If the designated area 104 is not selected on the work processing hearing screen 101 shown in FIG. 14 above, no processing is specified for the workpieces between the intray and the outtray. In other words, it is specified that no processing is to be performed on the workpieces between the intray and the outtray. In this case, only one type of workpiece is handled by the robot operation, and the display unit 4 can display only the first object details hearing screen 110, as shown in FIG. 48, as a screen for hearing details about the workpiece. In this case, even if a first object is not specified on the work processing hearing screen 101, the first object details hearing screen 110 can be displayed. The user can specify the shape and material of the object in the intray on the first object details hearing screen 110.

[0320] When the designation area 104 is not selected on the work processing listening screen 101 and the instruction area 46 included in the detailed listening screen 50 is selected, the reference listening screen 180 shown in Figures 49 and 50 is displayed on the main screen 11. Figure 49 shows an example of how a work-based listening screen 181 is displayed on the reference listening screen 180. Figure 50 shows an example of how a tray-based listening screen 200 is displayed on the reference listening screen 180.

[0321] The workpiece reference listening screen 181 shown in Fig. 49 allows the user to specify the holding reference position and placement reference position of an object in the in-tray. The tray reference listening screen 200 shown in Fig. 50 allows the display of a first in-tray reference listening screen 210 and an out-tray reference listening screen 230. When only one type of workpiece is handled in the robot operation, the number of in-trays is one.

[0322] In the above example, the asset candidate set proposed to the user is composed of only multiple pieces of hardware, but the multiple assets constituting the asset candidate set proposed to the user may also include software. In this case, the multiple assets constituting the asset candidate set may include software for performing object recognition on workpieces, software for performing object recognition on in-trays, software for performing object recognition on out-trays, or other software.

[0323] Although the information processing device has been described in detail above, the above description is merely illustrative in all respects, and this disclosure is not limited thereto. Furthermore, the various examples described above can be combined and applied as long as they are not mutually contradictory. It is understood that countless examples not illustrated can be envisioned without departing from the scope of this disclosure.

[0324] This disclosure includes the following:

[0325] In one embodiment, (1) an information processing device includes a display unit, an input unit that accepts user input, and a control unit, and the control unit causes the display unit to display a layout of a candidate set of multiple hardware items required for robot operation and an evaluation index for the operation corresponding to the layout, and when the input unit accepts a first input related to editing the layout, calculates the evaluation index based on the first input and displays it again on the display unit.

[0326] (2) In the information processing device of (1) above, the control unit acquires a second input related to the work of the robot from the input unit, and calculates the evaluation index based on the second input.

[0327] (3) In the information processing device of (2) above, the second input includes information on the type of processing for the work object, and the control unit calculates the evaluation index based on the information.

[0328] (4) An information processing device according to any one of (1) to (3) above, wherein the control unit acquires a third input related to the work of the robot from the input unit, acquires information on at least one candidate set for the plurality of hardware based on the third input, displays the information on the display unit so that it can be selected via the input unit, and displays the layout of the selected candidate set from the at least one candidate set on the display unit.

[0329] (5) In the information processing device of (4), the third input includes priority details for the work, and the control unit acquires the information of the at least one candidate set based on the priority details.

[0330] (6) In an information processing device as described in (4) or (5) above, the control unit determines a recommendation level for the at least one candidate set based on the third input, and causes the display unit to display the information and recommendation level for the at least one candidate set.

[0331] (7) In the information processing device according to any one of (1) to (6) above, the control unit causes the display unit to display the layout in a manner that allows a display viewpoint to be changed.

[0332] (8) In the information processing device according to any one of (1) to (7) above, the control unit causes the display unit to display the layout from the viewpoint of a person standing at a specific point.

[0333] Also, in one embodiment, (9) an information processing system includes any one of the information processing devices (1) to (8) above and a display device communicatively connected to the information processing device, wherein the information processing device is capable of outputting information about the layout to the display device, and the display device virtually reproduces and displays the layout based on the information about the layout.

[0334] In one embodiment, the program (10) is a program for causing a computer device to function as any one of the information processing devices (1) to (8) above or the information processing system (9) above.

[0335] REFERENCE SIGNS LIST 1 Information processing device 2 Control unit 4 Display unit 5 Input unit 3a Program

Claims

1. An information processing device comprising: a display unit; an input unit for accepting user input; and a control unit, wherein the control unit causes the display unit to display a layout of a candidate set of multiple hardware components required for robot operation and an evaluation index for the operation corresponding to the layout, and when the input unit accepts a first input related to editing the layout, determines the evaluation index based on the first input and displays it again on the display unit.

2. An information processing device according to claim 1, wherein the control unit acquires a second input related to the work of the robot from the input unit, and calculates the evaluation index based on the second input.

3. An information processing device according to claim 2, wherein the second input includes information on a type of processing for a work object, and the control unit determines the evaluation index based on the information.

4. An information processing device as described in any one of claims 1 to 3, wherein the control unit: acquires a third input related to the work of the robot from the input unit; acquires information of at least one candidate set for the plurality of hardware pieces based on the third input; displays the information on the display unit selectably via the input unit; and displays the layout of the selected candidate set from the at least one candidate set on the display unit.

5. An information processing device according to claim 4, wherein the third input includes a priority content for the task, and the control unit acquires the information of the at least one candidate set based on the priority content.

6. An information processing device as described in claim 4 or claim 5, wherein the control unit determines a recommendation level of the at least one candidate set based on the third input, and causes the display unit to display the information and the recommendation level of the at least one candidate set.

7. An information processing device according to any one of claims 1 to 6, wherein the control unit causes the display unit to display the layout in a manner that allows the display viewpoint to be changed.

8. An information processing device according to any one of claims 1 to 7, wherein the control unit causes the display unit to display the layout from the viewpoint of a person standing at a specific point.

9. An information processing system comprising: an information processing device according to any one of claims 1 to 8; and a display device communicatively connected to said information processing device, wherein said information processing device is capable of outputting information relating to said layout to said display device, and said display device virtually reproduces and displays said layout based on said information relating to said layout.

10. A program for causing a computer device to function as the information processing device according to any one of claims 1 to 8 or the information processing system according to claim 9.

Images