Processing device, program, display device, and processing system

Abstract

This processing device is provided with a simulation unit that simulates the work of a robot on a virtual space. The simulation unit selects at least one virtual work environment among a plurality of types of virtual work environments on the basis of the work time of each one of the plurality of types of virtual work environments, the work time being acquired by the simulation.

Description

Processing device, program, display device, and processing system

[0001] The present disclosure relates to simulation of robotic tasks.

[0002] Patent Document 1 describes a simulation technique.

[0003] Japanese Patent Application Laid-Open No. 2022-173888

[0004] A processing device, a program, a display device, and a processing system are disclosed. In one embodiment, the processing device includes a simulation unit that simulates robot work in a virtual space. The simulation unit selects at least one virtual work environment from among a plurality of types of virtual work environments based on the work time for each of the plurality of types of virtual work environments obtained by the simulation.

[0005] In one embodiment, the program is a program for causing a computer device to function as the processing device.

[0006] In one embodiment, the display device displays the results of the simulation performed by the processing device.

[0007] In one embodiment, a processing system includes the processing device described above and a robot controller capable of communicating with the processing device.

[0008] FIG. 1 is a schematic diagram showing an example of the configuration of a processing device. FIG. 2 is a schematic diagram showing an example of the configuration of a processing system. FIG. 3 is a schematic diagram showing an example of the configuration of a sensor device. FIG. 4 is a schematic diagram showing an example of a robot mechanism. FIG. 5 is a schematic diagram showing an example of a robot mechanism. FIG. 6 is a flowchart showing an example of the operation of a simulation unit. FIG. 7 is a schematic diagram showing an example of a display of a processing device. FIG. 8 is a schematic diagram showing an example of a display of a processing device.

[0009] 1 is a schematic diagram showing an example of the configuration of a processing device 1. The processing device 1 is, for example, a computer device. The processing device 1 is capable of simulating the work of a robot 10 in a virtual space on the computer. It can also be said that the processing device 1 is capable of simulating the movement of the robot 10 in the virtual space.

[0010] Fig. 2 is a schematic diagram showing an example of a processing system 50 including a processing device 1 and a robot 10. As shown in Fig. 2, the processing system 50 includes, for example, the processing device 1, a plurality of robots 10, a sensor device 20, and a server device 30. The processing system 50 can also be called a robot system.

[0011] The processing device 1, the multiple robots 10, the sensor device 20, and the server device 30 are connected to a network 40 including, for example, the Internet, and are capable of communicating with each other through the network 40. The processing device 1 can simulate the work of a single robot 10, and can also simulate the work of two or more robots 10 performed simultaneously.

[0012] <About the Robot> Each robot 10 includes, for example, a robot mechanism 12 and a robot controller 11 that controls the robot mechanism 12. The robot controller 11 (also simply referred to as the controller 11) is, for example, a computer device. The controller 11 is connected to a network 40. The controller 11 can communicate with the processing device 1, the sensor device 20, and the server device 30 via the network 40. Furthermore, the controller 11 of each robot 10 may communicate with the controllers 11 of other robots 10 via the network 40.

[0013] The multiple robotic mechanisms 12 included in the processing system 50 may include multiple robotic mechanisms 12 of the same type, or multiple robotic mechanisms 12 of different types. The multiple robotic mechanisms 12 included in the processing system 50 may include, for example, arm-type robotic mechanisms. In this case, the arm-type robotic mechanism serving as the robotic mechanism 12 may be a single-arm robotic mechanism or a double-arm arm-type robotic mechanism. The multiple robotic mechanisms 12 may include, for example, multiple arm-type robotic mechanisms with different numbers of joints, or multiple arm-type robots manufactured by different manufacturers. The multiple robotic mechanisms 12 may include self-propelled robotic mechanisms such as AGVs (Automatic Guided Vehicles) or AMRs (Autonomous Mobile Robots). The multiple robotic mechanisms 12 may also include humanoid robotic mechanisms. The multiple robotic mechanisms 12 may also include collaborative robotic mechanisms capable of working collaboratively with humans.

[0014] <Regarding the Server Device> The server device 30 manages the operation of the entire processing system 50. The server device 30 can be, for example, a computer device. The server device 30 can also be, for example, a processing device. The server device 30 can be, for example, a cloud server. The server device 30 can be configured as one computer device or multiple computer devices.

[0015] The server device 30 manages various information. The information stored in the server device 30 also includes programs (in other words, software). For example, the server device 30 stores, for each robot 10, robot-related information related to that robot 10. The robot-related information includes, for example, an operation program installed in the controller 11. The operation program included in the robot-related information is an operation program used by the robot 10 in the actual work environment, and can be said to be an operation program executed by the controller 11 in the actual work environment. Furthermore, the robot 10 can perform a predetermined task by, for example, operating in accordance with the operation program.

[0016] The robot-related information also includes robot mechanism information related to the robot mechanism 12. The robot mechanism information includes, for example, robot shape information that represents the three-dimensional shape of the robot mechanism 12. The robot shape information may also include color information of the robot mechanism 12. The robot shape information may be, for example, a URDF file that represents the structure of the robot mechanism 12. URDF is an abbreviation for Unified Robot Description Format. The URDF file describes the three-dimensional shape and other information of the robot mechanism 12. The robot shape information may also be a 3D model of the robot mechanism 12. 3D means three dimensions. The 3D model of the robot mechanism 12 may be generated based on CAD data. CAD is an abbreviation for Computer Aided Design.

[0017] The server device 30 may also store environmental information indicating the work environment in which the robot 10 performs work. The environmental information may include, for example, obstacle information regarding each obstacle that impedes the operation of the robot 10. The obstacle information may include, for example, obstacle shape information that represents the three-dimensional shape of the obstacle. The obstacle shape information may include color information of the obstacle. The obstacle shape information may be a 3D model of the obstacle. The 3D model of the obstacle may be generated based on CAD data. The multiple obstacles may include, for example, a workbench on which the robot 10 performs work, or may include structures such as walls and pillars. The multiple obstacles may also include at least one of a chair, a desk, a shelf, a partition, and a lighting fixture. The multiple obstacles may also include peripheral devices such as a camera.

[0018] The environmental information may also include object information of an object that is the work target of the robot 10. The object is, for example, an object that is to be held by the robot 10. The object information may include, for example, object shape information that represents the three-dimensional shape of the object. The object shape information may include color information of the object. The object shape information may be a 3D model of the object. The 3D model of the object may be generated based on CAD data. The object may include multiple types of objects.

[0019] The server device 30 may also store performance information indicating the performance of the robot 10 or peripheral devices located around the robot 10. The performance information may include, for example, the maximum output of the robot 10, such as the speed or holding force, or information such as the resolution of a camera. The server device 30 may also store simulation results of the simulations described below.

[0020] 1, the processing device 1 includes, for example, a control unit 2, a storage unit 3, a display unit 4, an input unit 5, and an interface 6. The processing device 1 can also be considered, for example, a processing circuit.

[0021] The interface 6 is capable of communicating with the network 40. The interface 6 is capable of communicating with the network 40 using at least one of wired communication and wireless communication. The interface 6 can also be called, for example, an interface circuit, a communication unit, or a communication circuit. The interface 6 inputs information received from the network 40 to the control unit 2. The interface 6 also transmits information from the control unit 2 to the network 40.

[0022] The control unit 2 can generally manage the operation of the processing device 1 by controlling the other components of the processing device 1. The control unit 2 can also be referred to as a control circuit, for example. The control unit 2 includes at least one processor to provide control and processing power for performing various functions, as described in more detail below.

[0023] According to various embodiments, the at least one processor may be implemented as a single integrated circuit (IC) or as multiple communicatively connected integrated circuits ICs and / or discrete circuits. The at least one processor may be implemented according to various known techniques.

[0024] In one embodiment, a processor includes one or more circuits or units configured to perform one or more data computational procedures or processes, for example, by executing instructions stored in associated memory. In other embodiments, a processor may be firmware (e.g., discrete logic components) configured to perform one or more data computational procedures or processes.

[0025] According to various embodiments, the processor may include one or more processors, controllers, microprocessors, microcontrollers, application specific integrated circuits (ASICs), digital signal processors, programmable logic devices, field programmable gate arrays, or any combination of these devices or configurations, or other known devices and configurations, to perform the functions described below.

[0026] The control unit 2 may include, for example, a CPU (Central Processing Unit) as a processor. The storage unit 3 may include a non-transitory recording medium readable by the CPU of the control unit 2, such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The storage unit 3 stores, for example, a program 3a for controlling the processing device 1. Various functions of the control unit 2 are realized, for example, by the CPU of the control unit 2 executing the program 3a in the storage unit 3.

[0027] The configuration of the control unit 2 is not limited to the above example. For example, the control unit 2 may include multiple CPUs. The control unit 2 may also include at least one DSP (Digital Signal Processor). All or some of the functions of the control unit 2 may be realized by a hardware circuit that does not require software to realize the function. The storage unit 3 may also include a computer-readable non-transitory recording medium other than ROM and RAM. The storage unit 3 may also include, for example, a small hard disk drive or SSD (Solid State Drive).

[0028] The input unit 5 is capable of accepting various inputs from the user. The input unit 5 may include, for example, a mouse and a keyboard. The input unit 5 may also include a touch sensor that accepts touch operations by the user. The input unit 5 may also include a microphone that accepts voice input by the user. The control unit 2 recognizes the content of the user input accepted by the input unit 5 based on the output signal from the input unit 5.

[0029] The display unit 4 is capable of displaying various types of information, such as simulation results or robot-related information, under the control of the control unit 2. The display unit 4 has a display surface that displays various types of information. The display unit 4 may be, for example, a liquid crystal display, an organic electroluminescence (EL) display, or a plasma display. Furthermore, if the input unit 5 includes a touch sensor, the touch sensor and the display surface of the display unit 4 may form a touch panel display having a display function and a touch detection function. In this case, the input unit 5 can detect a touch operation on the display surface of the display unit 4. The input unit 5 and the display unit 4 form a user interface.

[0030] The control unit 2 includes, for example, a simulation unit 2a as a functional block. The simulation unit 2a can simulate the work of the robot 10 in a virtual space. The simulation unit 2a, for example, recreates an actual work environment of the robot 10 in the virtual space. The simulation unit 2a can simulate the work of the robot 10 while changing a virtual work environment in which the actual work environment of the robot 10 is recreated in the virtual space. The simulation unit 2a can execute a simulation of the work of the robot 10 in each of a plurality of types of virtual work environments. The operation of the simulation unit 2a will be described in detail later. Note that all or some of the functions of the simulation unit 2a may be realized by a hardware circuit that does not require software to realize the function.

[0031] The server device 30 may have the same configuration as the processing device 1. In this case, the server device 30 may not have at least one of a display unit and an input unit. Furthermore, the robot controller 11 may have the same configuration as the processing device 1. In this case, the robot controller 11 may not have at least one of a display unit and an input unit. Furthermore, in the above example, the processing device 1 has the display unit 4 and the input unit 5, but at least one of the display unit 4 and the input unit 5 may exist independently of the processing device 1. In this case, at least one of the display unit 4 and the input unit 5 may be capable of communicating with other components of the processing system 50 via the network 40. The display unit 4 and the input unit 5 may be referred to as a display device and an input device, respectively.

[0032] <About the Sensor Device> The sensor device 20 performs three-dimensional measurement of the actual work environment of the robot 10 and generates environmental measurement data indicating the results of the three-dimensional measurement of the actual work environment. The environmental measurement data indicates, for example, three-dimensional shape and color information of the actual work environment. The environmental measurement data is, for example, point cloud data indicating the three-dimensional shape and color information of the actual work environment.

[0033] FIG. 3 is a schematic diagram showing an example of the configuration of the sensor device 20. The sensor device 20 is, for example, a portable computer. The sensor device 20 may be, for example, a tablet terminal, a laptop computer, or another type of portable device. The sensor device can also be considered a processing device. The user carries the sensor device 20 to the actual work environment of the robot 10. The user then operates the sensor device 20 located in the actual work environment to cause the sensor device 20 to measure the actual work environment of the robot 10. When the sensor device 20 is located in the actual work environment, the sensor device 20 can generate point cloud data representing the three-dimensional shape and color information of the actual work environment. Note that the sensor device 20 may also be a stationary device placed in the actual work environment of the robot 10.

[0034] 3, the sensor device 20 includes, for example, a control unit 22, a storage unit 23, a display unit 24, an input unit 25, an interface 26, a distance sensor 27, and a color camera 28. The sensor device 20 can also be referred to as, for example, a sensor circuit.

[0035] The interface 26 is capable of communicating with the network 40. The interface 26 is capable of communicating with the network 40 using at least one of wired communication and wireless communication. The interface 26 may also be referred to as, for example, an interface circuit, a communication unit, or a communication circuit. The interface 26 inputs information received from the network 40 to the control unit 22. The interface 26 also transmits information from the control unit 22 to the network 40.

[0036] The control unit 22 can control the other components of the sensor device 20 to provide overall management of the operation of the sensor device 20. The control unit 22 can also be referred to as, for example, a control circuit. The control unit 22 includes at least one processor to provide control and processing capabilities for performing various functions, as described in more detail below. The above description of the at least one processor included in the control unit 2 of the processing device 1 can also be applied to the at least one processor included in the control unit 22.

[0037] The control unit 22 may include, for example, a CPU as a processor. The storage unit 23 may include a non-transitory recording medium such as a ROM or RAM that can be read by the CPU of the control unit 22. The storage unit 23 stores, for example, a program 23a for controlling the sensor device 20. The various functions of the control unit 22 are realized, for example, by the CPU of the control unit 22 executing the program 23a in the storage unit 23.

[0038] The configuration of the control unit 22 is not limited to the above example. For example, the control unit 22 may include multiple CPUs. The control unit 22 may also include at least one DSP. All or some of the functions of the control unit 22 may be implemented by a hardware circuit that does not require software to realize the function. Similarly to the memory unit 3 of the processing device 1, the memory unit 23 may include a computer-readable non-transitory recording medium other than ROM and RAM.

[0039] The input unit 25 is capable of accepting various inputs from the user. The control unit 2 recognizes the content of the user input accepted by the input unit 25 based on an output signal from the input unit 25. The configuration of the input unit 25 may be the same as the configuration of the input unit 5 of the processing device 1, for example. The display unit 24 is capable of displaying various information under the control of the control unit 22. The configuration of the display unit 24 may be the same as the configuration of the display unit 4 of the processing device 1, for example.

[0040] The distance sensor 27 is capable of measuring the distance to an object present in the actual work environment. The distance sensor 27 is capable of acquiring distance information to an object present in the actual work environment. The distance sensor 27 acquires the distance information using, for example, LiDAR. LiDAR is an abbreviation for Light Detection And Ranging. The distance sensor 27 may acquire the distance information using, for example, a TOF method or an FMCW method. ToF is an abbreviation for Time of Flight, and FMCW is an abbreviation for Frequency Modulated Continuous Wave. The distance sensor 27 may acquire the distance information using a stereo camera.

[0041] The color camera 28 can acquire color information of objects present in the actual work environment. The color camera 28 captures an image of the actual work environment and generates color image data that represents a color image of the actual work environment. Each of the multiple pixel data that make up the color image data can be considered to be color information of the actual work environment. The pixel data includes, for example, an R component (red component), a G component (green component), and a B component (blue component). The color image data is also called RGB image data.

[0042] The control unit 22 generates point cloud data representing the three-dimensional shape and color information of the actual work environment as environmental measurement data, based on the distance measurement results from the distance sensor 27 and the color image data generated by the color camera 28. Hereinafter, the point cloud data representing the three-dimensional shape and color information of the actual work environment may be referred to as environmental point cloud data.

[0043] Environmental point cloud data is data that represents a point cloud (also called a measurement point cloud), which is a collection of multiple measurement points in the actual work environment. The multiple measurement points are located on the surfaces of objects that exist in the actual work environment. Environmental point cloud data can also be said to be data that represents objects that exist in the actual work environment as multiple points.

[0044] The environment point cloud data includes, for example, three-dimensional coordinate information and color information of each measurement point in the actual work environment. The environment point cloud data is also called colored point cloud data. The environment point cloud data can also be said to be data representing the surface shape and color of an object that exists in the actual work environment. The environment point cloud data includes coordinate information that represents the position of a point on the surface of an object that exists in the actual work environment, and color information that represents the color of the point. The color information included in the environment point cloud data includes, for example, an R component, a G component, and a B component.

[0045] In the environmental point cloud data, for each measurement point, coordinate information for that measurement point and color information for that measurement point are associated with each other. In the environmental point cloud data, for each point constituting the point cloud, coordinate information for that point and color information for that point are associated with each other. In other words, in the environmental point cloud data, for each of multiple points on the surface of an object present in the actual work environment, coordinate information (which can also be considered position information) representing the position of that point is associated with color information representing the color of that point.

[0046] In the sensor device 20, the interface 26 transmits the environmental measurement data (in other words, environmental point cloud data) generated by the control unit 22 to, for example, the server device 30 via the network 40. The server device 30 stores and manages the received environmental measurement data. When executing a simulation, the simulation unit 2a of the processing device 1 acquires the environmental measurement data from the server device 30 via the interface 6. The simulation unit 2a sets a virtual work environment for the robot 10 in a virtual space based on the acquired environmental measurement data. The simulation unit 2a then simulates the work of the robot 10 in the virtual work environment.

[0047] 4 is a schematic diagram showing an example of a single-arm robot mechanism 12 (also referred to as a single-arm robot mechanism 12A). The single-arm robot mechanism 12A includes, for example, an arm 120 and an end effector 121 connected to the arm 120.

[0048] The arm 120 includes, for example, a plurality of joints. The arm 120 can change its posture by, for example, rotating at least one of the plurality of joints.

[0049] The end effector 121 is capable of holding an object 80 on which work is to be performed. The end effector 121 is capable of, for example, suctioning and holding the object 80. The end effector 121 has a suction pad 122 that suctions the object 80. The posture of the end effector 121 changes depending on the posture of the arm 120. Note that the end effector 121 may have multiple fingers. In this case, the end effector 121 grips the object 80 with the multiple fingers and holds the object 80.

[0050] The single-arm robot mechanism 12A can move the object 80 held by the end effector 121 by moving the arm 120. Specifically, the single-arm robot mechanism 12A can move the object 80 held by the end effector 121 by changing the posture of the arm 120.

[0051] The single-arm robot mechanism 12A holds, for example, the object 80 in a first tray 91 on the work table 90 with the end effector 121. Then, the single-arm robot mechanism 12A moves the arm 120 to move the object 80 held by the end effector 121 to a second tray 92 on the work table 90.

[0052] The controller 11 of the robot 10 (also referred to as the single-arm robot 10A) equipped with the single-arm robot mechanism 12A can control the attitude of the arm 120. The controller 11 can control the position of the end effector 121 by controlling the attitude of the arm 120. The controller 11 can also control the holding operation of the end effector 121. The controller 11 can cause the end effector 121 to hold an object 80 or release the object 80. The controller 11 can also control the holding force of the end effector 121 to hold the object 80. Note that the controller 11 may be provided as separate controllers, one for controlling the arm 120 and one for controlling the end effector 121. The controller 11 can also be considered a processing device.

[0053] As shown in FIG. 4 , the processing system 50 includes a first camera 71 and a second camera 72. The first camera 71 is, for example, a color camera. The first camera 71 is capable of capturing images of the actual work environment of the robot 10 from above. The position of the first camera 71 is fixed. In the example of FIG. 4 , the first camera 71 is attached, for example, to the tip of a holding arm extending upward from the work table 90. In the example of FIG. 4 , the first camera 71 is capable of capturing images of the entire single-arm robot 10A, the work table 90, and the area around the work table 90 from above. Therefore, the first camera 71 is capable of capturing images of the first tray 91 and the second tray 92 on the work table 90 from above. The color image data generated by the first camera 71 is input to the controller 11. Each of the multiple pixel data constituting the color image data generated by the first camera 71 includes, for example, an R component, a G component, and a B component. Note that the first camera 71 may be a 3D camera.

[0054] The second camera 72 is, for example, a color camera. The second camera 72 is, for example, fixed to the end effector 121. The color image data generated by the second camera 72 is input to the controller 11. Each of the multiple pixel data constituting the color image data generated by the second camera 72 includes, for example, an R component, a G component, and a B component. The posture of the second camera 72 changes depending on the posture of the end effector 121. Therefore, the imaging range of the second camera 72 changes depending on the posture of the end effector 121. Since the posture of the end effector 121 changes depending on the posture of the arm 120, it can also be said that the imaging range of the second camera 72 changes depending on the posture of the arm 120. Note that the second camera 72 may be a 3D camera.

[0055] The server device 30 stores first camera information related to the first camera 71 and second camera information related to the second camera 72. The first camera information includes, for example, position information indicating the placement position of the first camera 71 in the actual work environment, first camera shape information representing the three-dimensional shape of the first camera 71, and performance information indicating the performance of the first camera 71. The second camera information includes, for example, position information indicating the attachment position of the second camera 72 to the end effector 121, second camera shape information representing the three-dimensional shape of the second camera 72, and performance information indicating the performance of the second camera 72. The first camera shape information and the second camera shape information may be, for example, CAD data or a three-dimensional model.

[0056] The server device 30 stores first tray information related to the first tray 91 and second tray information related to the second tray 92. The first tray information includes, for example, position information indicating the placement position of the first tray 91 in the actual work environment and first tray shape information representing the three-dimensional shape of the first tray 91. The second tray information includes, for example, position information indicating the placement position of the second tray 92 in the actual work environment and second tray shape information representing the three-dimensional shape of the second tray 92. The first tray shape information and the second tray shape information may be, for example, CAD data or a three-dimensional model.

[0057] The server device 30 stores object information about the object 80. The object region includes, for example, object shape information representing the three-dimensional shape of the object. The object shape information may be, for example, CAD data or a 3D model.

[0058] <Example of Robot Operation> The single-arm robot 10A (also simply referred to as the robot 10A) holds, for example, an object 80 in a first tray 91 placed on a work table 90 with the end effector 121. Next, the robot 10A moves the arm 120 to move the object 80 to another second tray 92 on the work table 90. Then, the robot 10A causes the end effector 121 to release the object 80 from its hold and place the object 80 at a predetermined location in the second tray 92.

[0059] For example, a plurality of objects 80 are stacked in a random manner in the first tray 91. The robot 10A performs a moving and placing operation to move the plurality of objects 80 in the first tray 91 and place them in the second tray 92. In the moving and placing operation, for example, the robot 10A holds the plurality of objects 80 in the tray 95 one by one, moves them to the second tray 92, and places them in a predetermined location in the second tray 92. Note that the plurality of objects 80 do not have to be stacked in a random manner in the first tray 91.

[0060] When the moving and placing operation starts, the controller 11 causes the end effector 121 to hold one of the objects 80 in the first tray 91, for example, based on color image data generated by the first camera 71 or the second camera 72. Next, the controller 11 moves the arm 120 toward the second tray 92 so that the single-arm robot mechanism 12A (also simply referred to as the robot mechanism 12A) does not interfere with any obstacles, and moves the object 80 held by the end effector 121 to the second tray 92. In the example of Fig. 4, the first tray 91, the second tray 92, and the work table 90 are obstacles.

[0061] When moving the arm 120, the controller 11 identifies an area in the actual work environment where an obstacle exists (obstacle existence area) based on, for example, color image data generated by the first camera 71. Based on the identified obstacle existence area, the controller 11 moves the arm 120 toward the second tray 92 so that the robot mechanism 12A does not interfere with the obstacle. Thereafter, the controller 11 causes the end effector 121 to release its hold on the object 80. As a result, one object 80 is placed at a predetermined location in the second tray 92.

[0062] After the object 80 held by the end effector 121 is placed in the second tray 92, the controller 11 moves the arm 120 toward the first tray 91 so that the robot mechanism 12A does not interfere with obstacles, and moves the end effector 121 above the second tray 92. Then, the controller 11 causes the end effector 121 to hold one of the objects 80 in the first tray 91 again. Thereafter, the controller 11 moves the arm 120 toward the second tray 92 so that the robot mechanism 12A does not interfere with obstacles, and moves the object 80 to the second tray 92. Then, the controller 11 causes the end effector 121 to release its hold on the object 80. Thereafter, the robot 10A operates in the same manner until all of the objects 80 in the first tray 91 are placed in the second tray 92, and the moving and placing operation is completed.

[0063] When the controller 11 causes the end effector 121 to hold one object 80 in the first tray 91, the controller 11 executes an object recognition program for recognizing the object 80 based on color image data generated by the first camera 71 or the second camera 72. The controller 11 then executes an object holding program for holding the object 80 based on the color image data generated by the first camera 71 or the second camera 72, and causes the end effector 121 to hold one object 80 in the first tray 91 based on the recognition result of the object 80. The object recognition program may be, for example, a machine learning program. The object holding program may also be, for example, a machine learning program. The object recognition program and the object holding program can be considered to be operation programs used by the robot 10 in an actual work environment.

[0064] The controller 11 also executes an object release program for releasing the object 80, causing the end effector 121 to release the object 80. The object release program, like the object recognition program and the object holding program, can be said to be an operation program used by the robot 10 in the actual work environment.

[0065] Furthermore, the controller 11 executes a target motion creation program that creates a target motion for the arm 120 when the arm 120 is moved toward the second tray 92 with the end effector 121 holding the object 80, and when the arm 120 is moved toward the first tray 91 with the end effector 121 not holding the object 80. The target motion for the arm 120 means the posture that the arm 120 should take at each time. The posture of the arm 120 is determined by the amount of rotation of each joint of the arm 120. The controller 11 controls the rotation of each joint of the arm 120 so that the motion of the arm 120 becomes the created target motion. The target motion creation program, like the object recognition program and the object holding program, can be said to be an operation program used by the robot 10 in an actual work environment.

[0066] Here, among the operations of the single-arm robot 10A during operation (also referred to as single-arm robot operations), the operation in which the end effector 121 holds the object 80 in the first tray 91 is referred to as a holding operation. The holding operation can also be referred to as a holding task. Furthermore, among the single-arm robot operations, the operation in which the arm 120 moves toward the second tray 92 while the object 80 is held by the end effector 121 is referred to as a holding and moving operation. The holding and moving operation can also be referred to as a holding and moving task. Furthermore, among the single-arm robot operations, the operation in which the end effector 121 releases its hold on the object 80 and places the object 80 on the second tray 92 is referred to as a releasing operation. The releasing operation can also be referred to as a releasing task. Furthermore, the operation in which the end effector 121 places the object 80 on the second tray 92 and then the arm 120 moves toward the first tray 91 while the end effector 121 is not holding the object 80 is referred to as a non-holding and moving operation. The non-holding and moving operation can also be referred to as a non-holding and moving task. The holding movement operation and non-holding movement operation of the single-arm robot 10A can also be said to be posture change operations of the arm 120. The holding movement and release movement are operations of the end effector 121 (in other words, the work of the end effector 121), and the holding movement operation and non-holding movement operation are operations of the arm 120 (in other words, the work of the arm 120).

[0067] The controller 11 executes the target motion creation program to create a target motion of the arm 120 from a start position (also referred to as a start position of movement) of the movement motion of the arm 120 to an end position (also referred to as an end position of movement) of the movement motion of the arm 120. When causing the arm 120 to execute a holding movement motion, the controller 11 executes the target motion creation program to create a target motion of the arm 120 from the start position of movement in the holding movement motion to an end position of movement in the holding movement motion, such that the robot mechanism 12A does not interfere with an obstacle. Furthermore, when causing the arm 120 to execute a non-holding movement motion, the controller 11 executes the target motion creation program to create a target motion of the arm 120 from the start position of movement in the non-holding movement motion to an end position of movement in the non-holding movement motion, such that the robot mechanism 12A does not interfere with an obstacle.

[0068] The movement start position of the arm 120 in the holding movement operation is the position of the arm 120 when the holding movement operation starts, and the movement end position of the arm 120 in the holding movement operation is the position of the arm 120 when the holding movement operation ends. The movement start position of the arm 120 in the non-holding movement operation is the position of the arm 120 when the non-holding movement operation starts, and the movement end position of the arm 120 in the non-holding movement operation is the position of the arm 120 when the non-holding movement operation ends. In this example, the movement start position of the arm 120 in the holding movement operation is the same as the movement end position of the arm 120 in the non-holding movement operation. Also, the movement end position of the arm 120 in the holding movement operation is the same as the movement start position of the arm 120 in the non-holding movement operation.

[0069] Hereinafter, the movement start position in the holding movement operation of the arm 120 and the movement end position in the non-holding movement operation of the arm 120 may be collectively referred to as the first movement reference position. The first movement reference position is both the movement start position in the holding movement operation of the arm 120 and the movement end position in the non-holding movement operation of the arm 120.

[0070] The movement end position of the holding movement operation of the arm 120 and the movement start position of the non-holding movement operation of the arm 120 are sometimes collectively referred to as the second movement reference position. The second movement reference position is both the movement end position of the holding movement operation of the arm 120 and the movement start position of the non-holding movement operation of the arm 120.

[0071] In the above example, one robot 10A performs the moving and placing task of moving multiple objects 80 in the first tray 91 and placing them in the second tray 92, but multiple robots 10A may perform the moving and placing task of moving multiple objects 80 in the first tray 91 and placing them in the second tray 92. Also, multiple robots 10A may transfer one object 80 from the first tray 91 to the second tray 92. By having multiple robots 10A perform the moving and placing task, it is possible to shorten the work time for the moving and placing task.

[0072] 5 is a schematic diagram showing an example of two robots 10A performing a transfer and placement task. Each of the robots 10A holds an object 80 in a first tray 91 with an end effector 121. Then, each of the robots 10A moves its arm 120 to move the object 80 held by the end effector 121 to a second tray 92 and place the object 80 in the second tray 92.

[0073] For example, the start position of the movement of the arm 120 in the holding movement operation is the same among multiple robots 10A. Also, the end position of the movement of the arm 120 in the holding movement operation is the same among multiple robots 10A. Similarly, the start position of the movement of the arm 120 in the non-holding movement operation is the same among multiple robots 10A. Also, the end position of the movement of the arm 120 in the non-holding movement operation is the same among multiple robots 10A.

[0074] <Example of Operation of Simulation Unit> Figure 6 is a flowchart showing an example of the operation of the processing device 1 when the simulation unit 2a simulates the above-mentioned movement and placement work in a virtual space. The simulation unit 2a may, for example, perform the simulation while changing at least one of the position and number of the robot 10A in the virtual work environment. The simulation unit 2a may also, for example, perform the simulation while changing at least one of the position and number of obstacles in the virtual work environment. Below, a case will be described in which the simulation unit 2a performs a simulation while changing the positions of the robot 10A and obstacles in the virtual work environment. In the following example, the positions of the robot 10A and obstacles differ between multiple types of virtual work environments in which the simulation is performed.

[0075] In step s1, the simulation unit 2a acquires information necessary for the simulation (also referred to as required information). The required information is managed, for example, by the server device 30. The simulation unit 2a acquires the required information from the server device 30 via the interface 6.

[0076] The necessary information includes, for example, environmental measurement data generated by the sensor device 20. The necessary information also includes, for example, work area information indicating a work area (also referred to as a work space) in which movement and placement work can be performed in the actual work environment. The work area can also be said to be the range in the actual work environment in which one robot 10A or multiple robots 10A can perform movement and placement work. For example, if a safety fence is installed in the actual work environment, the work area may be set inside the safety fence. The work area information may be expressed, for example, by coordinate values ​​in a robot coordinate system. The origin of the robot coordinate system is set, for example, at the base portion (in other words, the base end) of the arm 120 provided in the single-arm robot 10A.

[0077] The required information includes, for example, the number of robot mechanisms 12A in the actual work environment (also referred to as the robot mechanism count) and robot mechanism information related to the robot mechanisms 12A in the actual work environment. The required information also includes, for example, the number of obstacles in the actual work environment (also referred to as the obstacle count) and obstacle information about the obstacles in the actual work environment.

[0078] The required information includes, for example, first tray information about the first tray 91 in the actual work environment, second tray information about the second tray 92 in the actual work environment, and object information about the object 80 in the actual work environment. The required information also includes, for example, first camera information about the first camera 71 in the actual work environment, and second camera information about the second camera 72 in the actual work environment.

[0079] The necessary information includes, for example, first movement reference position information indicating a first movement reference position in the movement and placement work of the simulation object, and second movement reference position information indicating a second movement reference position in the movement and placement work of the simulation object. The first movement reference position information and the second movement reference position information are expressed, for example, by coordinate values ​​in the robot coordinate system.

[0080] The necessary information includes, for example, an operation program used by the robot 10A when it performs a movement and placement operation in the actual work environment, including, for example, an object recognition program, an object holding program, an object release program, and a target operation creation program.

[0081] If the content of the work to be simulated changes, the content of the required information will also change. For example, if the content of the work to be simulated changes, the operation program etc. included in the required information will change. The simulation unit 2a obtains the required information corresponding to the content of the work to be simulated from the server device 30.

[0082] Furthermore, if the type of robot 10 or peripheral device that performs the task to be simulated changes, the content of the required information will also change. For example, if the type of robot 10 that performs the task to be simulated changes, the robot mechanism information, operation program, etc. included in the required information will change. The simulation unit 2a obtains the required information according to the type of robot 10 that performs the task to be simulated from the server device 30.

[0083] Furthermore, if the number of robots 10 used in the work to be simulated changes, the content of the required information will also change. For example, if the number of robots 10 used in the work changes, the operation programs and the like included in the required information will change. The simulation unit 2a obtains the required information according to the number of robots 10 used in the work to be simulated from the server device 30.

[0084] At least a part of the necessary information may be input by the user to the processing device 1 through the input unit 5 (in other words, through a user interface). For example, the user may input first movement reference position information and second movement reference position information to the processing device 1 through the input unit 5. In the processing device 1, the control unit 2 stores at least a part of the necessary information input by the user in the memory unit 3. The simulation unit 2a reads out and acquires at least a part of the necessary information from the memory unit 3.

[0085] After acquiring the necessary information in step s1, the simulation unit 2a acquires the number of attempts of the simulation in step s2. In this example, the number of attempts is represented by N (N is an integer equal to or greater than 2). For example, the user may input the number of attempts N to the processing device 1 through the input unit 5. In this case, in the processing device 1, the control unit 2 stores the number of attempts N input by the user in the memory unit 3. The simulation unit 2a reads and acquires the number of attempts N from the memory unit 3. Furthermore, if the server device 30 manages the number of attempts N, the simulation unit 2a may acquire the number of attempts N from the server device 30.

[0086] After step s2, in step s3, the simulation unit 2a sets a virtual work environment in a virtual space on the computer based on the necessary information acquired in step s1.

[0087] In step s3, the simulation unit 2a sets up a partial virtual work environment in the virtual space that reproduces the actual work environment excluding the robot 10A, obstacles, object 80, first tray 91, second tray 92, first camera 71, and second camera 72, based on the environmental measurement data (e.g., environmental point cloud data) included in the necessary information.

[0088] Next, the simulation unit 2a places a virtual first tray, which is a reproduction of the first tray 91 in the actual work environment, in the partial virtual work environment based on the first tray information included in the required information. Furthermore, the simulation unit 2a places a virtual second tray, which is a reproduction of the second tray 92 in the actual work environment, in the partial virtual work environment based on the second tray information included in the required information. Furthermore, the simulation unit 2a places a virtual object, which is a reproduction of the object 80 in the actual work environment, in the virtual first tray based on the object information included in the required information. At this time, multiple virtual objects are randomly stacked in the virtual first tray. The virtual first tray, virtual second tray, and virtual object are, for example, 3D models of the first tray 91, second tray 92, and object 80, respectively.

[0089] Furthermore, the simulation unit 2a sets a virtual work area of ​​the same size as the work area in the partial virtual work environment based on the work area information included in the required information. Then, the simulation unit 2a places virtual robot mechanisms that replicate the robot mechanism 12A in the virtual work area based on the robot mechanism information of the robot mechanism 12A included in the required information. At this time, the simulation unit 2a places the same number of virtual robot mechanisms as the number of robot mechanisms included in the required information in the virtual work area. Furthermore, the simulation unit 2a randomly determines the placement positions of the virtual robot mechanisms. In other words, the simulation unit 2a randomly places the same number of virtual robot mechanisms as the number of robot mechanisms. The virtual robot mechanisms are, for example, 3D models of the robot mechanisms 12.

[0090] Furthermore, the simulation unit 2a places virtual obstacles that are reproductions of obstacles in the actual work environment within the virtual work area based on the obstacle information included in the necessary information. At this time, the simulation unit 2a places the same number of virtual obstacles within the virtual work area as the number of obstacles included in the necessary information. Furthermore, the simulation unit 2a randomly determines the placement positions of the virtual obstacles. In other words, the simulation unit 2a randomly places the same number of virtual obstacles as the number of obstacles. The virtual obstacles are, for example, 3D models of obstacles.

[0091] Furthermore, the simulation unit 2a places a virtual first camera that reproduces the first camera 71 in the actual work environment in the partial virtual work environment based on the first camera information included in the necessary information. Furthermore, the simulation unit 2a places a virtual second camera that reproduces the second camera 72 in the actual work environment in the partial virtual work environment based on the second camera information included in the necessary information. The virtual first camera and virtual second camera are, for example, 3D models of the first camera 71 and the second camera 72, respectively.

[0092] The setting of the virtual work environment in the virtual space on the computer is completed by placing the first virtual camera, the second virtual camera, the first virtual tray, the second virtual tray, the virtual object, the virtual robot mechanism, and the virtual obstacle in the partial virtual work environment.

[0093] In the partial virtual work environment, for example, the lighting environment in the real work environment may be reproduced. In this case, in the virtual lighting environment, for example, virtual lighting fixtures and virtual windows may be reproduced based on information on the light from each lighting fixture and each window in the real work environment, thereby reproducing the lighting environment in the real work environment. Note that the virtual lighting fixtures and virtual windows are, for example, 3D models of the lighting fixtures and windows, respectively.

[0094] Although the above description is of an example in which the simulation unit 2a sets a partial virtual work environment in the virtual space, the simulation unit 2a may set a virtual work environment in the virtual space that reproduces a real work environment including at least one of the robot 10A, obstacles, object 80, first tray 91, second tray 92, first camera 71, and second camera 72. In this case, the simulation unit 2a may perform object recognition on at least one of the robot 10A, obstacles, object 80, first tray 91, second tray 92, first camera 71, and second camera 72 from image data or point cloud data constituting the virtual work environment, and replace the recognized object with virtual data such as a 3D model based on the recognition results. Note that object recognition may be performed using, for example, AI, which performs machine learning, or a technique such as template matching. AI is an abbreviation for artificial intelligence.

[0095] After step s3, the simulation unit 2a executes step s4, in which the simulation unit 2a performs a simulation of the movement and placement work in the virtual work environment set in step s3.

[0096] When the simulation starts, the simulation unit 2a first causes a virtual end effector of the virtual robot mechanism to hold one virtual object in the virtual first tray based on virtual color image data generated by the virtual first camera or the virtual second camera. At this time, the simulation unit 2a executes an object recognition program included in the necessary information to recognize the virtual object in the virtual first tray based on the virtual color image data. Then, the simulation unit 2a executes an object holding program included in the necessary information to cause the virtual end effector to hold one virtual object in the virtual first tray based on the virtual object recognition result and the virtual color image data.

[0097] Next, the simulation unit 2a moves the virtual arm of the virtual robot mechanism toward the virtual second tray so as not to interfere with the virtual obstacle, thereby moving the virtual object held by the virtual end effector to the virtual second tray. At this time, the simulation unit 2a sets a virtual first movement reference position in the virtual work environment to a position identical to the first movement reference position indicated by the first movement reference position information included in the necessary information. The simulation unit 2a also sets a virtual second movement reference position in the virtual work environment to a position identical to the second movement reference position indicated by the second movement reference position information included in the necessary information. The simulation unit 2a then executes a desired motion generation program included in the necessary information to generate a desired motion for the virtual arm from the virtual first movement reference position to the virtual second movement reference position. The simulation unit 2a then controls the rotation of each virtual joint of the virtual arm so that the motion of the virtual arm becomes the generated desired motion, thereby moving the virtual arm toward the virtual second tray.

[0098] When the virtual arm moves to the virtual second movement reference position, the simulation unit 2a executes the object release program included in the necessary information to cause the virtual end effector to release its hold on the virtual object and place the virtual object at a predetermined location in the virtual second tray.

[0099] After the virtual object held by the virtual end effector is placed in the virtual second tray, the simulation unit 2a moves the virtual arm toward the virtual first tray and moves the virtual end effector above the virtual second tray so that the virtual robot mechanism does not interfere with the virtual obstacle. At this time, the simulation unit 2a executes the desired motion creation program included in the necessary information to create a desired motion for the virtual arm from the virtual second movement reference position to the virtual first movement reference position. Then, the simulation unit 2a controls the rotation of each virtual joint of the virtual arm to move the virtual arm toward the virtual first tray so that the motion of the virtual arm becomes the created target motion.

[0100] When the virtual arm moves to the virtual first movement reference position, the simulation unit 2a causes the virtual end effector to hold one virtual object in the virtual first tray again in the same manner as described above. Thereafter, the simulation unit 2a operates in the same manner. If there are multiple virtual robot mechanisms in the virtual work environment, the simulation unit 2a performs the same process for each of the multiple virtual robot mechanisms.

[0101] When all the virtual objects in the virtual first tray are placed in the virtual second tray, the simulation unit 2a ends the simulation of the moving and placing work.

[0102] When the simulation of the moving and placing task is completed, in step s5, the simulation unit 2a calculates the task time for the moving and placing task in the virtual work environment set in step s3. That is, the simulation unit 2a calculates the time required to virtually perform the moving and placing task in the virtual work environment set in step s3. In step s3, the simulation unit 2a determines the time from the start to the end of the simulation as the task time for the moving and placing task in the virtual work environment set in step s3. The task time may be, for example, a cycle time. Note that the task time may also be, for example, a takt time. Note that the cycle time is, for example, the time required to complete the moving and placing task of moving all of the objects 80, and the takt time is, for example, the time required for the robot 10A to pick up one object 80 from the first tray 91, move the held object 80 from the first tray 91 to the second tray 92, and place the object 80 in the second tray 92.

[0103] Once the task time is calculated, in step s6, the simulation unit 2a determines whether the simulation of the movement and placement task has been performed N times. That is, the simulation unit 2a determines whether the simulation has been performed N times. If the determination in step s6 is YES, step s7 is executed. On the other hand, if the determination in step s6 is NO, step s3 is executed again to reset the virtual work environment. In step s3 again, the simulation unit 2a changes the virtual work environment set in the previous step s3 and resets the virtual work environment. For example, the simulation unit 2a resets the virtual work environment by changing only the placement positions of at least one of the virtual robot mechanisms and virtual obstacles in the virtual work environment set in the previous step s3. If multiple virtual robot mechanisms exist, the placement positions of at least one of the multiple virtual robot mechanisms may be changed. Furthermore, if multiple virtual obstacles exist, the placement positions of at least one of the multiple virtual obstacles may be changed. The reset virtual work environment will be different from the previous virtual work environment. The simulation unit 2a may reset the virtual work environment by, for example, randomly arranging the virtual robot mechanism and virtual obstacles within the virtual work area. In this case, the new virtual work environment is the same as the virtual work environment set in step s3, but with the positions of both the virtual robot mechanism and virtual obstacles changed.

[0104] Once the virtual work environment is reset in step s3, the simulation unit 2a performs a simulation of the movement and placement work in the reset virtual work environment in step s4. Thereafter, the simulation unit 2a operates in the same manner. Until the determination in step s6 is YES, the simulation unit 2a repeatedly performs the simulation of the movement and placement work while changing the positions of the virtual robot mechanism and virtual obstacles.

[0105] When the simulation unit 2a is repeatedly executing a simulation, the display unit 4 of the processing device 1 may display the state of the simulation in the simulation unit 2a.

[0106] FIG. 7 is a schematic diagram showing an example of how the display unit 4 displays the state of a simulation performed by the simulation unit 2a. In the example of FIG. 7, the display unit 4 displays a simulation video 400 showing N simulations performed by the simulation unit 2a. The simulation video 400 is a CG video. CG is an abbreviation for Computer Graphics. The simulation video 400 can also be called a simulation CG video 400. The control unit 2 generates the simulation video 400 and displays it on the display unit 4. The simulation video 400 may be stored and saved in the storage unit 3. The processing device 1 may also transmit the generated simulation video 400 to the server device 30, and the server device 30 may manage the simulation video 400.

[0107] In the simulation video 400, an overall video 410 showing the overall state of the virtual work environment during the simulation is shown on the left side. In addition, in the simulation video 400, a virtual captured video 420 obtained by virtually capturing images with the virtual second camera 72v is shown on the right side. The overall video 410 and the virtual captured video 420 are CG videos.

[0108] In the example of Figure 7, the virtual second camera 72v is photographing the virtual first tray 91v from above, so the virtual photographed video 420 shows the virtual first tray 91v and the virtual object 80v inside the virtual first tray 91v.

[0109] The overall video 410 shows a virtual object 80v, a virtual first tray 91v, a virtual second tray 92v, a virtual first camera 71v, a virtual second camera 72v, and a virtual robot mechanism 12Av. The overall video 410 also shows a virtual work table 90v that reproduces a work table 90 in a real work environment. The overall video 410 shows the virtual robot mechanism 12Av holding the virtual object 80v in the virtual first tray 91v with a virtual end effector 121v and moving the virtual arm 120v to move the held virtual object 80v to the virtual second tray 92v. The overall video 410 shows the virtual robot mechanism 12Av virtually performing a movement and placement task.

[0110] By displaying the simulation video 400 during the simulation by the simulation unit 2a, the user can visually check the state of the simulation currently being performed by the simulation unit 2a.

[0111] In step s7, which is executed after the simulation has been executed N times, the display unit 4 of the processing device 1 displays the results of the simulation executed by the simulation unit 2 a, thereby allowing the user to visually confirm the results of the simulation.

[0112] FIG. 8 is a schematic diagram showing an example of the display on the display unit 4 in step s7. In the example of FIG. 8, the display unit 4 displays the minimum value 500 of the N work times calculated by the process of FIG. 6 as a result of the simulation. The minimum value 500 can be said to be the smallest value when comparing the work times in the N types of virtual work environments in which the simulation was performed. The simulation unit 2a selects the virtual work environment with the smallest work time based on the work times for each of the multiple types of virtual work environments obtained by the simulation. The display unit 4 displays the work time in the virtual work environment selected by the simulation unit 2a, i.e., the minimum value 500.

[0113] 8, the display unit 4 displays information 510 (also referred to as first virtual environment information 510) about the virtual work environment selected by the simulation unit 2a, i.e., the virtual work environment in which the work time is the minimum value of 500 (also referred to as the minimum work time virtual work environment). For example, if the work time obtained in the second simulation run is the minimum value of 500 out of N work times obtained by running N simulations, the virtual work environment in which the second simulation was run will be the minimum work time virtual work environment.

[0114] The first virtual environment information 510 includes position information 511 indicating the placement position of a virtual robot mechanism in the minimum work time virtual work environment, and position information 512 indicating the placement position of a virtual obstacle in the minimum work time virtual work environment. If multiple virtual robot mechanisms exist in the minimum work time virtual work environment, the position information 511 of each of the multiple virtual robot mechanisms is included in the first virtual environment information 510. Furthermore, if multiple virtual obstacles exist in the minimum work time virtual work environment, the position information 512 of each of the multiple virtual obstacles is included in the first virtual environment information 510. The position information 511 and the position information 512 may be expressed, for example, as coordinate values ​​in a robot coordinate system.

[0115] In this way, the display unit 4 displays the minimum value 500 of the work time, so that the user can visually confirm the minimum value 500 of the work time.

[0116] In step s7, the display unit 4 may display, as the minimum work time simulation video, only a portion of the simulation video 400 stored in the storage unit 3 that shows the state of the simulation performed in the minimum work time virtual work environment. The control unit 2 extracts the minimum work time simulation video from the simulation video 400 that shows the state of N simulations stored in the storage unit 3, and displays it on the display unit 4. The minimum work time simulation video can be said to be the result of the simulation.

[0117] Furthermore, in step s7, the display unit 4 may display a list of the N work times obtained by the process of Fig. 6. Fig. 9 is a schematic diagram showing an example of such a state. In the example of Fig. 9, the display unit 4 displays, for each of the N simulations, the order 540 in which the simulation was performed, the work time 550 obtained by performing the simulation, and information 560 about the virtual work environment in which the simulation was performed (also referred to as second virtual environment information 560), in association with one another.

[0118] The second virtual environment information 560 includes position information 561 indicating the placement position of the virtual robot mechanism in the virtual work environment, and position information 562 indicating the placement position of a virtual obstacle in the virtual work environment. If multiple virtual robot mechanisms exist in the virtual work environment, the position information 561 of each of the multiple virtual robot mechanisms is included in the second virtual environment information 560. If multiple virtual obstacles exist in the minimum work time virtual work environment, the position information 562 of each of the multiple virtual obstacles is included in the second virtual environment information 560. The position information 561 and the position information 562 may be expressed, for example, as coordinate values ​​in the robot coordinate system.

[0119] In this way, the display unit 4 displays a list of N work times calculated by the process of Figure 6 as a result of the simulation, allowing the user to visually confirm the N work times.

[0120] 9, the display unit 4 may display an individual simulation video showing the state of each of the N simulations. An individual simulation video is a video showing the state of one simulation. The individual simulation video can be acquired from the simulation video 400 stored in the storage unit 3.

[0121] 6 is executed, the interface 6 of the processing device 1 may output operation information of the virtual robot in the minimum work time virtual work environment and information about the minimum work time virtual work environment to the outside of the processing device 1. Hereinafter, the information about the minimum work time virtual work environment output by the interface 6 will be referred to as third virtual environment information. Also, the operation information of the virtual robot in the minimum work time virtual work environment output by the interface 6 will be referred to as third robot operation information.

[0122] The third robot operation information includes, for example, first movement reference position information and second movement reference position information. The third robot operation information also includes target movement information indicating a target movement of the virtual robot mechanism created by simulation in the minimum work time virtual work environment. The target movement information includes first target movement information indicating a target movement of the virtual arm from a virtual first movement reference position corresponding to the first movement reference position to a virtual second movement reference position corresponding to the second movement reference position, and second target movement information indicating a target movement of the virtual arm from the virtual second movement reference position to the virtual first movement reference position. Each of the first target movement information and the second target movement information may be expressed as a rotation amount at each time for each virtual joint of the virtual arm of the virtual robot mechanism.

[0123] The third virtual environment information includes, for example, information indicating the placement position of the virtual robot mechanism in the minimum work time virtual work environment (also referred to as third virtual robot position information). The third virtual environment information also includes, for example, information regarding a virtual obstacle in the minimum work time virtual work environment (also referred to as third virtual obstacle information). The third virtual obstacle information includes, for example, virtual obstacle range information indicating the range in which the virtual obstacle exists in the minimum work time virtual work environment. As the virtual obstacle range information, for example, position information indicating the positions of each vertex of a polygonal virtual obstacle may be used.

[0124] The interface 6 may transmit the third robot operation information and the third virtual environment information to at least one of the server device 30, the sensor device 20, and the robot 10A that performs the movement and placement work simulated by the simulation unit 2a, via the network 40. Hereinafter, the robot 10A that performs the movement and placement work simulated by the simulation unit 2a will be referred to as the robot 10A to be simulated.

[0125] The server device 30 may store and manage the received third robot operation information and third virtual environment information. Furthermore, the server device 30 may display the third robot operation information and the third virtual environment information. This allows the user to confirm the movement of the arm 120 that may minimize the operation time when the robot 10A performs the movement and placement task. Furthermore, the user can confirm the range of obstacles that may minimize the operation time when the robot 10A performs the movement and placement task. Furthermore, the server device 30 may transmit the third robot operation information and the third virtual environment information to the robot 10A that is the target of the simulation.

[0126] The controller 11 of the robot 10A may display the received third robot operation information and third virtual environment information. Furthermore, the sensor device 20 may display the received third robot operation information and third virtual environment information on the display unit 24. Furthermore, the processing device 1 may display the third robot operation information and third virtual environment information on the display unit 4.

[0127] Furthermore, the robot 10A may perform a movement and placement task based on the received third robot operation information. This allows verification of the results of the simulation by the simulation unit 2a. That is, for example, the robot 10A is controlled based on an operation program supplemented with the third robot operation information.

[0128] When the robot 10A performs a movement and placement task based on the third robot operation information, the robot mechanism 12A of the robot 10A is placed at the same position as the position indicated by the third virtual robot position information included in the third virtual environment information. The controller 11 of the robot 10A then causes the end effector 121 to hold the target object 80 on the first tray 91, and then moves the arm 120 from the first movement reference position indicated by the first movement reference position information included in the third robot operation information to the second movement reference position indicated by the second movement reference position information included in the third robot operation information. At this time, the controller 11 controls the rotation of each joint of the arm 120 so that the movement of the arm 120 becomes the target movement indicated by the first target movement information included in the third robot operation information. Furthermore, after placing the object 80 held by the end effector 121 in the second tray 92, the controller 11 moves the arm 120 from the second movement reference position indicated by the second movement reference position information included in the third robot operation information to the first movement reference position indicated by the first movement reference position information included in the third robot operation information. At this time, the controller 11 controls the rotation of each joint of the arm 120 so that the movement of the arm 120 becomes the target movement indicated by the second target movement information included in the third robot operation information.

[0129] The user can verify the results of the simulation in the simulation unit 2a by checking whether the movement of the robot mechanism 12A of the robot 10A performing the movement and placement task is appropriate based on the third robot operation information.

[0130] If multiple virtual robot mechanisms exist in the virtual work environment, third robot operation information is generated individually for each of the multiple virtual robot mechanisms. In this case, each of the multiple robots 10A corresponding to the multiple virtual robot mechanisms may receive the corresponding third robot operation information from the server device 30 or the processing device 1, and perform movement and placement work based on the received third robot operation information.

[0131] While the robot 10A is performing the movement and placement task based on the third robot operation information derived from the simulation results, the first camera 71 may capture video of the actual work environment. The first camera 71 may then transmit video data (also referred to as actual work environment video data) representing the captured video of the actual work environment while the robot 10A is performing the movement and placement task based on the third robot operation information to the controller 11. The captured video represented by the actual work environment video data captures the movement of the entire robot mechanism 12A and the state of the first tray 91 and the second tray 92 while the robot 10A is performing the movement and placement task based on the third robot operation information.

[0132] The controller 11 may transmit the actual work environment video data received from the first camera 71 to the processing device 1. In this case, the control unit 2 of the processing device 1 may generate a simulation CG video showing, in CG, the state of the simulation in the minimum work time virtual work environment, and cause the display unit 4 to simultaneously display the generated simulation CG video and the captured video represented by the actual work environment video data. The simulation CG video shows the movement of the entire virtual robot mechanism in the minimum work time virtual work environment during the simulation, as well as the state of the virtual first tray and the virtual second tray. The simulation CG video may be a virtual captured video obtained by virtually capturing video with the virtual first camera during the simulation in the minimum work time virtual work environment. The display unit 4 may simultaneously display the simulation CG video and the captured video represented by the actual work environment video data so that the start timing of the work of the virtual robot mechanism in the simulation CG video is the same as the start timing of the work of the robot mechanism 12A in the captured video represented by the actual work environment video data.

[0133] The processing device 1 may transmit simulation CG video data representing the generated simulation CG video to the controller 11 of the robot 10A. The controller 11 may then simultaneously display the simulation CG video represented by the received simulation CG video data and the captured video represented by the actual working environment video data received from the first camera 71. The sensor device 20 may also receive the simulation CG video data from the processing device 1 and the actual working environment video data from the controller 11, and simultaneously display the simulation CG video represented by the simulation CG video data and the captured video represented by the actual working environment video data on the display unit 24. The server device 30 may also receive the simulation CG video data from the processing device 1 and the actual working environment video data from the controller 11, and simultaneously display the simulation CG video represented by the simulation CG video data and the captured video represented by the actual working environment video data on the display unit 24.

[0134] The display unit 4 may simultaneously display the minimum work time simulation video obtained from the simulation video 400 displayed during the simulation and the captured video indicated by the actual work environment video data. Furthermore, at least one of the controller 11, the sensor device 20, and the server device 30 may simultaneously display the minimum work time simulation video and the captured video indicated by the actual work environment video data. In this case, the robot 10 may synchronize the simulation video and the actual work environment video data based on the first movement reference position.

[0135] In addition, if there is a difference in the movement or environmental settings of the robot mechanism 12 between the simulation CG video and the captured video shown by the actual work environment video data, the control unit 2 may detect the difference and highlight and display the difference on the display unit 4.

[0136] As described above, in this example, the simulation unit 2a of the processing device 1 simulates the work of the robot 10 in each of a plurality of types of virtual work environments and calculates the work time for that work in the plurality of types of virtual work environments. This makes it possible to obtain the work time in a plurality of types of virtual work environments, thereby improving the convenience of the processing device 1.

[0137] Furthermore, as in the above example, when the simulation unit 2a performs a simulation based on an operation program used by the robot 10 in the actual work environment, the accuracy of the simulation can be improved.

[0138] In the above example, the simulation unit 2a selected the virtual work environment with the shortest working time based on the working time for each of the multiple types of virtual work environments obtained by simulation, but it may also select at least one of the multiple types of virtual work environments. In this way, the processing device 1 selects at least one virtual work environment based on the working time for each of the multiple types of virtual work environments, thereby improving the convenience of the processing device 1.

[0139] Furthermore, the results of the simulation may be verified by having the robot 10A perform a movement and placement task in a real work environment set based on the third virtual environment information. In this case, in the real work environment, the robot 10A is placed at a position indicated by the third virtual robot position information included in the third virtual environment information. Furthermore, in the real work environment, an obstacle is placed within the same range as the virtual obstacle indicated by the third virtual obstacle information included in the third virtual environment information. In this way, in the real work environment set based on the third virtual environment information, the controller 11 causes the end effector 121 to hold the object 80, then executes the desired motion creation program to create a desired motion for the arm 120 from the first movement reference position to the second movement reference position, and controls the rotation of each joint of the arm 120 so that the motion of the arm 120 becomes the created target motion. Furthermore, after the controller 11 causes the end effector 121 to place the object 80 in the second tray 92, it executes the target motion creation program to create a target motion for the arm 120 from the second movement reference position to the first movement reference position, and controls the rotation of each joint of the arm 120 so that the motion of the arm 120 becomes the created target motion. When the robot 10A performs movement and placement work in the actual work environment set based on the third virtual environment information, the processing device 1 may output only the third virtual environment information out of the third robot operation information and the third virtual environment information related to the virtual work environment with the minimum work time to the outside of the processing device 1.

[0140] The processing device 1 may output, for each of the N types of virtual work environments in which a simulation was performed, the work time in that virtual work environment, operation information of the virtual robot in that virtual work environment (also referred to as fourth robot operation information), and information about that virtual work environment (fourth virtual environment information) to the outside of the processing device 1. In this case, the processing device 1 outputs N work times, N pieces of fourth robot operation information, and N pieces of fourth virtual environment information to the outside of the processing device 1. The interface 6 of the processing device 1 may transmit the N work times, N pieces of fourth robot operation information, and N pieces of fourth virtual environment information to at least one of the server device 30, the sensor device 20, and the robot 10A that is the target of the simulation. The content of the fourth robot operation information is the same as the content of the third robot operation information described above, and the content of the fourth virtual environment information is the same as the third virtual environment information described above. The fourth robot operation information and the fourth virtual environment information related to the virtual work environment with the minimum work time are identical to the third robot operation information and the third virtual environment information, respectively. Hereinafter, the N task times, the N fourth robot operation information, and the N fourth virtual environment information will be collectively referred to as list information.

[0141] The server device 30 may store and manage the received list information. The server device 30 may also transmit the list information to the robot 10A that is the target of the simulation.

[0142] The server device 30 may also display list information. This allows the user to visually confirm, for each virtual work environment in which a simulation was performed, the work time in that virtual work environment, operation information of the virtual robot in that virtual work environment, and information about that virtual work environment. The user can also identify the minimum work time based on the displayed list information. From the list information, the user can visually confirm the virtual work environment in which the work time is minimum, i.e., the operation information of the virtual robot in the minimum work time virtual work circle, and information about the minimum work time virtual work circle. This allows the user to confirm the movement of the arm 120 that may minimize the work time when the robot 10A performs a movement and placement task. The user can also confirm the range of obstacles that may minimize the work time when the robot 10A performs a movement and placement task.

[0143] The controller 11 of the robot 10A may display the received list information. The sensor device 20 may display the received list information on the display unit 24. The processing device 1 may display the list information on the display unit 4.

[0144] The server device 30 may also obtain the minimum work time from the received list information. Then, the server device 30 may obtain fourth robot operation information and fourth virtual environment information related to the virtual work environment with the minimum work time from the list information.

[0145] The server device 30 may display the minimum work time, the fourth robot operation information and the fourth virtual environment information related to the virtual work environment with the minimum work time, which are obtained from the list information. The server device 30 may also transmit the minimum work time, the fourth robot operation information and the fourth virtual environment information related to the virtual work environment with the minimum work time, which are obtained from the list information, to the robot 10A that is the target of the simulation.

[0146] The controller 11 of the robot 10A may also obtain the minimum value of the work time from the received list information. Then, the controller 11 may obtain fourth robot operation information and fourth virtual environment information related to the virtual work environment with the minimum work time from the list information.

[0147] The controller 11 may display the minimum work time, and fourth robot operation information and fourth virtual environment information related to the virtual work environment with the minimum work time, which are acquired from the list information or the server device. The controller 11 may also perform a movement and placement task based on the fourth robot operation information related to the virtual work environment with the minimum work time, which is acquired from the list information or the server device. The robot 10A may also perform a movement and placement task in an actual work environment set based on the fourth virtual environment information related to the virtual work environment with the minimum work time.

[0148] The simulation unit 2a can also execute a simulation while changing both the position and the number of the robot 10A in the virtual work environment. The simulation unit 2a can also execute a simulation while changing both the positions and the number of obstacles in the virtual work environment.

[0149] For example, in step s3 described above, the simulation unit 2a randomly determines the number of virtual robot mechanisms to be placed in the virtual work area. Then, the simulation unit 2a places the randomly determined number of virtual robot mechanisms in the virtual work area. The simulation unit 2a also randomly determines the number of virtual obstacles to be placed in the virtual work area. Then, the simulation unit 2a places the randomly determined number of virtual obstacles in the virtual work area. By the simulation unit 2a operating in the same manner in each of the repeatedly executed steps s3, the simulation unit 2a performs a simulation while changing the position and number of the robot 10A in the virtual work environment and the positions and number of obstacles in the virtual work environment.

[0150] The simulation unit 2a may perform the simulation while changing only the number and positions of the robots 10A in the virtual work environment. For example, the simulation unit 2a may increase the number of robots 10A. The simulation unit 2a may also perform the simulation while changing only the number and positions of the obstacles in the virtual work environment. For example, the simulation unit 2a may decrease the number of obstacles. In this case, the positions of the robot mechanisms 12A and the obstacles in the real work environment are determined in advance, and the required information includes position information indicating the positions of the robot mechanisms 12A and the obstacles in the real work environment. In step s3, the simulation unit 2a may randomly determine the number of virtual robot mechanisms and virtual obstacles. Then, the simulation unit 2a places a randomly determined number of virtual robot mechanisms and virtual obstacles in the virtual work area based on the position information of the robot mechanisms 12A and the obstacles included in the required information.

[0151] When the simulation unit 2a performs a simulation while changing the number of robot 10A and obstacles in the virtual work environment, the first virtual environment information 510 and the third virtual environment information may include the number of virtual robot mechanisms and obstacles in the minimum work time virtual work environment. Similarly, the second virtual environment information 560 and the fourth virtual environment information may include the number of virtual robot mechanisms and obstacles in the virtual work environment.

[0152] The simulation unit 2a may also simulate only a part of the relocation and placement work. In this case, the simulation unit 2a calculates the work time of the part of the relocation and placement work that has been simulated in step s5.

[0153] For example, the simulation unit 2a may simulate only the first arm operation (i.e., the holding movement operation) of the movement and placement operation in which the arm 120 moves from the first movement reference position to the second movement reference position. Also, the simulation unit 2a may simulate only the second arm operation (i.e., the non-holding movement operation) of the movement and placement operation in which the arm 120 moves from the second movement reference position to the first movement reference position. Also, the simulation unit 2a may simulate only the first arm operation and the second arm operation of the movement and placement operation.

[0154] The control unit 2 of the processing device 1 may estimate the lighting environment in the actual work environment based on the environmental measurement data obtained by the sensor device 20. For example, the control unit 2 may estimate the position of each lighting device in the actual work environment, the brightness of the light from each lighting device, and the color of the light from each lighting device based on the environmental measurement data. The control unit 2 may also estimate the position of each window in the actual work environment, the brightness of the light entering through each window, and the color of the light entering through each window based on the environmental measurement data. Note that at least one of the server device 30, the sensor device 20, and the controller 11 may estimate the lighting environment in the actual work environment based on the environmental measurement data.

[0155] The sensor device 20 may also continuously measure the actual work environment during a specific time period of the day and generate environmental measurement data at each time during the specific time period. The specific time period may be, for example, a time period during which the robot 10 performs work. The control unit 2 of the processing device 1 may, similarly to the above, set up a virtual work environment in a virtual space that recreates the actual work environment at each time during the specific time period based on the environmental measurement data at each time during the specific time period generated by the sensor device 20. The control unit 2 may then generate training image data for training a machine learning program constituting, for example, an object recognition program or an object retention program, based on the virtual work environment at each time during the specific time period. For example, the control unit 2 may use virtual image data representing virtual images captured by a virtual first camera in the virtual work environment at each time period as training image data. The virtual images are CG images. This allows for obtaining multiple training image data corresponding to changes in the lighting environment during a specific time period in the actual work environment.

[0156] The simulation of the work of the robot 10 in the virtual space may be performed by the server device 30, the sensor device 20, or the robot controller 11. The generation of the learning image data may be performed by the server device 30, the sensor device 20, or the robot controller 11.

[0157] As described above, the processing device and the system including the same have been described in detail, but the above description is merely illustrative in all respects and does not limit the present disclosure. Furthermore, the various examples described above can be combined and applied as long as they are not mutually inconsistent. It is understood that countless examples not illustrated can be envisioned without departing from the scope of this disclosure.

[0158] This disclosure includes the following:

[0159] In one embodiment, (1) the processing device includes a simulation unit that simulates the work of a robot in a virtual space, and the simulation unit selects at least one virtual work environment from among the multiple types of virtual work environments based on the work time for each of the multiple types of virtual work environments obtained by the simulation.

[0160] (2) In the processing device of (1) above, at least one of the placement positions and the number of the robots differs among the plurality of types of virtual work environments.

[0161] (3) In the processing device of (1) or (2) above, at least one of the positions and the number of obstacles differs among the plurality of types of virtual work environments.

[0162] (4) In the processing device according to any one of (1) to (3) above, the simulation unit performs the simulation based on an operation program used by the robot in an actual work environment.

[0163] (5) The processing device according to any one of (1) to (4) above, further comprising a display unit that displays the results of the simulation.

[0164] (6) In the processing device of (5) above, the results displayed by the display unit include the operation time.

[0165] (7) In the processing device according to any one of (1) to (6) above, the simulation unit selects the first virtual work environment in which the work time is the shortest.

[0166] (8) In the processing device of (6) above, the results displayed by the display unit include a list of the work times in the multiple types of virtual work environments.

[0167] (9) The processing device of (7) above, further comprising an interface for outputting information relating to the first virtual work environment, among the plurality of types of virtual work environments, to the outside of the processing device.

[0168] (10) In the processing device of (9) above, the interface outputs operation information of the robot in the first virtual work environment to the outside of the processing device.

[0169] (11) A processing device according to any one of (1) to (10) above, comprising an interface for outputting the work time in the plurality of types of virtual work environments and information relating to the plurality of types of virtual work environments to the outside of the processing device.

[0170] (12) In the processing device of (11) above, the interface outputs operation information of the robot in the plurality of types of virtual work environments to the outside of the processing device.

[0171] (13) The program is for causing a computer device to function as any one of the processing devices (1) to (12) above.

[0172] (14) A display device displays the results of the simulation executed by any one of the processing devices (1) to (12) above.

[0173] (15) The processing system includes any one of the processing devices (1) to (12) above and a robot controller capable of communicating with the processing device.

[0174] REFERENCE SIGNS LIST 1 Processing device 2a Simulation unit 3a Program 4 Display unit (display device) 6 Interface 11 Robot controller 20 Sensor device 30 Server device 50 Processing system

Claims

1. It is equipped with a simulation unit that simulates robot operations in a virtual space, The simulation unit is a processing unit that selects at least one virtual work environment from among the multiple types of virtual work environments based on the work time of each of the multiple types of virtual work environments obtained by the simulation.

2. The apparatus according to claim 1, A processing device wherein at least one of the placement position and number of the robots differs among the multiple types of virtual work environments.

3. The apparatus according to claim 1, A processing device wherein at least one of the location and number of obstacles differs among the multiple types of virtual work environments.

4. The apparatus according to claim 1, The simulation unit is a processing unit that performs the simulation based on the motion program used by the robot in the actual work environment.

5. The apparatus according to claim 1, A processing apparatus comprising a display unit for displaying the results of the aforementioned simulation.

6. The processing apparatus according to claim 5, A processing apparatus wherein the results displayed by the display unit include the work time.

7. The apparatus according to claim 1, The simulation unit is a processing unit that selects a first virtual work environment that minimizes the work time.

8. The apparatus according to claim 6, A processing device wherein the results displayed by the display unit include a list of the work times in the multiple types of virtual work environments.

9. The apparatus according to claim 7, A processing device having an interface for outputting information regarding the first virtual work environment from among the multiple types of virtual work environments to the outside of the processing device.

10. The processing apparatus according to claim 9, The interface is a processing device that outputs the operation information of the robot in the first virtual work environment to the outside of the processing device.

11. The apparatus according to claim 1, A processing device comprising an interface for outputting the work time in the aforementioned multiple types of virtual work environments and information relating to the aforementioned multiple types of virtual work environments to an external device.

12. The apparatus according to claim 11, The interface is a processing device that outputs the operation information of the robot in the multiple types of virtual work environments to the outside of the processing device.

13. A program for causing a computer device to function as a processing device according to any one of claims 1 to 12.

14. A display device for displaying the results of the simulation performed by the processing device according to any one of claims 1 to 12.

15. A processing system comprising a processing device according to any one of claims 1 to 12, and a robot controller capable of communicating with the processing device.