Information processing device, robot system, method for processing information, method for manufacturing article, program and recording medium

Abstract

To effectively design movements of a robot and operations relating to imaging of an imaging device.SOLUTION: An information processing device includes a processing part for simulating so that a virtual robot and a virtual imaging device move while interlocking with each other in a virtual space. The processing part links setting information of the virtual imaging device with teaching points of the virtual robot.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] This disclosure relates to the technology of robots.

Background Art

[0002] A robot system in which a robot holds an imaging device and images an object is known. In such a robot system, it is necessary to teach the robot before actually performing work.

[0003] Patent Document 1 discloses that in a simulation device that performs simulation by a virtual robot, information input regarding the depth of field and the field of view of a virtual imaging unit is received, and the field of view is displayed on a display unit.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the method described in Patent Document 1 does not deal with the case where the settings of the imaging device are changed while the robot is operating, and it has been desired to efficiently design the operation of the robot and the operation related to the imaging of the imaging device.

[0006] This disclosure aims to enable efficient design of the operation of the robot and the operation related to the imaging of the imaging device.

Means for Solving the Problems

[0007] According to a first aspect of this disclosure, the information processing device includes a processing unit that simulates a virtual robot and a virtual imaging device moving in conjunction with each other in a virtual space, and the processing unit associates the setting information of the virtual imaging device with the teaching points of the virtual robot.

[0008] According to a second aspect of this disclosure, the information processing method is an information processing method for simulating the coordinated movement of a virtual robot and a virtual imaging device in a virtual space, characterized in that it associates the setting information of the virtual imaging device with the teaching points of the virtual robot. [Effects of the Invention]

[0009] According to this disclosure, the movements of the robot and the movements related to imaging of the imaging device can be designed efficiently. [Brief explanation of the drawing]

[0010] [Figure 1] This is an explanatory diagram of the robot system according to the first embodiment. [Figure 2] (a) is an explanatory diagram of the information processing device according to the first embodiment. (b) is a block diagram of the information processing device according to the first embodiment. [Figure 3] This is an explanatory diagram of the virtual space obtained by simulation of the information processing device according to the first embodiment. [Figure 4] This is an explanatory diagram of an example of a UI window according to the first embodiment. [Figure 5] This is an explanatory diagram of an example of a UI window according to the first embodiment. [Figure 6] This is an explanatory diagram of an example of a UI window according to the first embodiment. [Figure 7] This is an explanatory diagram of an example of a UI window according to the first embodiment. [Figure 8] This is an explanatory diagram of the robot system according to the second embodiment. [Figure 9] This is an explanatory diagram of the virtual space obtained by simulation of the information processing device according to the second embodiment. [Figure 10] It is an explanatory diagram of an example of a UI window according to the second embodiment. [Figure 11] It is an explanatory diagram of an example of a UI window according to the second embodiment. [Figure 12] It is an explanatory diagram of an example of a UI window according to the third embodiment. [Figure 13] It is an explanatory diagram of an example of a UI window according to the third embodiment. [Figure 14] It is an explanatory diagram of an example of a UI window according to the third embodiment. [Figure 15] It is an explanatory diagram of an example of a UI window according to the third embodiment. [Figure 16] It is an explanatory diagram of a robot system according to the fourth embodiment. [Figure 17] It is an explanatory diagram of a virtual space by simulation of an information processing apparatus according to the fourth embodiment. [Figure 18] It is an explanatory diagram of an example of a UI window according to the fourth embodiment. [Figure 19] It is an explanatory diagram of an example of a UI window according to the fourth embodiment. [Figure 20] It is an explanatory diagram of an example of a UI window according to the fourth embodiment. [Figure 21] It is an explanatory diagram of a robot system according to the fifth embodiment. [Figure 22] It is an explanatory diagram of a virtual space by simulation of an information processing apparatus according to the fifth embodiment. [Figure 23] It is an explanatory diagram of an example of a UI window according to the fifth embodiment. [Figure 24] It is an explanatory diagram of an example of a UI window according to the fifth embodiment. [Figure 25] It is an explanatory diagram of an example of a UI window according to the sixth embodiment. [Figure 26] It is an explanatory diagram of an example of a UI window according to the sixth embodiment. [Figure 27] It is an explanatory diagram of an example of a UI window according to the seventh embodiment. [Figure 28]This is an explanatory diagram of an example of a UI window according to the seventh embodiment. [Figure 29] This is an explanatory diagram of an example of a UI window according to the seventh embodiment. [Figure 30] This is an explanatory diagram of an example of a UI window according to the seventh embodiment. [Figure 31] This is an explanatory diagram of an example of a UI window according to the seventh embodiment. [Modes for carrying out the invention]

[0011] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the drawings.

[0012] [First Embodiment] Figure 1 is an explanatory diagram of a robot system 1000 according to the first embodiment. The robot system 1000 comprises a robot 100, an imaging device 500, a control device 200, and an information processing device 300.

[0013] Robot 100 is an industrial robot that can be used to hold an object. In Figure 1, robot 100 is holding an imaging device 500. Robot 100 is a manipulator. Robot 100 is positioned on a stand (not shown). Workpiece W1 is positioned around robot 100. Workpiece W1 has multiple imaging targets O1, O2. Imaging targets O1, O2 are information such as barcodes such as QR codes (registered trademarks) or part numbers, and are the imaging targets of the imaging device 500. In the first embodiment, the case in which imaging targets O1, O2 are imaged and information is read will be described as an example.

[0014] Robot 100 is, for example, a vertically articulated robotic arm. The fixed end, which is the base end of robot 100, is fixed to a stand (not shown). An imaging device 500 is attached to the free end, which is the tip of robot 100. Thus, the imaging device 500 is held by robot 100. Robot 100 has a base 110 and a plurality of links 111 to 116. These base 110 and links 111 to 116 are connected by joints J1 to J6, so that each link 111 to 116 can rotate in the direction indicated by the arrow at each joint J1 to J6. In this embodiment, link 116 is the tip of robot 100, and the imaging device 500 is fixed to link 116.

[0015] Each joint J1 to J6 of the robot 100 is equipped with a motor (not shown) as a power source, and each motor drives each joint J1 to J6, i.e., each link 111 to 116, allowing the robot 100 to assume various postures.

[0016] The robot 100 may have a robot hand attached to the link 116. For example, in a manufacturing line for manufacturing goods, the robot 100 can use its robot hand to grasp a workpiece and perform transport operations or assembly operations such as assembling it with other workpieces, or it can grasp a tool and perform workpiece machining operations. Alternatively, the robot 100 can also use actuators other than a robot hand attached to the link 116, depending on the work content of the manufacturing process.

[0017] The imaging device 500 is, for example, a digital camera and includes a lens 501 and an image sensor (not shown). The imaging device 500 captures subjects in the field of view, i.e., the imaging area, according to the set information, and transmits the image data obtained from this capture to the control device 200.

[0018] The workpiece W1 is positioned in the vicinity of the robot 100 at a predetermined position and orientation relative to the robot 100. The workpiece W1 is formed, for example, in a roughly rectangular parallelepiped shape. An imaging target O1 is placed on one face of the workpiece W1, and an imaging target O2 is placed on a different face of the workpiece W1 from the one on which imaging target O1 is placed. The outer shape of each imaging target O1, O2 is, for example, a rectangle in plan view.

[0019] The control device 200 shown in Figure 1 controls the operation of the robot 100 based on the robot's operation information, i.e., teaching data indicating the robot program, and also controls the imaging operation of the imaging device 500 in relation to the operation of the robot 100. The control device 200 acquires the teaching data from the information processing device 300. The teaching data includes information on commands for the robot 100, information on teaching points for the robot 100, and setting information for the imaging device 500. In this embodiment, the control device 200 operates the robot 100 based on the teaching data and causes the imaging device 500 to image each of the imaging targets O1 and O2. The control device 200 then acquires the image data in which each of the imaging targets O1 and O2 is captured from the imaging device 500.

[0020] The information processing device 300 is composed of a computer and functions as a simulator. In this embodiment, the information processing device 300 generates teaching data through computer simulation, i.e., offline teaching. The teaching data generated by the information processing device 300 is output to the control device 200. The method of outputting the teaching data to the control device 200 is not particularly limited. For example, the teaching data generated by the information processing device 300 may be output to the control device 200 via wired communication, wireless communication, or via a storage device (not shown).

[0021] Figure 2(a) is an explanatory diagram of the information processing device 300 according to the first embodiment. The information processing device 300 comprises a device body 301, a display 302 which is an example of a display unit connected to the device body 301, and a keyboard 303 and a mouse 304 which are examples of input units connected to the device body 301. The following description will use the case where the information processing device 300 is a general-purpose computer, such as a desktop PC, as an example, but is not limited to this. The information processing device 300 may be a general-purpose computer such as a laptop PC, tablet PC, or smartphone, or it may be a teaching pendant, or it may be a computer dedicated to a simulator. Furthermore, the information processing device 300 may be incorporated into a control device 200. That is, the control device 200 may have simulator functions. Also, instead of the display 302, keyboard 303, and mouse 304, there may be a touch panel display in which the display unit and input unit are integrated.

[0022] Figure 2(b) is a block diagram of the information processing device 300 according to the first embodiment. The main body 301 of the information processing device 300 includes a processor, a CPU (Central Processing Unit) 311. The CPU 311 is an example of a processing unit.

[0023] The main unit 301 also includes a ROM (Read Only Memory) 312, a RAM (Random Access Memory) 313, and an HDD (Hard Disk Drive) 314 as storage units. Furthermore, the main unit 301 includes a recording disk drive 315 and an I / O (Input / Output) interface 320. The CPU 311, ROM 312, RAM 313, HDD 314, recording disk drive 315, and I / O 320 are connected to each other via a bus 310, enabling communication between them.

[0024] ROM312 is a non-temporary storage device. ROM312 stores the basic program that is read by the CPU311 when the computer starts up. RAM313 is a temporary storage device used for the CPU311's arithmetic processing. HDD314 is a non-temporary storage device that stores various data, such as the results of the CPU311's arithmetic processing. In this embodiment, program 350 is stored in HDD314. Program 350 is application software. By executing program 350, the CPU311 functions as a processing unit capable of simulating the behavior of a virtual robot and a virtual imaging device in a virtual environment (virtual space), as described later.

[0025] The recording disk drive 315 can read various data and programs recorded on the recording disk 340. The I / O 320 functions as an external interface. The display 302, keyboard 303, and mouse 304 are connected to the I / O 320. The display 302, under the control of the CPU 311, displays UI images that serve as the user interface (UI), as well as images that reflect information entered by the user using the keyboard 303 and mouse 304. Teaching data, including information on teaching points, is created by the CPU 311, which executes the program 350.

[0026] In this embodiment, the non-temporary recording medium readable by the computer is the HDD 314, and the program 350 is recorded on the HDD 314, but this is not the only possible representation. The program 350 may be recorded on any non-temporary recording medium readable by the computer. Examples of recording media that can be used to supply the program 350 to the computer include flexible disks, optical disks, magneto-optical disks, magnetic tapes, non-volatile memory, and the like.

[0027] Figure 3 is an explanatory diagram of the virtual space R simulated by the information processing device 300 according to the first embodiment. The CPU 311 defines the virtual space R shown in Figure 3 as a virtual environment. Virtual objects in the virtual space R are defined by three-dimensional model data, such as CAD (Computer-Aided Design) data, and in Figure 3, for convenience, they are visualized and shown as structures.

[0028] The virtual objects defined in the virtual space R shown in Figure 3 will now be explained. The virtual space R defines a virtual robot 100V, a virtual imaging device 500V, and a virtual workpiece W1V. The virtual robot 100V is defined by a 3D model data that simulates the robot 100 shown in Figure 1. The virtual imaging device 500V is defined by a 3D model data that simulates the imaging device 500 shown in Figure 1. The virtual workpiece W1V is defined by a 3D model data that simulates the workpiece W1 shown in Figure 1. The virtual workpiece W1V includes virtual imaging targets O1V and O2V, which simulate the imaging targets O1 and O2 shown in Figure 1. The virtual space R shown in Figure 3 is displayed as a still image or video on the display 302 shown in Figure 2(a).

[0029] The virtual robot 100V has a virtual base 110V and multiple virtual links 111V to 116V. These virtual base 110V and virtual links 111V to 116V are defined to be connected by virtual joints J1V to J6V. This makes it possible to simulate each virtual link 111V to 116V rotating in the direction indicated by the arrow at each virtual joint J1V to J6V. The virtual imaging device 500V is linked to virtual link 116V and is defined to move in conjunction with virtual link 116V. That is, when the CPU 311 simulates the movement of virtual link 116V in virtual space R, it simulates the virtual imaging device 500V moving in conjunction with virtual link 116V while maintaining its relative position and orientation to virtual link 116V in virtual space R.

[0030] The CPU 311 executes the information processing method, i.e., simulation, described below, by running program 350. In the following description, it is assumed that data for three teaching points are pre-set in the information processing device 300 as multiple teaching points. That is, the data for the three teaching points is stored in HDD 314. In addition, the setting information for the imaging device 500, i.e., the setting information for the virtual imaging device 500V, is set in association with each of the multiple teaching points by the processing of the CPU 311 described later.

[0031] Here, the teaching points are used in the information processing device 300 to simulate the virtual operation of the virtual robot 100V, and in the control device 200 to control the actual operation of the robot 100. In addition, the setting information is used in the information processing device 300 to simulate the virtual imaging operation of the virtual imaging device 500V, and in the control device 200 to control the actual imaging operation of the imaging device 500. That is, the setting information set for the virtual imaging device 500V is the same as the setting information set by the functions of the actual imaging device 500.

[0032] Figures 4 to 6 are explanatory diagrams illustrating an example of a UI window UI1 according to the first embodiment. When the CPU 311 executes the program 350, it displays a UI window UI1, which is an example of a UI image, on the display 302, as shown in any of Figures 4 to 6. In the first embodiment, the CPU 311 accepts user input and selections through the UI window UI1. User input through the UI window UI1 can be performed using the keyboard 303 and mouse 304, but is not limited to these. Similarly, user selections through the UI window UI1 can be made using the mouse 304, but is not limited to these.

[0033] The UI window UI1 consists of a three-dimensional space display screen 10, a teaching point selection screen 20, a teaching point information display screen 30, and an imaging area display screen 40.

[0034] The three-dimensional space display screen 10 displays virtual objects defined in the virtual space R as images. Specifically, the three-dimensional space display screen 10 displays a robot image 100I corresponding to the virtual robot 100V, a camera image 500I corresponding to the virtual imaging device 500V, and a workpiece image W1I corresponding to the virtual workpiece W1V. The workpiece image W1I includes images O1I and O2I corresponding to the virtual imaging targets O1V and O2V.

[0035] The teaching point selection screen 20 displays a list of names assigned to the teaching points to be operated on the virtual robot 100V. Teaching points can be added or deleted on the teaching point selection screen 20. In the examples in Figures 4-6, the names of three registered teaching points, "P1," "P2," and "P3," are displayed on the teaching point selection screen 20. Teaching point names can be freely assigned by the user or automatically assigned. For the sake of explanation, the three teaching points will be referred to as teaching points P1, P2, and P3. On the teaching point selection screen 20, the name of the selected teaching point is highlighted.

[0036] In the example shown in Figure 4, teaching point P1 is selected. Teaching point P1 is the coordinate information of the basic posture of the virtual robot 100V. Teaching point P1 is created assuming that the actual robot 100 will wait until the workpiece W1 is transported to a predetermined position.

[0037] In the example shown in Figure 5, teaching point P2 is selected. Teaching point P2 is the coordinate information of the posture of the virtual robot 100V when the virtual imaging device 500V is virtually made to image the virtual imaging target O1V. Teaching point P2 is created with the assumption that the posture of the actual robot 100 will be controlled so that the imaging target O1 is within the field of view (i.e., imaging area) of the imaging device 500.

[0038] In the example in Figure 6, teaching point P3 is selected. Teaching point P3 is the coordinate information of the posture of the virtual robot 100V when the virtual imaging device 500V is virtually made to image the virtual imaging target O2V. Teaching point P3 is created with the assumption that the posture of the actual robot 100 will be controlled so that the imaging target O2 falls within the field of view (i.e., imaging area) of the imaging device 500.

[0039] Here, "virtually imaging" of the virtual imaging device 500V means that the CPU 311 calculates two-dimensional image data in the field of view of the virtual imaging device 500V, i.e., the imaging area. In other words, the virtual imaging operation of the virtual imaging device 500V is a simulation that assumes the actual imaging operation of the imaging device 500.

[0040] The teaching point information display screen 30 includes a number of boxes 31 into which the coordinate information of the teaching point selected on the teaching point selection screen 20 can be entered. The teaching point information display screen 30 also includes a box 32 into which setting information related to the virtual imaging operation of the virtual imaging device 500V, which is associated with the selected teaching point, can be entered. In other words, in the first embodiment, the teaching point information display screen 30 is an image that accepts the input of setting information. In the first embodiment, the box 32 can accept information on the field of view (zoom magnification) of the virtual imaging device 500V as setting information. It is possible to enter new information into each of the boxes 31 and 32, and it is also possible to edit the information already entered into each of the boxes 31 and 32.

[0041] In the teaching point information display screen 30 shown in Figures 4 to 6, the field containing multiple boxes 31 for inputting the coordinate information of the teaching point is labeled "Coordinate Information," and the field containing a box 32 for inputting setting information is labeled "Camera Information."

[0042] In the examples shown in Figures 4 to 6, the teaching point information display screen 30 allows input of the angle information of virtual joints J1V to J6V as the coordinate information for each teaching point P1 to P3. In the examples shown in Figures 4 to 6, the multiple boxes 31 into which the angle information of virtual joints J1V to J6V is input are named "J1" to "J6" for easier recognition by the user. Note that the coordinate information of the teaching points is not limited to the angle information of virtual joints J1V to J6V; for example, it may also be the position and orientation information of a tool center point (TCP) defined to be linked to the virtual link 116V.

[0043] In the example in Figure 4, the angle information for the virtual joints J1V to J6V at teaching point P1 is entered as follows: 0.000° in box 31 for J1, 0.000° in box 31 for J2, 0.000° in box 31 for J3, 0.000° in box 31 for J4, -90.000° in box 31 for J5, and 0.000° in box 31 for J6. In addition, in the example in Figure 4, the field of view information of the virtual imaging device 500V associated with teaching point P1, i.e., the zoom magnification, is entered as 1.0 in box 32.

[0044] In the example in Figure 5, the angle information for the virtual joints J1V to J6V at teaching point P2 is entered as follows: -31.617° in box 31 for J1, 25.852° in box 31 for J2, 46.742° in box 31 for J3, -35.882° in box 31 for J4, -75.749° in box 31 for J5, and -83.148° in box 31 for J6. In addition, in the example in Figure 5, the field of view information of the virtual imaging device 500V associated with teaching point P2, i.e., the zoom magnification, is entered as 1.0 in box 32.

[0045] In the example in Figure 6, the angle information for the virtual joints J1V to J6V at teaching point P3 is entered as follows: 59.331° in box 31 for J1, 12.298° in box 31 for J2, -17.452° in box 31 for J3, -86.950° in box 31 for J4, 59.468° in box 31 for J5, and -92.974° in box 31 for J6. In addition, in the example in Figure 6, the field of view information of the virtual imaging device 500V associated with teaching point P3, i.e., the zoom magnification, is entered as 1.0 in box 32.

[0046] The CPU 311 performs a simulation of operating the virtual robot 100V at the teaching point selected on the teaching point selection screen 20, and draws a corresponding robot image 100I on the three-dimensional space display screen 10 based on the posture of the virtual robot 100V. In the example in Figure 4, the robot image 100I is drawn on the three-dimensional space display screen 10 in a posture corresponding to the posture of the virtual robot 100V operating at teaching point P1. In the example in Figure 5, the robot image 100I is drawn on the three-dimensional space display screen 10 in a posture corresponding to the posture of the virtual robot 100V operating at teaching point P2. In the example in Figure 6, the robot image 100I is drawn on the three-dimensional space display screen 10 in a posture corresponding to the posture of the virtual robot 100V operating at teaching point P3.

[0047] Furthermore, when the CPU 311 moves the virtual imaging device 500V in conjunction with the virtual robot 100V which has been moved to the selected teaching point, it displays the image obtained by the virtual imaging operation of the virtual imaging device 500V based on the setting information on the imaging area display screen 40.

[0048] In the example shown in Figure 4, the CPU 311 moves the virtual imaging device 500V in conjunction with the virtual robot 100V that has been moved to the teaching point P1, and displays the image 41 obtained by the virtual imaging operation of the virtual imaging device 500V based on the zoom magnification information on the imaging area display screen 40.

[0049] In the example shown in Figure 5, the CPU 311 moves the virtual imaging device 500V in conjunction with the virtual robot 100V that has been moved to the teaching point P2, and displays the image 42 obtained by the virtual imaging operation of the virtual imaging device 500V based on the zoom magnification information on the imaging area display screen 40. Image 42 includes image O1I corresponding to the virtual imaging target O1V.

[0050] In the example shown in Figure 6, the CPU 311 moves the virtual imaging device 500V in conjunction with the virtual robot 100V operating at the teaching point P3, and displays the image 43 obtained by the virtual imaging operation of the virtual imaging device 500V based on the zoom magnification information of 1.0 on the imaging area display screen 40. Image 43 includes image O2I corresponding to the virtual imaging target O2V.

[0051] The information in each box 31, 32 can be changed by the user as described above. Figure 7 is an explanatory diagram of an example of the UI window UI1 according to the first embodiment. Figure 7 shows the UI window UI1 when the zoom magnification is changed from 1.0 to 2.0 in Figure 6. The CPU 311 moves the virtual imaging device 500V in conjunction with the virtual robot 100V that has been moved to the teaching point P3, and displays the image 44 obtained by the virtual imaging operation of the virtual imaging device 500V based on the information of zoom magnification 2.0 on the imaging area display screen 40. In this way, an image 44 is obtained in which the virtual imaging target O2V is captured at a sufficient size without changing the information of the teaching point P3. That is, image 44 contains an image O2I corresponding to the virtual imaging target O2V. The image O2I in image 44 is larger in size than the image O2I in image 43.

[0052] In this way, users can efficiently design imaging operations that meet their desired requirements by changing the settings of the virtual imaging device 500V. Furthermore, when performing offline teaching, users can design teaching data while simultaneously checking the robot image 100I, the camera image 500I, and each image 41-44 obtained from the virtual imaging operation. It is not necessary to perform imaging at all teaching points. Therefore, teaching data can be created by associating setting information such as zoom magnification with the teaching points where imaging is required, for example, each teaching point P2, P3.

[0053] The teaching points of the virtual robot 100V and the setting information of the virtual imaging device 500V associated with those teaching points, created in this way by the information processing device 300, can be saved as teaching data, for example, on the HDD 314. This teaching data can also be sent to the control device 200. In the control device 200, this teaching data is used as the teaching points of the robot 100 and the setting information of the imaging device 500 associated with those teaching points. This allows the control device 200 to control the imaging device 500 in correspondence with the operation of the robot 100. For example, the control device 200 can control the robot 100 based on teaching point P2 and control the imaging device 500 based on setting information, for example, a zoom magnification of 1.0, associated with teaching point P2. Alternatively, the control device 200 can control the robot 100 based on teaching point P3 and control the imaging device 500 based on setting information, for example, a zoom magnification of 2.0, associated with teaching point P3.

[0054] As described above, according to the first embodiment, in the simulation, the CPU 311 associates the setting information of the virtual imaging device 500V with the teaching points of the virtual robot 100V, so that the user can efficiently design the operation of the robot 100 and the operation of the imaging device 500.

[0055] Furthermore, since the CPU 311 accepts setting information to be associated with the selected teaching points via the UI window UI1, the user can design the movements of the robot 100 and the imaging device 500 more efficiently.

[0056] Furthermore, in the first embodiment, the CPU 311 displays images 41 to 44 on the display 302, specifically on the UI window UI1 within the display 302. Therefore, the user can easily confirm what kind of image is obtained in relation to the teaching points of the virtual robot 100V and the setting information of the virtual imaging device 500V, and the operation of the robot 100 and the operation of the imaging device 500 can be designed more efficiently.

[0057] Furthermore, in the first embodiment, the CPU 311 displays a robot image 100I corresponding to the virtual robot 100V operating at the selected teaching point on the display 302, i.e., the UI window UI1 on the display 302. Preferably, a workpiece image W1I corresponding to the virtual workpiece W1V is also displayed. This makes it easier for the user to confirm what posture the robot 100 will take relative to the workpiece W1, and enables more efficient design.

[0058] Furthermore, in the first embodiment, the CPU 311 also displays the camera image 500I corresponding to the virtual imaging device 500V, which is linked to the virtual robot 100V, on the display 302, specifically on the UI window UI1 in the display 302. This makes it easier for the user to confirm the orientation of the imaging device 500 relative to the workpiece W1, and enables more efficient design.

[0059] In addition, multiple teaching points are often set. In the first embodiment, the CPU 311 can associate unique setting information with each of the multiple teaching points P1 to P3. For example, as shown in Figure 7, it is possible to set the zoom magnification of the virtual imaging device 500V to a different value from the other teaching points P1 and P2, for example, to 2.0, for teaching point P3. This allows the user to design more efficiently. Although unique setting information is associated with each of the multiple teaching points P1 to P3, the content of each setting information can be freely set by the user. Therefore, the content of each setting information may be different from each other or the same from each other.

[0060] [Second Embodiment] A second embodiment will now be described. Figure 8 is an explanatory diagram of the robot system 1000A according to the second embodiment. In the second embodiment, the explanation of matters similar to those in the first embodiment will be simplified or omitted. The configuration of the robot system 1000A in the second embodiment is the same as that described in the first embodiment, except for the imaging device. That is, in the second embodiment, the imaging device 500A shown in Figure 1 is replaced with an imaging device 500A. In addition, in the first embodiment described above, the case in which an object to be imaged is imaged and information such as a barcode is read was used as an example, but the invention is not limited to this. In the second embodiment, the case in which an object to be imaged is imaged and the shape of an article is recognized will be used as an example.

[0061] The imaging device 500A has two imaging units 501A and 502A as an example of multiple imaging units. Each imaging unit 501A and 502A has a different imaging direction. In the second embodiment, the field of view, i.e., the imaging area, is different. Imaging unit 502A can image the area opposite to imaging unit 501A. Imaging unit 501A is an example of a first camera, and imaging unit 502A is an example of a second camera. The imaging device 500A is, for example, a mobile terminal such as a smartphone, where imaging unit 501A is a so-called front camera and imaging unit 502A is a so-called rear camera. If the direction in which the front camera images is defined as the first direction, the rear camera images in a second direction which is the opposite direction to the first direction. The imaging device 500A can selectively image different imaging areas with its two imaging units 501A and 502A.

[0062] Each imaging unit 501A and 502A of the imaging device 500A is a digital camera, and each has a lens, an image sensor, etc. Each imaging unit 501A and 502A captures a subject in the field of view, i.e., the imaging area, according to the set information, and transmits the image data obtained from this capture to the control device 200.

[0063] A workpiece W2 is positioned around the robot 100. The workpiece W2 is positioned near the robot 100 in a predetermined position and orientation relative to the robot 100. The workpiece W2 has multiple imaging targets O3 and O4. The imaging targets O3 and O4 are a pair of side plates spaced apart and facing each other. The imaging target O3 is a side plate with a triangular shape in plan view, and the imaging target O4 is a side plate with a rectangular shape in plan view.

[0064] Figure 9 is an explanatory diagram of the virtual space R obtained by simulation of the information processing device 300 according to the second embodiment. The CPU 311 defines the virtual space R shown in Figure 9 as a virtual environment. Virtual objects in the virtual space R are defined by three-dimensional model data, such as CAD data, and in Figure 9, for convenience, they are visualized and illustrated as structures.

[0065] The virtual objects defined in the virtual space R shown in Figure 9 will now be explained. The virtual space R defines a virtual robot 100V, a virtual imaging device 500AV mounted on a virtual mobile terminal, and a virtual workpiece W2V. The virtual robot 100V is defined by a 3D model data that simulates the robot 100 shown in Figure 1. The virtual imaging device 500AV is defined by a 3D model data that simulates the imaging device 500A shown in Figure 8. The virtual workpiece W2V is defined by a 3D model data that simulates the workpiece W2 shown in Figure 8. The virtual workpiece W2V includes virtual imaging targets O3V and O4V, which simulate the imaging targets O3 and O4 shown in Figure 8. The virtual space R shown in Figure 9 is displayed as a still image or video on the display 302 shown in Figure 2(a).

[0066] The virtual imaging device 500AV is defined to be linked to the virtual link 116V of the virtual robot 100V and to move in conjunction with the virtual link 116V. In other words, when the CPU 311 simulates the movement of the virtual link 116V in the virtual space R, it simulates the virtual imaging device 500AV moving in conjunction with the virtual link 116V while maintaining its relative position and orientation to the virtual link 116V in the virtual space R.

[0067] The virtual imaging device 500AV has two virtual imaging units 501AV and 502AV as an example of multiple virtual imaging units. Virtual imaging unit 501AV is an example of a first virtual camera, and virtual imaging unit 502AV is an example of a second virtual camera. Each virtual imaging unit 501AV and 502AV has a different virtual imaging direction. In the second embodiment, the virtual field of view, i.e., the virtual imaging area, is different. Virtual imaging unit 502AV can virtually image the opposite side from virtual imaging unit 501AV. The virtual imaging device 500AV is, for example, a virtual smartphone, where virtual imaging unit 501AV is a virtual front camera and virtual imaging unit 502AV is a virtual rear camera. If the direction in which virtual imaging unit 501AV virtually images is defined as the first direction, then virtual imaging unit 502AV virtually images the second direction, which is the opposite direction to the first direction. The virtual imaging device 500AV can selectively virtually image different virtual imaging regions using its two virtual imaging units 501AV and 502AV.

[0068] The CPU 311 executes the program 350 to perform the information processing method, i.e., simulation, described below. In the following description, it is assumed that the data for two teaching points is pre-set in the information processing device 300 as multiple teaching points. That is, the data for the two teaching points is stored in the HDD 314. In addition, the setting information for the imaging device 500A, i.e., the setting information for the virtual imaging device 500AV, is set in association with each of the multiple teaching points by the processing of the CPU 311 described later.

[0069] Figures 10 and 11 are explanatory diagrams illustrating an example of a UI window UI2 according to the second embodiment. When the CPU 311 executes the program 350, it displays a UI window UI2, which is an example of a UI image as shown in either Figure 10 or Figure 11, on the display 302. In the second embodiment, the CPU 311 accepts user input and selections via the UI window UI2. User input via the UI window UI2 can be performed using the keyboard 303 and mouse 304, but is not limited to these. Similarly, user selections via the UI window UI2 can be made using the mouse 304, but is not limited to these.

[0070] UI window UI2, like UI window UI1 described in the first embodiment, is configured to include a three-dimensional space display screen 10, a teaching point selection screen 20, a teaching point information display screen 30, and an imaging area display screen 40.

[0071] The three-dimensional spatial display screen 10 shows a robot image 100I corresponding to the virtual robot 100V, a camera image 500AI corresponding to the virtual imaging device 500AV, and a workpiece image W2I corresponding to the virtual workpiece W2V. The workpiece image W2I includes images O3I and O4I corresponding to the virtual imaging targets O3V and O4V.

[0072] The teaching point selection screen 20 displays a list of names assigned to the teaching points to be operated on the virtual robot 100V. In the examples in Figures 10 and 11, the names of two registered teaching points, "P4" and "P5," are displayed on the teaching point selection screen 20. For the sake of explanation, these two teaching points will be referred to as teaching points P4 and P5. On the teaching point selection screen 20, the name of the currently selected teaching point is highlighted.

[0073] In the example shown in Figure 10, teaching point P4 is selected. Teaching point P4 is the coordinate information of the posture of the virtual robot 100V when the virtual imaging device 500AV is virtually made to image the virtual imaging target O3V. Teaching point P4 is created with the assumption that the posture of the actual robot 100 will be controlled so that the imaging target O3 falls within the field of view (i.e., imaging area) of the imaging unit 501A of the imaging device 500A.

[0074] In the example in Figure 11, teaching point P5 is selected. Teaching point P5 is the coordinate information of the posture of the virtual robot 100V when the virtual imaging device 500AV is virtually made to image the virtual imaging target O4V. Teaching point P5 is created assuming that the posture of the actual robot 100 will be controlled so that the imaging target O4 is within the field of view (i.e., imaging area) of the imaging unit 502A of the imaging device 500A. In the examples in Figures 10 and 11, the information of teaching point P4 and the information of teaching point P5 are the same value, but they may be different values.

[0075] The teaching point information display screen 30 includes, as in the first embodiment, a number of boxes 31 into which the coordinate information of the teaching point selected on the teaching point selection screen 20 can be entered. The teaching point information display screen 30 also includes boxes 32 and 33 into which setting information related to the virtual imaging operation of the virtual imaging device 500AV, which is associated with the selected teaching point, can be entered. The setting information set for the virtual imaging device 500AV is the same as the setting information set by the functions of the actual imaging device 500A.

[0076] In the second embodiment, box 33 can be used to input information about the imaging direction of the virtual imaging device 500AV as setting information. The imaging direction information of the virtual imaging device 500AV indicates which of the virtual imaging units 501AV and 502AV will be used. Specifically, the imaging direction information of the virtual imaging device 500AV indicates which of the virtual imaging units 501AV and 502AV will be used to perform virtual imaging. In the examples in Figures 10 and 11, the setting information is configured such that "IN" selects the virtual imaging unit 501AV and "OUT" selects the virtual imaging unit 502AV.

[0077] Box 32 allows input of setting information, specifically the field of view (zoom magnification) of the virtual imaging unit selected in Box 33. New information can be entered into each of Boxes 31, 32, and 33, and information already entered in each of Boxes 31, 32, and 33 can also be edited.

[0078] The CPU 311 performs a simulation to operate the virtual robot 100V at the teaching point selected on the teaching point selection screen 20, and draws a corresponding robot image 100I on the three-dimensional space display screen 10 based on the posture of the virtual robot 100V. In the example in Figure 10, the robot image 100I is drawn on the three-dimensional space display screen 10 in a posture corresponding to the posture of the virtual robot 100V operating at teaching point P4. In the example in Figure 11, the robot image 100I is drawn on the three-dimensional space display screen 10 in a posture corresponding to the posture of the virtual robot 100V operating at teaching point P5.

[0079] Furthermore, when the CPU 311 moves the virtual imaging device 500AV in conjunction with the virtual robot 100V which has been moved to the selected teaching point, it displays the image obtained by the virtual imaging operation of the virtual imaging device 500AV based on the setting information on the imaging area display screen 40.

[0080] In the example shown in Figure 10, the CPU 311 moves the virtual imaging device 500AV in conjunction with the virtual robot 100V operating at the teaching point P4, and displays the image 45 obtained by the virtual imaging operation of the virtual imaging unit 501AV of the virtual imaging device 500AV based on the zoom magnification information of 1.0 on the imaging area display screen 40. Image 45 includes image O3I corresponding to the virtual imaging target O3V. Image O3I is identified as a triangular image.

[0081] In the example shown in Figure 11, the CPU 311 moves the virtual imaging device 500AV in conjunction with the virtual robot 100V operating at the teaching point P5, and displays the image 46 obtained by the virtual imaging operation of the virtual imaging unit 502AV of the virtual imaging device 500AV based on the zoom magnification information of 1.0 on the imaging area display screen 40. Image 46 includes image O4I corresponding to the virtual imaging target O4V. Image O4I is identified as a rectangular image.

[0082] The teaching points of the virtual robot 100V and the setting information of the virtual imaging device 500AV associated with those teaching points, created in this way by the information processing device 300, can be saved as teaching data, for example, on the HDD 314. This teaching data can also be sent to the control device 200. In the control device 200, this teaching data is used as the teaching points of the robot 100 and the setting information of the imaging device 500A associated with those teaching points. This allows the control device 200 to control the imaging device 500A in correspondence with the operation of the robot 100. For example, the control device 200 can control the robot 100 based on the teaching point P4 and control the imaging device 500A based on setting information, for example, that indicates that the imaging unit 501A is to be used for imaging and setting information of a zoom magnification of 1.0, which are associated with the teaching point P4. Furthermore, for example, the control device 200 can control the robot 100 based on the teaching point P5, and control the imaging device 500A based on setting information associated with the teaching point P5, such as setting information indicating that the imaging unit 502A is to be used for imaging and setting information of a zoom magnification of 1.0.

[0083] As described above, according to the second embodiment, the user can efficiently design the operation of the robot 100 and the operation of the imaging device 500A. Furthermore, the various embodiments and modifications described above may be implemented in combination with this embodiment and / or this modification.

[0084] [Third Embodiment] A third embodiment will now be described. Figure 12 is an explanatory diagram of an example of a UI window UI3 according to the third embodiment. In the third embodiment, the explanation of matters similar to those in the first embodiment will be simplified or omitted. The configuration of the robot system 1000 in the third embodiment is the same as described in the first embodiment. When the CPU 311 executes the program 350, it displays a UI window UI3, which is an example of a UI image as shown in Figure 12, on the display 302. In the third embodiment, the CPU 311 accepts user input and selections via the UI window UI3. User input via the UI window UI3 can be performed using the keyboard 303 and mouse 304, but is not limited to these. Similarly, user selections via the UI window UI3 can be made using the mouse 304, but is not limited to these.

[0085] UI window UI3, like UI window UI1 described in the first embodiment, is configured to include a three-dimensional space display screen 10, a teaching point selection screen 20, a teaching point information display screen 30, and an imaging area display screen 40.

[0086] In the third embodiment, the teaching point information display screen 30 includes, in addition to boxes 31 and 32, an import button 34, which is an example of a first button, and an export button 35, which is an example of a second button. The import button 34 is a button used to read setting information from a storage unit, for example, an HDD 314. The export button 35 is a button used to save setting information to a storage unit, for example, an HDD 314.

[0087] When the import button 34 is pressed, the CPU 311 displays the UI window UI4 shown in Figure 13 on the display 302. The UI window UI4 includes an area 51 that displays the directory from which the setting information is read, and an area 52 that displays a list of files stored in the directory displayed in area 51. When the user selects a file name associated with a setting candidate displayed in area 52, the CPU 311 displays the selected file name in area 53. For example, if the user selects the file name "cameraA.ini" in area 52 that indicates a setting candidate, the CPU 311 displays "cameraA.ini" in area 53.

[0088] When the user presses the execute button 54, the CPU 311 reads the selected file from the HDD 314. When the user presses the cancel button 55, the CPU 311 returns to the process of displaying the UI window UI3 shown in Figure 12 without reading the file.

[0089] Figure 14 is an explanatory diagram of an example of a UI window UI3 according to the third embodiment. When the execute button 54 in Figure 13 is pressed, the CPU 311 reflects the values ​​registered in the read file into the setting information and changes the value of the zoom magnification box 32 as shown in Figure 14. In the example in Figure 14, the zoom magnification "2.5" is displayed in box 32.

[0090] Furthermore, when the export button 35 is pressed, the UI window UI5 shown in Figure 15 is displayed on the display 302. The UI window UI5 includes an area 56 that displays the directory where the settings information is saved, and an area 57 that displays a list of files saved in the directory displayed in area 56.

[0091] When the user enters a file name in area 58 and presses the execute button 59, the CPU 311 saves a file containing the zoom magnification information as setting information entered on the teaching point information display screen 30 to the HDD 314 with the file name entered in area 58. In this case, the file is saved in the directory shown in area 57. For example, a file with the file name "cameraD.ini" is saved as a setting candidate. When the user presses the cancel button 59A, the CPU 311 returns to the process of displaying the UI window UI3 shown in Figure 14 without saving the file.

[0092] As described above, according to the third embodiment, the user can efficiently design the operation of the robot 100 and the operation of the imaging device 500. Furthermore, the various embodiments and modifications described above may be implemented in combination with this embodiment and / or this modification.

[0093] [Fourth Embodiment] A fourth embodiment will now be described. Figure 16 is an explanatory diagram of the robot system 1000B according to the fourth embodiment. In the fourth embodiment, the explanation of matters similar to those described in the first to third embodiments above will be simplified or omitted. The configuration of the robot system 1000B in the fourth embodiment is the same as described in the embodiments above, except for the imaging device. That is, in the fourth embodiment, instead of the imaging devices 500 and 500A shown in Figures 1 and 8, the imaging device 500B is mounted on the robot 100.

[0094] As shown in Figure 16, the imaging device 500B includes an imaging unit 501B having multiple lenses. The focal lengths of the multiple lenses are different from each other. In the fourth embodiment, the multiple lenses of the imaging unit 501B include a first lens 511B, a second lens 512B having a longer focal length than the first lens 511B, and a third lens 513B having a longer focal length than the second lens 512B. As a result, the imaging unit 501B can image areas with different angles of view by changing the lens used during imaging. Furthermore, by switching between these lenses 511B to 513B, it is possible to perform imaging with a zoom magnification of 1.0x or less, for example, 0.5x imaging. An imaging target O5 is arranged around the robot 100. The imaging target O5 is a side plate with an outer shape that is triangular in plan view.

[0095] Figure 17 is an explanatory diagram of the virtual space R obtained by simulation of the information processing device 300 according to the fourth embodiment. The CPU 311 defines the virtual space R shown in Figure 17 as a virtual environment. Virtual objects in the virtual space R are defined by three-dimensional model data, such as CAD data, and in Figure 17, they are visualized and shown as structures for convenience.

[0096] The virtual objects defined in the virtual space R shown in Figure 17 will now be explained. The virtual space R defines a virtual robot 100V, a virtual imaging device 500BV, and a virtual imaging target O5V. The virtual robot 100V is defined by a 3D model data that simulates robot 100 shown in Figure 16. The virtual imaging device 500BV is defined by a 3D model data that simulates imaging device 500B shown in Figure 16. The first virtual lens 511BV, the second virtual lens 512BV, and the third virtual lens 513BV are defined by 3D model data that simulate the first lens 511B, the second lens 512B, and the third lens 513B, respectively. The virtual imaging target O5V is defined by a 3D model data that simulates imaging target O5 shown in Figure 16. The virtual space R shown in Figure 17 is displayed as a still image or video on the display 302 shown in Figure 2(a).

[0097] The virtual imaging device 500BV is defined to be linked to the virtual link 116V of the virtual robot 100V and to move in conjunction with the virtual link 116V. In other words, when the CPU 311 simulates the movement of the virtual link 116V in the virtual space R, it simulates the virtual imaging device 500BV moving in conjunction with the virtual link 116V while maintaining its relative position and orientation to the virtual link 116V in the virtual space R.

[0098] The virtual imaging device 500BV is, for example, a virtual smartphone and has a virtual imaging unit 501BV that includes a plurality of virtual lenses 511BV to 513BV. In the fourth embodiment, by switching between these virtual lenses 511BV to 513BV, it is possible to capture images with a field of view, i.e., zoom magnification of 1.0x or less, for example, 0.5x.

[0099] The CPU 311 executes the information processing method, i.e., simulation, described below, by running program 350. In the following description, it is assumed that the data for two teaching points is pre-set in the information processing device 300 as multiple teaching points. That is, the data for the two teaching points is stored in the HDD 314. In addition, the setting information for the imaging device 500B, i.e., the setting information for the virtual imaging device 500BV, is set in association with each of the multiple teaching points by the processing of the CPU 311 described later.

[0100] Figures 18 and 19 are explanatory diagrams of an example of a UI window UI6 according to the fourth embodiment. When the CPU 311 executes the program 350, it displays a UI window UI6, which is an example of a UI image as shown in either Figure 18 or Figure 19, on the display 302. In the fourth embodiment, the CPU 311 accepts user input and selections via the UI window UI6. User input via the UI window UI6 can be performed using the keyboard 303 and mouse 304, but is not limited to these. Similarly, user selections via the UI window UI4 can be made using the mouse 304, but is not limited to these.

[0101] The UI window UI6, like the UI window UI1 described in the first embodiment, is configured to include a three-dimensional space display screen 10, a teaching point selection screen 20, a teaching point information display screen 30, and an imaging area display screen 40. The three-dimensional space display screen 10 displays a robot image 100I corresponding to the virtual robot 100V, a camera image 500BI corresponding to the virtual imaging device 500BV, and an image O5I corresponding to the virtual imaging target O5V.

[0102] The teaching point selection screen 20 displays a list of names assigned to the teaching points to be operated on the virtual robot 100V. In the examples in Figures 18 and 19, the names of two registered teaching points, "P6" and "P7," are displayed on the teaching point selection screen 20. For the sake of explanation, these two teaching points will be referred to as teaching points P6 and P7. On the teaching point selection screen 20, the name of the currently selected teaching point is highlighted.

[0103] In the example in Figure 18, teaching point P6 is selected. Teaching point P6 is the coordinate information of the posture of the virtual robot 100V when the virtual imaging device 500BV is virtually made to image the virtual target O5V. Teaching point P6 is created assuming that the posture of the actual robot 100 will be controlled so that the target O5 is within the imaging area of ​​the imaging unit 501B of the imaging device 500B. However, in the example in Figure 18, it is confirmed that the entire image of the virtual target O5V is not displayed on the imaging area display screen 40 because the distance between the virtual imaging device 500BV and the virtual target O5V is too close. At this time, the zoom magnification is set to 1.0 in box 32 on the teaching point information display screen 30.

[0104] In the example shown in Figure 19, a zoom magnification of 0.5 is set in box 32 within the teaching point information display screen 30, and it can be confirmed that the entire image of the virtual imaging target O5V is displayed within the imaging area on the imaging area display screen 40. By the user inputting a zoom magnification of 0.5 into box 32 in this way, the CPU 311 processes the task of associating a zoom magnification of 0.5 with the teaching point P6. This enables the control device 200 to accurately image the imaging target O5 with the imaging device 500B when controlling the actual robot 100. Furthermore, since the user can accurately image the imaging target O5 with the imaging device 500B without having to recreate the posture data of the robot 100, it becomes possible to efficiently design the operation of the robot 100 and the operation of the imaging device 500B.

[0105] Furthermore, as shown in Figure 20, the user may be able to select one lens to be used for imaging from among the multiple lenses 511B to 513B included in the imaging unit 501B. In the example in Figure 20, the third lens 513B is selected in box 33. In this way, it is possible to easily set which of the multiple lenses 511B to 513B to be used by the imaging device 500B to take an image, in correspondence with the teaching points set on the robot 100.

[0106] As described above, according to the fourth embodiment, the user can efficiently design the operation of the robot 100 and the operation of the imaging device 500B. Furthermore, the various embodiments and modifications described above may be implemented in combination with this embodiment and / or this modification.

[0107] [Fifth Embodiment] A fifth embodiment will now be described. Figure 21 is an explanatory diagram of the robot system 1000B according to the fifth embodiment. In the fifth embodiment, the explanation of matters similar to those in the fourth embodiment will be simplified or omitted. The configuration of the robot system 1000B in the fifth embodiment is as described in the fourth embodiment above. In the fifth embodiment, instead of the imaging target O5 shown in Figure 16, imaging targets O6 and O7 are arranged around the robot 100. The imaging device 500B described in the fourth embodiment is attached to the robot 100. Both imaging targets O6 and O7 are side plates that are rectangular in plan view. For imaging target O6, the longitudinal direction of the rectangle is vertical. For imaging target O7, the longitudinal direction of the rectangle is horizontal. In other words, imaging target O6 is placed vertically, and imaging target O7 is placed horizontally.

[0108] Figure 22 is an explanatory diagram of the virtual space R obtained by simulation of the information processing device 300 according to the fifth embodiment. The CPU 311 defines the virtual space R shown in Figure 22 as a virtual environment. Virtual objects in the virtual space R are defined by three-dimensional model data, such as CAD data, and in Figure 22, for convenience, they are visualized and shown as structures.

[0109] The virtual objects defined in the virtual space R shown in Figure 22 will now be explained. The virtual space R defines a virtual robot 100V, a virtual imaging device 500BV, and virtual imaging targets O6V and O7V. The virtual robot 100V is defined by a 3D model data that simulates robot 100 shown in Figure 21. The virtual imaging device 500BV is defined by a 3D model data that simulates imaging device 500B shown in Figure 21. The virtual imaging targets O6V and O7V are defined by 3D model data that simulate imaging targets O6 and O7 shown in Figure 21. The virtual space R shown in Figure 22 is displayed as a still image or video on the display 302 shown in Figure 2(a).

[0110] The virtual imaging device 500BV is defined to be linked to the virtual link 116V of the virtual robot 100V and to move in conjunction with the virtual link 116V. In other words, when the CPU 311 simulates the movement of the virtual link 116V in the virtual space R, it simulates the virtual imaging device 500BV moving in conjunction with the virtual link 116V while maintaining its relative position and orientation to the virtual link 116V in the virtual space R.

[0111] The virtual imaging device 500BV is, for example, a virtual smartphone and has a virtual imaging unit 501BV that includes a plurality of virtual lenses 511BV to 513BV. In the fifth embodiment, the aspect ratio of the virtual image obtained by virtual imaging of the virtual imaging device 500BV is switched depending on the orientation of the virtual imaging device 500BV.

[0112] The CPU 311 executes the information processing method, i.e., simulation, described below, by running program 350. In the following description, it is assumed that the data for two teaching points is pre-set in the information processing device 300 as multiple teaching points. That is, the data for the two teaching points is stored in the HDD 314. In addition, the setting information for the imaging device 500B, i.e., the setting information for the virtual imaging device 500BV, is set in association with each of the multiple teaching points by the processing of the CPU 311 described later.

[0113] Figures 23 and 24 are explanatory diagrams of an example of a UI window UI7 according to the fifth embodiment. When the CPU 311 executes the program 350, it displays a UI window UI7, which is an example of a UI image as shown in either Figure 23 or Figure 24, on the display 302. In the fifth embodiment, the CPU 311 accepts user input and selections via the UI window UI7. User input via the UI window UI7 can be performed using the keyboard 303 and mouse 304, but is not limited to these. Similarly, user selections via the UI window UI7 can be made using the mouse 304, but is not limited to these.

[0114] UI window UI7, like UI windows UI1 to UI3 described in the various embodiments above, is configured to include a three-dimensional space display screen 10, a teaching point selection screen 20, a teaching point information display screen 30, and an imaging area display screen 40. The three-dimensional space display screen 10 includes a robot image 100I corresponding to the virtual robot 100V, a camera image 500BI corresponding to the virtual imaging device 500BV, and images O6I and O7I corresponding to the virtual imaging targets O6V and O7V.

[0115] The teaching point selection screen 20 displays a list of names assigned to the teaching points to be operated on the virtual robot 100V. In the examples in Figures 23 and 24, the names of two registered teaching points, "P8" and "P9," are displayed on the teaching point selection screen 20. For the sake of explanation, these two teaching points will be referred to as teaching points P8 and P9. On the teaching point selection screen 20, the name of the currently selected teaching point is highlighted.

[0116] In the example shown in Figure 23, teaching point P8 is selected. Teaching point P8 is the coordinate information of the orientation of the virtual robot 100V when the virtual imaging device 500BV is virtually used to image the virtual target O6V. At this time, the virtual imaging device 500BV is oriented so that its longitudinal direction is vertical, just like the virtual target O6V. In other words, the orientation of the virtual imaging device 500BV is vertical. In this case, the imaging area display screen 40 is automatically updated to become a vertically oriented display area, as shown in Figure 23. This allows the user to accurately understand how the virtual target O6V is being imaged, even when the virtual imaging device 500BV is oriented vertically.

[0117] In the example shown in Figure 24, teaching point P9 is selected. Teaching point P9 is the coordinate information of the virtual robot 100V's orientation when the virtual imaging device 500BV virtually images the virtual imaging target O7V. At this time, the virtual imaging device 500BV is oriented so that its longitudinal direction is horizontal, just like the virtual imaging target O7V. In other words, the virtual imaging device 500BV is oriented sideways. In this case, the imaging area display screen 40 is automatically updated to display a horizontally elongated area, as shown in Figure 24. This allows the user to accurately understand how the virtual imaging target O7V is being imaged, even when the virtual imaging device 500BV is oriented sideways.

[0118] As described above, according to the fifth embodiment, in the case of an imaging device mounted on a mobile terminal such as a smartphone, where the orientation of the captured image changes depending on the posture of the imaging device, the aspect ratio of the display of the imaging area is also linked on the simulator. As a result, even if the orientation of the captured image changes, the user can accurately understand how the image is being captured on the simulator, and the operation of the robot 100 and the operation of the imaging device 500B can be efficiently designed. Furthermore, the various embodiments and modifications described above may be implemented in combination with this embodiment and / or this modification.

[0119] [Sixth Embodiment] A sixth embodiment will now be described. Figures 25 and 26 are explanatory diagrams of an example of a UI window UI8 according to the sixth embodiment. In the sixth embodiment, the explanation of matters similar to those in the various embodiments described above will be simplified or omitted. The configuration of the robot system in the sixth embodiment is the same as described in the first embodiment. When the CPU 311 executes the program 350, it displays a UI window UI8, which is an example of a UI image as shown in Figure 25 or Figure 26, on the display 302. In the sixth embodiment, the CPU 311 accepts user input and selection via the UI window UI8. User input via the UI window UI8 can be performed using the keyboard 303 and mouse 304, but is not limited to these. Similarly, user selection via the UI window UI8 can be performed using the mouse 304, but is not limited to these. The UI window UI8 is configured to include a three-dimensional space display screen 10, a teaching point selection screen 20, a teaching point information display screen 30, and an imaging area display screen 40, similar to the UI windows UI1 to UI3 and UI6 to UI7 described in the various embodiments described above.

[0120] In the sixth embodiment, the teaching point information display screen 30 includes a camera type selection box 36 in addition to boxes 31 and 32. The camera type selection box 36 allows the user to select a camera to be used in the robot system from a predetermined set of cameras. In the example in Figure 25, the virtual imaging device 500BV is selected, and the camera image 500BI corresponding to the virtual imaging device 500BV is displayed. The virtual imaging device 500BV has a virtual imaging unit 501BV. The virtual imaging unit 501BV includes a first virtual lens 511BV, a second virtual lens 512BV, and a third virtual lens 513BV, each with different focal lengths. The three-dimensional space display screen 10 shown in Figure 25 displays the camera images 501BI, 511BI, 512BI, and 513BI, which correspond to the virtual imaging unit 501BV, the first virtual lens 511BV, the second virtual lens 512BV, and the third virtual lens 513BV, respectively.

[0121] In the example shown in Figure 26, a virtual imaging device (not shown) different from the virtual imaging device 500BV is selected, and the camera image 500CI corresponding to the virtual imaging device (not shown) is displayed. The virtual imaging device (not shown) has a virtual imaging unit (not shown) configured within it. The virtual imaging unit (not shown) has a fourth virtual lens (not shown) and a fifth virtual lens (not shown) with different focal lengths. The three-dimensional spatial display screen 10 shown in Figure 26 displays camera images 501CI, 511CI, and 513CI corresponding to the virtual imaging unit (not shown), the fourth virtual lens (not shown), and the fifth virtual lens (not shown), respectively.

[0122] In the example shown in Figure 25, "Smartphone B" is selected in the camera type selection box 36. At this time, the three-dimensional space display screen 10 displays a camera image 500BI that mimics the virtual imaging device 500BV corresponding to "Smartphone B," and the imaging area display screen 40 displays the virtual imaging area captured by "Smartphone B."

[0123] In the example shown in Figure 26, "Smartphone C" is selected in the camera type selection box 36. At this time, the three-dimensional space display screen 10 displays a camera image 500CI that mimics a virtual imaging device corresponding to "Smartphone C," and the imaging area display screen 40 displays the virtual imaging area captured by "Smartphone C."

[0124] As described above, according to the sixth embodiment, even when there are multiple imaging devices mounted on the robot or mobile terminals equipped with imaging devices, the imaging device can be easily changed on the robot simulator, and the user can easily understand how imaging is performed. Therefore, when multiple imaging devices can be considered, the user can efficiently design the operation of the robot 100 and the operation of each imaging device in the robot system. Furthermore, the various embodiments and modifications described above may be implemented in combination with this embodiment and / or this modification.

[0125] [Seventh Embodiment] A seventh embodiment will now be described. Figures 27 to 30 are explanatory diagrams of an example of a UI window UI9 according to the seventh embodiment. In the seventh embodiment, the explanation of matters similar to those in the various embodiments described above will be simplified or omitted. The configuration of the robot system in the seventh embodiment is the same as described in the first embodiment. When the CPU 311 executes the program 350, it displays a UI window UI9, which is an example of a UI image as shown in Figures 27 to 30, on the display 302. In the seventh embodiment, the CPU 311 accepts user input and selection via the UI window UI9. User input via the UI window UI9 can be performed using the keyboard 303 and mouse 304, but is not limited to these. Similarly, user selection via the UI window UI9 can be performed using the mouse 304, but is not limited to these. The UI window UI9 is ​​configured to include a three-dimensional space display screen 10, a teaching point selection screen 20, a teaching point information display screen 30, and an imaging area display screen 40, similar to the UI windows UI1 to UI3 and UI6 to UI8 described in the various embodiments described above. The configuration of the robot, imaging device, and imaging target in the seventh embodiment is the same as that of the robot 100, imaging device 500A, and imaging targets O3 and O4 in the second embodiment described above.

[0126] As shown in Figure 27, in the seventh embodiment, the imaging information setting screen 60 is displayed in the UI window UI9 as a screen for confirming and setting camera information at each teaching point. The imaging information setting screen 60 is positioned, for example, below the imaging area display screen 40. In the seventh embodiment, the imaging information setting screen 60 is displayed below the imaging area display screen 40, but it is not limited to this. For example, the imaging information setting screen 60 may be displayed above the imaging area display screen 40.

[0127] The imaging information setting screen 60 has an imaging setting UI similar to the imaging setting UI of the imaging device 500A installed in the smartphone. The imaging information setting screen 60 displays an imaging mode setting bar 61, a zoom magnification setting button 62, a flash setting button 63, a live imaging setting button 64, an aspect ratio setting button 65, a timer setting button 66, an in / out switching button 67, and a registration button 68.

[0128] The imaging mode setting bar 61 is a UI that sets the imaging mode to be set on the imaging device 500A when the imaging device 500A is to perform imaging, and this mode is set on the simulator, i.e., the information processing device. As shown in Figure 27, in the seventh embodiment, the user can set the imaging mode to normal mode, slow mode, or panoramic mode on the simulator by operating the imaging mode setting bar 61. In the example in Figure 27, normal mode is set, and the "normal" part of the imaging mode setting bar 61 is highlighted in grayscale. When "slow" is clicked in the state shown in Figure 27, the "slow" part is highlighted as shown in Figure 28. The same applies to panoramic mode. By operating the imaging mode setting bar 61, the user can easily set the imaging mode corresponding to each teaching point and easily understand what imaging mode is set.

[0129] The zoom magnification setting button 62 is a UI for setting the zoom magnification when the imaging device 500A takes an image. In the example shown in Figure 27, the zoom magnification is set to 1.5x. When the zoom magnification setting button 62 is clicked once in the state shown in Figure 27, it is set to 2.0x as shown in Figure 28. After a predetermined number of clicks, the zoom magnification returns to the minimum magnification. The minimum and maximum magnifications that can be set, and the range of change in zoom magnification with each click can be freely set. By operating the zoom magnification setting button 62 in the UI window UI9, the user can easily set the zoom magnification corresponding to each teaching point, and can also easily understand how the zoom magnification is set.

[0130] The flash setting button 63 is a UI that sets whether or not to fire the flash when the imaging device 500A takes an image. In the example shown in Figure 27, the flash setting is enabled, and when the flash setting button 63 is clicked in the state shown in Figure 27, a diagonal line appears on the flash setting button 63, as shown in Figure 28. In the state shown in Figure 28, the flash setting is disabled. By operating the flash setting button 63, the user can easily set whether or not to fire the flash in relation to each teaching point, and can also easily see whether or not the flash is being fired.

[0131] The Live Imaging Setting button 64 is a UI that sets whether or not to record video and audio together for a predetermined time when the imaging device 500A performs imaging. In the seventh embodiment, the predetermined time is set to 1.5 seconds, but it is not limited to this and can be freely set. In the example shown in Figure 27, the Live Imaging Setting is enabled, and when the Live Imaging Setting button 64 is clicked in the state shown in Figure 27, a diagonal line is displayed on the Live Imaging Setting button 64, as shown in Figure 28. In the state shown in Figure 28, the Live Imaging Setting is disabled. The user can easily set whether or not to perform live imaging in association with each teaching point, and can also easily understand whether or not to perform live imaging.

[0132] The aspect ratio setting button 65 is a UI for setting the aspect ratio of the captured image when the imaging device 500A performs imaging. In the example shown in Figure 27, the aspect ratio is set to 1:1. When the aspect ratio setting button 65 is clicked in the state shown in Figure 27, the aspect ratio setting bar 65a is displayed as shown in Figure 29. In the seventh embodiment, the user can select and set the aspect ratio of the captured image from 1:1, 4:3, or 16:9 using the aspect ratio setting bar 65a. In the example shown in Figure 28, since the aspect ratio is set to 1:1, "1:1" on the aspect ratio setting bar 65a is highlighted in grayscale. When "4:3" or "16:9" on the aspect ratio setting bar 65a is clicked, the text indicating the clicked aspect ratio is highlighted in grayscale, and the aspect ratio of the image on the imaging area display screen 40 is changed.

[0133] Figure 30 illustrates the UI window UI9 when the aspect ratio is set to 4:3. The "4:3" on the aspect ratio setting bar 65a is highlighted in grayscale, and the aspect ratio of the image on the imaging area display screen 40 is set to 4:3. When a part of the imaging information setting screen 60 other than the aspect ratio setting button 65 and the aspect ratio setting bar 65a is clicked in the state shown in Figure 30, the selected aspect ratio is set, and the screen returns to the screen shown in Figure 27. The user can easily set the aspect ratio corresponding to each teaching point, and can also easily understand how the aspect ratio is set.

[0134] The timer setting button 66 is a UI that sets whether or not to set a timer, which creates a time difference between the timing of instructing the imaging device 500A to take an image and the timing of the imaging device 500A actually taking an image. When the timer setting button 66 is clicked in the state shown in Figure 27, the timer setting bar 66a is displayed as shown in Figure 31. In the seventh embodiment, the user can select from "OFF" (indicating timer off), "3s" (indicating 3 seconds), or "10s" (indicating 10 seconds) using the timer setting bar 66a to turn off the timer or set the timer duration. In the example shown in Figure 31, since timer off is set, "OFF" on the timer setting bar 66a is highlighted in grayscale. When "3s" or "10s" on the timer setting bar 66a is clicked, the text indicating the clicked timer is highlighted in grayscale. In the state shown in Figure 31, if any part of the imaging information setting screen 60 other than the timer setting button 66 and timer setting bar 66a is clicked, the selected timer is set, and the screen returns to the one shown in Figure 27. The user can easily set timers in association with each teaching point, and can also easily understand how the timers are set.

[0135] The in-out switching button 67 is a UI that sets whether to use the in-camera or the out-camera when taking images with the imaging device 500A. In the example shown in Figure 27, the out-camera is set, and the imaging area display screen 40 displays image O3I, which corresponds to the virtual imaging target O3V captured by the virtual imaging device 500AV. When the in-out switching button 67 is clicked in the state shown in Figure 27, the in-camera is set, as shown in Figure 28. In the imaging area display screen 40 shown in Figure 28, image O4I, which corresponds to the virtual imaging target O4V captured by the virtual imaging device 500AV, is displayed. The user can easily set whether to use the in-camera or the out-camera at each teaching point, and can easily understand which setting is made by looking at the display on the imaging area display screen 40.

[0136] The registration button 68 is a UI that registers information set by the imaging mode setting bar 61, zoom magnification setting button 62, flash setting button 63, live imaging setting button 64, aspect ratio setting button 65, timer setting button 66, and in / out switching button 67, associating it with the teaching point.

[0137] As described above, according to the seventh embodiment, when the imaging device is a mobile terminal such as a smartphone, the UI for imaging settings on the robot simulator is partially the same as the UI of the mobile terminal. This allows the user to intuitively configure imaging settings. Furthermore, by checking the imaging information setting screen 60, the user can easily understand what imaging settings are made at each teaching point. Thus, the user can efficiently design the operation of the robot 100 and the operation of the imaging device 500A. In addition, the various embodiments and modifications described above may be implemented in combination with this embodiment and / or this modification.

[0138] This disclosure is not limited to the embodiments described above, and many variations are possible within the technical concept of this disclosure. For example, different embodiments described above may be combined and implemented. Furthermore, the effects described in this embodiment are merely a list of the most preferred effects arising from the embodiments of this disclosure, and the effects of the embodiments of this disclosure are not limited to those described in this embodiment.

[0139] The embodiments described above describe a case where the robot is a vertically articulated robot arm, but the invention is not limited to this. The robot may be various types of robots, such as a horizontally articulated robot arm, a parallel link robot arm, or a Cartesian robot. Furthermore, this disclosure is applicable to machines that can automatically perform movements such as extension and retraction, bending and straightening, vertical movement, horizontal movement, or rotation, or combinations thereof, based on information stored in a memory device provided in the control device.

[0140] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0141] The above disclosure of embodiments includes the following sections.

[0142] (Section 1) It includes a processing unit that simulates the coordinated movement of a virtual robot and a virtual imaging device in a virtual space. The processing unit associates the setting information of the virtual imaging device with the teaching points of the virtual robot. An information processing device characterized by the following:

[0143] (Section 2) The processing unit receives input of the setting information to correspond to the teaching point. The information processing apparatus according to item 1, characterized in that

[0144] (Section 3) The processing unit displays an image on the display unit that receives input of the setting information corresponding to the teaching point. The information processing apparatus according to item 1 or 2, characterized in that

[0145] (Section 4) The image that accepts the input of the aforementioned setting information includes a first button used to read the aforementioned setting information from the storage unit. The information processing apparatus according to item 3, characterized in that

[0146] (Section 5) The image that accepts the input of the aforementioned setting information includes a second button used to save the aforementioned setting information to the storage unit. The information processing apparatus according to item 4, characterized in that

[0147] (Section 6) The processing unit, while the virtual imaging device is moved in conjunction with the virtual robot that has been moved to the teaching point, displays the image obtained by the virtual imaging operation of the virtual imaging device based on the setting information on the display unit. An information processing apparatus according to any one of claims 1 to 5, characterized in that

[0148] (Section 7) The image corresponding to the virtual robot that has been operated on the aforementioned teaching point is displayed on the display unit. An information processing apparatus according to any one of items 1 to 6, characterized in that

[0149] (Section 8) Multiple teaching points are provided, each of which is a teaching point, and the setting information is uniquely associated with each of the multiple teaching points. An information processing apparatus according to any one of items 1 to 7, characterized in that

[0150] (Section 9) The aforementioned setting information includes information on the field of view in the virtual imaging device. An information processing apparatus according to any one of claims 1 to 8, characterized in that

[0151] (Section 10) The virtual imaging device is installed in a virtual mobile terminal. The processing unit associates the setting information, which is set by the functions of the virtual mobile terminal, with the teaching point. An information processing apparatus according to any one of items 1 to 9, characterized in that

[0152] (Section 11) The aforementioned setting information includes information on the imaging direction in the virtual imaging device. The information processing apparatus according to item 10, characterized in that

[0153] (Section 12) The virtual imaging device comprises a first virtual camera that virtually images facing a first direction, and a second virtual camera that virtually images facing a second direction opposite to the first direction. The processing unit associates information indicating whether to use the first virtual camera or the second virtual camera with the teaching point. The information processing apparatus according to item 10 or 11, characterized in that

[0154] (Section 13) The first virtual camera is a virtual front camera, and the second virtual camera is a virtual rear camera. The information processing apparatus according to item 12, characterized in that

[0155] (Section 14) The processing unit can set the zoom magnification of the virtual imaging device to 1.0x or less. An information processing apparatus according to any one of claims 10 to 13, characterized in that

[0156] (Section 15) The processing unit modifies the image obtained by the virtual imaging operation of the virtual imaging device in accordance with the change in the orientation of the virtual mobile terminal when the orientation of the virtual mobile terminal is changed. An information processing apparatus according to any one of claims 10 to 14, characterized in that

[0157] (Section 16) The processing unit can configure multiple types of the virtual mobile terminals. An information processing apparatus according to any one of claims 10 to 15, characterized in that

[0158] (Section 17) The aforementioned virtual mobile terminal is a virtual smartphone, The processing unit displays the user interface for causing the virtual imaging device of the virtual smartphone to perform a virtual imaging operation, along with the image obtained from the virtual imaging operation of the virtual imaging device of the virtual smartphone. An information processing apparatus according to any one of claims 10 to 16, characterized in that

[0159] (Section 18) The processing unit can configure at least one of the following via the user interface: setting the imaging mode, setting the zoom magnification, setting the flash, setting the live imaging, setting the aspect ratio, setting the timer, and setting the front or rear camera. The information processing apparatus according to item 17, characterized in that

[0160] (Section 19) Robots and, The imaging device held by the robot, A control device for controlling the robot and the imaging device, A data processing device comprising: A robotic system characterized by the following features.

[0161] (Section 20) The control device controls the robot based on the teaching points and controls the imaging device based on the setting information associated with the teaching points. The robot system according to item 19, characterized in that

[0162] (Section 21) An information processing method that simulates a virtual robot and a virtual imaging device moving in conjunction with each other in a virtual space, The setting information of the virtual imaging device is associated with the teaching points of the virtual robot. An information processing method characterized by the following:

[0163] (Section 22) A robotic system used to manufacture articles as described in item 19 or 20, A method for manufacturing an article, characterized by the following:

[0164] (Section 23) A program that causes a computer to execute the information processing method described in item 21.

[0165] (Section 24) A computer-readable recording medium on which the program described in item 23 is recorded. [Explanation of symbols]

[0166] 100...Robot, 100V...Virtual robot, 300...Information processing device, 311...CPU (Processing unit), 500...Imaging device, 500V...Virtual imaging device, 1000...Robot system

Claims

1. An information processing method for programming the operation of a robot, comprising: setting an imaging condition of an imaging device that moves together with the robot in association with a teaching point for the robot; 1. An information processing method comprising:

2. Accepting input of the imaging conditions to be associated with the teaching points.

2. The information processing method according to claim 1,

3. An image for accepting input of the imaging conditions to be associated with the teaching point is displayed on a display unit.

2. The information processing method according to claim 1,

4. the image for accepting input of the imaging conditions includes a first button used to read out the imaging conditions from a storage unit; 4. The information processing method according to claim 3.

5. the image for accepting input of the imaging conditions includes a second button used to save the imaging conditions in a storage unit; 5. The information processing method according to claim 4.

6. The robot's movements can be programmed by simulation; In the simulation, in a state where the imaging device is moved in conjunction with the robot operated to the teaching point, an image obtained by a virtual imaging operation of the imaging device based on the imaging conditions is displayed on a display unit.

2. The information processing method according to claim 1,

7. The robot's movements can be programmed by simulation; an image corresponding to the robot operated to the teaching point in the simulation is displayed on a display unit; 2. The information processing method according to claim 1,

8. a plurality of teaching points are provided, each of which is the teaching point, and the information on the imaging conditions is uniquely associated with each of the plurality of teaching points; 2. The information processing method according to claim 1,

9. the imaging conditions include information on an angle of view of the imaging device; 2. The information processing method according to claim 1,

10. The robot's movements can be programmed by simulation; In the simulation, the imaging device is installed in a mobile terminal.

2. The information processing method according to claim 1,

11. The imaging conditions include information on an imaging direction of the imaging device.

11. The information processing method according to claim 10.

12. the imaging device includes a first camera that faces a first direction and captures an image, and a second camera that faces a second direction different from the first direction and captures an image; the imaging condition indicating the camera to be used out of the first camera and the second camera is associated with the teaching point; 11. The information processing method according to claim 10.

13. A display unit displays whether the first camera or the second camera is set at the teaching point.

13. The information processing method according to claim 12.

14. The first camera is an in-camera, and the second camera is an out-camera.

13. The information processing method according to claim 12.

15. The zoom magnification of the imaging device can be set to 1.0x or less.

11. The information processing method according to claim 10.

16. When the attitude of the mobile terminal is changed, an image obtained by the imaging operation of the imaging device is changed in accordance with the change in attitude.

11. The information processing method according to claim 10.

17. A plurality of types of the mobile terminal can be set.

11. The information processing method according to claim 10.

18. the mobile terminal is a smartphone, a user interface for causing the imaging device of the smartphone to perform an imaging operation, together with an image obtained by the imaging operation of the imaging device of the smartphone; 11. The information processing method according to claim 10.

19. The user interface allows the user to set at least one of the following: imaging mode setting, zoom magnification setting, flash setting, live imaging setting, aspect ratio setting, timer setting, and in-camera or out-camera setting.

20. The information processing method according to claim 18,

20. An information processing device for programming the operation of a robot, comprising: setting an imaging condition of an imaging device that moves together with the robot in association with a teaching point for the robot; 1. An information processing device comprising:

21. A robot in real space, an imaging device in real space held by the robot in real space; a control device that controls the robot in the real space and the imaging device in the real space; and the information processing device according to claim 20. A robot system characterized by:

22. the control device controls the robot in the real space based on the teaching points, and controls the imaging device in the real space based on the imaging conditions associated with the teaching points.

22. The robotic system of claim 21.

23. 20. An information processing method according to claim 1, wherein a robot in real space is controlled based on data programmed using the information processing method according to claim 1 to manufacture an article. A method for manufacturing an article.

24. A program for causing a computer to execute the information processing method according to any one of claims 1 to 19.

25. A computer-readable recording medium on which the program according to claim 24 is recorded.

Images