Robot controlling system, method and program

Abstract

To provide a system capable of shortening a cycle time.SOLUTION: A system according to the invention includes a robot, a three-dimensional camera, a robot controller, a determination unit, and an output unit. The determination unit determines one or more imaging position candidates of the three-dimensional camera while the robot is picking up and placing a target object. The output unit outputs, to the robot controller, an inquiry as to whether or not the three-dimensional camera can be moved to each of the one or more imaging position candidates while the robot is picking up and placing the target object. The robot controller determines a next imaging position of the three-dimensional camera in response to a confirmation result for the inquiry while the robot is picking and placing the target object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a system, method, and program for controlling a robot. [Background technology]

[0002] Conventionally, there is known a system that measures the three-dimensional shapes of multiple workpieces based on images obtained by a three-dimensional camera and controls the picking and placing of the target workpieces by a robot based on the measurement results. Such a system repeats a cycle including capturing images with a three-dimensional camera, measuring the three-dimensional shapes, determining the target workpieces, and picking and placing the target workpieces.

[0003] Japanese Patent Application Laid-Open No. 2020-040143 (Patent Document 1) discloses a manipulator including a robot arm that performs a gripping operation on an object and an imaging unit provided on the robot arm. The manipulator analyzes an image obtained by the imaging unit and determines the next imaging position before the next gripping operation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-040143 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the system described in Patent Document 1, the next imaging position is determined each time from an image obtained from the imaging unit, so the process of determining the imaging position is included in the above cycle. Therefore, the process of determining the imaging position may affect the time required for the above cycle (hereinafter referred to as the "cycle time"). For example, the cycle time may become longer.

[0006] The present disclosure has been made in consideration of the above-mentioned circumstances, and its purpose is to provide a system, method, and program that can shorten the cycle time from pick-and-placement of one object to pick-and-placement of the next object. [Means for solving the problem]

[0007] According to an example of the present disclosure, a system includes a robot, a three-dimensional camera, a robot controller, an object recognition unit, a command unit, a determination unit, and an output unit. The robot has an end effector that picks up and places an object, and an arm that moves the end effector. The three-dimensional camera is attached to the arm. The robot controller controls the operation of the robot. The object recognition unit recognizes the position and orientation of one or more objects in an input image based on an input image obtained based on shooting by the three-dimensional camera. The command unit selects an object from among the one or more objects, and outputs a command to the robot controller to pick up the object. The determination unit determines one or more next imaging position candidates of the three-dimensional camera based on the input image while the robot is picking up and placing the object in response to the command. The output unit outputs a first inquiry to the robot controller for each of the one or more imaging position candidates as to whether or not the three-dimensional camera can be moved to the imaging position candidate while the robot is picking up and placing the object. The robot controller or the output unit determines the next imaging position of the three-dimensional camera according to the confirmation result of the first inquiry while the robot is picking and placing the object.

[0008] According to this disclosure, the next imaging position of the 3D camera 200 is determined while the robot is picking and placing the object. This allows the system to immediately start moving the 3D camera to the next imaging position after the pick and place of the workpiece to be picked is completed. As a result, the system can reduce the cycle time from pick and place of one object to pick and place of the next object.

[0009] In the above disclosure, the one or more imaging location candidates include a plurality of imaging location candidates, and the first query is appended with a list of the plurality of imaging location candidates.

[0010] According to this disclosure, the robot controller can determine whether or not to move the three-dimensional camera to each of a plurality of candidate imaging positions based on the list.

[0011] In the above disclosure, the one or more imaging position candidates include a plurality of imaging position candidates. The system further includes a setting unit that sets a priority for each of the plurality of imaging position candidates based on the input image. The output unit transmits the first inquiry to the robot controller in descending order of priority.

[0012] According to this disclosure, the system can quickly determine whether or not to move the three-dimensional camera to a candidate imaging position with high priority.

[0013] In the above disclosure, when the three-dimensional camera cannot be moved to any of the one or more imaging position candidates, the output unit transmits a second inquiry to the robot controller as to whether the three-dimensional camera can be moved to a midpoint between the latest imaging position of the three-dimensional camera and any of the one or more imaging position candidates. The robot controller or the output unit determines the next imaging position according to the confirmation result of the second inquiry.

[0014] According to this disclosure, even if the 3D camera cannot be moved to one or more of the imaging position candidates, the system can determine an intermediate point between the latest imaging position and the imaging position candidates as an alternative imaging position candidate.

[0015] In the above disclosure, when the three-dimensional camera cannot be moved to any of one or more imaging position candidates, the robot controller or the output unit determines the latest imaging position of the three-dimensional camera as the next imaging position.

[0016] According to this disclosure, the system can quickly determine the next imaging position even if the 3D camera cannot be moved to one or more candidate imaging positions.

[0017] In the above disclosure, the robot controller, the object recognition unit, the command unit, the decision unit, and the output unit may be realized by the same processor.

[0018] According to another example of the present disclosure, a method for controlling a robot includes the following first to fifth steps. The robot has an end effector for picking and placing an object, an arm for moving the end effector, and a three-dimensional camera attached to the arm. The first step is for one or more processors to recognize positions and orientations of one or more objects in an input image based on an input image obtained based on shooting by the three-dimensional camera. The second step is for one or more processors to select an object from among the one or more objects and control the robot to pick up the object. The third step is for one or more processors to determine one or more next imaging position candidates for the three-dimensional camera based on the input image while the robot is picking and placing the object. The fourth step is for one or more processors to check, for each of the one or more imaging position candidates while the robot is picking and placing the object, whether or not the three-dimensional camera can be moved to the imaging position candidate while the robot is picking and placing the object. The fifth step is for one or more processors to determine a next imaging position for the three-dimensional camera according to a result of checking while the robot is picking and placing the object.

[0019] According to yet another example of the present disclosure, a program causes a computer to execute the above method. These disclosures also allow the method or program to reduce the cycle time from pick-and-place of one object to pick-and-place of the next object. Effect of the Invention

[0020] In accordance with the present disclosure, systems, methods and programs can reduce the cycle time from pick and place of one object to pick and place of the next object. [Brief description of the drawings]

[0021] [Figure 1] 1 is a schematic diagram showing an overall configuration of a system according to an embodiment; [Diagram 2] 2 is a schematic diagram illustrating an example of a hardware configuration of the image processing device illustrated in FIG. 1. [Diagram 3] FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of a robot controller according to the present embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of an image processing device and a robot controller. [Diagram 5] FIG. 11 is a diagram illustrating a method for generating a model image. [Figure 6] 10 is a diagram illustrating an example of a recognition result stored in a recognition result storage unit. FIG. [Figure 7] FIG. 13 is a diagram illustrating an example of a model image set. [Figure 8] 2 is a flowchart showing an example of a process flow of the system shown in FIG. 1. [Figure 9] 9 is a flowchart showing a process flow of a first embodiment of a subroutine of step S8 shown in FIG. 8. [Figure 10] FIG. 2 is a diagram illustrating an example of a plurality of virtual viewpoints. [Figure 11] 11 is a diagram showing an example of a virtual depth map seen from the virtual viewpoint shown in FIG. 10. FIG. [Figure 12] 10 is a flowchart showing a second embodiment of the subroutine of step S8 shown in FIG. 8. [Figure 13] 11 is a diagram showing the positional relationship between a virtual viewpoint and a specific workpiece. FIG. [Figure 14] 11 is a diagram showing an example of a model image corresponding to the posture of a specific workpiece when viewed from a virtual viewpoint. FIG. [Figure 15] 13 is a diagram showing an example in which the posture of a specific workpiece is erroneously recognized; FIG. [Figure 16] FIG. 16 is a diagram showing an example of a first model image obtained from the erroneous recognition result shown in FIG. 15. [Figure 17] FIG. 13 is a diagram showing an example of a recognition result recorded in a third embodiment of the subroutine of step S8. [Figure 18] 10 is a flowchart showing a third embodiment of the subroutine of step S8 shown in FIG. 8. [Figure 19] 10 is a flowchart showing a fourth embodiment of the subroutine of step S8 shown in FIG. 8. [Figure 20] 10 is a flowchart showing a fifth embodiment of the subroutine of step S8 shown in FIG. 8. [Figure 21] 9 is a flowchart showing a first embodiment of a subroutine of step S9 shown in FIG. 8. [Figure 22] 22 is a flowchart showing a first embodiment of a subroutine of step S909 shown in FIG. 21. [Diagram 23] 22 is a flowchart showing a second embodiment of the subroutine of step S909 shown in FIG. 21. [Figure 24] 10 is a flowchart showing a second embodiment of the subroutine of step S9 shown in FIG. 8. [Diagram 25] 25 is a flowchart showing a second embodiment of the subroutine of step S936 shown in FIG. 24. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference characters and the description thereof will not be repeated.

[0023] §1 Examples of application An example of a situation in which the present invention is applied will be described with reference to FIG. 1. FIG. 1 is a schematic diagram showing the overall configuration of a system according to an embodiment. The system 1 measures the three-dimensional shapes of a plurality of workpieces 2 randomly piled on a tray 3. The workpieces 2 are an example of an "object" in the present disclosure. Based on the measurement results, the system 1 selects a workpiece 2 to be picked (hereinafter referred to as a "workpiece to be picked") from the plurality of workpieces 2, and performs pick-and-place for the workpiece to be picked. The workpiece to be picked is an example of an "object" in the present disclosure.

[0024] As shown in FIG. 1, the system 1 includes an image processing device 100, a three-dimensional camera 200, a robot 300, and a robot controller 400.

[0025] The three-dimensional camera 200 captures an image of a subject including a plurality of workpieces 2 randomly stacked on a tray 3. The three-dimensional camera 200 includes a projection unit 201 and an image capture unit 202. The projection unit 201 projects an arbitrary projection pattern light onto the subject in accordance with a projection instruction from the image processing device 100. The projection pattern is, for example, a pattern in which brightness changes periodically along a predetermined direction within an irradiation surface. The image capture unit 202 captures an image of the subject with the projection pattern light projected thereon in accordance with an image capture instruction from the image processing device 100.

[0026] The projection unit 201 includes, as main components, a light source such as an LED (Light Emitting Diode) or a halogen lamp, and a filter arranged on the side of the irradiation surface of the projection unit 201. The filter generates a projection pattern light required for measuring a three-dimensional shape as described later, and can arbitrarily change the light transmittance in the surface according to an instruction from the image processing device 100. The projection unit 201 may project a fixed projection pattern light that does not change over time, or may project a projection pattern light that changes over time. The projection unit 201 may be configured using a liquid crystal or a DMD (Digital Mirror Device) and a light source (such as an LED or laser light source).

[0027] The photographing unit 202 includes, as main components, an optical system such as a lens, and an imaging element such as a CCD (Coupled Charged Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0028] The robot 300 performs pick-and-place for the workpiece 2. The robot 300 includes a robot arm 301 and a base 302 that supports the robot arm 301. The robot arm 301 is an example of an "arm" in the present disclosure. A three-dimensional camera 200 and an end effector 304 are attached to the tip of the robot arm 301. The end effector 304 is configured to hold the workpiece 2, and includes, for example, a suction pad or a two-fingered hand. The operation of the robot 300 is controlled by a robot controller 400.

[0029] The robot controller 400 controls the operation of the robot 300 according to commands from the image processing device 100. The commands indicate, for example, the position and orientation of the tip of the robot arm 301 (hereinafter referred to as the "tool center point"). The robot controller 400 controls the robot arm 301 so that the position and orientation of the tool sensor point match the commands.

[0030] The image processing device 100 performs image processing on the captured image received from the three-dimensional camera 200. For example, the image processing device 100 may determine the position and orientation to be taken by the tool center point of the robot arm 301 based on the image processing result. The image processing device 100 may generate a command to move the tool center point to the determined position and orientation, and output the command to the robot controller 400.

[0031] The image processing device 100 includes a three-dimensional recognition unit 17, a picking position calculation unit 19, a determination unit 13, and a candidate list management unit 23.

[0032] The three-dimensional recognition unit 17 is an example of an "object recognition unit" in the present disclosure. The three-dimensional recognition unit 17 repeatedly executes an object recognition process that recognizes the position and orientation of the workpiece 2 from a real depth map obtained based on the image captured by the three-dimensional camera 200. The real depth map is an example of an "input image" in the present disclosure, and represents the three-dimensional shape of the subject of the three-dimensional camera 200. The depth map is also called a distance image, and represents the distance from the three-dimensional camera 200 for each pixel.

[0033] The picking position calculation unit 19 is an example of a "command unit" in the present disclosure. The picking position calculation unit 19 selects a workpiece to be picked from one or more objects whose positions and orientations have been recognized by the three-dimensional recognition unit 17, and outputs a command to the robot controller 400 to pick and place the workpiece to be picked. As a result, the robot controller 400 controls the robot 300 to pick and place the workpiece to be picked.

[0034] The determination unit 13 determines a candidate for the next imaging position of the three-dimensional camera 200 based on the actual depth map while the robot 300 is picking and placing the workpiece to be picked in response to a command.

[0035] The candidate list management unit 23 is an example of an "output unit" in the present disclosure. While the robot 300 is picking and placing the workpiece to be picked, the candidate list management unit 23 outputs a first inquiry to the robot controller 400 for each of one or more imaging position candidates as to whether or not the 3D camera 200 can be moved to the imaging position candidate.

[0036] While the robot 300 is picking and placing the workpiece to be picked, the robot controller 400 or the candidate list management unit 23 determines the next imaging position of the three-dimensional camera 200 in accordance with the confirmation result of the first inquiry.

[0037] Thus, according to this embodiment, the next imaging position of the three-dimensional camera 200 is determined while the robot 300 is picking and placing the workpiece to be picked. This allows the system 1 to immediately start moving the three-dimensional camera 200 to the next imaging position after the picking and placing of the workpiece to be picked is completed. As a result, the system 1 can shorten the cycle time from picking and placing of one workpiece to picking and placing of the next workpiece.

[0038] §2 Specific examples <Example of hardware configuration of image processing device> The image processing device 100 is typically a computer having a general-purpose architecture, and executes a pre-installed program (instruction code) to perform the image capture position candidate determination process according to this embodiment. Such a program is typically distributed in a state stored in various recording media, or is installed in the image processing device 100 via a network, etc.

[0039] When such a general-purpose computer is used, an OS (Operating System) for executing basic processing of the computer may be installed in addition to an application for executing the process of determining the imaging position candidate according to the present embodiment. In this case, the program according to the present embodiment may call necessary modules among program modules provided as part of the OS in a predetermined sequence at a predetermined timing to execute the processing. That is, the program according to the present embodiment itself may not include the above-mentioned modules, and may execute the processing in cooperation with the OS. The program according to the present embodiment may be in a form that does not include some of these modules.

[0040] Furthermore, the program according to the present embodiment may be provided by being incorporated into a part of another program. In this case, the program itself does not include the modules included in the other program to be combined as described above, and the processing is executed in cooperation with the other program. That is, the program according to the present embodiment may be in a form incorporated into such other program. Note that some or all of the functions provided by the execution of the program may be implemented as a dedicated hardware circuit.

[0041] Fig. 2 is a schematic diagram showing an example of a hardware configuration of the image processing device shown in Fig. 1. As shown in Fig. 2, the image processing device 100 includes a CPU (Central Processing Unit) 110, which is an example of a processor, a main memory 112, a storage 114, a camera interface 116, an input interface 118, a display controller 120, a communication interface 124, and a data reader / writer 126. These components are connected to each other via a bus 128 so as to be able to communicate data with each other.

[0042] The CPU 110 loads the programs 115 installed in the storage 114 into the main memory 112 and executes them in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a dynamic random access memory (DRAM) or a static random access memory (SRAM), and holds images acquired from the three-dimensional camera 200 in addition to the programs 115 read from the storage 114. Furthermore, the storage 114 is a non-volatile storage device such as a solid state drive (SSD) or a hard disk drive (HDD). The storage 114 stores various data, which will be described later. The storage 114 may include a semiconductor storage device such as a flash memory.

[0043] The camera interface 116 mediates data transmission between the CPU 110 and the three-dimensional camera 200. That is, the camera interface 116 is connected to the three-dimensional camera 200. The camera interface 116 provides projection commands and image capture commands to the three-dimensional camera 200 in accordance with internal commands generated by the CPU 110. The camera interface 116 includes an image buffer 116a for temporarily storing images from the three-dimensional camera 200. When a predetermined number of images are stored in the image buffer 116a, the camera interface 116 transfers the stored images to the main memory 112.

[0044] The input interface 118 mediates data transmission between the CPU 110 and the input device 160. That is, the input interface 118 accepts input information input by a user to the input device 160. The input device 160 is an example of a "user interface" in the present disclosure.

[0045] The display controller 120 is connected to a display device 150, and controls the screen of the display device 150 so as to notify the user of the processing results of the CPU 110, etc.

[0046] The communication interface 124 mediates data transmission between the CPU 110 and an external device such as the robot controller 400. The communication interface 124 typically includes Ethernet (registered trademark) or USB (Universal Serial Bus), or the like.

[0047] The data reader / writer 126 mediates data transmission between the CPU 110 and the memory card 106, which is a recording medium. That is, the memory card 106 is distributed in a state in which a program executed by the image processing device 100 and the like are stored, and the data reader / writer 126 reads the program from the memory card 106. In addition, in response to an internal command from the CPU 110, the data reader / writer 126 writes images captured by the three-dimensional camera 200 and / or processing results in the image processing device 100 to the memory card 106. The memory card 106 includes a general-purpose semiconductor storage device such as a Secure Digital (SD), a magnetic storage medium such as a flexible disk, or an optical storage medium such as a compact disk read only memory (CD-ROM).

[0048] <Example of robot controller hardware configuration> 3 is a schematic diagram showing an example of the hardware configuration of a robot controller according to this embodiment. As shown in FIG. 3, the robot controller 400 includes a network controller 410 and a control processing circuit 420.

[0049] The network controller 410 mainly exchanges data with the image processing device 100 via the network.

[0050] The control processing circuit 420 executes arithmetic processing required to drive the robot 300. As an example, the control processing circuit 420 includes a processor 430, a main memory 440, a storage 450, and an interface circuit 460. The interface circuit 460 exchanges data with the robot 300.

[0051] The processor 430 executes control calculations for driving the robot 300. The main memory 440 is configured with, for example, a volatile storage device such as a DRAM or an SRAM. The storage 450 is configured with, for example, a non-volatile storage device such as an SSD or an HDD.

[0052] The storage 450 stores a system program 452 for implementing control for driving the robot 300. The system program 452 includes instructions for executing control calculations related to the operation of the robot 300, and instructions related to the interface with the robot 300.

[0053] <Functional configuration> Fig. 4 is a diagram showing an example of the functional configuration of the image processing device and the robot controller. As shown in Fig. 4, the image processing device 100 includes a three-dimensional measuring unit 10, a viewpoint conversion unit 11, a missing area calculation unit 12, a determination unit 13, a model image generation unit 14, an assignment unit 15, a three-dimensional recognition unit 17, a recognition result storage unit 18, a picking position calculation unit 19, a range setting unit 20, a priority setting unit 21, a registration unit 22, a candidate list management unit 23, and a communication unit 24.

[0054] The three-dimensional measurement unit 10, viewpoint conversion unit 11, missing area calculation unit 12, determination unit 13, model image generation unit 14, assignment unit 15, three-dimensional recognition unit 17, picking position calculation unit 19, range setting unit 20, priority setting unit 21, registration unit 22, and candidate list management unit 23 are realized by the CPU 110 shown in Fig. 2 executing a program 115. The recognition result storage unit 18 is configured by the main memory 112 or the storage 114. The communication unit 24 is configured by a communication interface 124.

[0055] The three-dimensional measuring unit 10 outputs a projection instruction and an image capturing instruction to the three-dimensional camera 200 in response to the three-dimensional camera 200 reaching the next image capturing position. As a result, the three-dimensional camera 200 captures the multiple workpieces 2 onto which the projection pattern is projected. The three-dimensional measuring unit 10 generates a real depth map based on the captured images received from the three-dimensional camera 200. The three-dimensional measuring unit 10 generates a real depth map using, for example, a triangulation method. As the triangulation method, known techniques such as a fixed pattern projection method, a phase shift method, a stereo matching method, and a light section method can be adopted. For example, the phase shift method is a method that uses an image captured in a state in which a projection pattern light in which the shading is changed sinusoidally is projected onto a subject. A plurality of projection patterns with different shading change periods and phases are prepared, and the projection unit 201 of the three-dimensional camera 200 sequentially irradiates the multiple projection pattern lights onto the field of view of the image capturing unit 202. The three-dimensional measurement unit 10 measures the three-dimensional shape of the visual field area based on the change in brightness (luminance or brightness) in the captured image when multiple projection patterns are projected, and generates a real depth map indicating the measurement results. Note that the three-dimensional measurement process is not limited to the triangulation method, and may be performed using other methods (such as a coaxial method).

[0056] The model image generating unit 14 generates a model image set 16 including a plurality of model images 60 (see FIG. 7) showing a work model corresponding to the work 2, based on CAD (Computer Aided Design) data showing the three-dimensional shape of the work 2.

[0057] 5 is a diagram for explaining a method for generating a model image. The model image generating unit 14 places a work model 2M having a three-dimensional shape represented by CAD data in a virtual space. Furthermore, the model image generating unit 14 sets a plurality of points P on a spherical surface centered on the work model 2M. The distance between the plurality of points P and the work model 2M is constant. The model image generating unit 14 generates a depth map of the work model 2M when viewed from each of the plurality of points P as a model image 60.

[0058] 4, every time a real depth map is generated, the three-dimensional recognition unit 17 executes an object recognition process to recognize the position and orientation of the workpiece 2 from the real depth map by template matching using a plurality of model images 60. The three-dimensional recognition unit 17 stores the recognition result of the position and orientation of the workpiece 2 in the recognition result storage unit 18.

[0059] Template matching is a technique for searching for an area similar to a model image in a target image (a real depth map in this embodiment). The three-dimensional recognition unit 17 calculates feature amounts from each of the real depth map and the model image 60, and calculates a score (hereinafter referred to as a "matching score") that indicates the degree of match between the feature amounts. The three-dimensional recognition unit 17 recognizes the position and orientation of the workpiece 2 in the real depth map by searching for an area similar to the model image 60 in the real depth map based on the matching score.

[0060] Fig. 6 is a diagram showing an example of the recognition result stored in the recognition result storage unit. As shown in Fig. 6, the recognition result indicates, for example, for each workpiece 2, the recognized position and orientation, and a model image ID that identifies the model image having the highest matching score with the area including the workpiece 2 in the actual depth map.

[0061] 4, the assigning unit 15 assigns a score (hereinafter referred to as a "recognition ease score") representing the recognizability of the posture of the workpiece 2 in template matching to each of the multiple model images 60. The model image set 16 includes the recognizability score assigned to each model image 60.

[0062] Fig. 7 is a diagram showing an example of a model image set 16. As shown in Fig. 7, the model image set 16 includes a plurality of model images 60 to which recognizability scores have been assigned.

[0063] The recognizability is determined based on the uniqueness of the model image 60 and the area (size) of the work model 2M in the model image 60.

[0064] For example, when the work model 2M is spherical, the shape of the work model 2M viewed from multiple points P shown in FIG. 5 is substantially the same. In this case, the uniqueness of each model image 60 is small. When a model image 60 with low uniqueness is used, the three-dimensional recognition unit 17 may not be able to correctly recognize the posture of the work 2. Therefore, the assignment unit 15 evaluates the ease of recognition based on the uniqueness of the model image 60. The uniqueness of a certain model image 60 is greater the lower the similarity with other model images 60. Therefore, the assignment unit 15 calculates a score (hereinafter referred to as a "uniqueness score") that represents the degree of deviation between the feature amount calculated from the target model image 60 and the feature amount calculated from the other model images 60 as an index representing the uniqueness of the target model image 60.

[0065] When the area of ​​the work model 2M in the model image 60 is large, the feature amount calculated from the model image 60 better represents the features of the work model 2M. Therefore, when a model image 60 in which the area of ​​the work model 2M is large is used, the three-dimensional recognition unit 17 can more easily correctly recognize the posture of the work 2. Therefore, the assignment unit 15 evaluates the ease of recognition based on the area of ​​the work model 2M in the model image 60. The area of ​​the work model 2M is represented, for example, by the number of pixels in which the work model 2M exists in the model image 60.

[0066] The assigning unit 15 calculates the recognizability score by inputting the uniqueness score of each model image 60 and the area of ​​the work model 2M into a predetermined relational expression. The relational expression is determined so that the larger the uniqueness score, the larger the recognizability score, and the larger the area of ​​the work model 2M, the larger the recognizability score.

[0067] The assigning unit 15 may assign a recognizability score to each model image 60 according to a user input. In this case, the user may check a plurality of model images 60 and input a recognizability score for each model image 60 in consideration of the recognizability of the work 2 in template matching.

[0068] 4, the picking position calculation unit 19 selects a workpiece to be picked by the robot arm 301 based on the latest recognition result stored in the recognition result storage unit 18. The picking position calculation unit 19 calculates the position and orientation of the tool center point when picking up the workpiece to be picked (hereinafter referred to as the "picking position and orientation").

[0069] The picking position calculation unit 19 specifies the position and orientation of the workpiece to be picked from the recognition result. For example, when the recognition result indicates the positions and orientations of multiple works 2, the picking position calculation unit 19 specifies the workpiece 2 having the position and orientation with the highest matching score as the workpiece to be picked.

[0070] The picking position calculation unit 19 calculates the picking position and posture so that the relative position and posture of the end effector 304 with respect to the workpiece to be picked becomes a pre-registered position and posture (hereinafter referred to as the "registered position and posture"). The registered position and posture are registered in advance based on, for example, CAD data indicating the shape of the workpiece 2 and CAD data indicating the shape of the end effector 304.

[0071] The picking position calculation unit 19 generates a pick-and-place command to which the picking position and orientation are added, and outputs the pick-and-place command to the robot controller 400 via the communication unit 24.

[0072] The viewpoint conversion unit 11 converts the real depth map into a plurality of virtual depth maps viewed from a plurality of virtual viewpoints. The plurality of virtual viewpoints are arranged on a spherical surface so that the distance from a reference point (e.g., the center of the tray 3) is constant. The plurality of virtual viewpoints are arranged, for example, at equal intervals on the spherical surface.

[0073] The range setting unit 20 sets the movable range of the three-dimensional camera 200 in response to an input to the input device 160. When the movable range is set by the range setting unit 20, the viewpoint conversion unit 11 places a plurality of virtual viewpoints within the movable range.

[0074] For example, the user can set the movable range so as to exclude positions that are obviously inappropriate for photographing the workpieces 2 randomly piled on the tray 3, depending on the position of the tray 3. This prevents a position that is obviously inappropriate for photographing the workpieces 2 from being selected as the virtual viewpoint. If a position that is obviously inappropriate for photographing the workpieces 2 is selected as the virtual viewpoint, the generation of a virtual depth map corresponding to the virtual viewpoint and the processing of the virtual depth map corresponding to the virtual viewpoint are unnecessary. Therefore, by setting the movable range, the time required for such unnecessary processing can be reduced.

[0075] The missing area calculation unit 12 calculates the area of ​​a missing region where distance information is missing for each of the multiple virtual depth maps.

[0076] The determination unit 13 may determine one or more virtual viewpoints corresponding to one or more virtual depth maps having a relatively large area of ​​the missing region among the multiple virtual depth maps as one or more next imaging position candidates for the three-dimensional camera 200. Alternatively, the determination unit 13 may select one or more imaging position candidates based on the recognizability of the posture of the workpiece 2 when viewed from the virtual viewpoint. The recognizability is evaluated based on the recognizability score assigned to the model image 60 corresponding to the estimated posture of the workpiece 2 when viewed from the virtual viewpoint.

[0077] When an inappropriate position is registered by the registration unit 22, the determination unit 13 determines a position other than the inappropriate position as the imaging position candidate.

[0078] The registration unit 22 registers the inappropriate position. For example, the registration unit 22 registers the latest image capture position as an inappropriate position in response to receiving a notification from the robot controller 400 indicating that picking has failed. Furthermore, the registration unit 22 may register the inappropriate position based on the recognition result stored in the recognition result storage unit 18. A method of registering an inappropriate position based on the recognition result will be described later in detail.

[0079] The priority setting unit 21 sets a priority for each of the one or more imaging position candidates determined by the determination unit 13 based on one or more indices. The one or more indices may include the area of ​​the missing area. The one or more indices may include the movement amount of the robot arm 301 for moving the three-dimensional camera to the imaging position candidate. The movement amount is calculated according to the difference between the position and posture of the robot arm 301 after pick-and-place and the position and posture of the robot arm 301 when the three-dimensional camera 200 is located at the imaging position candidate. The one or more indices may include the recognizability of the workpiece 2 when viewed from the imaging position candidate. The recognizability is evaluated based on the recognizability score assigned to the model image 60 corresponding to the estimated posture of the workpiece 2 when viewed from the imaging position candidate.

[0080] The candidate list management unit 23 manages a list of one or more imaging position candidates. Furthermore, the candidate list management unit 23 manages a priority set for each imaging position candidate. The candidate list management unit 23 generates a first inquiry as to whether the three-dimensional camera 200 can move to the imaging position candidate, and outputs the inquiry to the robot controller 400 via the communication unit 24.

[0081] The candidate list management unit 23 may select one imaging position candidate from the list in descending order of priority and generate a first inquiry regarding the selected imaging position candidate. In this case, the candidate list management unit 23 selects the imaging position candidate with the highest priority from the list and generates a first inquiry regarding the selected imaging position candidate. Thereafter, the candidate list management unit 23 receives a notification indicating the confirmation result of whether or not movement is possible from the robot controller 400 via the communication unit 24. If the confirmation result indicates that movement is not possible, the candidate list management unit 23 may delete from the list the imaging position candidate to which the three-dimensional camera 200 cannot move, and generate a new first inquiry regarding the imaging position candidate with the highest priority among the remaining imaging position candidates. If the confirmation result indicates that movement is possible, the candidate list management unit 23 stops generating the new first inquiry.

[0082] Alternatively, the candidate list management unit 23 may generate a first inquiry regarding all of one or more imaging position candidates included in the list. In this case, the candidate list management unit 23 receives a notification indicating the confirmation result of whether or not movement is possible from the robot controller 400 via the communication unit 24. The notification indicates that movement is possible and indicates the imaging position candidate that has been determined as the next imaging position.

[0083] If the three-dimensional camera 200 cannot be moved to all of the one or more imaging position candidates, the candidate list management unit 23 may determine a position other than the one or more imaging position candidates as the next imaging position. In this case, the candidate list management unit 23 generates a movement command for moving the three-dimensional camera 200 to the next imaging position, and outputs the movement command to the robot controller 400 via the communication unit 24.

[0084] The communication unit 24 transmits and receives data to and from an external device. Specifically, the communication unit 24 transmits a pick-and-place command generated by the picking position calculation unit 19 to the robot controller 400. The communication unit 24 transmits an inquiry or a movement command generated by the candidate list management unit 23 to the robot controller 400. Furthermore, the communication unit 24 receives from the robot controller 400 a notification indicating that the three-dimensional camera 200 has reached the next imaging position, a notification indicating that picking has failed, or a notification indicating the confirmation result of whether or not movement is possible.

[0085] As shown in Fig. 4, the robot controller 400 includes a communication unit 41, a movement possibility confirmation unit 42, and a control unit 43. The communication unit 41 is configured by the network controller 410 shown in Fig. 3. The movement possibility confirmation unit 42 and the control unit 43 are configured by a control processing circuit 420.

[0086] The communication unit 41 transmits and receives data to and from an external device. Specifically, the communication unit 41 receives a pick-and-place command, an inquiry, or a movement command from the image processing device 100. When the three-dimensional camera 200 reaches the next imaging position, the communication unit 41 transmits a notification indicating that the three-dimensional camera 200 has reached the next imaging position to the image processing device 100. When the robot 300 fails to pick up, the communication unit 41 transmits a notification indicating that picking has failed to the image processing device 100. Furthermore, the communication unit 41 transmits a notification indicating the confirmation result by the movement feasibility confirmation unit 42 to the image processing device 100.

[0087] The movement feasibility confirmation unit 42 determines whether or not the three-dimensional camera 200 can be moved to the imaging position candidate in accordance with the inquiry received from the image processing device 100. The movement feasibility confirmation unit 42 determines whether or not the three-dimensional camera 200 can be moved to the imaging position candidate based on the kinematics of the robot 300. The movement feasibility confirmation unit 42 outputs the confirmation result to the image processing device 100 via the communication unit 41.

[0088] The movement possibility confirmation unit 42 determines a candidate imaging position to which the three-dimensional camera 200 can move as the next imaging position, and outputs a movement command to the control unit 43 to move the three-dimensional camera 200 to the determined imaging position.

[0089] The control unit 43 controls the operation of the robot 300. Specifically, the control unit 43 controls the robot 300 to perform a series of operations in response to a pick-and-place command, such as picking up a workpiece to be picked and placing the workpiece to be picked in a predetermined location.

[0090] The control unit 43 monitors the state of the end effector 304 and determines whether the picking operation has been successful or not according to the state of the end effector 304. In response to determining that picking has failed, the control unit 43 generates a notification indicating that picking has failed and outputs the notification to the image processing device 100 via the communication unit 41.

[0091] Furthermore, the control unit 43 controls the robot 300 so that the three-dimensional camera 200 moves to the next imaging position in response to a movement command received from the image processing device 100 or the movement feasibility confirmation unit 42. In response to the three-dimensional camera 200 reaching the next imaging position, the control unit 43 generates a notification indicating that the three-dimensional camera 200 has reached the next imaging position. The control unit 43 outputs the generated notification to the image processing device 100 via the communication unit 41.

[0092] <System processing flow> Fig. 8 is a flowchart showing an example of the flow of processing of the system shown in Fig. 1. As shown in Fig. 8, first, in step S1, the CPU 110 operating as the range setting unit 20 of the image processing device 100 sets the movable range of the three-dimensional camera 200 in response to a user input to the input device 160.

[0093] In the next step S2, the CPU 110 outputs a command to the robot controller 400 to move the three-dimensional camera 200 to a predetermined initial position. The initial position is, for example, vertically above the center of the tray 3. The control processing circuit 420 operating as the control unit 43 of the robot controller 400 controls the robot 300 so that the three-dimensional camera 200 moves to the initial position. As a result, the three-dimensional camera 200 moves to the initial position.

[0094] In the next step S3, the CPU 110 operating as the three-dimensional measuring unit 10 of the image processing device 100 outputs a projection instruction and an image capturing instruction to the three-dimensional camera 200. As a result, the three-dimensional camera 200 captures an image of the subject (which may include the workpieces 2 randomly piled on the tray 3) onto which the projection pattern is projected.

[0095] In the next step S4, CPU 110 measures the three-dimensional shape of the subject based on the captured image obtained by three-dimensional camera 200. CPU 110 generates a real depth map indicating the three-dimensional shape of the subject.

[0096] In the next step S5, the CPU 110 operating as the three-dimensional recognition unit 17 of the image processing device 100 performs object recognition processing to recognize the position and orientation of the workpiece 2 from the real depth map by template matching using a plurality of model images 60. The CPU 110 records the recognition result as shown in FIG.

[0097] In the next step S6, the CPU 110 operating as the picking position calculation unit 19 of the image processing device 100 judges whether or not the positions and orientations of one or more workpieces 2 have been recognized. Specifically, the CPU 110 judges whether or not the latest recognition result indicates the positions and orientations of one or more workpieces 2.

[0098] If the position and orientation of one or more workpieces 2 cannot be recognized (NO in step S6), the system 1 ends the process.

[0099] If one or more positions and orientations of the workpieces 2 have been recognized (YES in step S6), the system 1 executes step S7 and steps S8 and S9 in parallel.

[0100] In step S7, the CPU 110 selects a workpiece to be picked from one or more workpieces 2 whose positions and orientations have been recognized. The CPU 110 calculates the position and orientation (picking position and orientation) of the tool center point when picking the workpiece to be picked, and outputs a pick-and-place command to which the picking position and orientation has been added, to the robot controller 400. The control processing circuit 420, which operates as the control unit 43 of the robot controller 400, controls the robot 300 to pick and place the workpiece to be picked in response to the pick-and-place command.

[0101] In step S8, the CPU 110 of the image processing device 100 determines one or more next imaging position candidates for the three-dimensional camera 200. Step S8 is executed while step S7 is being executed.

[0102] In the next step S9, the system 1 determines the next imaging position of the three-dimensional camera 200. Step S9 is also performed while step S7 is being performed.

[0103] When steps S7 to S9 are completed, the system 1 advances the process to step S10. In step S10, the control processing circuit 420 operating as the control unit 43 of the robot controller 400 controls the robot 300 to move the three-dimensional camera 200 to the next imaging position. After step S10, the system 1 returns the process to step S3.

[0104] 8, step S7 and steps S8 and S9 are executed in parallel, so that the system 1 can move the three-dimensional camera 200 to the next imaging position more quickly. As a result, the cycle time required to execute steps S3 to S10 is shortened.

[0105] <First Example of Subroutine of Step S8> A first embodiment of the subroutine of step S8 will be described with reference to Figures 9 to 11. Figure 9 is a flowchart showing the process flow of a first embodiment of the subroutine of step S8 shown in Figure 8. As shown in Figure 9, the first embodiment of the subroutine of step S8 includes steps S801 to S804.

[0106] First, in step S801, the CPU 110 operating as the viewpoint conversion unit 11 of the image processing device 100 obtains a real depth map corresponding to the latest imaging position.

[0107] In the next step S802, the CPU 110 converts the real depth map into a plurality of virtual depth maps viewed from a plurality of virtual viewpoints.

[0108] In the next step S803, the CPU 110 operating as the missing area calculation unit 12 of the image processing device 100 calculates the area of ​​a missing region where distance information is missing for each of the multiple virtual depth maps.

[0109] In the next step S804, the CPU 110 operating as the determination unit 13 of the image processing device 100 determines one or more virtual viewpoints corresponding to one or more virtual depth maps having a relatively large area of ​​the missing region as one or more next imaging position candidates for the three-dimensional camera 200. For example, the CPU 110 extracts a predetermined number of virtual depth maps having the top areas of the missing regions from among the multiple virtual depth maps, and determines the virtual viewpoints corresponding to each of the extracted virtual depth maps as the imaging position candidates.

[0110] Fig. 10 is a diagram showing an example of a plurality of virtual viewpoints, and Fig. 11 is a diagram showing an example of a virtual depth map viewed from the virtual viewpoints shown in Fig. 10.

[0111] As shown in FIG. 10, the CPU 110 arranges a plurality of virtual viewpoints VP so that the distance from a reference point OP (e.g., the center of the tray 3) coincides with the distance L. In the example shown in FIG. 10, the plurality of virtual viewpoints VP include virtual viewpoints VP_1 and VP_2. The distance L is the distance between the latest imaging position RP of the three-dimensional camera 200 and the reference point OP. As shown in FIG. 11, the CPU 110 converts the real depth map 50 into a virtual depth map 52 seen from each of the plurality of virtual viewpoints VP. In FIG. 11, the virtual depth map 52_1 is a virtual depth map seen from the virtual viewpoint VP_1. The virtual depth map 52_2 is a virtual depth map seen from the virtual viewpoint VP_2.

[0112] In the example shown in FIG. 10, workpieces 2_1 and 2_2 having an L-shaped cross section are present on the tray 3. The three-dimensional camera 200 located at the latest imaging position RP receives reflected light from the surfaces 2a and 2b of the workpiece 2_1 and reflected light from the surfaces 2c and 2d of the workpiece 2_2. Furthermore, the three-dimensional camera 200 receives reflected light from areas 3a, 3b, and 3c of the upper surface of the tray 3. Therefore, the CPU 110 can accurately measure the three-dimensional shapes of the surfaces 2a and 2b of the workpiece 2_1, the surfaces 2c and 2d of the workpiece 2_2, and the areas 3a, 3b, and 3c of the upper surface of the tray 3 based on the captured image received from the three-dimensional camera 200.

[0113] The three-dimensional camera 200 can receive reflected light from a surface 2z of the workpiece 2_1. However, the surface 2z is approximately parallel to an optical axis 250 of the three-dimensional camera 200. Therefore, the CPU 110 cannot measure the three-dimensional shape of the surface 2z.

[0114] When viewed from the three-dimensional camera 200 located at the latest imaging position RP, the area 3z on the upper surface of the tray 3 is hidden by the workpiece 2_1. Therefore, the CPU 110 cannot measure the three-dimensional shape of the area 3z.

[0115] As shown in Fig. 10, the region 3z is visible from the virtual viewpoint VP_1. However, the real depth map does not have distance information for the region 3z. Therefore, as shown in Fig. 11, the virtual depth map 52_1 corresponding to the virtual viewpoint VP_1 has a missing region 56_1 in which distance information is missing. The missing region 56_1 corresponds to the region 3z.

[0116] As shown in Fig. 10, the surface 2z of the workpiece 2_1 is visible from the virtual viewpoint VP_2. Also, when viewed from the virtual viewpoint VP_2, the region 3z is located on the back side of the surface 2z. However, the real depth map does not have distance information for the surface 2z and the region 3z. Therefore, as shown in Fig. 11, the virtual depth map 52_2 corresponding to the virtual viewpoint VP_2 has a missing region 56_2 where distance information is missing. The missing region 56_2 corresponds to the surface 2z (and the region 3z on the back side of the surface 2z).

[0117] The CPU 110 may calculate the areas of the defective regions 56_1 and 56_2, and determine the virtual viewpoint VP corresponding to the virtual depth map 52 having a relatively large area as the imaging position candidate.

[0118] 11, the virtual depth maps 52_1 and 52_2 may include an outside-frame region 54 outside a virtual frame line 53 that corresponds to the image frame of the real depth map 50. The outside-frame region 54 corresponds to the outside of the field of view of the three-dimensional camera 200 located at the imaging position RP that corresponds to the real depth map 50. Therefore, the CPU 110 does not include the outside-frame region 54 in the missing region.

[0119] The virtual depth maps 52_1 and 52_2 may include minute regions in which distance information is missing due to the influence of noise, etc. In order not to count such minute regions as missing regions, it is preferable that the CPU 110 determines, as missing regions, regions in which a predetermined number or more of pixels in which distance information is missing are connected.

[0120] <Second embodiment of the subroutine of step S8> Fig. 12 is a flow chart showing a second embodiment of the subroutine of step S8 shown in Fig. 8. As shown in Fig. 12, the second embodiment of the subroutine of step S8 includes steps S811 and S812.

[0121] First, in step S811, the CPU 110 operating as the determination unit 13 of the image processing device 100 acquires a recognition result corresponding to the latest real depth map (hereinafter, referred to as the "latest recognition result").

[0122] In the next step S812, the CPU 110 determines, based on the latest recognition result, one or more virtual viewpoints that increase the ease of recognition of one or more specific works as one or more imaging position candidates.

[0123] Details of step S812 will be described with reference to Fig. 13 and Fig. 14. Fig. 13 is a diagram showing the positional relationship between a virtual viewpoint and a specific workpiece. Fig. 14 is a diagram showing an example of a model image corresponding to the posture of the specific workpiece when viewed from the virtual viewpoint.

[0124] The CPU 110 identifies one or more workpieces 2 other than the workpiece to be picked as one or more specific workpieces 2S. For example, the CPU 110 identifies a workpiece 2 having a position and orientation corresponding to the highest matching score among one or more workpieces 2 other than the workpiece to be picked as the specific workpiece 2S. Alternatively, the CPU 110 identifies a workpiece 2 having a position and orientation that is in the top predetermined number of matching scores among one or more workpieces 2 other than the workpiece to be picked as the specific workpiece 2S.

[0125] The CPU 110 determines one or more virtual viewpoints VP that satisfy a specific condition as one or more candidate imaging positions for the three-dimensional camera 200. The specific condition is a condition that a first recognizability of the posture of one or more specific workpieces 2S when viewed from the virtual viewpoint VP is higher than a second recognizability of the posture of one or more specific workpieces 2S in the real depth map.

[0126] The first recognizability is determined based on the recognizability score of a model image (hereinafter referred to as the "first model image 61") among the multiple model images 60 that corresponds to the estimated posture of each of one or more specific works 2S when viewed from the virtual viewpoint VP.

[0127] The posture of each specific work 2S when viewed from the virtual viewpoint VP is estimated based on the posture of the specific work 2S recognized from the real depth map, the virtual viewpoint VP, and the latest imaging position RP (imaging position corresponding to the latest real depth map) of the three-dimensional camera 200. Specifically, the posture of each specific work 2S when viewed from the virtual viewpoint VP is estimated by applying a rotation transformation about a reference point OP to the recognized posture of the specific work 2S in order to match the virtual viewpoint VP with the latest imaging position RP.

[0128] The second recognizability is determined based on the recognizability score of a model image (hereinafter referred to as the “second model image 62”) among the multiple model images 60 that corresponds to the recognized posture of each of the one or more specific works 2S.

[0129] 13, the CPU 110 identifies one specific workpiece 2S. The CPU 110 identifies, as the second model image 62, the model image having the highest matching score with the area including the specific workpiece 2S in the actual depth map based on the recognition result shown in FIG.

[0130] The CPU 110 reads out the recognizability score corresponding to the second model image 62 from the model image set 16 shown in Fig. 7, and evaluates the second recognizability based on the read recognizability score. When one specific work 2S is identified, the CPU 110 may determine the recognizability score corresponding to the second model image 62 as the evaluation value of the second recognizability. When multiple specific works 2S are identified, the CPU 110 may identify multiple second model images 62 corresponding to the multiple specific works 2S, and determine a representative value (e.g., average value, median value, etc.) of the multiple recognizability scores corresponding to the multiple second model images 62 as the evaluation value of the second recognizability.

[0131] The CPU 110 estimates the posture of the specific work 2S as viewed from the virtual viewpoint VP_1 based on the recognized posture of the specific work 2S, the virtual viewpoint VP_1, and the latest imaging position RP, and identifies the first model image 61_1 corresponding to the estimated posture. The CPU 110 reads out the recognizability score corresponding to the first model image 61_1 from the model image set 16, and evaluates the first recognizability of the specific work 2S as viewed from the virtual viewpoint VP_1 based on the read recognizability score. The method of evaluating the first recognizability is the same as the method of evaluating the second recognizability.

[0132] Similarly, the CPU 110 estimates the posture of the specific work 2S when viewed from the virtual viewpoint VP_2 based on the recognized posture of the specific work 2S, the virtual viewpoint VP_2, and the latest imaging position RP, and identifies the first model image 61_2 corresponding to the estimated posture. The CPU 110 reads out the recognizability score corresponding to the first model image 61_2 from the model image set 16, and evaluates the first recognizability of the specific work 2S when viewed from the virtual viewpoint VP_2 based on the read out recognizability score.

[0133] 13 and 14, the uniqueness of the first model image 61_1 is greater than the uniqueness of the second model image 62. Therefore, the recognizability score corresponding to the first model image 61_1 is greater than the recognizability score corresponding to the second model image 62. Therefore, the CPU 110 determines that the virtual viewpoint VP_1 satisfies the above-mentioned specific condition, and determines the virtual viewpoint VP_1 as the next imaging position candidate.

[0134] The area of ​​the work model 2M in the first model image 61_2 is larger than the area of ​​the work model 2M in the second model image 62. Therefore, the recognizability score corresponding to the first model image 61_2 is larger than the recognizability score corresponding to the second model image 62. Therefore, the CPU 110 determines that the virtual viewpoint VP_2 satisfies the above specific condition, and determines the virtual viewpoint VP_2 as the next imaging position candidate.

[0135] <Third Example of the Subroutine of Step S8> The third embodiment of the subroutine of step S8 is a modification of the above-mentioned second embodiment. In the object recognition process, the posture of the specific workpiece 2S may be erroneously recognized.

[0136] 15 is a diagram showing an example in which the posture of a specific workpiece 2S is erroneously recognized. In the example shown in FIG.

[0137] FIG. 16 is a diagram showing an example of a first model image obtained from the erroneous recognition result shown in FIG. 15. In FIG. 16, the first model images 61_3, 61_4 and the second model image 62_1 are identified by applying the erroneous recognition result shown in FIG. 15 to the second embodiment of the subroutine of step S8. That is, the second model image 62_1 corresponds to the erroneously recognized posture. The first model image 61_3 corresponds to the estimated posture of the specific workpiece 2S when viewed from the virtual viewpoint VP_1. The first model image 61_4 corresponds to the estimated posture of the specific workpiece 2S when viewed from the virtual viewpoint VP_2.

[0138] The recognizability score corresponding to the first model image 61_3 is greater than the recognizability score corresponding to the second model image 62_1. On the other hand, the recognizability score corresponding to the first model image 61_4 is less than the recognizability score corresponding to the second model image 62_1. Therefore, the virtual viewpoint VP_1 is determined to satisfy the above specific condition, and is determined as the next imaging position candidate.

[0139] When the virtual viewpoint VP_1 is determined as the next imaging position and a new real depth map is acquired, the posture of the specific workpiece 2S is recognized from the new real depth map. At this time, if the posture of the specific workpiece 2S is correctly recognized, the model image 60 corresponding to the recognized posture is different from the first model image 61_3. In this way, after the posture of the specific workpiece 2S is erroneously recognized in the k-th object recognition process, the posture of the specific workpiece 2S may be correctly recognized in the k+1-th object recognition process. In this case, the first model image 61 corresponding to the posture of the specific workpiece 2S estimated from the recognition result by the k-th object recognition process does not match the second model image 62 corresponding to the posture of the specific workpiece 2S recognized in the k+1-th object recognition process.

[0140] Conversely, there may be cases where the posture of the specific work 2S is correctly recognized in the k-th object recognition process, and then the posture of the specific work 2S is erroneously recognized in the k+1-th object recognition process. In this case as well, the first model image 61 corresponding to the posture of the specific work 2S estimated from the recognition result of the k-th object recognition process does not match the second model image 62 corresponding to the posture of the specific work 2S recognized in the k+1-th object recognition process.

[0141] In this way, if the first model image 61 corresponding to the posture of the specific work 2S estimated from the recognition result by the kth object recognition process does not match the second model image 62 corresponding to the posture of the specific work 2S recognized in the k+1th object recognition process, the posture of the specific work 2S has been erroneously recognized in one of the object recognition processes. A virtual viewpoint that may erroneously recognize the posture of the specific work 2S is inappropriate as a candidate for the next shooting position. Therefore, the third embodiment of the subroutine of step S8 includes a process of registering a virtual viewpoint that may erroneously recognize the posture of the specific work 2S as an inappropriate position.

[0142] In a third embodiment of the subroutine of step S8, for each specific workpiece 2S, a first model image 61 corresponding to a posture estimated from the previous recognition result is compared with a second model image 62 corresponding to a posture estimated from the latest recognition result. Therefore, the CPU 110 records a model image ID that identifies the first model image 61 corresponding to the next imaging position.

[0143] Fig. 17 is a diagram showing an example of the recognition result recorded in the third embodiment of the subroutine of step S8. As shown in Fig. 17, the CPU 110 records, in the recognition result, a model image ID for identifying the first model image 61 corresponding to the estimated orientation when viewed from the next imaging position for each specific workpiece 2S. In the example shown in Fig. 17, the workpiece 2 with the position and orientation "X2, Y2, Z2, Xr2, Yr2, Zr2" is identified as the specific workpiece 2S, and a model image ID for identifying the first model image 61 for the specific workpiece 2S is recorded.

[0144] When the CPU 110 determines a plurality of imaging position candidates, it records the model image ID that identifies the first model image 61 corresponding to each specific workpiece 2S for each imaging position candidate. After that, when the next imaging position is determined from among the plurality of imaging position candidates, it is sufficient to delete the model image ID that identifies the first model image 61 corresponding to the imaging position candidate other than the next imaging position from the recognition result.

[0145] Fig. 18 is a flow chart showing a third embodiment of the subroutine of step S8 shown in Fig. 8. As shown in Fig. 18, the third embodiment of the subroutine of step S8 includes steps S812, and S821 to S824.

[0146] First, in step S821, the CPU 110 of the image processing device 100 acquires a recognition result corresponding to the previous real depth map (hereinafter referred to as the "previous recognition result") and a recognition result corresponding to the latest real depth map (hereinafter referred to as the "latest recognition result").

[0147] In the next step S822, the CPU 110 operating as the registration unit 22 of the image processing device 100 determines whether or not the first model image 61 recorded in the previous recognition result matches the second model image 62 recorded in the latest recognition result for each of one or more specific works 2S identified from the previous recognition result.

[0148] If the first model image 61 and the second model image 62 do not match for any of the one or more specific workpieces 2S (NO in step S822), the system 1 advances the process to step S823. If the first model image 61 and the second model image 62 match for all of the one or more specific workpieces 2S (YES in step S822), the system 1 advances the process to step S812.

[0149] In step S823, the CPU 110 registers the previous imaging position and the latest imaging position as inappropriate positions. After step S823, the system 1 advances the process to step S812.

[0150] In step S812, the CPU 110 determines, based on the latest recognition result, one or more virtual viewpoints that increase the ease of recognition of one or more specific works 2S, as one or more imaging position candidates.

[0151] In step S824 after step S812, the CPU 110 records, for each of the one or more imaging position candidates, a model image ID that identifies the first model image 61 corresponding to each of the one or more specific workpieces 2S in the latest recognition result. At this time, the CPU 110 records the model image ID that identifies the first model image 61 in association with information on the imaging position candidate. The information on the imaging position candidate indicates, for example, the coordinates of the imaging position candidate.

[0152] In addition, when a plurality of imaging position candidates are determined in step S824, the CPU 110 records, for each imaging position candidate, a model image ID that identifies the first model image 61 corresponding to each specific workpiece 2S in association with the imaging position candidate information. In the subsequent step S9, at the stage where the next imaging position is determined from among the plurality of imaging position candidates, the model image ID that identifies the first model image 61 corresponding to the imaging position candidate other than the next imaging position is deleted from the recognition result.

[0153] <Fourth Example of the Subroutine of Step S8> The fourth embodiment of the subroutine of step S8 is a combination of the above-mentioned first and second embodiments. That is, in the fourth embodiment, one or more virtual viewpoints corresponding to one or more virtual depth maps having relatively large areas of missing regions are determined as one or more imaging position candidates, and then the imaging position candidates are narrowed down in consideration of the ease of recognition of the work 2.

[0154] Fig. 19 is a flow chart showing a fourth embodiment of the subroutine of step S8 shown in Fig. 8. As shown in Fig. 19, the fourth embodiment of the subroutine of step S8 includes steps S801 to S804, S811, and S812a.

[0155] As in the first embodiment of the subroutine of step S8, the CPU 110 performs the processes of steps S801 to S804. As a result, one or more virtual viewpoints corresponding to one or more virtual depth maps having relatively large areas of missing regions among the multiple virtual depth maps are temporarily determined as one or more imaging position candidates.

[0156] After step S804, the CPU 110 continues to perform the processes of steps S811 and S812a. Step S812a corresponds to step S812. In step S812a, the CPU 110 narrows down the one or more imaging position candidates determined in step S804 to imaging position candidates that increase the ease of recognition of one or more specific workpieces 2S based on the latest recognition result. Specifically, the CPU 110 narrows down the one or more imaging position candidates determined in step S804 to one or more imaging position candidates corresponding to one or more virtual viewpoints that satisfy a specific condition.

[0157] <Fifth Example of the Subroutine of Step S8> The fifth embodiment of the subroutine of step S8 is a combination of the first and third embodiments. That is, in the fifth embodiment, like the fourth embodiment, one or more virtual viewpoints corresponding to one or more virtual depth maps having relatively large areas of missing areas are determined as one or more imaging position candidates, and then the imaging position candidates are narrowed down in consideration of the ease of recognition of the work 2. Furthermore, the fifth embodiment registers inappropriate positions according to the recognition results.

[0158] Fig. 20 is a flow chart showing a fifth embodiment of the subroutine of step S8 shown in Fig. 8. As shown in Fig. 20, the fifth embodiment of the subroutine of step S8 includes steps S801 to S804, S812a, and S821 to S824.

[0159] As in the first embodiment of the subroutine of step S8, the CPU 110 performs the processes of steps S801 to S804. As a result, one or more virtual viewpoints corresponding to one or more virtual depth maps having relatively large areas of missing regions among the multiple virtual depth maps are temporarily determined as one or more imaging position candidates.

[0160] After step S804, CPU 110 performs the processes of steps S821, S822, S823, S812a, and S824. In step S823, an inappropriate position is registered, as in the third embodiment.

[0161] Details of step S812a are as described in Example 4. As a result, the one or more imaging position candidates determined in step S804 are narrowed down to imaging position candidates that increase the ease of recognition of one or more specific workpieces 2S.

[0162] <First Example of Subroutine of Step S9> Fig. 21 is a flow chart showing a first embodiment of the subroutine of step S9 shown in Fig. 8. The first embodiment of the subroutine of step S9 includes steps S901 to S909.

[0163] First, in step S901, the CPU 110 operating as the priority setting unit 21 of the image processing device 100 sets a priority for each of the determined one or more imaging position candidates based on one or more indices. For example, for each imaging position candidate, the CPU 110 calculates at least one of the score Sa determined from the area of ​​the missing area in the corresponding virtual depth map, the score Sb determined from the movement amount of the robot arm 301 for moving the three-dimensional camera 200 to the imaging position candidate, and the score Sc determined from the recognition ease of the workpiece 2 when viewed from the imaging position candidate. The CPU 110 sets the priority based on the calculated score. When the CPU 110 calculates only one of the scores Sa, Sb, and Sc, the CPU 110 sets the priority in the order of the calculated score values. When the CPU 110 calculates two or more scores of the scores Sa, Sb, and Sc, the CPU 110 calculates an integrated score from the values ​​of the calculated two or more scores, and sets the priority in the order of the integrated score values.

[0164] In the next step S902, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 selects the imaging position candidate with the highest priority.

[0165] In the next step S903, the CPU 110 generates a first inquiry as to whether or not the three-dimensional camera 200 can be moved to the selected imaging position candidate, and outputs the first inquiry to the robot controller 400.

[0166] In the next step S904, the control processing circuit 420 operating as the movement possibility confirmation unit 42 of the robot controller 400 judges whether or not the three-dimensional camera 200 can be moved to the selected imaging position candidate.

[0167] If the three-dimensional camera 200 can be moved to the imaging position candidate (YES in step S904), in step S905 the control processing circuit 420 determines the selected imaging position candidate as the next imaging position. Furthermore, the control processing circuit 420 outputs a confirmation result indicating that the three-dimensional camera 200 can be moved to the imaging position candidate and that the imaging position candidate will be the next imaging position to the image processing device 100. After step S905, the system 1 completes step S9.

[0168] If the three-dimensional camera 200 cannot be moved to the imaging position candidate (NO in step S904), in step S906 the control processing circuit 420 notifies the image processing device 100 of the confirmation result indicating that movement is not possible.

[0169] In step S907 after step S906, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 determines whether or not there is an unselected imaging position candidate.

[0170] If there is an unselected imaging position candidate (YES in step S907), the CPU 110 selects the imaging position candidate with the next highest priority in step S908. After step S908, the system 1 returns the process to step 903.

[0171] If there is no unselected imaging position candidate (NO in step S907), in step S909, the system 1 determines a position other than the one or more imaging position candidates as the next imaging position. After step S909, the system 1 completes step S9.

[0172] <First Example of Subroutine of Step S909> Fig. 22 is a flow chart showing a first embodiment of the subroutine of step S909 shown in Fig. 21. The first embodiment of the subroutine of step S909 includes step S911.

[0173] In step S911, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 determines the latest imaging position as the next imaging position. The CPU 110 outputs a movement command to the robot controller 400 to move the three-dimensional camera 200 to the next imaging position.

[0174] <Second embodiment of the subroutine of step S909> Fig. 23 is a flow chart showing a second embodiment of the subroutine of step S909 shown in Fig. 21. The second embodiment of the subroutine of step S909 includes steps S921 to S927 and S911.

[0175] First, in step S921, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 calculates the midpoint between the imaging position candidate with the highest priority and the latest imaging position.

[0176] In the next step S922, the CPU 110 generates a second inquiry as to whether the three-dimensional camera 200 can be moved to the intermediate point, and outputs the second inquiry to the robot controller 400.

[0177] In the next step S923, the control processing circuit 420 operating as the movement possibility confirmation unit 42 of the robot controller 400 judges whether or not the three-dimensional camera 200 can be moved to the midpoint.

[0178] If the three-dimensional camera 200 can be moved to the midpoint (YES in step S923), in step S924 the control processing circuit 420 determines the midpoint as the next imaging position. Furthermore, the control processing circuit 420 outputs to the image processing device 100 a confirmation result indicating that the three-dimensional camera 200 can be moved to the midpoint and that the midpoint will be the next imaging position. After step S924, the system 1 completes step S9.

[0179] If the three-dimensional camera 200 cannot be moved to the intermediate point (NO in step S923), then in step S925 the control processing circuit 420 outputs a confirmation result indicating that movement is not possible to the image processing device 100.

[0180] In step S926 after step S925, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 determines whether or not there is an imaging position candidate for which the midpoint with respect to the latest imaging position has not been calculated.

[0181] If there is an imaging position candidate for which the midpoint has not been calculated (YES in step S926), in step S927, the CPU 110 calculates the midpoint between the imaging position candidate with the next highest priority and the latest imaging position. After step S927, the system 1 returns the process to step 922.

[0182] If there is no imaging position candidate for which the midpoint has not been calculated (NO in step S926), in step S911, the CPU 110 determines the latest imaging position as the next imaging position and outputs a movement command to the robot controller 400 to move the three-dimensional camera 200 to the next imaging position.

[0183] <Second embodiment of the subroutine of step S9> Fig. 24 is a flow chart showing a second embodiment of the subroutine of step S9 shown in Fig. 8. The second embodiment of the subroutine of step S9 includes steps S901, and S931 to S936.

[0184] As in the first embodiment shown in FIG. 21, first, in step S901, the CPU 110 operating as the priority setting unit 21 of the image processing device 100 sets a priority for each of the one or more determined imaging position candidates based on one or more indicators.

[0185] In the next step S931, the CPU 110 generates a first inquiry as to whether or not the three-dimensional camera 200 can move to all of one or more imaging position candidates (plurality of imaging position candidates) included in the list, and outputs the first inquiry to the robot controller 400. The CPU 110 may generate the first inquiry to which the list is added.

[0186] In the next step S932, the control processing circuit 420 operating as the movement possibility confirmation unit 42 of the robot controller 400 confirms whether or not the three-dimensional camera 200 can move to each of the one or more imaging position candidates.

[0187] In the next step S933, the control processing circuit 420 determines whether or not there is a candidate imaging position to which the three-dimensional camera 200 can move.

[0188] If there is a candidate imaging position to which the three-dimensional camera 200 can move (YES in step S933), in step S934 the control processing circuit 420 determines the imaging position candidate with the highest priority among the one or more candidate imaging positions to which the three-dimensional camera 200 can move as the next imaging position. Furthermore, the control processing circuit 420 outputs a confirmation result indicating that there is a candidate imaging position to which the three-dimensional camera 200 can move and indicating the next imaging position to the image processing device 100. After step S934, the system 1 completes step S9.

[0189] If there are no candidate imaging positions to which the three-dimensional camera 200 can move (NO in step S933), in step S935, the control processing circuit 420 outputs a confirmation result indicating that there are no candidate imaging positions to which the three-dimensional camera 200 can move to the image processing device 100.

[0190] In step S936 after step S935, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 determines a position other than the one or more imaging position candidates as the next imaging position. After step S936, the system 1 completes step S9.

[0191] <First Example of Subroutine of Step S936> The first embodiment of the subroutine of step S936 includes step S911 shown in Fig. 22, similarly to the first embodiment of the subroutine of step S909. That is, in step S911, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 determines the latest imaging position as the next imaging position. The CPU 110 outputs a movement command to the robot controller 400 to move the 3D camera 200 to the next imaging position.

[0192] <Second embodiment of the subroutine of step S936> Fig. 25 is a flow chart showing a second embodiment of the subroutine of step S936 shown in Fig. 24. The second embodiment of the subroutine of step S936 includes steps S941 to S946 and S911.

[0193] First, in step S941, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 calculates the midpoint between each of one or more imaging position candidates included in the list and the latest imaging position.

[0194] In the next step S942, the CPU 110 generates a second inquiry as to whether the three-dimensional camera 200 can move to each waypoint, and outputs the second inquiry to the robot controller 400. Each waypoint is labeled with the priority of the corresponding imaging position candidate.

[0195] In the next step S943, the control processing circuit 420 operating as the movement possibility confirmation unit 42 of the robot controller 400 confirms whether or not the three-dimensional camera 200 can be moved to each of the intermediate points.

[0196] In the next step S944, the control processing circuit 420 determines whether or not there is an intermediate point to which the three-dimensional camera 200 can be moved.

[0197] If there is an intermediate point to which the three-dimensional camera 200 can move (YES in step S944), in step S945 the control processing circuit 420 determines the intermediate point corresponding to the imaging position candidate with the highest priority among the one or more intermediate points to which the three-dimensional camera 200 can move as the next imaging position. Furthermore, the control processing circuit 420 outputs a confirmation result indicating the existence of an intermediate point to which the three-dimensional camera 200 can move and the next imaging position to the image processing device 100. After step S945, the system 1 completes step S936.

[0198] If there is no intermediate point to which the three-dimensional camera 200 can move (NO in step S944), in step S946, the control processing circuit 420 outputs a confirmation result indicating that there is no intermediate point to which the three-dimensional camera 200 can move to the image processing device 100.

[0199] In step S911 after step S946, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 determines the latest imaging position as the next imaging position. The CPU 110 outputs a movement command to the robot controller 400 to move the three-dimensional camera 200 to the next imaging position.

[0200] <Modification> In the above description, when the robot controller 400 determines in response to the inquiry that the three-dimensional camera 200 can move to an imaging position candidate, it determines the imaging position candidate as the next imaging position. However, when the robot controller 400 determines that the three-dimensional camera 200 can move to an imaging position candidate, it may notify the image processing device 100 of a confirmation result indicating that the movement is possible. The CPU 110 (candidate list management unit 23) of the image processing device 100 may determine one imaging position candidate to which it is determined that the three-dimensional camera 200 can move as the next imaging position. Thereafter, the CPU 110 may generate a movement command for moving the three-dimensional camera 200 to the next imaging position, and output the movement command to the robot controller 400.

[0201] The image processing device 100 may include a plurality of devices. Alternatively, the image processing device 100 may have a plurality of processors. In this case, the three-dimensional measurement unit 10, the viewpoint conversion unit 11, the missing area calculation unit 12, the determination unit 13, the model image generation unit 14, the assignment unit 15, the three-dimensional recognition unit 17, the picking position calculation unit 19, the range setting unit 20, the priority setting unit 21, the registration unit 22, and the candidate list management unit 23 may be realized by the same processor or by different processors.

[0202] In the above description, the robot controller 400 and the image processing device 100 are separate entities. However, the functions of the robot controller 400 may be incorporated into the image processing device 100. In this case, the robot controller 400, the three-dimensional measurement unit 10, the viewpoint conversion unit 11, the missing area calculation unit 12, the determination unit 13, the model image generation unit 14, the assignment unit 15, the three-dimensional recognition unit 17, the picking position calculation unit 19, the range setting unit 20, the priority setting unit 21, the registration unit 22, and the candidate list management unit 23 may be realized by the same processor.

[0203] §3 Supplementary Note As described above, the present embodiment includes the following disclosure.

[0204] (Configuration 1) A system (1), A robot (300) having an end effector (304) for picking and placing an object and an arm (301) for moving the end effector (304); A three-dimensional camera (200) attached to the arm (301); A robot controller (400) for controlling the operation of the robot (300); an object recognition unit (17, 110) that recognizes the position and orientation of one or more objects (2) in an input image based on an input image captured by the three-dimensional camera (200); a command unit (19, 110) that selects the target object from the one or more objects (2) and outputs a command to the robot controller (400) to pick up the target object; a determination unit (13, 110) that determines one or more next imaging position candidates for the three-dimensional camera (200) based on the input image while the robot (300) is picking and placing the object in response to the command; an output unit (23, 110) that outputs a first inquiry to the robot controller (400) as to whether or not the three-dimensional camera (200) can be moved to each of the one or more imaging position candidates while the robot (300) is picking up and placing the object; A system (1) in which the robot controller (400) or the output section (23, 110) determines the next imaging position of the 3D camera (200) depending on the confirmation result of the first query while the robot (300) is picking and placing the object.

[0205] (Configuration 2) the one or more imaging position candidates include a plurality of imaging position candidates, 2. The system (1) according to configuration 1, wherein the first query is accompanied by a list of the plurality of imaging position candidates.

[0206] (Configuration 3) the one or more imaging position candidates include a plurality of imaging position candidates, The system (1) comprises: a setting unit (21, 110) that sets a priority for each of the plurality of imaging position candidates based on the input image, The system (1) according to configuration 1, wherein the output unit (23, 110) transmits the first inquiry to the robot controller (400) in descending order of priority.

[0207] (Configuration 4) when the three-dimensional camera (200) cannot be moved to any of the one or more imaging position candidates, the output unit (23, 110) transmits to the robot controller (400) a second inquiry as to whether or not the three-dimensional camera (200) can be moved to a midpoint between the latest imaging position of the three-dimensional camera (200) and any of the one or more imaging position candidates; The system (1) according to any one of configurations 1 to 3, wherein the robot controller (400) or the output section (23, 110) determines the next imaging position depending on a confirmation result for the second inquiry.

[0208] (Configuration 5) The system (1) described in any one of configurations 1 to 3, wherein when the three-dimensional camera (300) cannot move to any of the one or more candidate imaging positions, the robot controller (400) or the output unit (23, 110) determines the latest imaging position of the three-dimensional camera (200) as the next imaging position.

[0209] (Configuration 6) A system (1) described in any one of configurations 1 to 5, wherein the robot controller (400), the object recognition unit (17, 110), the command unit (19, 110), the decision unit (13, 110), and the output unit (23, 110) are realized by a same processor.

[0210] (Configuration 7) A method for controlling a robot (300), the robot (300) having an end effector (304) for picking and placing an object, an arm (301) for moving the end effector (304), and a three-dimensional camera (200) attached to the arm, the method comprising: One or more processors recognize the position and orientation of one or more objects (2) in an input image based on an input image captured by the 3D camera (200); the one or more processors selecting the target object from among the one or more objects (2) and controlling the robot (300) to pick up the target object; determining, by the one or more processors, one or more next imaging positions for the 3D camera (200) based on the input images while the robot (300) is picking and placing the object; the one or more processors, while the robot (300) is picking and placing the object, for each of the one or more imaging position candidates, checking whether the 3D camera (200) can be moved to the imaging position candidate; and determining, by the one or more processors, a next imaging position for the three-dimensional camera (200) depending on a result of the checking while the robot (300) is picking and placing the object.

[0211] (Configuration 8) A program (115) for causing a computer to execute a method for controlling a robot (300), the robot having an end effector (304) for picking and placing an object, an arm (301) for moving the end effector (304), and a three-dimensional camera (200) attached to the arm (301), the method comprising: Based on an input image obtained based on the photographing by the 3D camera (200), a position and orientation of one or more objects (2) in the input image is recognized; selecting the target object from the one or more objects (2) and controlling the robot (300) to pick the target object; determining, based on the input images, one or more next imaging positions for the 3D camera (200) while the robot (300) is picking and placing the object; While the robot (300) is picking and placing the object, for each of the one or more imaging position candidates, confirming whether or not the 3D camera (200) can be moved to the imaging position candidate; determining a next imaging position of the three-dimensional camera (200) according to a result of the checking while the robot (300) is picking and placing the object.

[0212] Although the embodiment of the present invention has been described, the embodiment disclosed herein should be considered as illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, and it is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0213] 1 system, 2 work, 2M work model, 2S specific work, 2a to 2c, 2z surface, 3 tray, 3a to 3c, 3z area, 10 three-dimensional measurement unit, 11 viewpoint conversion unit, 12 missing area calculation unit, 13 determination unit, 14 model image generation unit, 15 assignment unit, 16 model image set, 17 three-dimensional recognition unit, 18 recognition result storage unit, 19 picking position calculation unit, 20 range setting unit, 21 priority setting unit, 22 registration unit, 23 candidate list management unit, 24, 41 communication unit, 42 movement possibility confirmation unit, 43 control unit, 50 real depth map, 52 virtual depth map, 53 virtual frame line, 54 outside frame area, 56_1, 56_2 missing area, 60, ID model image, 61 first model image, 62 second model image, 100 image processing device, 106 Memory card, 110 CPU, 112,440 main memory, 114,450 storage, 115 program, 116 camera interface, 116a image buffer, 118 input interface, 120 display controller, 124 communication interface, 126 data reader / writer, 128 bus, 150 display device, 160 input device, 200 three-dimensional camera, 201 projection unit, 202 shooting unit, 250 optical axis, 300 robot, 301 robot arm, 302 base, 304 end effector, 400 robot controller, 410 network controller, 420 control processing circuit, 430 processor, 452 system program, 460 interface circuit, RP imaging position, VP virtual viewpoint.

Claims

1. 1. A system comprising: A robot having an end effector for picking and placing an object and an arm for moving the end effector; A three-dimensional camera attached to the arm; A robot controller for controlling the operation of the robot; an object recognition unit that recognizes a position and orientation of one or more objects in an input image based on the input image captured by the three-dimensional camera; a command unit that selects the target object from the one or more objects and outputs a command to the robot controller to pick up the target object; a determination unit that determines one or more next imaging position candidates for the three-dimensional camera based on the input image while the robot is picking and placing the object in response to the command; an output unit that outputs a first inquiry to the robot controller regarding whether or not the three-dimensional camera can be moved to each of the one or more imaging position candidates while the robot is picking up and placing the object; The system, wherein the robot controller or the output unit determines the next imaging position of the three-dimensional camera depending on the confirmation result of the first query while the robot is picking and placing the object.

2. the one or more imaging position candidates include a plurality of imaging position candidates, The system of claim 1 , wherein the first query is accompanied by a list of the plurality of potential imaging locations.

3. the one or more imaging position candidates include a plurality of imaging position candidates, The system comprises: a setting unit that sets a priority for each of the plurality of imaging position candidates based on the input image, The system according to claim 1 , wherein the output unit transmits the first inquiry to the robot controller in the order of decreasing priority.

4. when the three-dimensional camera cannot be moved to any of the one or more imaging position candidates, the output unit transmits a second inquiry to the robot controller as to whether or not the three-dimensional camera can be moved to a midpoint between a latest imaging position of the three-dimensional camera and any of the one or more imaging position candidates; The system according to claim 1 , wherein the robot controller or the output unit determines the next imaging position depending on a confirmation result for the second inquiry.

5. The system according to claim 1 , wherein the robot controller or the output unit determines the latest imaging position of the three-dimensional camera as the next imaging position when the three-dimensional camera cannot be moved to any of the one or more candidate imaging positions.

6. The system according to claim 1 , wherein the robot controller, the object recognition unit, the command unit, the decision unit, and the output unit are realized by a same processor.

7. A method for controlling a robot, the robot having an end effector for picking and placing an object, an arm for moving the end effector, and a three-dimensional camera attached to the arm, the method comprising: One or more processors recognize the position and orientation of one or more objects in an input image based on an input image captured by the three-dimensional camera; the one or more processors selecting the target object from among the one or more objects and controlling the robot to pick the target object; determining, by the one or more processors, one or more next imaging positions for the 3D camera based on the input images while the robot is picking and placing the object; the one or more processors, while the robot is picking and placing the object, for each of the one or more imaging position candidates, checking whether the 3D camera can be moved to the imaging position candidate; and determining, by the one or more processors, a next imaging position for the three-dimensional camera depending on a result of the checking while the robot is picking and placing the object.

8. A program for causing a computer to execute a method for controlling a robot, the robot having an end effector for picking and placing an object, an arm for moving the end effector, and a three-dimensional camera attached to the arm, the method comprising: Recognizing a position and orientation of one or more objects in an input image obtained based on the photographing of the three-dimensional camera; selecting the target object from among the one or more objects and controlling the robot to pick the target object; determining, while the robot is picking and placing the object, one or more next possible imaging positions for the 3D camera based on the input images; While the robot is picking and placing the object, for each of the one or more imaging position candidates, checking whether the 3D camera can be moved to the imaging position candidate; determining a next imaging position for the three-dimensional camera according to a result of the checking while the robot is picking and placing the object.

Images